[jira] [Created] (MAPREDUCE-5271) Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
lijuanhzou created MAPREDUCE-5271: - Summary: Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs Key: MAPREDUCE-5271 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5271 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 2.0.0-alpha Environment: linux 64 Reporter: lijuanhzou Assignee: lijuanhzou (hadoop-2.0.0-cdh4.2.0)I encounter following question when do a simple words-count working.small input data is ok,there is problem when using big datas: Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:379) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$CompressAwarePath cannot be cast to java.lang.Comparable at java.util.TreeMap.put(TreeMap.java:542) at java.util.TreeSet.add(TreeSet.java:238) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.closeOnDiskFile(MergeManagerImpl.java:340) at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:495) at org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5245: --- Attachment: MAPREDUCE-5245.2.patch Thanks, Mayank! The patch is updated, with assertion on mapreduce property names added A number of public static variables are removed from JobConf Key: MAPREDUCE-5245 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5245.1.patch, MAPREDUCE-5245.2.patch A number of public static variables are removed from JobConf: boolean DEFAULT_MAPREDUCE_RECOVER_JOB String MAPREDUCE_RECOVER_JOB String WORKFLOW_ADJACENCY_PREFIX_PATTERN String WORKFLOW_ADJACENCY_PREFIX_STRING String WORKFLOW_ID String WORKFLOW_NAME String WORKFLOW_NODE_NAME String WORKFLOW_TAGS The workflow related variables are moved to MRJobConfig. The follwing public static variables becomes default: String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY The variables there are no longer referred internally in 2.x, but they might be used by users as they were public. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5245) A number of public static variables are removed from JobConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666061#comment-13666061
]
Hadoop QA commented on MAPREDUCE-5245:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584655/MAPREDUCE-5245.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3675//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3675//console
This message is automatically generated.
A number of public static variables are removed from JobConf
Key: MAPREDUCE-5245
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5245
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5245.1.patch, MAPREDUCE-5245.2.patch
A number of public static variables are removed from JobConf:
boolean DEFAULT_MAPREDUCE_RECOVER_JOB
String MAPREDUCE_RECOVER_JOB
String WORKFLOW_ADJACENCY_PREFIX_PATTERN
String WORKFLOW_ADJACENCY_PREFIX_STRING
String WORKFLOW_ID
String WORKFLOW_NAME
String WORKFLOW_NODE_NAME
String WORKFLOW_TAGS
The workflow related variables are moved to MRJobConfig.
The follwing public static variables becomes default:
String MAPRED_JOB_MAP_MEMORY_MB_PROPERTY
String MAPRED_JOB_REDUCE_MEMORY_MB_PROPERTY
The variables there are no longer referred internally in 2.x, but they might
be used by users as they were public.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5246) Adding application type to submission context
[ https://issues.apache.org/jira/browse/MAPREDUCE-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666198#comment-13666198 ] Hudson commented on MAPREDUCE-5246: --- Integrated in Hadoop-Yarn-trunk #219 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/219/]) YARN-563. Add the concept of an application-type for each application. Contributed by Mayank Bansal. MAPREDUCE-5246. Specify application-type at the time of job submission after YARN-563. Contributed by Mayank Bansal. (Revision 1485790) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485790 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestApplicatonReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java *
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666196#comment-13666196 ] Hudson commented on MAPREDUCE-5230: --- Integrated in Hadoop-Yarn-trunk #219 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/219/]) MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary compatibility with mapred in 1.x Contributed by Mayank Bansal. (Revision 1485906) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485906 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/NLineInputFormat.java createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666284#comment-13666284 ] Hudson commented on MAPREDUCE-5230: --- Integrated in Hadoop-Hdfs-trunk #1409 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1409/]) MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary compatibility with mapred in 1.x Contributed by Mayank Bansal. (Revision 1485906) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485906 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/NLineInputFormat.java createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5246) Adding application type to submission context
[ https://issues.apache.org/jira/browse/MAPREDUCE-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666286#comment-13666286 ] Hudson commented on MAPREDUCE-5246: --- Integrated in Hadoop-Hdfs-trunk #1409 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1409/]) YARN-563. Add the concept of an application-type for each application. Contributed by Mayank Bansal. MAPREDUCE-5246. Specify application-type at the time of job submission after YARN-563. Contributed by Mayank Bansal. (Revision 1485790) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485790 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestApplicatonReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java *
[jira] [Commented] (MAPREDUCE-5271) Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666300#comment-13666300
]
Jason Lowe commented on MAPREDUCE-5271:
---
This looks to be the same issue reported in MAPREDUCE-5034 and MAPREDUCE-4965.
The problem has since been fixed by having {{CompressAwarePath}} extend
{{Path}}.
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
Key: MAPREDUCE-5271
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5271
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 2.0.0-alpha
Environment: linux 64
Reporter: lijuanhzou
Assignee: lijuanhzou
Labels: hadoop
(hadoop-2.0.0-cdh4.2.0)I encounter following question when do a simple
words-count working.small input data is ok,there is problem when using big
datas:
Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in InMemoryMerger - Thread to merge in-memory shuffled map-outputs
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:121)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:379)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
Caused by: java.lang.ClassCastException:
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$CompressAwarePath
cannot be cast to java.lang.Comparable
at java.util.TreeMap.put(TreeMap.java:542)
at java.util.TreeSet.add(TreeSet.java:238)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.closeOnDiskFile(MergeManagerImpl.java:340)
at
org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl$InMemoryMerger.merge(MergeManagerImpl.java:495)
at
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run(MergeThread.java:94)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5230) createFileSplit is removed from NLineInputFormat of mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666331#comment-13666331 ] Hudson commented on MAPREDUCE-5230: --- Integrated in Hadoop-Mapreduce-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1435/]) MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary compatibility with mapred in 1.x Contributed by Mayank Bansal. (Revision 1485906) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485906 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/NLineInputFormat.java createFileSplit is removed from NLineInputFormat of mapred -- Key: MAPREDUCE-5230 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5230 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Fix For: 2.0.5-beta Attachments: MAPREDUCE-5230-trunk-1.patch createFileSplit is removed from NLineInputFormat of mapred, because it's no longer used in the new getSplit implementation. However, since function is protected before, there is still the potential risk that the user defined format class which extends old NLineInputFormat uses the protected function. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5246) Adding application type to submission context
[ https://issues.apache.org/jira/browse/MAPREDUCE-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666333#comment-13666333 ] Hudson commented on MAPREDUCE-5246: --- Integrated in Hadoop-Mapreduce-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1435/]) YARN-563. Add the concept of an application-type for each application. Contributed by Mayank Bansal. MAPREDUCE-5246. Specify application-type at the time of job submission after YARN-563. Contributed by Mayank Bansal. (Revision 1485790) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485790 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/TestApplicatonReport.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java *
[jira] [Resolved] (MAPREDUCE-4042) unit test TestControlledMapReduceJob.testControlledMapReduceJob fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-4042. --- Resolution: Not A Problem No longer an issue since TestControlledMapReduceJob was removed as part of MAPREDUCE-4266. unit test TestControlledMapReduceJob.testControlledMapReduceJob fails -- Key: MAPREDUCE-4042 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4042 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves unit test TestControlledMapReduceJob.testControlledMapReduceJob fails. This is an ant test: ant test -Dtestcase=TestControlledMapReduceJob error: Timeout occurred. Please note the time in the report does not reflect the time until the timeout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4041) TestMapredGroupMappingServiceRefresh unit test failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-4041. --- Resolution: Not A Problem TestMapredGroupMappingServiceRefresh was removed as part of MAPREDUCE-4266. TestMapredGroupMappingServiceRefresh unit test failures --- Key: MAPREDUCE-4041 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4041 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Thomas Graves On branch-0.23 the following unit tests fail: org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testGroupMappingRefresh org.apache.hadoop.security.TestMapredGroupMappingServiceRefresh.testRefreshSuperUserGroupsConfiguration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4235) Killing app can lead to inconsistent app status between RM and HS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-4235. --- Resolution: Not A Problem This is an inherent race between the RM and the AM. The AM can succeed or fail *just* as the RM tries to kill it. Whether the RM reports the job as successful or killed is based on which way the race is resolved. In practice clients should not be surprised if a kill request ends up with the application in a non-killed terminal state such as FAILED/SUCCEEDED because of this race Killing app can lead to inconsistent app status between RM and HS - Key: MAPREDUCE-4235 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4235 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe If a client tries to kill an application that is about to complete, the application states between the ResourceManager's web UI and the history server can be inconsistent. When the problem occurs, the ResourceManager shows the Status/FinalStatus as KILLED/KILLED and the history link will redirect to a broken link. The history link still references the ApplicationMaster which is now missing. The history server entry will show the application state as SUCCEEDED. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666460#comment-13666460
]
Zhijie Shen commented on MAPREDUCE-5237:
Hm... Though MR 1.x applications will not use INVALID explicitly, there's still
the risk of being broken. For example,
{code}
switch(clusterStatus.getJobTrackerState()) {
case INITIALIZING:
...;
break;
case RUNNING:
...;
break;
default:
throw new Exception(Unexpected State);
}
{code}
Thanks, Mayank! I'll modify the code accordingly.
ClusterStatus incompatiblity issues with MR1
Key: MAPREDUCE-5237
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch
The three functions are:
CollectionString getGraylistedTrackerNames()
int getGraylistedTrackers()
State getJobTrackerState()
The're tracker related, such that they are no longer used in MR2. Maybe we
should add them and throw UnsupportedOperationException?
In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5237: --- Attachment: MAPREDUCE-5237.3.patch JobTracker.State.INVALID is removed from the lates patch ClusterStatus incompatiblity issues with MR1 Key: MAPREDUCE-5237 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch, MAPREDUCE-5237.3.patch The three functions are: CollectionString getGraylistedTrackerNames() int getGraylistedTrackers() State getJobTrackerState() The're tracker related, such that they are no longer used in MR2. Maybe we should add them and throw UnsupportedOperationException? In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666524#comment-13666524
]
Zhijie Shen commented on MAPREDUCE-5228:
1. FileInputFormat.Counter and FileOutputFormat.Counter in mapreduce should be
added as well.
2. depricatedCounterMap, initDepricatedMap() and getNewGroupKey() is better to
be defined in AbstractCounters
3. Whenever FileInputFormatCounter and FileOutputFormatCounter are referred,
the mapping policy needs to be applied. Now it's only applied in one place.
4. Is it simpler to do the following in AbstractCounters:
{code}
...
public synchronized C findCounter(Enum? key) {
+ key = oldToNew(key);
C counter = cache.get(key);
...
+ private Enum? oldToNew(Enum? key) {
+ if (key == FileInputFormat.Counter) {
+ return FileInputFormatCounter;
+ } else if (key == FileOutputFormat.Counter) {
+ return FileOutputFormatCounter
+ }
+ // more mapping rules
+ return key;
+ }
...
{code}
AbstractCounters#findCounter seems to be the ultimate method to be called.
Moreover, since there won't be a lot of mapping rules, the sequential lookup
through if else should be okay.
Enum Counter is removed from FileInputFormat and FileOutputFormat of both
mapred and mapreduce
--
Key: MAPREDUCE-5228
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Mayank Bansal
Attachments: MAPREDUCE-5228-trunk-1.patch
The enum was used by findCounter(Enum key) to find a specific counter object.
Now it seems to be replaced by FileInputFormatCounter and
FileOutputFormatCounter. Now the enum seems to be only used internally, but
not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5228: --- Status: Open (was: Patch Available) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce -- Key: MAPREDUCE-5228 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Mayank Bansal Attachments: MAPREDUCE-5228-trunk-1.patch The enum was used by findCounter(Enum key) to find a specific counter object. Now it seems to be replaced by FileInputFormatCounter and FileOutputFormatCounter. Now the enum seems to be only used internally, but not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5237) ClusterStatus incompatiblity issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666531#comment-13666531
]
Hadoop QA commented on MAPREDUCE-5237:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584723/MAPREDUCE-5237.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3676//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3676//console
This message is automatically generated.
ClusterStatus incompatiblity issues with MR1
Key: MAPREDUCE-5237
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5237
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5237.1.patch, MAPREDUCE-5237.2.patch,
MAPREDUCE-5237.3.patch
The three functions are:
CollectionString getGraylistedTrackerNames()
int getGraylistedTrackers()
State getJobTrackerState()
The're tracker related, such that they are no longer used in MR2. Maybe we
should add them and throw UnsupportedOperationException?
In addition, UNINITIALIZED_MEMORY_VALUE changes from long to int
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1303#comment-1303 ] Andrey Klochkov commented on MAPREDUCE-4980: Vinod, can you please point me to downstream components depending on MiniMRClientClusterFactory? Can't find such by quick grepping through Core, HBase, Ping, Hive. Parallel test execution of hadoop-mapreduce-client-core --- Key: MAPREDUCE-4980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated MAPREDUCE-4980: --- Attachment: MAPREDUCE-4980--n5.patch rebasing and refreshing the patch Parallel test execution of hadoop-mapreduce-client-core --- Key: MAPREDUCE-4980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1321#comment-1321
]
Hadoop QA commented on MAPREDUCE-4980:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584744/MAPREDUCE-4980--n5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 119
new or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3677//console
This message is automatically generated.
Parallel test execution of hadoop-mapreduce-client-core
---
Key: MAPREDUCE-4980
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch,
MAPREDUCE-4980--n4.patch, MAPREDUCE-4980--n5.patch, MAPREDUCE-4980.patch
The maven surefire plugin supports parallel testing feature. By using it, the
tests can be run more faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5272) A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
Zhijie Shen created MAPREDUCE-5272:
--
Summary: A Minor Error in Javadoc of TestMRWithDistributedCache in
Branch-1
Key: MAPREDUCE-5272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Trivial
{code}
/**
* Tests the use of the
* {@link org.apache.hadoop.mapreduce.filecache.DistributedCache} within the
* full MR flow as well as the LocalJobRunner. This ought to be part of the
* filecache package, but that package is not currently in mapred, so cannot
* depend on MR for testing.
*/
{code}
It should be org.apache.hadoop.filecache.DistributedCache instead. Branch-1
doesn't have org.apache.hadoop.mapreduce.filecache.DistributedCache
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5272) A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5272:
---
Attachment: MAPREDUCE-5272-branch-1.patch
Changed the javadoc
A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
--
Key: MAPREDUCE-5272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Trivial
Attachments: MAPREDUCE-5272-branch-1.patch
{code}
/**
* Tests the use of the
* {@link org.apache.hadoop.mapreduce.filecache.DistributedCache} within the
* full MR flow as well as the LocalJobRunner. This ought to be part of the
* filecache package, but that package is not currently in mapred, so cannot
* depend on MR for testing.
*/
{code}
It should be org.apache.hadoop.filecache.DistributedCache instead. Branch-1
doesn't have org.apache.hadoop.mapreduce.filecache.DistributedCache
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5272) A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated MAPREDUCE-5272:
---
Status: Patch Available (was: Open)
A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
--
Key: MAPREDUCE-5272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Trivial
Attachments: MAPREDUCE-5272-branch-1.patch
{code}
/**
* Tests the use of the
* {@link org.apache.hadoop.mapreduce.filecache.DistributedCache} within the
* full MR flow as well as the LocalJobRunner. This ought to be part of the
* filecache package, but that package is not currently in mapred, so cannot
* depend on MR for testing.
*/
{code}
It should be org.apache.hadoop.filecache.DistributedCache instead. Branch-1
doesn't have org.apache.hadoop.mapreduce.filecache.DistributedCache
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5272) A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1370#comment-1370
]
Hadoop QA commented on MAPREDUCE-5272:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584757/MAPREDUCE-5272-branch-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3678//console
This message is automatically generated.
A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1
--
Key: MAPREDUCE-5272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5272
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Priority: Trivial
Attachments: MAPREDUCE-5272-branch-1.patch
{code}
/**
* Tests the use of the
* {@link org.apache.hadoop.mapreduce.filecache.DistributedCache} within the
* full MR flow as well as the LocalJobRunner. This ought to be part of the
* filecache package, but that package is not currently in mapred, so cannot
* depend on MR for testing.
*/
{code}
It should be org.apache.hadoop.filecache.DistributedCache instead. Branch-1
doesn't have org.apache.hadoop.mapreduce.filecache.DistributedCache
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce
Zhijie Shen created MAPREDUCE-5273: -- Summary: Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce Key: MAPREDUCE-5273 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen Two protected variables are removed from CombineFileRecordReader in both mapred and mapreduce: * FileSystem fs * ClassRecordReaderK, V rrClass -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar reassigned MAPREDUCE-5251: - Assignee: Ashwin Shankar Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5274) String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce
Zhijie Shen created MAPREDUCE-5274: -- Summary: String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce Key: MAPREDUCE-5274 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5274 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce after upgrading to M/R 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
Zhijie Shen created MAPREDUCE-5275: -- Summary: Mapreduce API: TokenCache incompatibility issues with MR1 Key: MAPREDUCE-5275 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen There're following incompatibility issues: * TokenDelegationTokenIdentifier getDelegationToken(Credentials, String) is removed * Credentials loadTokens(String, Configuration) changes to Credentials loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5274) Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils
[ https://issues.apache.org/jira/browse/MAPREDUCE-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5274: --- Summary: Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils (was: String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce) Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils -- Key: MAPREDUCE-5274 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5274 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Zhijie Shen Assignee: Zhijie Shen String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce after upgrading to M/R 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5078) TestMRAppMaster fails on Windows due to mismatched path separators
[
https://issues.apache.org/jira/browse/MAPREDUCE-5078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated MAPREDUCE-5078:
---
Target Version/s: (was: 3.0.0)
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
TestMRAppMaster fails on Windows due to mismatched path separators
--
Key: MAPREDUCE-5078
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5078
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-5078.1.patch
The failing test is {{TestMRAppMaster#testMRAppMasterForDifferentUser}}.
There is an assertion about the AM staging directory, but the expected value
is constructed with a mix of forward and back slashes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4885) Streaming tests have multiple failures on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4885: --- Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Streaming tests have multiple failures on Windows - Key: MAPREDUCE-4885 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/streaming, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.0.5-beta Attachments: MAPREDUCE-4885.1.patch, MAPREDUCE-4885.2.patch, MAPREDUCE-4885.3.patch There are multiple test failures due to Queue configuration missing child queue names for root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated MAPREDUCE-4987:
---
Target Version/s: (was: 3.0.0)
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
TestMRJobs#testDistributedCache fails on Windows due to classpath problems
and unexpected behavior of symlinks
--
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch,
MAPREDUCE-4987.3.patch, MAPREDUCE-4987.4.patch, MAPREDUCE-4987.5.patch,
MAPREDUCE-4987.6.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5191) TestQueue#testQueue fails with timeout on Windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated MAPREDUCE-5191:
---
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
TestQueue#testQueue fails with timeout on Windows
-
Key: MAPREDUCE-5191
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5191
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 2.0.5-beta
Attachments: MAPREDUCE-5191.2.patch, MAPREDUCE-5191.3.patch,
MAPREDUCE-5191.patch
Test times out on my machine after 5 seconds always on the below stack:
{code}
testQueue(org.apache.hadoop.mapred.TestQueue) Time elapsed: 5009 sec
ERROR!
java.lang.Exception: test timed out after 5000 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
sun.security.provider.SeedGenerator$ThreadedSeedGenerator.getSeedByte(SeedGenerator.java:330)
at
sun.security.provider.SeedGenerator$ThreadedSeedGenerator.getSeedBytes(SeedGenerator.java:319)
at
sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:117)
at
sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
at
sun.security.provider.SecureRandom.engineNextBytes(SecureRandom.java:171)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
at java.security.SecureRandom.next(SecureRandom.java:455)
at java.util.Random.nextLong(Random.java:284)
at java.io.File.generateFile(File.java:1682)
at java.io.File.createTempFile(File.java:1791)
at java.io.File.createTempFile(File.java:1828)
at org.apache.hadoop.mapred.TestQueue.writeFile(TestQueue.java:221)
at org.apache.hadoop.mapred.TestQueue.testQueue(TestQueue.java:53)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5177) Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/MAPREDUCE-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-5177: --- Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute Key: MAPREDUCE-5177 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5177 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 2.0.5-beta Attachments: MAPREDUCE-5177.commonutils.2.patch, MAPREDUCE-5177.commonutils.patch Move to using common utils described in HADOOP-9413 that work well cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5270: --- Status: Open (was: Patch Available) Straight forward patch, mostly looks good. Except it needs minor upmerge against trunk. Please upload an updated patch. Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Attachment: MAPREDUCE-5270.3.patch the new patch fix conflicts with YARN-571 Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5270: --- Status: Patch Available (was: Open) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666889#comment-13666889
]
Hadoop QA commented on MAPREDUCE-5270:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584793/MAPREDUCE-5270.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 16 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3679//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3679//console
This message is automatically generated.
Migrate from using BuilderUtil factory methods to individual record factory
method on MapReduce side
Key: MAPREDUCE-5270
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch,
MAPREDUCE-5270.3.patch
Migrate the factory method on map reduce side.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5270: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Tests in hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-app/ successfully ran. Checked this in to trunk and branch-2. Thanks Jian! Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side Key: MAPREDUCE-5270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5270 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.0.5-beta Attachments: MAPREDUCE-5270.1.patch, MAPREDUCE-5270.2.patch, MAPREDUCE-5270.3.patch Migrate the factory method on map reduce side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5270) Migrate from using BuilderUtil factory methods to individual record factory method on MapReduce side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666920#comment-13666920 ] Hudson commented on MAPREDUCE-5270: --- Integrated in Hadoop-trunk-Commit #3789 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3789/]) MAPREDUCE-5270. Migrated MR app from using BuilderUtil factory methods to individual record factory methods. Contributed by Jian He. (Revision 1486271) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1486271 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHSWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientServiceDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobsWithHistoryService.java Migrate from using BuilderUtil factory methods to
[jira] [Commented] (MAPREDUCE-259) Rack-aware Shuffle
[
https://issues.apache.org/jira/browse/MAPREDUCE-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666926#comment-13666926
]
nemon lou commented on MAPREDUCE-259:
-
Interesting featrue. Still working on this?
Rack-aware Shuffle
--
Key: MAPREDUCE-259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-259
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy
We could try and experiment with *rack-aware* scheduling of fetches
per-reducer. Given the disparities between in-rack and off-rack bandwidth it
could be a improvement to do something along these lines:
{noformat}
if (no. of known map-output locations than no. of copier threads) {
try to schedule 75% of copies off-rack
try schedule 25% of copies in-rack
}
{noformat}
This could lead to better utilization of both in-rack switch b/w...
Clearly we want to schedule more cross-switch than in-rack since off-rack
copies will take significantly more time; hence the 75-25 split.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-259) Rack-aware Shuffle
[
https://issues.apache.org/jira/browse/MAPREDUCE-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666927#comment-13666927
]
nemon lou commented on MAPREDUCE-259:
-
MAPREDUCE-2038 is doing the same thing,but is not updated for a long time,too.
Rack-aware Shuffle
--
Key: MAPREDUCE-259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-259
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Arun C Murthy
We could try and experiment with *rack-aware* scheduling of fetches
per-reducer. Given the disparities between in-rack and off-rack bandwidth it
could be a improvement to do something along these lines:
{noformat}
if (no. of known map-output locations than no. of copier threads) {
try to schedule 75% of copies off-rack
try schedule 25% of copies in-rack
}
{noformat}
This could lead to better utilization of both in-rack switch b/w...
Clearly we want to schedule more cross-switch than in-rack since off-rack
copies will take significantly more time; hence the 75-25 split.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
