[jira] [Updated] (MAPREDUCE-5481) Enable uber jobs to have multiple reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5481: -- Resolution: Fixed Fix Version/s: 2.3.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Xuan and Jason for figuring this out and Alejandro for reviewing. Enable uber jobs to have multiple reducers --- Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0, 2.3.0 Reporter: Jason Lowe Assignee: Sandy Ryza Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5481-1.patch, MAPREDUCE-5481.patch, MAPREDUCE-5481.patch, syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console This is caused by changes made in MAPREDUCE-434 breaking Uber AMs. The easiest fix is to make similar changes in Uber AMs to those in MAPREDUCE-434 to allow multiple reducers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Enable uber jobs to have multiple reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822244#comment-13822244 ] Hudson commented on MAPREDUCE-5481: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4734 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4734/]) MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541844) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java Enable uber jobs to have multiple reducers --- Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0, 2.3.0 Reporter: Jason Lowe Assignee: Sandy Ryza Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5481-1.patch, MAPREDUCE-5481.patch, MAPREDUCE-5481.patch, syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console This is caused by changes made in MAPREDUCE-434 breaking Uber AMs. The easiest fix is to make similar changes in Uber AMs to those in MAPREDUCE-434 to allow multiple reducers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Enable uber jobs to have multiple reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822331#comment-13822331 ] Hudson commented on MAPREDUCE-5481: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #391 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/391/]) MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541844) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java Enable uber jobs to have multiple reducers --- Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0, 2.3.0 Reporter: Jason Lowe Assignee: Sandy Ryza Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5481-1.patch, MAPREDUCE-5481.patch, MAPREDUCE-5481.patch, syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console This is caused by changes made in MAPREDUCE-434 breaking Uber AMs. The easiest fix is to make similar changes in Uber AMs to those in MAPREDUCE-434 to allow multiple reducers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822325#comment-13822325 ] Hudson commented on MAPREDUCE-5610: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #391 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/391/]) MAPREDUCE-5610. TestSleepJob fails in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541566) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobCreator.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java TestSleepJob fails in jdk7 -- Key: MAPREDUCE-5610 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 Project: Hadoop Map/Reduce Issue Type: Test Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 3.0.0, 2.3.0, 0.23.10 Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch In jdk7 tests methods in a class do not run in file order, but rather in random order. TestSleepJob hosts are not initialized and a NullPointerException is thrown unless testRandomLocation was run first. This can be easily seen by running tests individually. org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5624) move grizzly-test and junit dependencies to test scope
[ https://issues.apache.org/jira/browse/MAPREDUCE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822321#comment-13822321 ] Hudson commented on MAPREDUCE-5624: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #391 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/391/]) MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541702) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-archives/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-extras/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/pom.xml move grizzly-test and junit dependencies to test scope -- Key: MAPREDUCE-5624 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5624 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Steve Loughran Assignee: Ted Yu Attachments: mapreduce-5624.txt stop the the grizzly dependences Junit getting into everything downstream by moving them to test scope -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5431) Missing pom dependency in MR-client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822327#comment-13822327 ] Hudson commented on MAPREDUCE-5431: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #391 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/391/]) MAPREDUCE-5431 Missing pom dependency in MR-client (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541682) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml Missing pom dependency in MR-client --- Key: MAPREDUCE-5431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5431 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Assignee: Timothy St. Clair Labels: maven Fix For: 2.3.0 Attachments: HADOOP-2.2.0-9610.patch, HADOOP-9610.patch There is a missing dependencies in the mr-client pom.xml that is exposed when running a mvn-rpmbuild against system dependencies. Regular mvn build bypasses the issue via its default classpath. patch provided by pmack...@redhat.com -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822339#comment-13822339 ] Hudson commented on MAPREDUCE-5610: --- FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #790 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/790/]) svn merge -c 1541566 FIXES: MAPREDUCE-5610. TestSleepJob fails in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541590) * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobCreator.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java TestSleepJob fails in jdk7 -- Key: MAPREDUCE-5610 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 Project: Hadoop Map/Reduce Issue Type: Test Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 3.0.0, 2.3.0, 0.23.10 Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch In jdk7 tests methods in a class do not run in file order, but rather in random order. TestSleepJob hosts are not initialized and a NullPointerException is thrown unless testRandomLocation was run first. This can be easily seen by running tests individually. org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822416#comment-13822416 ] Hudson commented on MAPREDUCE-5610: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1608 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1608/]) MAPREDUCE-5610. TestSleepJob fails in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541566) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobCreator.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java TestSleepJob fails in jdk7 -- Key: MAPREDUCE-5610 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 Project: Hadoop Map/Reduce Issue Type: Test Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 3.0.0, 2.3.0, 0.23.10 Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch In jdk7 tests methods in a class do not run in file order, but rather in random order. TestSleepJob hosts are not initialized and a NullPointerException is thrown unless testRandomLocation was run first. This can be easily seen by running tests individually. org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5624) move grizzly-test and junit dependencies to test scope
[ https://issues.apache.org/jira/browse/MAPREDUCE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822412#comment-13822412 ] Hudson commented on MAPREDUCE-5624: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1608 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1608/]) MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541702) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-archives/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-extras/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/pom.xml move grizzly-test and junit dependencies to test scope -- Key: MAPREDUCE-5624 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5624 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Steve Loughran Assignee: Ted Yu Attachments: mapreduce-5624.txt stop the the grizzly dependences Junit getting into everything downstream by moving them to test scope -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Enable uber jobs to have multiple reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822422#comment-13822422 ] Hudson commented on MAPREDUCE-5481: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1608 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1608/]) MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541844) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java Enable uber jobs to have multiple reducers --- Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0, 2.3.0 Reporter: Jason Lowe Assignee: Sandy Ryza Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5481-1.patch, MAPREDUCE-5481.patch, MAPREDUCE-5481.patch, syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console This is caused by changes made in MAPREDUCE-434 breaking Uber AMs. The easiest fix is to make similar changes in Uber AMs to those in MAPREDUCE-434 to allow multiple reducers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5431) Missing pom dependency in MR-client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822418#comment-13822418 ] Hudson commented on MAPREDUCE-5431: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1608 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1608/]) MAPREDUCE-5431 Missing pom dependency in MR-client (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541682) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml Missing pom dependency in MR-client --- Key: MAPREDUCE-5431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5431 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Assignee: Timothy St. Clair Labels: maven Fix For: 2.3.0 Attachments: HADOOP-2.2.0-9610.patch, HADOOP-9610.patch There is a missing dependencies in the mr-client pom.xml that is exposed when running a mvn-rpmbuild against system dependencies. Regular mvn build bypasses the issue via its default classpath. patch provided by pmack...@redhat.com -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5481) Enable uber jobs to have multiple reducers
[ https://issues.apache.org/jira/browse/MAPREDUCE-5481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822447#comment-13822447 ] Hudson commented on MAPREDUCE-5481: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1582/]) MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza) (sandy: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541844) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestUberAM.java Enable uber jobs to have multiple reducers --- Key: MAPREDUCE-5481 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5481 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 3.0.0, 2.3.0 Reporter: Jason Lowe Assignee: Sandy Ryza Priority: Blocker Fix For: 2.3.0 Attachments: MAPREDUCE-5481-1.patch, MAPREDUCE-5481.patch, MAPREDUCE-5481.patch, syslog TestUberAM has been timing out on trunk for some time now and surefire then fails the build. I'm not able to reproduce it locally, but the Jenkins builds have been seeing it fairly consistently. See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1529/console This is caused by changes made in MAPREDUCE-434 breaking Uber AMs. The easiest fix is to make similar changes in Uber AMs to those in MAPREDUCE-434 to allow multiple reducers. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5431) Missing pom dependency in MR-client
[ https://issues.apache.org/jira/browse/MAPREDUCE-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822443#comment-13822443 ] Hudson commented on MAPREDUCE-5431: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1582/]) MAPREDUCE-5431 Missing pom dependency in MR-client (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541682) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml Missing pom dependency in MR-client --- Key: MAPREDUCE-5431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5431 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 3.0.0, 2.1.0-beta Reporter: Timothy St. Clair Assignee: Timothy St. Clair Labels: maven Fix For: 2.3.0 Attachments: HADOOP-2.2.0-9610.patch, HADOOP-9610.patch There is a missing dependencies in the mr-client pom.xml that is exposed when running a mvn-rpmbuild against system dependencies. Regular mvn build bypasses the issue via its default classpath. patch provided by pmack...@redhat.com -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822441#comment-13822441 ] Hudson commented on MAPREDUCE-5610: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1582/]) MAPREDUCE-5610. TestSleepJob fails in jdk7. Contributed by Jonathan Eagles (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541566) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/JobCreator.java * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestSleepJob.java TestSleepJob fails in jdk7 -- Key: MAPREDUCE-5610 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 Project: Hadoop Map/Reduce Issue Type: Test Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 3.0.0, 2.3.0, 0.23.10 Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, MAPREDUCE-5610.patch In jdk7 tests methods in a class do not run in file order, but rather in random order. TestSleepJob hosts are not initialized and a NullPointerException is thrown unless testRandomLocation was run first. This can be easily seen by running tests individually. org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5624) move grizzly-test and junit dependencies to test scope
[ https://issues.apache.org/jira/browse/MAPREDUCE-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822437#comment-13822437 ] Hudson commented on MAPREDUCE-5624: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1582/]) MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope (stevel: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1541702) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-mapreduce-project * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-archives/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-datajoin/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-extras/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-gridmix/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-rumen/pom.xml * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/pom.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/pom.xml move grizzly-test and junit dependencies to test scope -- Key: MAPREDUCE-5624 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5624 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build Reporter: Steve Loughran Assignee: Ted Yu Attachments: mapreduce-5624.txt stop the the grizzly dependences Junit getting into everything downstream by moving them to test scope -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5611) CombineFileInputFormat creates more rack-local tasks due to less split location info.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandra Prakash Bhagtani updated MAPREDUCE-5611:
Attachment: CombineFileInputFormat-trunk.patch
attaching patch against trunk
CombineFileInputFormat creates more rack-local tasks due to less split
location info.
-
Key: MAPREDUCE-5611
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5611
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chandra Prakash Bhagtani
Assignee: Chandra Prakash Bhagtani
Attachments: CombineFileInputFormat-trunk.patch
I have come across an issue with CombineFileInputFormat. Actually I ran a
hive query on approx 1.2 GB data with CombineHiveInputFormat which internally
uses CombineFileInputFormat. My cluster size is 9 datanodes and
max.split.size is 256 MB
When I ran this query with replication factor 9, hive consistently creates
all 6 rack-local tasks and with replication factor 3 it creates 5 rack-local
and 1 data local tasks.
When replication factor is 9 (equal to cluster size), all the tasks should
be data-local as each datanode contains all the replicas of the input data,
but that is not happening i.e all the tasks are rack-local.
When I dug into CombineFileInputFormat.java code in getMoreSplits method, I
found the issue with the following snippet (specially in case of higher
replication factor)
{code:title=CombineFileInputFormat.java|borderStyle=solid}
for (IteratorMap.EntryString,
ListOneBlockInfo iter = nodeToBlocks.entrySet().iterator();
iter.hasNext();) {
Map.EntryString, ListOneBlockInfo one = iter.next();
nodes.add(one.getKey());
ListOneBlockInfo blocksInNode = one.getValue();
// for each block, copy it into validBlocks. Delete it from
// blockToNodes so that the same block does not appear in
// two different splits.
for (OneBlockInfo oneblock : blocksInNode) {
if (blockToNodes.containsKey(oneblock)) {
validBlocks.add(oneblock);
blockToNodes.remove(oneblock);
curSplitSize += oneblock.length;
// if the accumulated split size exceeds the maximum, then
// create this split.
if (maxSize != 0 curSplitSize = maxSize) {
// create an input split and add it to the splits array
addCreatedSplit(splits, nodes, validBlocks);
curSplitSize = 0;
validBlocks.clear();
}
}
}
{code}
First node in the map nodeToBlocks has all the replicas of input file, so the
above code creates 6 splits all with only one location. Now if JT doesn't
schedule these tasks on that node, all the tasks will be rack-local, even
though all the other datanodes have all the other replicas.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated MAPREDUCE-5052: --- Attachment: MAPREDUCE-5052.patch Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Attachments: MAPREDUCE-5052.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated MAPREDUCE-5052: --- Target Version/s: 2.3.0 Status: Patch Available (was: Open) updated startTime to submitTime on RM WebUI page add submitTime column on the JobHistory page Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Attachments: MAPREDUCE-5052.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5616) MR Client-AppMaster RPC max retries on socket timeout is too high.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-5616: - Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Target Version/s: 3.0.0, 2.3.0 (was: 3.0.0, 2.2.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review, Bikas. I've committed this to trunk and branch-2. MR Client-AppMaster RPC max retries on socket timeout is too high. -- Key: MAPREDUCE-5616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5616 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.2.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 2.3.0 Attachments: MAPREDUCE-5616.1.patch MAPREDUCE-3811 introduced a separate config key for overriding the max retries applied to RPC connections from the MapReduce Client to the MapReduce Application Master. This was done to make failover from the AM to the MapReduce History Server faster in the event that the AM completes while the client thinks it's still running. However, the RPC client uses a separate setting for socket timeouts, and this one is not overridden. The default for this is 45 retries with a 20-second timeout on each retry. This means that in environments subject to connection timeout instead of connection refused, the client waits 15 minutes for failover. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5616) MR Client-AppMaster RPC max retries on socket timeout is too high.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822707#comment-13822707 ] Hudson commented on MAPREDUCE-5616: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4739 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4739/]) MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too high. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1542001) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java MR Client-AppMaster RPC max retries on socket timeout is too high. -- Key: MAPREDUCE-5616 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5616 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.2.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 3.0.0, 2.3.0 Attachments: MAPREDUCE-5616.1.patch MAPREDUCE-3811 introduced a separate config key for overriding the max retries applied to RPC connections from the MapReduce Client to the MapReduce Application Master. This was done to make failover from the AM to the MapReduce History Server faster in the event that the AM completes while the client thinks it's still running. However, the RPC client uses a separate setting for socket timeouts, and this one is not overridden. The default for this is 45 retries with a 20-second timeout on each retry. This means that in environments subject to connection timeout instead of connection refused, the client waits 15 minutes for failover. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[
https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822714#comment-13822714
]
Hadoop QA commented on MAPREDUCE-5052:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613882/MAPREDUCE-5052.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
org.apache.hadoop.mapreduce.v2.jobhistory.TestFileNameIndexUtils
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4199//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4199//console
This message is automatically generated.
Job History UI and web services confusing job start time and job submit time
Key: MAPREDUCE-5052
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, webapps
Affects Versions: 0.23.6
Reporter: Kendall Thrapp
Assignee: Chen He
Priority: Critical
Attachments: MAPREDUCE-5052.patch
The Start Time column shown on Job History server's main webpage
(http://host:port/jobhistory) is actually showing the *submit* time for
jobs. However, when you drill down to an individual job's page, there the
Start Time really does refer to when the job actually started.
This also true for the web services REST API, where the Jobs listing returns
the submit times as startTime, but the single Job API returns the start
time as startTime.
The two different times being referred to by the same name is confusing.
However, it is useful to have both times, as the difference between the
submit time and start time can show how long a job was stuck waiting in a
queue. The column on the main job history page should be changed to Submit
Time and the individual job's page should show both the submit time and
start time. The web services REST API should be updated with these changes
as well.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated MAPREDUCE-5052: --- Status: Open (was: Patch Available) Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Attachments: MAPREDUCE-5052.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13822833#comment-13822833 ] Chen He commented on MAPREDUCE-5052: I will figure out the test failures and resubmit today. Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Attachments: MAPREDUCE-5052.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Status: Patch Available (was: Open)
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-trunk.patch
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5627) TaskLogServlet could not get syslog
yangjun created MAPREDUCE-5627:
--
Summary: TaskLogServlet could not get syslog
Key: MAPREDUCE-5627
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5627
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.1
Environment: Linux version 2.6.18-238.9.1.el5
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
hadoop-1.2.1
Reporter: yangjun
Priority: Minor
When multiply tasks use one jvm and generated logs.
eg.
./attempt_201211220735_0001_m_00_0:
log.index
./attempt_201211220735_0001_m_01_0:
log.index
./attempt_201211220735_0001_m_02_0:
log.index stderr stdout syslog
get from http://:50060/tasklog?attemptid=
attempt_201211220735_0001_m_00_0
could get stderr,stdout,but not the others,include syslog.
see TaskLogServlet.haveTaskLog() method, not check from local log.index, but
check the original path.
resolve:
modify TaskLogServlet haveTaskLog method
private boolean haveTaskLog(TaskAttemptID taskId, boolean isCleanup,
TaskLog.LogName type) throws IOException {
File f = TaskLog.getTaskLogFile(taskId, isCleanup, type);
if (f.exists() f.canRead()) {
return true;
} else {
File indexFile = TaskLog.getIndexFile(taskId, isCleanup);
if (!indexFile.exists()) {
return false;
}
BufferedReader fis;
try {
fis = new BufferedReader(new InputStreamReader(
SecureIOUtils.openForRead(indexFile,
TaskLog.obtainLogDirOwner(taskId;
} catch (FileNotFoundException ex) {
LOG.warn(Index file for the log of + taskId
+ does not exist.);
// Assume no task reuse is used and files exist on attemptdir
StringBuffer input = new StringBuffer();
input.append(LogFileDetail.LOCATION
+ TaskLog.getAttemptDir(taskId, isCleanup) + \n);
for (LogName logName : TaskLog.LOGS_TRACKED_BY_INDEX_FILES) {
input.append(logName + :0 -1\n);
}
fis = new BufferedReader(new StringReader(input.toString()));
}
try {
String str = fis.readLine();
if (str == null) { // thefile doesn't have anything
throw new IOException(Index file for the log of + taskId
+ is empty.);
}
String loc = str.substring(str.indexOf(LogFileDetail.LOCATION)
+ LogFileDetail.LOCATION.length());
File tf = new File(loc, type.toString());
return tf.exists() tf.canRead();
} finally {
if (fis != null)
fis.close();
}
}
}
workaround:
url add filter=SYSLOG could print syslog also.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5626) TaskLogServlet could not get syslog
yangjun created MAPREDUCE-5626:
--
Summary: TaskLogServlet could not get syslog
Key: MAPREDUCE-5626
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5626
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.1
Environment: Linux version 2.6.18-238.9.1.el5
Java(TM) SE Runtime Environment (build 1.6.0_43-b01)
hadoop-1.2.1
Reporter: yangjun
Priority: Minor
When multiply tasks use one jvm and generated logs.
eg.
./attempt_201211220735_0001_m_00_0:
log.index
./attempt_201211220735_0001_m_01_0:
log.index
./attempt_201211220735_0001_m_02_0:
log.index stderr stdout syslog
get from http://:50060/tasklog?attemptid=
attempt_201211220735_0001_m_00_0
could get stderr,stdout,but not the others,include syslog.
see TaskLogServlet.haveTaskLog() method, not check from local log.index, but
check the original path.
resolve:
modify TaskLogServlet haveTaskLog method
private boolean haveTaskLog(TaskAttemptID taskId, boolean isCleanup,
TaskLog.LogName type) throws IOException {
File f = TaskLog.getTaskLogFile(taskId, isCleanup, type);
if (f.exists() f.canRead()) {
return true;
} else {
File indexFile = TaskLog.getIndexFile(taskId, isCleanup);
if (!indexFile.exists()) {
return false;
}
BufferedReader fis;
try {
fis = new BufferedReader(new InputStreamReader(
SecureIOUtils.openForRead(indexFile,
TaskLog.obtainLogDirOwner(taskId;
} catch (FileNotFoundException ex) {
LOG.warn(Index file for the log of + taskId
+ does not exist.);
// Assume no task reuse is used and files exist on attemptdir
StringBuffer input = new StringBuffer();
input.append(LogFileDetail.LOCATION
+ TaskLog.getAttemptDir(taskId, isCleanup) + \n);
for (LogName logName : TaskLog.LOGS_TRACKED_BY_INDEX_FILES) {
input.append(logName + :0 -1\n);
}
fis = new BufferedReader(new StringReader(input.toString()));
}
try {
String str = fis.readLine();
if (str == null) { // thefile doesn't have anything
throw new IOException(Index file for the log of + taskId
+ is empty.);
}
String loc = str.substring(str.indexOf(LogFileDetail.LOCATION)
+ LogFileDetail.LOCATION.length());
File tf = new File(loc, type.toString());
return tf.exists() tf.canRead();
} finally {
if (fis != null)
fis.close();
}
}
}
workaround:
url add filter=SYSLOG could print syslog also.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823238#comment-13823238
]
Hadoop QA commented on MAPREDUCE-3310:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613990/MAPREDUCE-3310-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1546 javac
compiler warnings (more than the trunk's current 1545 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestJobCleanup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4200//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4200//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4200//console
This message is automatically generated.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5628) Tracking ids only for HDFS tokens should be included in jobconf
Karthik Kambatla created MAPREDUCE-5628: --- Summary: Tracking ids only for HDFS tokens should be included in jobconf Key: MAPREDUCE-5628 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5628 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 2.2.0 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Critical MAPREDUCE-5379 adds the ability to track HDFS accesses of an MR job by adding the tracking-ids from all the tokens to the jobconf. However, only HDFS delegation tokens have a tracking-id. Trying to fetch tracking-ids from other tokens can lead to an NPE. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-trunk.patch
test failure seems unrelated.
uploading patch that fixes the javac warning (was in a testcase)
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-branch-1.patch
patch for branch-1
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-3310:
--
Attachment: MAPREDUCE-3310-trunk.patch
reuploading patch for trunk so jenkins do not pickup the branch-1 patch.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823318#comment-13823318
]
Hadoop QA commented on MAPREDUCE-3310:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614008/MAPREDUCE-3310-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestJobCleanup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4201//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4201//console
This message is automatically generated.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13823331#comment-13823331
]
Hadoop QA commented on MAPREDUCE-3310:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614010/MAPREDUCE-3310-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4202//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4202//console
This message is automatically generated.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
