[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13547831#comment-13547831 ] Kihwal Lee commented on MAPREDUCE-1700: --- +1 The patch looks good. I hope people try this with many different use cases. User supplied dependencies may conflict with MapReduce system JARs -- Key: MAPREDUCE-1700 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081]. The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548372#comment-13548372 ] Hudson commented on MAPREDUCE-4278: --- Integrated in Hadoop-Yarn-trunk #91 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/91/]) MAPREDUCE-4278. Cannot run two local jobs in parallel from the same gateway. Contributed by Sandy Ryza. (Revision 1430363) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430363 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobID.java cannot run two local jobs in parallel from the same gateway. Key: MAPREDUCE-4278 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Araceli Henley Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha Attachments: MAPREDUCE-4278-2-branch1.patch, MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch I cannot run two local mode jobs from Pig in parallel from the same gateway, this is a typical use case. If I re-run the tests sequentially, then the test pass. This seems to be a problem from Hadoop. Additionally, the pig harness, expects to be able to run Pig-version-undertest against Pig-version-stable from the same gateway. To replicate the error: I have two clusters running from the same gateway. If I run the Pig regression suites nightly.conf in local mode in paralell - once on each cluster. Conflicts in M/R local mode result in failures in the tests. ERROR1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) at org.apache.hadoop.mapred.Task.done(Task.java:875) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) --- ERROR2: 2012-05-17 20:25:36,762 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache. hadoop.util.LinuxResourceCalculatorPlugin@ffa490e 2012-05-17 20:25:36,837 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java :153) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm at.java:106) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) 2012-05-17 20:25:41,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4520) Add experimental support for MR AM to schedule CPUs along-with memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548374#comment-13548374 ] Hudson commented on MAPREDUCE-4520: --- Integrated in Hadoop-Yarn-trunk #91 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/91/]) MAPREDUCE-4520. Added support for MapReduce applications to request for CPU cores along-with memory post YARN-2. Contributed by Arun C. Murthy. (Revision 1430688) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java Add experimental support for MR AM to schedule CPUs along-with memory - Key: MAPREDUCE-4520 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4520 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4520.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4810) Add admin command options for ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548376#comment-13548376 ] Hudson commented on MAPREDUCE-4810: --- Integrated in Hadoop-Yarn-trunk #91 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/91/]) MAPREDUCE-4810. Added new admin command options for MR AM. Contributed by Jerry Chen. (Revision 1430707) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430707 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java Add admin command options for ApplicationMaster --- Key: MAPREDUCE-4810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4810 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Jason Lowe Assignee: Jerry Chen Priority: Minor Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4810.patch, MAPREDUCE-4810.patch It would be nice if the MR ApplicationMaster had the notion of admin options in addition to the existing user options much like we have for map and reduce tasks, e.g.: mapreduce.admin.map.child.java.opts vs. mapreduce.map.java.opts. This allows site-wide configuration options for MR AMs but still allows a user to easily override the heap size of the AM without worrying about dropping other admin-specified options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548479#comment-13548479 ] Hudson commented on MAPREDUCE-4278: --- Integrated in Hadoop-Hdfs-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1280/]) MAPREDUCE-4278. Cannot run two local jobs in parallel from the same gateway. Contributed by Sandy Ryza. (Revision 1430363) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430363 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobID.java cannot run two local jobs in parallel from the same gateway. Key: MAPREDUCE-4278 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Araceli Henley Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha Attachments: MAPREDUCE-4278-2-branch1.patch, MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch I cannot run two local mode jobs from Pig in parallel from the same gateway, this is a typical use case. If I re-run the tests sequentially, then the test pass. This seems to be a problem from Hadoop. Additionally, the pig harness, expects to be able to run Pig-version-undertest against Pig-version-stable from the same gateway. To replicate the error: I have two clusters running from the same gateway. If I run the Pig regression suites nightly.conf in local mode in paralell - once on each cluster. Conflicts in M/R local mode result in failures in the tests. ERROR1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) at org.apache.hadoop.mapred.Task.done(Task.java:875) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) --- ERROR2: 2012-05-17 20:25:36,762 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache. hadoop.util.LinuxResourceCalculatorPlugin@ffa490e 2012-05-17 20:25:36,837 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java :153) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm at.java:106) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) 2012-05-17 20:25:41,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4520) Add experimental support for MR AM to schedule CPUs along-with memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548481#comment-13548481 ] Hudson commented on MAPREDUCE-4520: --- Integrated in Hadoop-Hdfs-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1280/]) MAPREDUCE-4520. Added support for MapReduce applications to request for CPU cores along-with memory post YARN-2. Contributed by Arun C. Murthy. (Revision 1430688) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java Add experimental support for MR AM to schedule CPUs along-with memory - Key: MAPREDUCE-4520 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4520 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4520.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4810) Add admin command options for ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548483#comment-13548483 ] Hudson commented on MAPREDUCE-4810: --- Integrated in Hadoop-Hdfs-trunk #1280 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1280/]) MAPREDUCE-4810. Added new admin command options for MR AM. Contributed by Jerry Chen. (Revision 1430707) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430707 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java Add admin command options for ApplicationMaster --- Key: MAPREDUCE-4810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4810 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Jason Lowe Assignee: Jerry Chen Priority: Minor Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4810.patch, MAPREDUCE-4810.patch It would be nice if the MR ApplicationMaster had the notion of admin options in addition to the existing user options much like we have for map and reduce tasks, e.g.: mapreduce.admin.map.child.java.opts vs. mapreduce.map.java.opts. This allows site-wide configuration options for MR AMs but still allows a user to easily override the heap size of the AM without worrying about dropping other admin-specified options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548495#comment-13548495 ] Hudson commented on MAPREDUCE-4278: --- Integrated in Hadoop-Mapreduce-trunk #1308 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1308/]) MAPREDUCE-4278. Cannot run two local jobs in parallel from the same gateway. Contributed by Sandy Ryza. (Revision 1430363) Result = FAILURE tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430363 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalJobRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobID.java cannot run two local jobs in parallel from the same gateway. Key: MAPREDUCE-4278 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.20.205.0 Reporter: Araceli Henley Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha Attachments: MAPREDUCE-4278-2-branch1.patch, MAPREDUCE-4278-3-branch1.patch, MAPREDUCE-4278-branch1.patch, MAPREDUCE-4278-trunk.patch, MAPREDUCE-4278-trunk.patch I cannot run two local mode jobs from Pig in parallel from the same gateway, this is a typical use case. If I re-run the tests sequentially, then the test pass. This seems to be a problem from Hadoop. Additionally, the pig harness, expects to be able to run Pig-version-undertest against Pig-version-stable from the same gateway. To replicate the error: I have two clusters running from the same gateway. If I run the Pig regression suites nightly.conf in local mode in paralell - once on each cluster. Conflicts in M/R local mode result in failures in the tests. ERROR1: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find output/file.out in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56) at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944) at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924) at org.apache.hadoop.mapred.Task.done(Task.java:875) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374) --- ERROR2: 2012-05-17 20:25:36,762 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001 2012-05-17 20:25:36,778 [Thread-3] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache. hadoop.util.LinuxResourceCalculatorPlugin@ffa490e 2012-05-17 20:25:36,837 [Thread-3] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001 java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java :153) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm at.java:106) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.init(MapTask.java:489) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) 2012-05-17 20:25:41,291 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4520) Add experimental support for MR AM to schedule CPUs along-with memory
[ https://issues.apache.org/jira/browse/MAPREDUCE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548497#comment-13548497 ] Hudson commented on MAPREDUCE-4520: --- Integrated in Hadoop-Mapreduce-trunk #1308 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1308/]) MAPREDUCE-4520. Added support for MapReduce applications to request for CPU cores along-with memory post YARN-2. Contributed by Arun C. Murthy. (Revision 1430688) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430688 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java Add experimental support for MR AM to schedule CPUs along-with memory - Key: MAPREDUCE-4520 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4520 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4520.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4810) Add admin command options for ApplicationMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-4810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548499#comment-13548499 ] Hudson commented on MAPREDUCE-4810: --- Integrated in Hadoop-Mapreduce-trunk #1308 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1308/]) MAPREDUCE-4810. Added new admin command options for MR AM. Contributed by Jerry Chen. (Revision 1430707) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430707 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestYARNRunner.java Add admin command options for ApplicationMaster --- Key: MAPREDUCE-4810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4810 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Jason Lowe Assignee: Jerry Chen Priority: Minor Fix For: 2.0.3-alpha Attachments: MAPREDUCE-4810.patch, MAPREDUCE-4810.patch It would be nice if the MR ApplicationMaster had the notion of admin options in addition to the existing user options much like we have for map and reduce tasks, e.g.: mapreduce.admin.map.child.java.opts vs. mapreduce.map.java.opts. This allows site-wide configuration options for MR AMs but still allows a user to easily override the heap size of the AM without worrying about dropping other admin-specified options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4892) CombineFileInputFormat node input split can be skewed on small clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-4892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4892: --- Status: Open (was: Patch Available) I can think of a case where this breaks locality: Nodes: N1, N2, N3 Block allocation: 9 on N1, 1 on N2, 8 on N3 Rack: default Num Blocks needed: 9 (numMaps) Blocks(N1) = Blocks(N2) union Blocks(N3) Average # blocks per node determined by patch = 3 (9 numMaps / 3 nodes) Assignment before patch: 9 splits on N1. (suboptimal spread, but fully local) Final assignment after patch: 3 on N1, 1 on N2, 3 on N3 and the final 2 on rack. Assuming my analysis is correct: I think we should first note the total number of max possible local splits (9 here) and start giving one split per node, circle through all nodes and keep doing it while all max possible splits are created. This loop will end up being similar to the rack loop that we have after the node local assignments. CombineFileInputFormat node input split can be skewed on small clusters --- Key: MAPREDUCE-4892 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4892 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 3.0.0 Attachments: MAPREDUCE-4892.1.patch The CombineFileInputFormat split generation logic tries to group blocks by node in order to create splits. It iterates through the nodes and creates splits on them until there aren't enough blocks left on a node that can be grouped into a valid split. If the first few nodes have a lot of blocks on them then they can end up getting a disproportionately large share of the total number of splits created. This can result in poor locality of maps. This problem is likely to happen on small clusters where its easier to create a skew in the distribution of blocks on nodes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4850) Job recovery may fail if staging directory has been deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved MAPREDUCE-4850. -- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I ran test-patch and it came back clean. I just committed this. Job recovery may fail if staging directory has been deleted --- Key: MAPREDUCE-4850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4850 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Tom White Assignee: Tom White Fix For: 1.2.0 Attachments: MAPREDUCE-4850.patch, MAPREDUCE-4850.patch The job staging directory is deleted in the job cleanup task, which happens before the job-info file is deleted from the system directory (by the JobInProgress garbageCollect() method). If the JT shuts down between these two operations, then when the JT restarts and tries to recover the job, it fails since the job.xml and splits are no longer available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4893) MR AppMaster can do sub-optimal assignment of containers to map tasks leading to poor node locality
[
https://issues.apache.org/jira/browse/MAPREDUCE-4893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-4893:
---
Status: Open (was: Patch Available)
Can you please upmerge the patch to latest code, the test isn't applying
anymore at all.
Patch looks good overall, some minor nits:
- +792 after patch: The condition should be a simple else: {{ else if
(!PRIORITY_MAP.equals(priority)) { }}
- +1024 after patch is dead code: LinkedListTaskAttemptId list =
mapsHostMapping.get(host);
Will review test changes after your update.
MR AppMaster can do sub-optimal assignment of containers to map tasks leading
to poor node locality
---
Key: MAPREDUCE-4893
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4893
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
Fix For: 3.0.0
Attachments: MAPREDUCE-4893.1.patch
Say the MR AppMaster asks the RM for 3 containers on nodes n1, n2 and n3.
There are 10 node n1-n10 in the same rack. The RM can give it allocated
containers in the list order n5, n2, n1. The way AM map-container assignment
happens, the AM will try to assign node local maps to n5, failing which it
will assign rack local maps to n5. These rack local maps could be node local
on n2 and n1 and would have been assigned to containers on n1 and n2 if the
AM had not made an early rack local match for them on n5. This can lead to
poor locality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548617#comment-13548617 ] Arun C Murthy commented on MAPREDUCE-4305: -- Karthik Mayank - CS already has delay scheduling built-in, one area for improvement is to backport something like YARN-80 to branch-1. Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1700: - Resolution: Fixed Fix Version/s: 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks to everyone who provided feedback. User supplied dependencies may conflict with MapReduce system JARs -- Key: MAPREDUCE-1700 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081]. The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548644#comment-13548644 ] Hudson commented on MAPREDUCE-1700: --- Integrated in Hadoop-trunk-Commit #3203 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3203/]) MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce system JARs. (Revision 1430929) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1430929 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml User supplied dependencies may conflict with MapReduce system JARs -- Key: MAPREDUCE-1700 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Reporter: Tom White Assignee: Tom White Fix For: 2.0.3-alpha Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081]. The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4921:
---
Attachment: MAPREDUCE-4921.patch
Patch universally applies to trunk - 23. I moved {{TestYARNRunner}} so I
could change a private {{YARNRunner}} method to package scope instead of public.
JobClient should acquire HS token with RM principal
---
Key: MAPREDUCE-4921
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch
The job client may acquire a history server token during job submission. The
renewer is specified in a config value that the user must supply (for new
api, a bit different for old api). If this value is not the RM's principal,
then the RM cannot renew the token and long running jobs will fail. Since
the token is implicitly acquired for the job, the HS token's renewer should
always be the RM's principal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4921: --- Status: Patch Available (was: Open) JobClient should acquire HS token with RM principal --- Key: MAPREDUCE-4921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch The job client may acquire a history server token during job submission. The renewer is specified in a config value that the user must supply (for new api, a bit different for old api). If this value is not the RM's principal, then the RM cannot renew the token and long running jobs will fail. Since the token is implicitly acquired for the job, the HS token's renewer should always be the RM's principal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548721#comment-13548721 ] Tom White commented on MAPREDUCE-4808: -- Asokan, thanks for addressing my feedback. The latest patch looks good to me. Allow reduce-side merge to be pluggable --- Key: MAPREDUCE-4808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: 2.0.3-alpha Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf Allow reduce-side merge to be pluggable for MAPREDUCE-2454 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548724#comment-13548724
]
Hadoop QA commented on MAPREDUCE-4921:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563966/MAPREDUCE-4921.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3217//console
This message is automatically generated.
JobClient should acquire HS token with RM principal
---
Key: MAPREDUCE-4921
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch
The job client may acquire a history server token during job submission. The
renewer is specified in a config value that the user must supply (for new
api, a bit different for old api). If this value is not the RM's principal,
then the RM cannot renew the token and long running jobs will fail. Since
the token is implicitly acquired for the job, the HS token's renewer should
always be the RM's principal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4928) Use token request messages defined in hadoop common
Suresh Srinivas created MAPREDUCE-4928: -- Summary: Use token request messages defined in hadoop common Key: MAPREDUCE-4928 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4928 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, security Reporter: Suresh Srinivas Assignee: Suresh Srinivas MapReduce changes related to HADOOP-9192 to reuse the protobuf messages defined in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4928) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4928: --- Attachment: MR-4928.patch Use token request messages defined in hadoop common Key: MAPREDUCE-4928 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4928 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, security Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: MR-4928.patch MapReduce changes related to HADOOP-9192 to reuse the protobuf messages defined in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4928) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4928: --- Attachment: MR-4928.patch Use token request messages defined in hadoop common Key: MAPREDUCE-4928 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4928 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, security Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: MR-4928.patch, MR-4928.patch MapReduce changes related to HADOOP-9192 to reuse the protobuf messages defined in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4928) Use token request messages defined in hadoop common
[ https://issues.apache.org/jira/browse/MAPREDUCE-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated MAPREDUCE-4928: --- Hadoop Flags: Incompatible change Use token request messages defined in hadoop common Key: MAPREDUCE-4928 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4928 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, security Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: MR-4928.patch, MR-4928.patch MapReduce changes related to HADOOP-9192 to reuse the protobuf messages defined in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548825#comment-13548825 ] Sandy Ryza commented on MAPREDUCE-4896: --- The test requires a running RM ask for the information. I noticed that TestJobClient and TestMRJobClient, which test other mapreduce command line tools, are both @Ignored. Are these ignored because they've gone stale, or because using MiniMRCluster makes tests run too slow? If the latter is the case, is there another good way to write a test for this? mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
Sandy Ryza created MAPREDUCE-4929: - Summary: mapreduce.task.timeout is ignored Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Sandy Ryza In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned MAPREDUCE-4929: - Assignee: Sandy Ryza mapreduce.task.timeout is ignored - Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Sandy Ryza Assignee: Sandy Ryza In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4929: -- Attachment: MAPREDUCE-4929-branch-1.patch mapreduce.task.timeout is ignored - Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4929-branch-1.patch In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4929: -- Target Version/s: 1.1.2 Affects Version/s: 1.1.1 Status: Patch Available (was: Open) mapreduce.task.timeout is ignored - Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4929-branch-1.patch In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[
https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548855#comment-13548855
]
Hadoop QA commented on MAPREDUCE-4929:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563998/MAPREDUCE-4929-branch-1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3218//console
This message is automatically generated.
mapreduce.task.timeout is ignored
-
Key: MAPREDUCE-4929
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4929-branch-1.patch
In MR1, only mapred.task.timeout works. Both should be made to work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548858#comment-13548858 ] Karthik Kambatla commented on MAPREDUCE-4929: - Sandy, just curious, what should the behavior be if a user sets both but to different values? Do we give higher priority to mapred/mapreduce? mapreduce.task.timeout is ignored - Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4929-branch-1.patch In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2931) CLONE - LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-2931: -- Assignee: Sandy Ryza (was: Aaron Kimball) Status: Patch Available (was: Open) CLONE - LocalJobRunner should support parallel mapper execution --- Key: MAPREDUCE-2931 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2931 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Forest Tan Assignee: Sandy Ryza Attachments: MAPREDUCE-1367-branch1.patch The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs. It is necessary to patch back MAPREDUCE-1367 into Hadoop 0.20.X version. Also, MapReduce-434 should be submitted together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2931) CLONE - LocalJobRunner should support parallel mapper execution
[ https://issues.apache.org/jira/browse/MAPREDUCE-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-2931: -- Attachment: MAPREDUCE-1367-branch1.patch CLONE - LocalJobRunner should support parallel mapper execution --- Key: MAPREDUCE-2931 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2931 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Forest Tan Assignee: Sandy Ryza Attachments: MAPREDUCE-1367-branch1.patch The LocalJobRunner currently supports only a single execution thread. Given the prevalence of multi-core CPUs, it makes sense to allow users to run multiple tasks in parallel for improved performance on small (local-only) jobs. It is necessary to patch back MAPREDUCE-1367 into Hadoop 0.20.X version. Also, MapReduce-434 should be submitted together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4929) mapreduce.task.timeout is ignored
[ https://issues.apache.org/jira/browse/MAPREDUCE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548868#comment-13548868 ] Sandy Ryza commented on MAPREDUCE-4929: --- As mapred.task.timeout is the one mentioned in the documentation, I made it so that it gets preference. Do you think that's the right behavior? mapreduce.task.timeout is ignored - Key: MAPREDUCE-4929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4929 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4929-branch-1.patch In MR1, only mapred.task.timeout works. Both should be made to work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548867#comment-13548867 ] Mayank Bansal commented on MAPREDUCE-4305: -- Hi Arun, In Hadoop-1 we have something called scheduling opportunities counter for a job which gets incremented every time is get the opportunity to get scheduled. In Yarn-80 we are using the same counter for delaying the schedule for Node. In this patch I used the scheduling opportunity counter as well as time outs for staging the jobs for different scheduling categories. Initially jobs will be eligible to schedule only on node local and after the (scheduling opportunity are greater then the number of nodes in cluster or timeout for node which ever comes first) will be graduated for Next Level which is Rack. Again Job will be waiting for (scheduling opportunity are greater then the number of nodes in cluster or timeout for node+ rack which ever comes first) And will be graduated for the next level which is off rack. And once the job is off-rack and it will scheduled immediately based on prior logic. Actually this approach is the combination of Yarn-80 and Fair scheduler delay scheduling algo which gives us the flexibility of staging the jobs between different levels and the same time using the scheduling opportunity counter which was already there. Please review the approach and let me know if this needs some improvement. Thanks, Mayank Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2931) CLONE - LocalJobRunner should support parallel mapper execution
[
https://issues.apache.org/jira/browse/MAPREDUCE-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548871#comment-13548871
]
Hadoop QA commented on MAPREDUCE-2931:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564000/MAPREDUCE-1367-branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3219//console
This message is automatically generated.
CLONE - LocalJobRunner should support parallel mapper execution
---
Key: MAPREDUCE-2931
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2931
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Forest Tan
Assignee: Sandy Ryza
Attachments: MAPREDUCE-1367-branch1.patch
The LocalJobRunner currently supports only a single execution thread. Given
the prevalence of multi-core CPUs, it makes sense to allow users to run
multiple tasks in parallel for improved performance on small (local-only)
jobs.
It is necessary to patch back MAPREDUCE-1367 into Hadoop 0.20.X version.
Also, MapReduce-434 should be submitted together.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4315) jobhistory.jsp throws 500 when a .txt file is found in /done
[ https://issues.apache.org/jira/browse/MAPREDUCE-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4315: -- Attachment: MAPREDUCE-4315.patch jobhistory.jsp throws 500 when a .txt file is found in /done Key: MAPREDUCE-4315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4315 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.20.2 Reporter: Alexander Alten-Lorenz Attachments: MAPREDUCE-4315.patch, MAPREDUCE-4315.patch if a .txt file located in /done the parser throws an 500 error. Trace: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:295) at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:279) at java.util.Arrays.mergeSort(Arrays.java:1270) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.sort(Arrays.java:1210) at org.apache.hadoop.mapred.jobhistory_jsp._jspService(jobhistory_jsp.java:279) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:864) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Reproduce: cd ../done touch test.txt reload jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4315) jobhistory.jsp throws 500 when a .txt file is found in /done
[ https://issues.apache.org/jira/browse/MAPREDUCE-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza reassigned MAPREDUCE-4315: - Assignee: Sandy Ryza jobhistory.jsp throws 500 when a .txt file is found in /done Key: MAPREDUCE-4315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4315 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.20.2 Reporter: Alexander Alten-Lorenz Assignee: Sandy Ryza Attachments: MAPREDUCE-4315.patch, MAPREDUCE-4315.patch if a .txt file located in /done the parser throws an 500 error. Trace: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:295) at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:279) at java.util.Arrays.mergeSort(Arrays.java:1270) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.sort(Arrays.java:1210) at org.apache.hadoop.mapred.jobhistory_jsp._jspService(jobhistory_jsp.java:279) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:864) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Reproduce: cd ../done touch test.txt reload jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4315) jobhistory.jsp throws 500 when a .txt file is found in /done
[
https://issues.apache.org/jira/browse/MAPREDUCE-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548891#comment-13548891
]
Hadoop QA commented on MAPREDUCE-4315:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564004/MAPREDUCE-4315.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3220//console
This message is automatically generated.
jobhistory.jsp throws 500 when a .txt file is found in /done
Key: MAPREDUCE-4315
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4315
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 0.20.2
Reporter: Alexander Alten-Lorenz
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4315.patch, MAPREDUCE-4315.patch
if a .txt file located in /done the parser throws an 500 error.
Trace:
java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:295)
at
org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:279)
at java.util.Arrays.mergeSort(Arrays.java:1270)
at java.util.Arrays.mergeSort(Arrays.java:1282)
at java.util.Arrays.mergeSort(Arrays.java:1282)
at java.util.Arrays.mergeSort(Arrays.java:1282)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.mergeSort(Arrays.java:1282)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.sort(Arrays.java:1210)
at
org.apache.hadoop.mapred.jobhistory_jsp._jspService(jobhistory_jsp.java:279)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:864)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Reproduce:
cd ../done
touch test.txt
reload jobhistory
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548987#comment-13548987 ] Karthik Kambatla commented on MAPREDUCE-4305: - Thanks Arun. My understanding is that Mayank's patch is mostly just backporting YARN-80 to MR1, along with other MR1 specific changes. Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4921: --- Status: Open (was: Patch Available) JobClient should acquire HS token with RM principal --- Key: MAPREDUCE-4921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch The job client may acquire a history server token during job submission. The renewer is specified in a config value that the user must supply (for new api, a bit different for old api). If this value is not the RM's principal, then the RM cannot renew the token and long running jobs will fail. Since the token is implicitly acquired for the job, the HS token's renewer should always be the RM's principal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4921: --- Attachment: MAPREDUCE-4921.branch-23.patch JobClient should acquire HS token with RM principal --- Key: MAPREDUCE-4921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch The job client may acquire a history server token during job submission. The renewer is specified in a config value that the user must supply (for new api, a bit different for old api). If this value is not the RM's principal, then the RM cannot renew the token and long running jobs will fail. Since the token is implicitly acquired for the job, the HS token's renewer should always be the RM's principal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548995#comment-13548995 ] Mayank Bansal commented on MAPREDUCE-4305: -- HI Karthik, There is more to it then YARN-80, Please followup with my previous comments. Its combination of YARN-80 and part of fair scheduler delay scheduling with timeouts. Thanks, Mayank Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4921: --- Status: Patch Available (was: Open) JobClient should acquire HS token with RM principal --- Key: MAPREDUCE-4921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch, MAPREDUCE-4921.patch The job client may acquire a history server token during job submission. The renewer is specified in a config value that the user must supply (for new api, a bit different for old api). If this value is not the RM's principal, then the RM cannot renew the token and long running jobs will fail. Since the token is implicitly acquired for the job, the HS token's renewer should always be the RM's principal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated MAPREDUCE-4921: --- Attachment: MAPREDUCE-4921.patch JobClient should acquire HS token with RM principal --- Key: MAPREDUCE-4921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch, MAPREDUCE-4921.patch The job client may acquire a history server token during job submission. The renewer is specified in a config value that the user must supply (for new api, a bit different for old api). If this value is not the RM's principal, then the RM cannot renew the token and long running jobs will fail. Since the token is implicitly acquired for the job, the HS token's renewer should always be the RM's principal. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549008#comment-13549008 ] Karthik Kambatla commented on MAPREDUCE-4305: - Didn't mean to undermine the patch's scope, completely agree it is a combination of YARN-80 and delay scheduling with timeouts from FS. Personally, I like this approach better, may be we can augment the one in YARN where applicable. Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549073#comment-13549073
]
Hadoop QA commented on MAPREDUCE-4921:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564040/MAPREDUCE-4921.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3221//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3221//console
This message is automatically generated.
JobClient should acquire HS token with RM principal
---
Key: MAPREDUCE-4921
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-4921.branch-23.patch,
MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch, MAPREDUCE-4921.patch
The job client may acquire a history server token during job submission. The
renewer is specified in a config value that the user must supply (for new
api, a bit different for old api). If this value is not the RM's principal,
then the RM cannot renew the token and long running jobs will fail. Since
the token is implicitly acquired for the job, the HS token's renewer should
always be the RM's principal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4848) TaskAttemptContext cast error during AM recovery
[
https://issues.apache.org/jira/browse/MAPREDUCE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549100#comment-13549100
]
Jason Lowe commented on MAPREDUCE-4848:
---
+1 lgtm.
TaskAttemptContext cast error during AM recovery
Key: MAPREDUCE-4848
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4848
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 0.23.4
Reporter: Jason Lowe
Assignee: Jerry Chen
Fix For: trunk
Attachments: MAPREDUCE-4848.patch
Recently saw an AM that failed and tried to recover, but the subsequent
attempt quickly exited with its own failure during recovery:
{noformat}
2012-12-05 02:33:36,752 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ClassCastException:
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl cannot be cast to
org.apache.hadoop.mapred.TaskAttemptContext
at
org.apache.hadoop.mapred.OutputCommitter.recoverTask(OutputCommitter.java:284)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.handle(RecoveryService.java:361)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1211)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1177)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:958)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:926)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:918)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
2012-12-05 02:33:36,752 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{noformat}
The RM then launched a third AM attempt which succeeded. The third attempt
saw basically no progress after parsing the history file from the second
attempt and ran the job again from scratch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549108#comment-13549108 ] Sandy Ryza commented on MAPREDUCE-4907: --- Attached patch for trunk TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4907: -- Attachment: MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549137#comment-13549137
]
Hadoop QA commented on MAPREDUCE-4907:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564046/MAPREDUCE-4907-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 2019 javac
compiler warnings (more than the trunk's current 2013 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3222//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3222//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3222//console
This message is automatically generated.
TrackerDistributedCacheManager issues too many getFileStatus calls
--
Key: MAPREDUCE-4907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, tasktracker
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk.patch
TrackerDistributedCacheManager issues a number of redundant getFileStatus
calls when determining the timestamps and visibilities of files in the
distributed cache. 300 distributed cache files deep in the directory
structure can hammer HDFS with a couple thousand requests.
A couple optimizations can reduce this load:
1. determineTimestamps and determineCacheVisibilities both call getFileStatus
on every file. We could cache the results of the former and use them for the
latter.
2. determineCacheVisibilities needs to check that all ancestor directories of
each file have execute permissions for everyone. This currently entails a
getFileStatus on each ancestor directory for each file. The results of these
getFileStatus calls could be cached as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549141#comment-13549141 ] Alejandro Abdelnur commented on MAPREDUCE-4907: --- Sandy, looks good, a couple of minor things: * the patch for trunk has 2 getFileStatus() with duplicate logic for adding an entry to the cache, we should have one delegating to the other so the cache addition logic is not dup. For branch-1 it is OK because the filesystem access is done in another class for one of the getFileStatus() (not to change the behavior I assume). * javac warnings, are they related? TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549145#comment-13549145 ] Sandy Ryza commented on MAPREDUCE-4907: --- The compiler warnings are because DistributedCache in mapred.filecache is deprecated. I'll change it to use the one in org.apache.hadoop.filecache and remove the 2nd trunk getFileStatus. TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4907: -- Attachment: MAPREDUCE-4907-trunk-1.patch TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4848) TaskAttemptContext cast error during AM recovery
[
https://issues.apache.org/jira/browse/MAPREDUCE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549161#comment-13549161
]
Hudson commented on MAPREDUCE-4848:
---
Integrated in Hadoop-trunk-Commit #3208 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3208/])
MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery.
Contributed by Jerry Chen (Revision 1431131)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1431131
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/recover/RecoveryService.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java
TaskAttemptContext cast error during AM recovery
Key: MAPREDUCE-4848
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4848
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 0.23.4
Reporter: Jason Lowe
Assignee: Jerry Chen
Fix For: trunk
Attachments: MAPREDUCE-4848.patch
Recently saw an AM that failed and tried to recover, but the subsequent
attempt quickly exited with its own failure during recovery:
{noformat}
2012-12-05 02:33:36,752 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ClassCastException:
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl cannot be cast to
org.apache.hadoop.mapred.TaskAttemptContext
at
org.apache.hadoop.mapred.OutputCommitter.recoverTask(OutputCommitter.java:284)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.handle(RecoveryService.java:361)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1211)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1177)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:958)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:926)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:918)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
2012-12-05 02:33:36,752 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{noformat}
The RM then launched a third AM attempt which succeeded. The third attempt
saw basically no progress after parsing the history file from the second
attempt and ran the job again from scratch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4848) TaskAttemptContext cast error during AM recovery
[
https://issues.apache.org/jira/browse/MAPREDUCE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4848:
--
Resolution: Fixed
Fix Version/s: (was: trunk)
0.23.6
2.0.3-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Jerry! I committed this to trunk, branch-2, and branch-0.23.
Note that in the future, the Fix Version/s field is reserved for recording
where the issue was actually fixed and should only be set when the fix is
checked in. The Target Version/s field is better suited for indicating where
a patch should be integrated if accepted.
TaskAttemptContext cast error during AM recovery
Key: MAPREDUCE-4848
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4848
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 0.23.4
Reporter: Jason Lowe
Assignee: Jerry Chen
Fix For: 2.0.3-alpha, 0.23.6
Attachments: MAPREDUCE-4848.patch
Recently saw an AM that failed and tried to recover, but the subsequent
attempt quickly exited with its own failure during recovery:
{noformat}
2012-12-05 02:33:36,752 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ClassCastException:
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl cannot be cast to
org.apache.hadoop.mapred.TaskAttemptContext
at
org.apache.hadoop.mapred.OutputCommitter.recoverTask(OutputCommitter.java:284)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.handle(RecoveryService.java:361)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1211)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1177)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:958)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:926)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:918)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
2012-12-05 02:33:36,752 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{noformat}
The RM then launched a third AM attempt which succeeded. The third attempt
saw basically no progress after parsing the history file from the second
attempt and ran the job again from scratch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549166#comment-13549166 ] Alejandro Abdelnur commented on MAPREDUCE-4907: --- LGTM, +1 pending jenkins. TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549178#comment-13549178
]
Hadoop QA commented on MAPREDUCE-4907:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564053/MAPREDUCE-4907-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapreduce.filecache.TestClientDistributedCacheManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3223//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3223//console
This message is automatically generated.
TrackerDistributedCacheManager issues too many getFileStatus calls
--
Key: MAPREDUCE-4907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, tasktracker
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch,
MAPREDUCE-4907-trunk.patch
TrackerDistributedCacheManager issues a number of redundant getFileStatus
calls when determining the timestamps and visibilities of files in the
distributed cache. 300 distributed cache files deep in the directory
structure can hammer HDFS with a couple thousand requests.
A couple optimizations can reduce this load:
1. determineTimestamps and determineCacheVisibilities both call getFileStatus
on every file. We could cache the results of the former and use them for the
latter.
2. determineCacheVisibilities needs to check that all ancestor directories of
each file have execute permissions for everyone. This currently entails a
getFileStatus on each ancestor directory for each file. The results of these
getFileStatus calls could be cached as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4907: -- Attachment: MAPREDUCE-4907-trunk-1.patch TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549202#comment-13549202 ] Sandy Ryza commented on MAPREDUCE-4907: --- Attached a patch that fixes test TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4315) jobhistory.jsp throws 500 when a .txt file is found in /done
[ https://issues.apache.org/jira/browse/MAPREDUCE-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549210#comment-13549210 ] Alejandro Abdelnur commented on MAPREDUCE-4315: --- Sandy, why not use a filter when reading files from the done/ dir to pick up only files with the right pattern? then you don't have to overload the comparator logic and you don't have to break the loop when the file does not match the pattern. jobhistory.jsp throws 500 when a .txt file is found in /done Key: MAPREDUCE-4315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4315 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 0.20.2 Reporter: Alexander Alten-Lorenz Assignee: Sandy Ryza Attachments: MAPREDUCE-4315.patch, MAPREDUCE-4315.patch if a .txt file located in /done the parser throws an 500 error. Trace: java.lang.ArrayIndexOutOfBoundsException: 1 at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:295) at org.apache.hadoop.mapred.jobhistory_jsp$2.compare(jobhistory_jsp.java:279) at java.util.Arrays.mergeSort(Arrays.java:1270) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.mergeSort(Arrays.java:1282) at java.util.Arrays.mergeSort(Arrays.java:1281) at java.util.Arrays.sort(Arrays.java:1210) at org.apache.hadoop.mapred.jobhistory_jsp._jspService(jobhistory_jsp.java:279) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:864) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Reproduce: cd ../done touch test.txt reload jobhistory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549222#comment-13549222
]
Hadoop QA commented on MAPREDUCE-4907:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564059/MAPREDUCE-4907-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapreduce.filecache.TestClientDistributedCacheManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3224//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3224//console
This message is automatically generated.
TrackerDistributedCacheManager issues too many getFileStatus calls
--
Key: MAPREDUCE-4907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, tasktracker
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch,
MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch
TrackerDistributedCacheManager issues a number of redundant getFileStatus
calls when determining the timestamps and visibilities of files in the
distributed cache. 300 distributed cache files deep in the directory
structure can hammer HDFS with a couple thousand requests.
A couple optimizations can reduce this load:
1. determineTimestamps and determineCacheVisibilities both call getFileStatus
on every file. We could cache the results of the former and use them for the
latter.
2. determineCacheVisibilities needs to check that all ancestor directories of
each file have execute permissions for everyone. This currently entails a
getFileStatus on each ancestor directory for each file. The results of these
getFileStatus calls could be cached as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4907: -- Attachment: MAPREDUCE-4907-trunk-1.patch TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4305) Implement delay scheduling in capacity scheduler for improving data locality
[ https://issues.apache.org/jira/browse/MAPREDUCE-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4305: - Attachment: PATCH-MAPREDUCE-4305-MR1-1.patch Fixing small bug Thanks, Mayank Implement delay scheduling in capacity scheduler for improving data locality Key: MAPREDUCE-4305 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4305 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Mayank Bansal Assignee: Mayank Bansal Attachments: MAPREDUCE-4305, MAPREDUCE-4305-1.patch, PATCH-MAPREDUCE-4305-MR1-1.patch, PATCH-MAPREDUCE-4305-MR1.patch Capacity Scheduler data local tasks are about 40%-50% which is not good. While my test with 70 node cluster i consistently get data locality around 40-50% on a free cluster. I think we need to implement something like delay scheduling in the capacity scheduler for improving the data locality. http://radlab.cs.berkeley.edu/publication/308 After implementing the delay scheduling on Hadoop 22 I am getting 100 % data locality in free cluster and around 90% data locality in busy cluster. Thanks, Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[
https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549246#comment-13549246
]
Hadoop QA commented on MAPREDUCE-4907:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564062/MAPREDUCE-4907-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3225//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3225//console
This message is automatically generated.
TrackerDistributedCacheManager issues too many getFileStatus calls
--
Key: MAPREDUCE-4907
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1, tasktracker
Affects Versions: 1.1.1
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch,
MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch,
MAPREDUCE-4907-trunk.patch
TrackerDistributedCacheManager issues a number of redundant getFileStatus
calls when determining the timestamps and visibilities of files in the
distributed cache. 300 distributed cache files deep in the directory
structure can hammer HDFS with a couple thousand requests.
A couple optimizations can reduce this load:
1. determineTimestamps and determineCacheVisibilities both call getFileStatus
on every file. We could cache the results of the former and use them for the
latter.
2. determineCacheVisibilities needs to check that all ancestor directories of
each file have execute permissions for everyone. This currently entails a
getFileStatus on each ancestor directory for each file. The results of these
getFileStatus calls could be cached as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549251#comment-13549251 ] Alejandro Abdelnur commented on MAPREDUCE-4907: --- +1 TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4907: -- Resolution: Fixed Fix Version/s: 2.0.3-alpha 1.2.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Sandy. Committed to trunk, branch-2 and branch-1. TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4907) TrackerDistributedCacheManager issues too many getFileStatus calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549256#comment-13549256 ] Hudson commented on MAPREDUCE-4907: --- Integrated in Hadoop-trunk-Commit #3210 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3210/]) MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus calls. (sandyr via tucu) (Revision 1431166) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1431166 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java TrackerDistributedCacheManager issues too many getFileStatus calls -- Key: MAPREDUCE-4907 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4907 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, tasktracker Affects Versions: 1.1.1 Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha Attachments: MAPREDUCE-4907.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk-1.patch, MAPREDUCE-4907-trunk.patch TrackerDistributedCacheManager issues a number of redundant getFileStatus calls when determining the timestamps and visibilities of files in the distributed cache. 300 distributed cache files deep in the directory structure can hammer HDFS with a couple thousand requests. A couple optimizations can reduce this load: 1. determineTimestamps and determineCacheVisibilities both call getFileStatus on every file. We could cache the results of the former and use them for the latter. 2. determineCacheVisibilities needs to check that all ancestor directories of each file have execute permissions for everyone. This currently entails a getFileStatus on each ancestor directory for each file. The results of these getFileStatus calls could be cached as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4921) JobClient should acquire HS token with RM principal
[
https://issues.apache.org/jira/browse/MAPREDUCE-4921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549289#comment-13549289
]
Hadoop QA commented on MAPREDUCE-4921:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564040/MAPREDUCE-4921.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3226//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3226//console
This message is automatically generated.
JobClient should acquire HS token with RM principal
---
Key: MAPREDUCE-4921
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4921
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-4921.branch-23.patch,
MAPREDUCE-4921.branch-23.patch, MAPREDUCE-4921.patch, MAPREDUCE-4921.patch
The job client may acquire a history server token during job submission. The
renewer is specified in a config value that the user must supply (for new
api, a bit different for old api). If this value is not the RM's principal,
then the RM cannot renew the token and long running jobs will fail. Since
the token is implicitly acquired for the job, the HS token's renewer should
always be the RM's principal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4848) TaskAttemptContext cast error during AM recovery
[
https://issues.apache.org/jira/browse/MAPREDUCE-4848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549296#comment-13549296
]
Jerry Chen commented on MAPREDUCE-4848:
---
Thansk for your comment and help checked in, Jason. I will follow the these
guidelines for future work.
TaskAttemptContext cast error during AM recovery
Key: MAPREDUCE-4848
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4848
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 0.23.4
Reporter: Jason Lowe
Assignee: Jerry Chen
Fix For: 2.0.3-alpha, 0.23.6
Attachments: MAPREDUCE-4848.patch
Recently saw an AM that failed and tried to recover, but the subsequent
attempt quickly exited with its own failure during recovery:
{noformat}
2012-12-05 02:33:36,752 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.ClassCastException:
org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl cannot be cast to
org.apache.hadoop.mapred.TaskAttemptContext
at
org.apache.hadoop.mapred.OutputCommitter.recoverTask(OutputCommitter.java:284)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$InterceptingEventHandler.handle(RecoveryService.java:361)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1211)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1177)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:958)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:926)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:918)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.realDispatch(RecoveryService.java:285)
at
org.apache.hadoop.mapreduce.v2.app.recover.RecoveryService$RecoveryDispatcher.dispatch(RecoveryService.java:281)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
2012-12-05 02:33:36,752 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{noformat}
The RM then launched a third AM attempt which succeeded. The third attempt
saw basically no progress after parsing the history file from the second
attempt and ran the job again from scratch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549347#comment-13549347 ] Arun C Murthy commented on MAPREDUCE-4896: -- bq. This is because the capacity scheduler's getQueueInfo throws an IOException when the queue isn't found, and the fair scheduler's returns null. That seems like bad beaviour on part of FS - returning null is a silent failure, an exception needs to be thrown in error cases. mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-4808: - Status: Open (was: Patch Available) Asokan, sorry I've been away traveling home during the holidays and hence the delay. I have more comments, but I'll put some here to keep the discussion going. Thanks for the design doc, but I was looking for thoughts on *how* the plugin was going used for use-cases you've mentioned (hash-join etc.), alternatives on design etc. IAC, taking a step back, the 'goal' here is to make the 'merge' pluggable. Reduce-side has 2 pieces: # Shuffle - Move data from maps to the reduce. # Merge - Merge already sorted map-outputs. The rest (MergeManager etc.) are merely implementation details to manage memory etc., which are irrelevant in several scenarios as soon as we consider alternatives to the current HTTP-based shuffle (several alternatives exist such RDMA etc.). Your current approach tries to encapsulate and enshrine the current implementation of the reduce task, which I'm not wild about. By this I mean, you are focussing too much on the current state and trying to make interfaces which are unnecessary for now and might not suffice for the future. I really don't think we should be tying Shuffle Merge as you have done by introducing yet another new interface (regardless of whether it's public or not). As I've noted above, adding a simple 'Merge' interface with one 'merge' call will address all of the use-cases you have outlined. If not, let's discuss. Allow reduce-side merge to be pluggable --- Key: MAPREDUCE-4808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: 2.0.3-alpha Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf Allow reduce-side merge to be pluggable for MAPREDUCE-2454 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549358#comment-13549358 ] Arun C Murthy commented on MAPREDUCE-4808: -- bq. As I've noted above, adding a simple 'Merge' interface with one 'merge' call will address all of the use-cases you have outlined. If not, let's discuss. Forgot to add - if you don't think this suffices, please help me understand why. We can discuss alternatives then. Thanks. Allow reduce-side merge to be pluggable --- Key: MAPREDUCE-4808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: 2.0.3-alpha Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf Allow reduce-side merge to be pluggable for MAPREDUCE-2454 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4808) Allow reduce-side merge to be pluggable
[ https://issues.apache.org/jira/browse/MAPREDUCE-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549361#comment-13549361 ] Arun C Murthy commented on MAPREDUCE-4808: -- As an example - with RDMA-based shuffle we don't need MergeManager at all since we can just do very-wide, network merges directly. Allow reduce-side merge to be pluggable --- Key: MAPREDUCE-4808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4808 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.2-alpha Reporter: Arun C Murthy Assignee: Mariappan Asokan Fix For: 2.0.3-alpha Attachments: COMBO-mapreduce-4809-4812-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, mapreduce-4808.patch, MergeManagerPlugin.pdf Allow reduce-side merge to be pluggable for MAPREDUCE-2454 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4883) Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
[
https://issues.apache.org/jira/browse/MAPREDUCE-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry Chen reassigned MAPREDUCE-4883:
-
Assignee: Jerry Chen
Reducer's Maximum Shuffle Buffer Size should be enlarged for 64bit JVM
--
Key: MAPREDUCE-4883
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4883
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.20.2, 1.0.3
Environment: Especially for 64bit JVM
Reporter: Lijie Xu
Assignee: Jerry Chen
Labels: patch
Original Estimate: 12h
Remaining Estimate: 12h
In hadoop-0.20.2, hadoop-1.0.3 or other versions, reducer's shuffle buffer
size cannot exceed 2048MB (i.e., Integer.MAX_VALUE). This is reasonable for
32bit JVM.
But for 64bit JVM, although reducer's JVM size can be set more than 2048MB
(e.g., mapred.child.java.opts=-Xmx4000m), the heap size used for shuffle
buffer is at most 2048MB * maxInMemCopyUse (default 0.7) not 4000MB *
maxInMemCopyUse.
So the pointed piece of code in ReduceTask.java needs modification for 64bit
JVM.
---
private final long maxSize;
private final long maxSingleShuffleLimit;
private long size = 0;
private Object dataAvailable = new Object();
private long fullSize = 0;
private int numPendingRequests = 0;
private int numRequiredMapOutputs = 0;
private int numClosed = 0;
private boolean closed = false;
public ShuffleRamManager(Configuration conf) throws IOException {
final float maxInMemCopyUse =
conf.getFloat(mapred.job.shuffle.input.buffer.percent, 0.70f);
if (maxInMemCopyUse 1.0 || maxInMemCopyUse 0.0) {
throw new IOException(mapred.job.shuffle.input.buffer.percent +
maxInMemCopyUse);
}
// Allow unit tests to fix Runtime memory
-- maxSize = (int)(conf.getInt(mapred.job.reduce.total.mem.bytes,
--(int)Math.min(Runtime.getRuntime().maxMemory(), Integer.MAX_VALUE))
-- * maxInMemCopyUse);
maxSingleShuffleLimit = (long)(maxSize *
MAX_SINGLE_SHUFFLE_SEGMENT_FRACTION);
LOG.info(ShuffleRamManager: MemoryLimit= + maxSize +
, MaxSingleShuffleLimit= + maxSingleShuffleLimit);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
