[jira] [Created] (MAPREDUCE-6605) Configuration options mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc-count.auto-incr NOT WORKING
Dong Zhen created MAPREDUCE-6605: Summary: Configuration options mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc-count.auto-incr NOT WORKING Key: MAPREDUCE-6605 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6605 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Dong Zhen As the default configuration file shows, https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml we can configure options mapreduce.map.skip.proc.count.autoincr mapreduce.reduce.skip.proc.count.autoincr in the mapred-default.xml. But they do not work because the expected keys in org.apache.hadoop.mapreduce.MRJobConfig.java public static final String MAP_SKIP_INCR_PROC_COUNT = "mapreduce.map.skip.proc-count.auto-incr" public static final String REDUCE_SKIP_INCR_PROC_COUNT = "mapreduce.reduce.skip.proc-count.auto-incr"; Or in org.apache.hadoop.mapreduce.util.ConfigUtil.java new DeprecationDelta("mapred.skip.map.auto.incr.proc.count", MRJobConfig.MAP_SKIP_INCR_PROC_COUNT), new DeprecationDelta("mapred.skip.reduce.auto.incr.proc.count", MRJobConfig.REDUCE_SKIP_INCR_PROC_COUNT), we can change them to mapreduce.map.skip.proc-count.auto-incr and mapreduce.reduce.skip.proc-count.auto-incr in the default configuration file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6605) Configuration options mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc-count.auto-incr NOT WORKING
[ https://issues.apache.org/jira/browse/MAPREDUCE-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Zhen updated MAPREDUCE-6605: - Affects Version/s: 2.7.1 > Configuration options mapreduce.map.skip.proc.count.autoincr and > mapreduce.reduce.skip.proc-count.auto-incr NOT WORKING > --- > > Key: MAPREDUCE-6605 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6605 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Dong Zhen > > As the default configuration file shows, > https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml > we can configure options > mapreduce.map.skip.proc.count.autoincr > mapreduce.reduce.skip.proc.count.autoincr > in the mapred-default.xml. But they do not work because the expected keys in > org.apache.hadoop.mapreduce.MRJobConfig.java > public static final String MAP_SKIP_INCR_PROC_COUNT = > "mapreduce.map.skip.proc-count.auto-incr" > public static final String REDUCE_SKIP_INCR_PROC_COUNT = > "mapreduce.reduce.skip.proc-count.auto-incr"; > Or > in org.apache.hadoop.mapreduce.util.ConfigUtil.java > new DeprecationDelta("mapred.skip.map.auto.incr.proc.count", > MRJobConfig.MAP_SKIP_INCR_PROC_COUNT), > new DeprecationDelta("mapred.skip.reduce.auto.incr.proc.count", > MRJobConfig.REDUCE_SKIP_INCR_PROC_COUNT), > we can change them to mapreduce.map.skip.proc-count.auto-incr and > mapreduce.reduce.skip.proc-count.auto-incr in the default configuration file. > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096769#comment-15096769 ] Jason Lowe commented on MAPREDUCE-6554: --- +1 latest patch lgtm. Could you also provide a patch for branch-2.7/branch-2.6? > MRAppMaster servicestart failing with NPE in > MRAppMaster#parsePreviousJobHistory > - > > Key: MAPREDUCE-6554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Critical > Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch, > 0003-MAPREDUCE-6554.patch > > > Create scenario so that MR app master gets preempted. > On next MRAppMaster launch tried to recover previous job history file > {{MRAppMaster#parsePreviousJobHistory}} > {noformat} > 2015-11-21 13:52:27,722 INFO [main] > org.apache.hadoop.service.AbstractService: Service > org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; > cause: java.lang.NullPointerException > java.lang.NullPointerException > at java.io.StringReader.(StringReader.java:50) > at org.apache.avro.Schema$Parser.parse(Schema.java:917) > at org.apache.avro.Schema.parse(Schema.java:966) > at > org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499) > 2015-11-21 13:52:27,725 INFO [main] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping > JobHistoryEventHandler. Size of the outstanding queue size is 0 > {noformat} > EventReader(EventReader stream) > {noformat} > this.version = in.readLine(); > ... > Schema myschema = new > SpecificData(Event.class.getClassLoader()).getSchema(Event.class); > this.schema = Schema.parse(in.readLine()); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097681#comment-15097681 ] Akira AJISAKA commented on MAPREDUCE-6595: -- Ran test-patch.sh for MAPREDUCE-6595.testing.patch with YETUS-271.03.patch. {noformat} | Vote | Subsystem | Runtime | Comment | +1 | @author | 0m 00s| The patch does not contain any @author | ||| tags. | -1 |test4tests | 0m 00s| The patch doesn't appear to include any | ||| new or modified tests. Please justify why | ||| no new tests are needed for this patch. | ||| Also please list what manual steps were | ||| performed to verify this patch. | 0 |mvndep | 1m 41s| Maven dependency ordering for branch | +1 |mvninstall | 7m 38s| trunk passed | -1 | compile | 2m 17s| hadoop-mapreduce-project in trunk | ||| failed. | +1 |checkstyle | 0m 22s| trunk passed | +1 | mvnsite | 3m 31s| trunk passed | +1 |mvneclipse | 1m 32s| trunk passed | 0 | findbugs | 0m 00s| Skipped branch modules with no Java | ||| source: hadoop-mapreduce-project | 0 | findbugs | 1m 38s| | ||| hadoop-mapreduce-project/hadoop-mapreduce- | ||| client/hadoop-mapreduce-client-core in | ||| trunk has 2 extant Findbugs warnings. | +1 | javadoc | 2m 31s| trunk passed | 0 |mvndep | 0m 41s| Maven dependency ordering for patch | +1 |mvninstall | 2m 25s| the patch passed | -1 | compile | 1m 37s| hadoop-mapreduce-project in the patch | ||| failed. | -1 | javac | 1m 37s| hadoop-mapreduce-project in the patch | ||| failed. | +1 |checkstyle | 0m 25s| the patch passed | +1 | mvnsite | 2m 54s| the patch passed | +1 |mvneclipse | 1m 04s| the patch passed | +1 |whitespace | 0m 00s| Patch has no whitespace issues. | +1 | xml | 0m 01s| The patch has no ill-formed XML file. | 0 | findbugs | 0m 00s| Skipped patch modules with no Java | ||| source: hadoop-mapreduce-project | +1 | findbugs | 1m 32s| the patch passed | +1 | javadoc | 2m 05s| the patch passed | -1 |asflicense | 0m 38s| Patch generated 9 ASF License warnings. | || 36m 13s | {noformat} The output shows that the 2 findbugs warnings are fixed. (actually ignored) > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6595) Fix findbugs warnings in OutputCommitter and FileOutputCommitter
[ https://issues.apache.org/jira/browse/MAPREDUCE-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097683#comment-15097683 ] Akira AJISAKA commented on MAPREDUCE-6595: -- Filed MAPREDUCE-6600. > Fix findbugs warnings in OutputCommitter and FileOutputCommitter > > > Key: MAPREDUCE-6595 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6595 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: MAPREDUCE-6595.01.patch, MAPREDUCE-6595.testing.patch, > findbugsHtml.html > > > There are 2 findbugs warnings in hadoop-mapreduce-client-core module. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6237/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6540) TestMRTimelineEventHandling fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-6540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-6540: --- Fix Version/s: (was: 2.7.3) (was: 2.8.0) 2.7.2 > TestMRTimelineEventHandling fails > - > > Key: MAPREDUCE-6540 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6540 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Sangjin Lee >Assignee: Sangjin Lee > Fix For: 2.7.2, 2.6.3 > > Attachments: MAPREDUCE-6540.001.patch > > > TestMRTimelineEventHandling fails after YARN-2859 is merged because it > changed the port the AHS binds to in a mini cluster. > {noformat} > Running org.apache.hadoop.mapred.TestMRTimelineEventHandling > Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 184.38 sec > <<< FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling > testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling) > Time elapsed: 70.528 sec <<< ERROR! > java.io.IOException: Job didn't finish in 30 seconds > at > org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622) > at > org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:99) > testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling) > Time elapsed: 84.312 sec <<< ERROR! > java.io.IOException: Job didn't finish in 30 seconds > at > org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622) > at > org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:162) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097241#comment-15097241 ] Hudson commented on MAPREDUCE-6549: --- FAILURE: Integrated in Hadoop-trunk-Commit #9102 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9102/]) MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause (vinodkv: rev 9b33a03900c9ffdb9f4b4c59161fdce39ca8c73d) * hadoop-mapreduce-project/CHANGES.txt > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.7.2, 2.6.3 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6377) JHS sorting on state column not working in webUi
[ https://issues.apache.org/jira/browse/MAPREDUCE-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-6377: --- Fix Version/s: (was: 2.7.3) 2.7.2 Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same. > JHS sorting on state column not working in webUi > > > Key: MAPREDUCE-6377 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6377 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.0 > Environment: 2 NM, JHS >Reporter: Bibin A Chundatt >Assignee: zhihai xu >Priority: Minor > Fix For: 2.7.2, 2.6.3 > > Attachments: MAPREDUCE-6377.000.patch, Sorting Issue.png, > state_sorted1.pdf, state_sorted2.pdf > > > Steps to reproduce > > 1. Install and setup HA cluster with JHS > 2.Create state in in JHS where few jobs are killed and Success > Check sorting State in JHS WebUI > Actual > = > Sorting on state column not working in JHS > Expected > == > Sorting on state column should be working -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5883) "Total megabyte-seconds" in job counters is slightly misleading
[ https://issues.apache.org/jira/browse/MAPREDUCE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5883: --- Fix Version/s: (was: 2.7.3) 2.7.2 Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same. > "Total megabyte-seconds" in job counters is slightly misleading > --- > > Key: MAPREDUCE-5883 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5883 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.0.0, 2.4.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts >Priority: Minor > Labels: BB2015-05-TBR > Fix For: 2.7.2, 2.6.3 > > Attachments: MAPREDUCE-5883.patch > > > The following counters are in milliseconds so "megabyte-seconds" might be > better stated as "megabyte-milliseconds" > MB_MILLIS_MAPS.name= Total megabyte-seconds taken by all map > tasks > MB_MILLIS_REDUCES.name=Total megabyte-seconds taken by all reduce > tasks > VCORES_MILLIS_MAPS.name= Total vcore-seconds taken by all map tasks > VCORES_MILLIS_REDUCES.name=Total vcore-seconds taken by all reduce > tasks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6549) multibyte delimiters with LineRecordReader cause duplicate records
[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-6549: --- Fix Version/s: (was: 2.7.3) (was: 2.8.0) 2.7.2 Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same. > multibyte delimiters with LineRecordReader cause duplicate records > -- > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 2.7.2 >Reporter: Dustin Cote >Assignee: Wilfred Spiegelenburg > Fix For: 2.7.2, 2.6.3 > > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch, > MAPREDUCE-6549.3.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097401#comment-15097401 ] Akira AJISAKA commented on MAPREDUCE-6363: -- +1, committing this. > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Vlad Sharanhovich >Priority: Critical > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6363: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.6.4 2.7.3 2.8.0 Status: Resolved (was: Patch Available) Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6. Thanks [~bibinchundatt] for the contribution! > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Bibin A Chundatt >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.6.4 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6465) NNBench result wrong when number of reducers greater than 1
[ https://issues.apache.org/jira/browse/MAPREDUCE-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA resolved MAPREDUCE-6465. -- Resolution: Duplicate This issue is fixed by MAPREDUCE-6363. Closing this issue. Thanks Bibin for reporting this and thanks Brahma for the comment. > NNBench result wrong when number of reducers greater than 1 > --- > > Key: MAPREDUCE-6465 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6465 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-HDFS-8977.patch, 0002-MAPREDUCE-6465.patch > > > Currently NNBench#analyzeResults consider only the part- for analysis > {code} > TPS: Create/Write/Close: 0 > Avg exec time (ms): Create/Write/Close: Infinity > Avg Lat (ms): Create/Write: Infinity > Avg Lat (ms): Close: NaN > {code} > Should consider all part files for output. > or disable reduces option -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097594#comment-15097594 ] Hudson commented on MAPREDUCE-6363: --- FAILURE: Integrated in Hadoop-trunk-Commit #9105 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9105/]) MAPREDUCE-6363. [NNBench] Lease mismatch error when running with (aajisaka: rev 7b0964f354e90968c2dac2f7acc17214732aed64) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java * hadoop-mapreduce-project/CHANGES.txt > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Bibin A Chundatt >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.6.4 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6363: - Assignee: Bibin A Chundatt (was: Vlad Sharanhovich) > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Bibin A Chundatt >Priority: Critical > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096245#comment-15096245 ] Bibin A Chundatt commented on MAPREDUCE-6554: - [~jlowe] Testcase/Findbug failures are not related to patch attached. Could you please review patch attached. > MRAppMaster servicestart failing with NPE in > MRAppMaster#parsePreviousJobHistory > - > > Key: MAPREDUCE-6554 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Critical > Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch, > 0003-MAPREDUCE-6554.patch > > > Create scenario so that MR app master gets preempted. > On next MRAppMaster launch tried to recover previous job history file > {{MRAppMaster#parsePreviousJobHistory}} > {noformat} > 2015-11-21 13:52:27,722 INFO [main] > org.apache.hadoop.service.AbstractService: Service > org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; > cause: java.lang.NullPointerException > java.lang.NullPointerException > at java.io.StringReader.(StringReader.java:50) > at org.apache.avro.Schema$Parser.parse(Schema.java:917) > at org.apache.avro.Schema.parse(Schema.java:966) > at > org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75) > at > org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087) > at > org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499) > 2015-11-21 13:52:27,725 INFO [main] > org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping > JobHistoryEventHandler. Size of the outstanding queue size is 0 > {noformat} > EventReader(EventReader stream) > {noformat} > this.version = in.readLine(); > ... > Schema myschema = new > SpecificData(Event.class.getClassLoader()).getSchema(Event.class); > this.schema = Schema.parse(in.readLine()); > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096564#comment-15096564 ] Bibin A Chundatt commented on MAPREDUCE-6363: - [~ajisakaa] Testcase failures are not related to patch attached. Could you please help in reviewing patch. > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Vlad Sharanhovich >Priority: Critical > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6606) Findbug issue in org.apache.hadoop.mapred.OutputCommitter
Bibin A Chundatt created MAPREDUCE-6606: --- Summary: Findbug issue in org.apache.hadoop.mapred.OutputCommitter Key: MAPREDUCE-6606 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6606 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bibin A Chundatt {noformat} org.apache.hadoop.mapred.OutputCommitter.isCommitJobRepeatable(JobContext) doesn't override method in superclass because parameter type org.apache.hadoop.mapred.JobContext doesn't match superclass parameter type org.apache.hadoop.mapreduce.JobContext Bug type NM_WRONG_PACKAGE_INTENTIONAL (click for details) In class org.apache.hadoop.mapred.OutputCommitter In method org.apache.hadoop.mapred.OutputCommitter.isCommitJobRepeatable(JobContext) superclass is org.apache.hadoop.mapreduce.OutputCommitter Did you intend to override org.apache.hadoop.mapreduce.OutputCommitter.isCommitJobRepeatable(JobContext) Actual type org.apache.hadoop.mapred.JobContext Expected org.apache.hadoop.mapreduce.JobContext Overrides org.apache.hadoop.mapred.OutputCommitter.isCommitJobRepeatable(JobContext) At OutputCommitter.java:[line 230] {noformat} https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html#Warnings_BAD_PRACTICE -- This message was sent by Atlassian JIRA (v6.3.4#6332)