[jira] [Updated] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6428:
Affects Version/s: 2.4.1
Job History can be lost if there is any issue in writing jhist file while AM
shuts down job
---
Key: MAPREDUCE-6428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6428
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Varun Saxena
Assignee: Varun Saxena
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.IOException: All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed.
Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94] org.apache.hadoop.util.ExitUtil:
Exiting with status 1
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618455#comment-14618455
]
Tsuyoshi Ozawa commented on MAPREDUCE-6038:
---
Thanks for the commit, Chris.
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
Varun Saxena created MAPREDUCE-6428:
---
Summary: Job History can be lost if there is any issue in writing
jhist file while AM shuts down job
Key: MAPREDUCE-6428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6428
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Varun Saxena
Assignee: Varun Saxena
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed. Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94] org.apache.hadoop.util.ExitUtil:
Exiting with status 1
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6428:
Description:
We found that while job succeeds (can be seen as successful in RM), but as
writing of jhist file fails, job cant be seen in JobHistory.
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed. Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94] org.apache.hadoop.util.ExitUtil:
Exiting with status 1
{noformat}
We can probably mark the job as failure if this happens. Thoughts ?
was:
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
[jira] [Commented] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618502#comment-14618502
]
Varun Saxena commented on MAPREDUCE-6428:
-
Same as MAPREDUCE-5703 it seems. But how will MAPREDUCE-5547 resolve it ? IIUC,
that talks about doing unregistration before jhist writing. But jhist writing
can still fail afterwards and there will be no way to tell RM about it.
Job History can be lost if there is any issue in writing jhist file while AM
shuts down job
---
Key: MAPREDUCE-6428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6428
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Varun Saxena
Assignee: Varun Saxena
We found that while job succeeds (can be seen as successful in RM), but as
writing of jhist file fails, job cant be seen in JobHistory.
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.IOException: All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed.
Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94]
[jira] [Commented] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618503#comment-14618503
]
Varun Saxena commented on MAPREDUCE-6428:
-
Pinging [~vinodkv]
Job History can be lost if there is any issue in writing jhist file while AM
shuts down job
---
Key: MAPREDUCE-6428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6428
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Varun Saxena
Assignee: Varun Saxena
We found that while job succeeds (can be seen as successful in RM), but as
writing of jhist file fails, job cant be seen in JobHistory.
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.io.IOException: All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed.
Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94] org.apache.hadoop.util.ExitUtil:
Exiting with status 1
{noformat}
We can probably mark the job as failure if this happens. Thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6428) Job History can be lost if there is any issue in writing jhist file while AM shuts down job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6428:
Description:
We found that while job succeeds (can be seen as successful in RM), but as
writing of jhist file fails, job cant be seen in JobHistory.
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,853 WARN [Thread-94]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Graceful stop failed. Exiting..
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleEvent(JobHistoryEventHandler.java:546)
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:340)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:157)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:131)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStop(MRAppMaster.java:1609)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1107)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.shutDownJob(MRAppMaster.java:554)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler$1.run(MRAppMaster.java:603)
Caused by: java.io.IOException: All datanodes 9.96.1.171:25009 are bad.
Aborting...
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1145)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:926)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2015-07-04 14:54:37,854 INFO [Thread-94] org.apache.hadoop.util.ExitUtil:
Exiting with status 1
{noformat}
We can probably mark the job as failure and inform RM if this happens. Thoughts
?
was:
We found that while job succeeds (can be seen as successful in RM), but as
writing of jhist file fails, job cant be seen in JobHistory.
{noformat}
2015-07-04 14:54:37,852 INFO [Thread-94]
org.apache.hadoop.service.AbstractService: Service
org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STOPPED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException:
All datanodes 9.96.1.171:25009 are bad. Aborting...
at
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618165#comment-14618165
]
Devaraj K commented on MAPREDUCE-6426:
--
Thanks [~zxu] for addressing comments and updating patch.
It looks good to me, I will commit it tomorrow if there are no other
comments/objections.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618159#comment-14618159
]
Hadoop QA commented on MAPREDUCE-6426:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 56s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 20s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 43s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 19s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 17m 22s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744157/MAPREDUCE-6426.001.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / d632574 |
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5874/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5874/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5874/console |
This message was automatically generated.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618376#comment-14618376
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #250 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/250/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
* hadoop-mapreduce-project/CHANGES.txt
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618390#comment-14618390
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Yarn-trunk #980 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/980/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618553#comment-14618553
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #248 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/248/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618579#comment-14618579
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2196 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2196/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618649#comment-14618649
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk #2177 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2177/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
* hadoop-mapreduce-project/CHANGES.txt
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618700#comment-14618700
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #238 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/238/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
* hadoop-mapreduce-project/CHANGES.txt
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Status: Patch Available (was: Open)
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, 0002-MAPREDUCE-2631.patch,
0003-MAPREDUCE-2631.patch, MAPREDUCE-2631.02.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sunil G updated MAPREDUCE-2631:
---
Attachment: 0003-MAPREDUCE-2631.patch
Uploading patch after correcting test failures.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, 0002-MAPREDUCE-2631.patch,
0003-MAPREDUCE-2631.patch, MAPREDUCE-2631.02.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6429) running M/R tasks under a security policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purvesh Patel updated MAPREDUCE-6429: - Attachment: Patch.pdf running M/R tasks under a security policy - Key: MAPREDUCE-6429 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6429 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf When running M/R tasks under a security policy without the 'java.lang.reflect.ReflectPermission suppressAccessChecks' granted to the user code, we are unable to instantiate new keys for a WritableComparator. This patch wraps the WritableComparator#newKey in a doPrivileged block handling non-public key constructors. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6430) new method to ReflectionUtils to test if the item is already public
[ https://issues.apache.org/jira/browse/MAPREDUCE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Purvesh Patel updated MAPREDUCE-6430: - Attachment: Patch.pdf new method to ReflectionUtils to test if the item is already public --- Key: MAPREDUCE-6430 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6430 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf It's noticed that it is common practice to call setAccessible on a method whether it is required or not. This patch provides a new method to ReflectionUtils to test if the item is already public and only call setAccessible if it is not public. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6430) new method to ReflectionUtils to test if the item is already public
Purvesh Patel created MAPREDUCE-6430: Summary: new method to ReflectionUtils to test if the item is already public Key: MAPREDUCE-6430 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6430 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf It's noticed that it is common practice to call setAccessible on a method whether it is required or not. This patch provides a new method to ReflectionUtils to test if the item is already public and only call setAccessible if it is not public. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6429) running M/R tasks under a security policy
Purvesh Patel created MAPREDUCE-6429: Summary: running M/R tasks under a security policy Key: MAPREDUCE-6429 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6429 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor When running M/R tasks under a security policy without the 'java.lang.reflect.ReflectPermission suppressAccessChecks' granted to the user code, we are unable to instantiate new keys for a WritableComparator. This patch wraps the WritableComparator#newKey in a doPrivileged block handling non-public key constructors. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated MAPREDUCE-6427:
--
Attachment: MAPREDUCE-6427.001.patch
Initial version. branch-2 and trunk versions appear identical.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619442#comment-14619442
]
Ray Chiang commented on MAPREDUCE-6427:
---
Done. Thanks for splitting out the new JIRA.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ray Chiang updated MAPREDUCE-6427:
--
Status: Patch Available (was: Open)
Submit for testing. Not sure about any potential incompatibilities.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619274#comment-14619274
]
Hadoop QA commented on MAPREDUCE-2631:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 2s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 22s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 0s | The applied patch generated 7
new checkstyle issues (total was 17, now 20). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 46s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 106m 33s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 150m 32s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744273/0003-MAPREDUCE-2631.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc6182d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5875/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5875/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5875/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5875/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5875/console |
This message was automatically generated.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, 0002-MAPREDUCE-2631.patch,
0003-MAPREDUCE-2631.patch, MAPREDUCE-2631.02.patch, MAPREDUCE-2631.1.patch,
MAPREDUCE-2631.2.patch, MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6426:
-
Attachment: MAPREDUCE-6426.001.patch
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618135#comment-14618135
]
zhihai xu commented on MAPREDUCE-6426:
--
thanks for the review [~devaraj.k]! I uploaded a new patch
MAPREDUCE-6426.001.patch, which addressed your comment. Please review it.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6430) new method to ReflectionUtils to test if the item is already public
[ https://issues.apache.org/jira/browse/MAPREDUCE-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619650#comment-14619650 ] Gabor Liptak commented on MAPREDUCE-6430: - [~ppatel] Please consider uploading a text patch file with your changes. http://wiki.apache.org/hadoop/HowToContribute new method to ReflectionUtils to test if the item is already public --- Key: MAPREDUCE-6430 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6430 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf It's noticed that it is common practice to call setAccessible on a method whether it is required or not. This patch provides a new method to ReflectionUtils to test if the item is already public and only call setAccessible if it is not public. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619712#comment-14619712
]
Ray Chiang commented on MAPREDUCE-6427:
---
Got it. That makes sense.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6429) running M/R tasks under a security policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619899#comment-14619899 ] Owen O'Malley commented on MAPREDUCE-6429: -- The Java sandbox is not a good match for the needs of Hadoop security. The performance penalty of the sandbox is huge and as you point out the code needs reflection and local file access. running M/R tasks under a security policy - Key: MAPREDUCE-6429 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6429 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf When running M/R tasks under a security policy without the 'java.lang.reflect.ReflectPermission suppressAccessChecks' granted to the user code, we are unable to instantiate new keys for a WritableComparator. This patch wraps the WritableComparator#newKey in a doPrivileged block handling non-public key constructors. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6429) running M/R tasks under a security policy
[ https://issues.apache.org/jira/browse/MAPREDUCE-6429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619651#comment-14619651 ] Gabor Liptak commented on MAPREDUCE-6429: - [~ppatel] Please consider uploading a text patch file with your changes. http://wiki.apache.org/hadoop/HowToContribute running M/R tasks under a security policy - Key: MAPREDUCE-6429 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6429 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 2.5.0 Reporter: Purvesh Patel Priority: Minor Attachments: Patch.pdf When running M/R tasks under a security policy without the 'java.lang.reflect.ReflectPermission suppressAccessChecks' granted to the user code, we are unable to instantiate new keys for a WritableComparator. This patch wraps the WritableComparator#newKey in a doPrivileged block handling non-public key constructors. Please see attached file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619559#comment-14619559
]
Hadoop QA commented on MAPREDUCE-6427:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 42s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 21s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 6s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 45m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744341/MAPREDUCE-6427.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 2e3d83f |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5876/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5876/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5876/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5876/console |
This message was automatically generated.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5951) Add support for the YARN Shared Cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619614#comment-14619614 ] Chris Trezzo commented on MAPREDUCE-5951: - Investigating TestLocalJobSubmission test failure. Add support for the YARN Shared Cache - Key: MAPREDUCE-5951 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5951 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Chris Trezzo Assignee: Chris Trezzo Labels: BB2015-05-TBR Attachments: MAPREDUCE-5951-trunk-v1.patch, MAPREDUCE-5951-trunk-v10.patch, MAPREDUCE-5951-trunk-v11.patch, MAPREDUCE-5951-trunk-v12.patch, MAPREDUCE-5951-trunk-v13.patch, MAPREDUCE-5951-trunk-v2.patch, MAPREDUCE-5951-trunk-v3.patch, MAPREDUCE-5951-trunk-v4.patch, MAPREDUCE-5951-trunk-v5.patch, MAPREDUCE-5951-trunk-v6.patch, MAPREDUCE-5951-trunk-v7.patch, MAPREDUCE-5951-trunk-v8.patch, MAPREDUCE-5951-trunk-v9.patch Implement the necessary changes so that the MapReduce application can leverage the new YARN shared cache (i.e. YARN-1492). Specifically, allow per-job configuration so that MapReduce jobs can specify which set of resources they would like to cache (i.e. jobjar, libjars, archives, files). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619698#comment-14619698
]
Chris Douglas commented on MAPREDUCE-6427:
--
bq. Initial version. branch-2 and trunk versions appear identical.
The branch-2 version should publish both the old metric _and_ the correctly
spelled label. Trunk can replace it outright. Please also add a release note.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
Attachments: MAPREDUCE-6427.001.patch
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
