Hadoop-Mapreduce-trunk - Build # 1905 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1905/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 29119 lines...] [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] hadoop-mapreduce-client-nativetask SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] Apache Hadoop MapReduce Streaming . SKIPPED [INFO] Apache Hadoop Distributed Copy SKIPPED [INFO] Apache Hadoop Archives SKIPPED [INFO] Apache Hadoop Rumen ... SKIPPED [INFO] Apache Hadoop Gridmix . SKIPPED [INFO] Apache Hadoop Data Join ... SKIPPED [INFO] Apache Hadoop Extras .. SKIPPED [INFO] Apache Hadoop Pipes ... SKIPPED [INFO] Apache Hadoop OpenStack support ... SKIPPED [INFO] Apache Hadoop Amazon Web Services support . SKIPPED [INFO] Apache Hadoop Azure support ... SKIPPED [INFO] Apache Hadoop Client .. SKIPPED [INFO] Apache Hadoop Mini-Cluster SKIPPED [INFO] Apache Hadoop Scheduler Load Simulator SKIPPED [INFO] Apache Hadoop Tools Dist .. SKIPPED [INFO] Apache Hadoop Tools ... SKIPPED [INFO] Apache Hadoop Distribution SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 23:52 min [INFO] Finished at: 2014-09-23T13:46:14+00:00 [INFO] Final Memory: 65M/839M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-common: There are test failures. [ERROR] [ERROR] Please refer to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/hadoop-common-project/hadoop-common/target/surefire-reports for the individual test results. [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-common Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating MAPREDUCE-5279 Updating HDFS-7001 Updating YARN-2584 Updating YARN-2539 Updating HDFS-7115 Updating MAPREDUCE-6095 Updating YARN-1959 Updating HADOOP-1 Updating YARN-2252 Updating YARN-1372 Updating HDFS-7106 Updating YARN-2540 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Created] (MAPREDUCE-6104) TestJobHistoryParsing.testPartialJob fails in branch-2
Mit Desai created MAPREDUCE-6104:
Summary: TestJobHistoryParsing.testPartialJob fails in branch-2
Key: MAPREDUCE-6104
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6104
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Mit Desai
Assignee: Mit Desai
TestJobHistoryParsing.testPartialJob intermittently fails with the following
timeout error.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.239 sec
FAILURE! - in org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
testPartialJob(org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing) Time
elapsed: 1.025 sec ERROR!
java.lang.Exception: test timed out after 1000 milliseconds
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at sun.misc.Resource.getBytes(Resource.java:124)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:444)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.init(Groups.java:77)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:257)
at
org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:234)
at
org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:773)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:758)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:631)
at
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing.testPartialJob(TestJobHistoryParsing.java:824)
Results :
Tests in error:
TestJobHistoryParsing.testPartialJob:824 » test timed out after 1000
millisec...
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] BUILD FAILURE
[INFO]
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Thinking ahead to hadoop-2.6
Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
I would like to see the shared cache (YARN-1492) make it in as well. We are going through the final review process (with Karthik and Vinod) and should be fairly close to complete. This is another feature that has been under development for a while. Thanks, Chris On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Thinking ahead to hadoop-2.6
+1 for end of next week. We have got all the patches for YARN-1051 committed to the branch. We are currently fixing test-patch plan to call a merge vote soon. Hopefully it will go through by end of next week. Thanks, Subru On Tue, Sep 23, 2014 at 4:31 PM, Karthik Kambatla ka...@cloudera.com wrote: Looking at the patches, we might be able to get most of YARN-1492 in by the end of next week. There would be a couple of security items still remaining, but we can may be call the feature alpha-ready without them. On Tue, Sep 23, 2014 at 4:25 PM, Chris Trezzo ctre...@gmail.com wrote: I would like to see the shared cache (YARN-1492) make it in as well. We are going through the final review process (with Karthik and Vinod) and should be fairly close to complete. This is another feature that has been under development for a while. Thanks, Chris On Tue, Sep 23, 2014 at 4:00 PM, Suresh Srinivas sur...@hortonworks.com wrote: I actually would like to see both archival storage and single replica memory writes to be in 2.6 release. Archival storage is in the final stages of getting ready for branch-2 merge as Nicholas has already indicated on the dev mailing list. Hopefully HDFS-6581 gets ready sooner. Both of these features are being in development for sometime. On Tue, Sep 23, 2014 at 3:27 PM, Andrew Wang andrew.w...@cloudera.com wrote: Hey Arun, Maybe we could do a quick run through of the Roadmap wiki and add/retarget things accordingly? I think the KMS and transparent encryption are ready to go. We've got a very few further bug fixes pending, but that's it. Two HDFS things that I think probably won't make the end of the week are archival storage (HDFS-6584) and single replica memory writes (HDFS-6581), which I believe are under the HSM banner. HDFS-6484 was just merged to trunk and I think needs a little more work before it goes into branch-2. HDFS-6581 hasn't even been merged to trunk yet, so seems a bit further off yet. Just my 2c as I did not work directly on these features. I just generally shy away from shipping bits quite this fresh. Thanks, Andrew On Tue, Sep 23, 2014 at 3:03 PM, Arun Murthy a...@hortonworks.com wrote: Looks like most of the content is in and hadoop-2.6 is shaping up nicely. I'll create branch-2.6 by end of the week and we can go from there to stabilize it - hopefully in the next few weeks. Thoughts? thanks, Arun On Tue, Aug 12, 2014 at 1:34 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.5 nearly done, it's time to start thinking ahead to hadoop-2.6. Currently, here is the Roadmap per the wiki: • HADOOP • Credential provider HADOOP-10607 • HDFS • Heterogeneous storage (Phase 2) - Support APIs for using storage tiers by the applications HDFS-5682 • Memory as storage tier HDFS-5851 • YARN • Dynamic Resource Configuration YARN-291 • NodeManager Restart YARN-1336 • ResourceManager HA Phase 2 YARN-556 • Support for admin-specified labels in YARN YARN-796 • Support for automatic, shared cache for YARN application artifacts YARN-1492 • Support NodeGroup layer topology on YARN YARN-18 • Support for Docker containers in YARN YARN-1964 • YARN service registry YARN-913 My suspicion is, as is normal, some will make the cut and some won't. Please do add/subtract from the list as appropriate. Ideally, it would be good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up a cadence. More importantly, as we discussed previously, we'd like hadoop-2.6 to be the *last* Apache Hadoop 2.x release which support JDK6. I'll start a discussion with other communities (HBase, Pig, Hive, Oozie etc.) and see how they feel about this. thanks, Arun -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this
[jira] [Created] (MAPREDUCE-6105) Inconsistent configuration in property mapreduce.reduce.shuffle.merge.percent
Dongwook Kwon created MAPREDUCE-6105:
Summary: Inconsistent configuration in property
mapreduce.reduce.shuffle.merge.percent
Key: MAPREDUCE-6105
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6105
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.4.1, 2.4.0
Reporter: Dongwook Kwon
Priority: Trivial
Similar to MAPREDUCE-5906, In MergeManagerImpl.java, the default value of
MRJobConfig.SHUFFLE_MERGE_PERCENT(mapreduce.reduce.shuffle.merge.percent)
should be 0.66
According to official doc.
https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
{code}
this.mergeThreshold = (long)(this.memoryLimit *
jobConf.getFloat(MRJobConfig.SHUFFLE_MERGE_PERCENT,
0.90f));
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
