Re: [VOTE] Release Apache Hadoop 2.2.0
+1 (non-binding) - Verified md5 checksums and signature. - Build from source, and run some example MR jobs in single node setup. On Sun, Oct 13, 2013 at 7:42 PM, Siddharth Seth ss...@apache.org wrote: +1 (binding) Verified checksums and signature. Built from source and ran some simple MR and Tez jobs. - Sid On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down the symlinks-related issues. I'll release note the fact that we have disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the last couple of weeks. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- - Tsuyoshi
[jira] [Resolved] (MAPREDUCE-5581) killing jobs which have failed causes log missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-5581. --- Resolution: Duplicate This is a duplicate of MAPREDUCE-5502. killing jobs which have failed causes log missing - Key: MAPREDUCE-5581 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5581 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 2.1.1-beta Reporter: Nemon Lou In hive code,when a job failed,they invoke the RunningJob.killJob() API immediately. From mapreduce client side,when job is at failed state,the YARNRunner will invoke resMgrDelegate.killApplication to kill that job.And this prevent AM from writing logs to job history server. -- This message was sent by Atlassian JIRA (v6.1#6144)
Hadoop-Mapreduce-trunk - Build # 1578 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1578/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 33671 lines...] TestEncryptedShuffle.encryptedShuffleWithoutClientCerts:169-encryptedShuffleWithCerts:156 null TestChild.testChild:151-submitAndValidateJob:137 null Tests in error: TestMiniMRWithDFSWithDistinctUsers.setUp:97 » YarnRuntime java.lang.OutOfMemor... TestMiniMRWithDFSWithDistinctUsers.setUp:97 » YarnRuntime java.lang.OutOfMemor... TestReduceFetchFromPartialMem.testReduceFromPartialMem:93-runJob:300 » IO Job... TestJobSysDirWithDFS.testWithDFS:130 » YarnRuntime java.lang.OutOfMemoryError:... TestReduceFetchFromPartialMem.testReduceFromPartialMem:93-runJob:300 » IO Job... TestLazyOutput.testLazyOutput:146 » YarnRuntime java.lang.OutOfMemoryError: un... TestSpecialCharactersInOutputPath.testJobWithDFS:112 » YarnRuntime java.lang.O... TestMapReduceLazyOutput.testLazyOutput:136 » YarnRuntime java.lang.OutOfMemory... TestSpeculativeExecution.setup:122 » IO Cannot run program stat: java.io.IOE... TestMRJobs.setup:130 » YarnRuntime java.lang.OutOfMemoryError: unable to creat... TestRMNMInfo.setup:84 » IO Cannot run program stat: java.io.IOException: err... TestUberAM.setup:45-TestMRJobs.setup:130 » YarnRuntime java.lang.OutOfMemoryE... Tests run: 455, Failures: 8, Errors: 12, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.515s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [45.525s] [INFO] hadoop-mapreduce-client-common SUCCESS [24.742s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [2.438s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [6:47.509s] [INFO] hadoop-mapreduce-client-hs SUCCESS [2:00.518s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [44:55.175s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 54:59.081s [INFO] Finished at: Mon Oct 14 14:14:28 UTC 2013 [INFO] Final Memory: 22M/84M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? [ERROR] Command was/bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient /home/jenkins/tools/java/jdk1.6.0_26/jre/bin/java -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefirebooter457716163653505892.jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire795825364178818457tmp /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/surefire/surefire_1128262753521404065768tmp [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Updating MAPREDUCE-5329 Updating MAPREDUCE-5463 Updating YARN-305 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[jira] [Resolved] (MAPREDUCE-5546) mapred.cmd on Windows set HADOOP_OPTS incorrectly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved MAPREDUCE-5546. -- Resolution: Fixed Fix Version/s: 2.2.1 3.0.0 I've committed this to trunk, branch-2, and branch-2.2. Chuan, thank you for the patch. mapred.cmd on Windows set HADOOP_OPTS incorrectly - Key: MAPREDUCE-5546 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5546 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.2.0 Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 3.0.0, 2.2.1 Attachments: MAPREDUCE-5546-trunk.patch The mapred command on Windows does not set HADOOP_OPTS correctly. As a result, some options and settings will miss in the final command, and this will lead to some desired behavior missing. One example is the logging file setting will miss, i.e. even if one set HADOOP_ROOT_LOGGER to DRFA, there is no history server log at HADOOP_LOGFILE. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5583) Ability to limit running map and reduce tasks
Jason Lowe created MAPREDUCE-5583: - Summary: Ability to limit running map and reduce tasks Key: MAPREDUCE-5583 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am, mrv2 Affects Versions: 2.1.1-beta, 0.23.9 Reporter: Jason Lowe It would be nice if users could specify a limit to the number of map or reduce tasks that are running simultaneously. Occasionally users are performing operations in tasks that can lead to DDoS scenarios if too many tasks run simultaneously (e.g.: accessing a database, web service, etc.). Having the ability to throttle the number of tasks simultaneously running would provide users a way to mitigate issues with too many tasks on a large cluster attempting to access a serivce at any one time. This is similar to the functionality requested by MAPREDUCE-224 and implemented by HADOOP-3412 but was dropped in mrv2. -- This message was sent by Atlassian JIRA (v6.1#6144)
streaming documentation in Hadoop 2?
Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't tied to a specific version, how does it work if new options are added? thanks, -Sandy
Re: streaming documentation in Hadoop 2?
It probably just needs doc, I'd go ahead and file a jira for it. The wiki content here could be a good starting point. On Mon, Oct 14, 2013 at 2:56 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't tied to a specific version, how does it work if new options are added? thanks, -Sandy
Re: streaming documentation in Hadoop 2?
Doc existed in MR1 http://hadoop.apache.org/docs/stable/streaming.html, but it looks like it and a bunch of other stuff (e.g. Rumen and the MapReduce Tutorial) weren't ported over. On Mon, Oct 14, 2013 at 3:20 PM, Eli Collins e...@cloudera.com wrote: It probably just needs doc, I'd go ahead and file a jira for it. The wiki content here could be a good starting point. On Mon, Oct 14, 2013 at 2:56 PM, Sandy Ryza sandy.r...@cloudera.com wrote: Hi All, I noticed that the hadoop streaming documentation does not exist in the Hadoop 2 source tree, and also cannot be found on the internet. Is this on purpose? I found this wiki page http://wiki.apache.org/hadoop/HadoopStreaming - is that where doc is supposed to go? As this page isn't tied to a specific version, how does it work if new options are added? thanks, -Sandy
