Question on mapred. vs mapreduce. settings for Hadoop trunk and 2.x
Regards By looking at the code in trunk and release 2.0.5 I noticed that while most settings related to MapReduce now have the mapreduce prefix (as in mapreduce.job.dir) there are still a few that have the mapred prefix, as in mapred.input.format.class, mapred.mapper.class, and others. Is there a reason for this? Or is this just legacy code? Thanks ./g PS: A related discussion on this topic is here: http://stackoverflow.com/questions/10986633/hadoop-configuration-mapred-vs-m apreduce ./g German Florez-Larrahondo, Ph.D. Distributed Computing Storage SW Engineer SARC Architecture Samsung Austin RD Center (SARC) 7300 RR Austin, TX 78730 Office: 512-425-3365 Mobile: 205-441-0266
Reduces Failing with 'Child Error'
Hi guys I keep getting my reduces to Fail and I can't get a clue of what is going on and it's quite frustrating. Could you help me? any ideas? I'm sending some info, let me know if you need more. Regards Tomas I'm running a cluster of 10 slaves, with EC2 m1.xlarge's and a attached volume of EBS of 80Gb, for data, plus the ephemeral nodes for local mapred *Instance Family**Instance Type**Processor Arch**vCPU**ECU**Memory (GiB)**Instance Storage (GB)**EBS-optimized Available**Network Performance* General purposem1.xlarge64-bit48154 x 420YesHigh Job Kind% CompleteNum TasksPendingRunningCompleteKilledFailed/Killed Task Attemptsmap100.00%245002212441 / 77reduce100.00%6800333555 / 43 attempt_201307181643_0007_r_00_0task_201307181643_0007_r_00 FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) task_201307181643_0007_r_01 FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) attempt_201307181643_0007_r_01_1task_201307181643_0007_r_01 FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) attempt_201307181643_0007_r_02_0task_201307181643_0007_r_02 FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Task process exit with nonzero status of 137. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) The tasktracker log is as follows: hadoop@cluster-slaves-00:/opt/hadoop/hadoop-1.0.3/logs$ grep attempt_201307181643_0007_r_63_0 hadoop-hadoop-tasktracker-cluster-slaves-00.log 2013-07-18 17:10:57,401 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201307181643_0007_r_63_0 task's state:UNASSIGNED 2013-07-18 17:10:57,401 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201307181643_0007_r_63_0 which needs 1 slots 2013-07-18 17:10:57,402 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201307181643_0007_r_63_0 which needs 1 slots 2013-07-18 17:10:57,549 INFO org.apache.hadoop.mapred.JvmManager: No new JVM spawned for jobId/taskid: job_201307181643_0007/attempt_201307181643_0007_r_63_0. Attempting to reuse: jvm_201307181643_0007_r_-901518427 2013-07-18 17:10:57,748 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201307181643_0007_r_-901518427 given task: attempt_201307181643_0007_r_63_0 2013-07-18 17:11:04,658 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.0% reduce copy 2013-07-18 17:11:11,139 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.02312925% reduce copy (17 of 245 at 7.61 MB/s) 2013-07-18 17:11:14,233 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.04217687% reduce copy (31 of 245 at 10.66 MB/s) 2013-07-18 17:11:17,274 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.0585034% reduce copy (43 of 245 at 12.29 MB/s) 2013-07-18 17:11:20,550 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.07755102% reduce copy (57 of 245 at 14.04 MB/s) 2013-07-18 17:11:24,342 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.10748299% reduce copy (79 of 245 at 16.99 MB/s) 2013-07-18 17:11:27,417 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.13469388% reduce copy (99 of 245 at 18.17 MB/s) 2013-07-18 17:11:30,502 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.14829932% reduce copy (109 of 245 at 18.03 MB/s) 2013-07-18 17:11:33,605 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.17687075% reduce copy (130 of 245 at 19.56 MB/s) 2013-07-18 17:11:37,243 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.20408162% reduce copy (150 of 245 at 20.46 MB/s) 2013-07-18 17:11:40,321 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.24081632% reduce copy (177 of 245 at 21.61 MB/s) 2013-07-18 17:11:43,395 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201307181643_0007_r_63_0 0.27074832% reduce copy (199 of 245 at 22.40 MB/s) 2013-07-18 17:11:46,497 INFO org.apache.hadoop.mapred.TaskTracker:
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
As I've mentioned in my previous email, if we get YARN-701 in, we should also get in the fix for unmanaged AMs in an un-secure setup in 2.1.0-beta. Else is a regression of a functionality it is already working. Because of that, to avoid continuing delaying the release, I'm suggesting to mention in the release notes the API changes and behavior changes that YARN-918 and YARN-701 will bring into the next beta or GA release. thx On Wed, Jul 17, 2013 at 4:14 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: On Jul 17, 2013, at 1:04 PM, Alejandro Abdelnur wrote: * YARN-701 It should be addressed before a GA release. Still, as it is this breaks unmanaged AMs and to me that would be a blocker for the beta. YARN-701 and the unmanaged AMs fix should be committed in tandem. * YARN-918 It is a consequence of YARN-701 and depends on it. YARN-918 is an API change. And YARN-701 is a behaviour change. We need both in 2.1.0. * YARN-926 It would be nice to have it addressed before GA release. Either ways. I'd get it in sooner than later specifically when we are trying to replace the old API with the new one. Thanks, +Vino
[jira] [Created] (MAPREDUCE-5403) Get rid of yarn.application.classpath
Sandy Ryza created MAPREDUCE-5403: - Summary: Get rid of yarn.application.classpath Key: MAPREDUCE-5403 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 2.0.5-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza yarn.application.classpath is a confusing property because it is used by MapReduce and not YARN, and MapReduce already has mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5404) MAPREDUCE-5265 should have used ephemeral port
Ted Yu created MAPREDUCE-5404:
-
Summary: MAPREDUCE-5265 should have used ephemeral port
Key: MAPREDUCE-5404
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5404
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ted Yu
I ran HBase trunk tests against 2.2.0-SNAPSHOT and many mapreduce jobs failed.
Here is one example:
{code}
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan2 Time elapsed:
0.001 sec ERROR!
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.service.ServiceStateException: java.net.BindException:
Problem binding to [0.0.0.0:10033] java.net.BindException: Address already in
use; For more details see: http://wiki.apache.org/hadoop/BindException
at
org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStart(MiniMRYarnCluster.java:177)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:80)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:183)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:171)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:163)
at org.apache.hadoop.mapred.MiniMRCluster.init(MiniMRCluster.java:124)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1751)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1692)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScanBase.setUpBeforeClass(TestTableInputFormatScanBase.java:84)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runners.Suite.runChild(Suite.java:127)
at org.junit.runners.Suite.runChild(Suite.java:26)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.hadoop.service.ServiceStateException:
java.net.BindException: Problem binding to [0.0.0.0:10033]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:88)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:91)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.MiniMRYarnCluster$JobHistoryServerWrapper.serviceStart(MiniMRYarnCluster.java:161)
... 30 more
Caused by: java.net.BindException: Problem binding to [0.0.0.0:10033]
java.net.BindException: Address already in use; For more details see:
http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:719)
at org.apache.hadoop.ipc.Server.bind(Server.java:398)
at org.apache.hadoop.ipc.Server$Listener.init(Server.java:501)
at org.apache.hadoop.ipc.Server.init(Server.java:2168)
at org.apache.hadoop.ipc.RPC$Server.init(RPC.java:997)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.init(ProtobufRpcEngine.java:505)
at
