Alex java -version -Xmx1024m is running fine
I was also able to create directory in /users/hadoop-0.20.203.0/log/userlogs But anyway I've doe a fresh install with the 0.20.205.0 version and the mapreduce job completed successfully ! Thank you for your help. Anthony > Date: Tue, 27 Dec 2011 17:06:28 +0100 > Subject: Re: Task process exit with nonzero status of 134 > From: [email protected] > To: [email protected] > > Anthoney, > > That could be when you try to allocate some dedicated space and it is > not available or present on your machine. > Did -Xmx has an dedicated memory value that you can provide to JVM on > your machine (e.g. "-Xmx1024m")? > > Second - filesystem full (mostly hadoop/userlogs). Ext3 has an limit > of 32k links per node. Try to create a directory in (mkdir > hadoop/userlogs/123), if you have too much directories an error will > occur. > > - Alex > > > > > > On Tue, Dec 27, 2011 at 3:18 PM, anthony garnier <[email protected]> > wrote: > > Alex > > I don't know why but now I got other error code : > > > > On Jobtracker : > > java.lang.Throwable: Child Error > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) > > Caused by: java.io.IOException: Task process exit with nonzero status of 1. > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > > > > On taskTracker : > > > > 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController: > > Output from DefaultTaskController's launchTask follows: > > 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController: > > 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not > > killed jvm_201112271514_0001_m_-1820714542 but just removed > > 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201112271514_0001_m_-1820714542 exited with exit code 1. Number of tasks > > it ran: 0 > > 2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner: > > attempt_201112271514_0001_m_000002_0 : Child Error > > java.io.IOException: Task process exit with nonzero status of 1. > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > > 2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 2 > > > > > > ________________________________ > > From: [email protected] > > To: [email protected]; [email protected] > > Subject: RE: Task process exit with nonzero status of 134 > > Date: Tue, 27 Dec 2011 13:54:22 +0000 > > > > > > Alex > > > > -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit > > JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) > > Anyway, I tested it but same result. > > Is it wise to test hadoop with the new jdk7_2 ? > > > > Anthony > > > > > >> Date: Tue, 27 Dec 2011 13:47:03 +0100 > >> Subject: Re: Task process exit with nonzero status of 134 > >> From: [email protected] > > > >> To: [email protected] > >> > >> Anthony, > >> > >> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest > >> update, could be a bug inside. Can you test SE 7u2? > >> Todd mentioned in a older post to use -XX:+UseCompressedOops > >> (hadoop-env.sh). Another option could be to take a closer look at > >> garbage collection with compressed option. > >> > >> - Alex > >> > >> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[email protected]> > >> wrote: > >> > Alex, > >> > > >> > Memory available on namenode / Jobtracker : > >> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie > >> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0%st > >> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers > >> > Swap: 2047M total, 1M used, 2046M free, 8833M cached > >> > > >> > On datanode / Tasktracker : > >> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, > >> > 0.26 > >> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie > >> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, > >> > 0.0%st > >> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers > >> > Swap: 2047M total, 0M used, 2047M free, 2771M cached > >> > > >> > src/mapred/mapred-default.xml : > >> > <property> > >> > <name>mapred.child.java.opts</name> > >> > <value>-Xmx200m</value> > >> > </property> > >> > > >> > > >> > So there should be enough memory > >> > > >> > Anthony > >> > > >> > > >> >> Date: Tue, 27 Dec 2011 11:58:46 +0100 > >> > > >> >> Subject: Re: Task process exit with nonzero status of 134 > >> >> From: [email protected] > >> >> To: [email protected] > >> > > >> >> > >> >> Anthony, > >> >> > >> >> How much memory you have available? Did the system going into swap? > >> >> > >> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize > >> >> (xmx). > >> >> - what says "top -Hc"? > >> >> > >> >> - Alex > >> >> > >> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier > >> >> <[email protected]> > >> >> wrote: > >> >> > Hi, > >> >> > > >> >> > I got Nothing in the dmesg > >> >> > I've checked the Tasktracker and this is what I got : > >> >> > > >> >> > /************************************************************ > >> >> > STARTUP_MSG: Starting TaskTracker > >> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 > >> >> > STARTUP_MSG: args = [] > >> >> > STARTUP_MSG: version = 0.20.203.0 > >> >> > STARTUP_MSG: build = > >> >> > > >> >> > > >> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > >> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > >> >> > ************************************************************/ > >> >> > 2011-12-23 15:11:02,275 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsConfig: > >> >> > loaded properties from hadoop-metrics2.properties > >> >> > 2011-12-23 15:11:02,330 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > MetricsSystem,sub=Stats registered. > >> >> > 2011-12-23 15:11:02,331 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > >> >> > period > >> >> > at 10 second(s). > >> >> > 2011-12-23 15:11:02,331 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker > >> >> > metrics > >> >> > system started > >> >> > 2011-12-23 15:11:02,597 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > ugi > >> >> > registered. > >> >> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to > >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > >> >> > org.mortbay.log.Slf4jLog > >> >> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added > >> >> > global > >> >> > filtersafety > >> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > >> >> > 2011-12-23 15:11:02,827 INFO > >> >> > org.apache.hadoop.mapred.TaskLogsTruncater: > >> >> > Initializing logs' truncater with mapRetainSize=-1 and > >> >> > reduceRetainSize=-1 > >> >> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > Starting > >> >> > tasktracker with owner as root > >> >> > 2011-12-23 15:11:02,870 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > jvm > >> >> > registered. > >> >> > 2011-12-23 15:11:02,871 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > TaskTrackerMetrics registered. > >> >> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting > >> >> > SocketReader > >> >> > 2011-12-23 15:11:02,900 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > RpcDetailedActivityForPort58709 registered. > >> >> > 2011-12-23 15:11:02,900 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > RpcActivityForPort58709 registered. > >> >> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > Responder: starting > >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > listener on 58709: starting > >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 0 on 58709: starting > >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 1 on 58709: starting > >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 2 on 58709: starting > >> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > TaskTracker up at: localhost/127.0.0.1:58709 > >> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > Starting > >> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> >> > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 3 on 58709: starting > >> >> > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > Using > >> >> > MemoryCalculatorPlugin : > >> >> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1 > >> >> > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > Starting > >> >> > thread: Map-events fetcher for all reduce tasks on > >> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> >> > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: > >> >> > setsid > >> >> > exited with exit code 0 > >> >> > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker: > >> >> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > >> >> > disabled. > >> >> > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: > >> >> > IndexCache > >> >> > created with max memory = 10485760 > >> >> > 2011-12-23 15:11:02,976 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > ShuffleServerMetrics registered. > >> >> > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port > >> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() > >> >> > is > >> >> > -1. > >> >> > Opening the listener on 50060 > >> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: > >> >> > listener.getLocalPort() returned 50060 > >> >> > webServer.getConnectors()[0].getLocalPort() returned 50060 > >> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty > >> >> > bound > >> >> > to port 50060 > >> >> > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26 > >> >> > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started > >> >> > [email protected]:50060 > >> >> > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231432_0002 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231121_0001 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231312_0002 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231142_0002 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112230933_0002 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231357_0001 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231052_0002 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> >> > Adding > >> >> > job_201112231426_0001 for user-log deletion with > >> >> > retainTimeStamp:1324735862967 > >> >> > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0 > >> >> > task's > >> >> > state:UNASSIGNED > >> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > Trying to > >> >> > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots > >> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In > >> >> > TaskLauncher, current free slots : 2 and trying to launch > >> >> > attempt_201112231511_0001_m_000002_0 which needs 1 slots > >> >> > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer: > >> >> > Initializing user root on this TT. > >> >> > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> >> > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In > >> >> > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 > >> >> > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM > >> >> > Runner > >> >> > jvm_201112231511_0001_m_-1653678851 spawned. > >> >> > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController: > >> >> > Writing commands to > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh > >> >> > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: > >> >> > JVM > >> >> > with > >> >> > ID: jvm_201112231511_0001_m_-1653678851 given task: > >> >> > attempt_201112231511_0001_m_000002_0 > >> >> > 2011-12-23 15:11:52,675 WARN > >> >> > org.apache.hadoop.mapred.DefaultTaskController: > >> >> > Exit code from task is : 134 > >> >> > 2011-12-23 15:11:52,675 INFO > >> >> > org.apache.hadoop.mapred.DefaultTaskController: > >> >> > Output from DefaultTaskController's launchTask follows: > >> >> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController: > >> >> > > >> >> > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM > >> >> > : > >> >> > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number > >> >> > of > >> >> > tasks it ran: 0 > >> >> > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > >> >> > attempt_201112231511_0001_m_000002_0 : Child Error > >> >> > java.io.IOException: Task process exit with nonzero status of 134. > >> >> > at > >> >> > org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > >> >> > > >> >> > > >> >> > > >> >> > I also checked the logs of the task : > >> >> > > >> >> > > >> >> > > >> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout > >> >> > > >> >> > A fatal error has been detected by the Java Runtime Environment: > >> >> > # > >> >> > # SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472 > >> >> > # > >> >> > # JRE version: 6.0_30-b12 > >> >> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode > >> >> > linux-amd64 compressed oops) > >> >> > # Problematic frame: > >> >> > # C [ld-linux-x86-64.so.2+0x882f] double+0xcf > >> >> > # > >> >> > # An error report file with more information is saved as: > >> >> > # > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log > >> >> > # > >> >> > # If you would like to submit a bug report, please visit: > >> >> > # http://java.sun.com/webapps/bugreport/crash.jsp > >> >> > # The crash happened outside the Java Virtual Machine in native code. > >> >> > # See problematic frame for where to report the bug. > >> >> > --------------- > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog > >> >> > > >> >> > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration: > >> >> > > >> >> > > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> >> > > >> >> > > >> >> > On the Job tracker : > >> >> > STARTUP_MSG: Starting JobTracker > >> >> > STARTUP_MSG: host = yval4950.inetpsa.com/10.68.76.61 > >> >> > STARTUP_MSG: args = [] > >> >> > STARTUP_MSG: version = 0.20.203.0 > >> >> > STARTUP_MSG: build = > >> >> > > >> >> > > >> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > >> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > >> >> > ************************************************************/ > >> >> > 2011-12-23 15:11:01,401 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsConfig: > >> >> > loaded properties from hadoop-metrics2.properties > >> >> > 2011-12-23 15:11:01,416 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > MetricsSystem,sub=Stats registered. > >> >> > 2011-12-23 15:11:01,417 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > >> >> > period > >> >> > at 10 second(s). > >> >> > 2011-12-23 15:11:01,417 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics > >> >> > system > >> >> > started > >> >> > 2011-12-23 15:11:01,727 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > ugi > >> >> > registered. > >> >> > 2011-12-23 15:11:01,733 INFO > >> >> > > >> >> > > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> >> > Updating the current master key for generating delegation tokens > >> >> > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Scheduler > >> >> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, > >> >> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) > >> >> > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader: > >> >> > Refreshing hosts (include/exclude) list > >> >> > 2011-12-23 15:11:01,745 INFO > >> >> > > >> >> > > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> >> > Starting expired delegation token remover thread, > >> >> > tokenRemoverScanInterval=60 min(s) > >> >> > 2011-12-23 15:11:01,745 INFO > >> >> > > >> >> > > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> >> > Updating the current master key for generating delegation tokens > >> >> > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Starting > >> >> > jobtracker with owner as root > >> >> > 2011-12-23 15:11:01,836 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > RpcDetailedActivityForPort9001 registered. > >> >> > 2011-12-23 15:11:01,837 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > RpcActivityForPort9001 registered. > >> >> > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting > >> >> > SocketReader > >> >> > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to > >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > >> >> > org.mortbay.log.Slf4jLog > >> >> > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added > >> >> > global > >> >> > filtersafety > >> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > >> >> > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port > >> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() > >> >> > is > >> >> > -1. > >> >> > Opening the listener on 50030 > >> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: > >> >> > listener.getLocalPort() returned 50030 > >> >> > webServer.getConnectors()[0].getLocalPort() returned 50030 > >> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty > >> >> > bound > >> >> > to port 50030 > >> >> > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26 > >> >> > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started > >> >> > [email protected]:50030 > >> >> > 2011-12-23 15:11:02,372 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > jvm > >> >> > registered. > >> >> > 2011-12-23 15:11:02,373 INFO > >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for > >> >> > source > >> >> > JobTrackerMetrics registered. > >> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > JobTracker > >> >> > up at: 9001 > >> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > JobTracker > >> >> > webserver: 50030 > >> >> > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Cleaning > >> >> > up the system directory > >> >> > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > History > >> >> > server being initialized in embedded mode > >> >> > 2011-12-23 15:11:02,581 INFO > >> >> > org.apache.hadoop.mapred.JobHistoryServer: > >> >> > Started job history server at: localhost:50030 > >> >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job > >> >> > History Server web address: localhost:50030 > >> >> > 2011-12-23 15:11:02,585 INFO > >> >> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store > >> >> > is > >> >> > inactive > >> >> > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Refreshing > >> >> > hosts information > >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> >> > Setting > >> >> > the includes file to > >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> >> > Setting > >> >> > the excludes file to > >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> >> > Refreshing hosts (include/exclude) list > >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Decommissioning 0 nodes > >> >> > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > Responder: starting > >> >> > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > listener on 9001: starting > >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 0 on 9001: starting > >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 1 on 9001: starting > >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 2 on 9001: starting > >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 3 on 9001: starting > >> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 4 on 9001: starting > >> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 6 on 9001: starting > >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 7 on 9001: starting > >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 9 on 9001: starting > >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 5 on 9001: starting > >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Starting > >> >> > RUNNING > >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> >> > handler 8 on 9001: starting > >> >> > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology: > >> >> > Adding a > >> >> > new node: /default-rack/ylal2960.inetpsa.com > >> >> > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Adding > >> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to > >> >> > host > >> >> > ylal2960.inetpsa.com > >> >> > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology: > >> >> > Adding a > >> >> > new node: /default-rack/ylal2950.inetpsa.com > >> >> > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Adding > >> >> > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to > >> >> > host > >> >> > ylal2950.inetpsa.com > >> >> > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration: > >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> >> > attempt to > >> >> > override final parameter: hadoop.tmp.dir; Ignoring. > >> >> > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration: > >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> >> > attempt to > >> >> > override final parameter: dfs.data.dir; Ignoring. > >> >> > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration: > >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> >> > attempt to > >> >> > override final parameter: dfs.name.dir; Ignoring. > >> >> > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > job_201112231511_0001: nMaps=1 nReduces=1 max=-1 > >> >> > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job > >> >> > job_201112231511_0001 added successfully for user 'root' to queue > >> >> > 'default' > >> >> > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger: > >> >> > USER=root IP=10.68.76.61 OPERATION=SUBMIT_JOB > >> >> > TARGET=job_201112231511_0001 RESULT=SUCCESS > >> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Initializing job_201112231511_0001 > >> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > Initializing job_201112231511_0001 > >> >> > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > jobToken generated and stored with users keys in > >> >> > /users/hadoop/mapred/system/job_201112231511_0001/jobToken > >> >> > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > Input > >> >> > size for job job_201112231511_0001 = 661630. Number of splits = 1 > >> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > tip:task_201112231511_0001_m_000000 has split on > >> >> > node:/default-rack/ylal2960.inetpsa.com > >> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5 > >> >> > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: > >> >> > Job > >> >> > job_201112231511_0001 initialized successfully with 1 map tasks and 1 > >> >> > reduce > >> >> > tasks. > >> >> > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Adding > >> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip > >> >> > task_201112231511_0001_m_000002, for tracker > >> >> > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > >> >> > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_0 is 151247 ms debug. > >> >> > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_0 is 351649 ms debug. > >> >> > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_0 is 551655 ms debug. > >> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Lost > >> >> > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > >> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress: > >> >> > Error > >> >> > from attempt_201112231511_0001_m_000002_0: Lost task tracker: > >> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Removing > >> >> > task 'attempt_201112231511_0001_m_000002_0' > >> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_0 is 752357 ms debug. > >> >> > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Launching > >> >> > task attempt_201112231511_0001_m_000002_0 timed out. > >> >> > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Adding > >> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip > >> >> > task_201112231511_0001_m_000002, for tracker > >> >> > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > >> >> > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_1 is 198155 ms debug. > >> >> > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_1 is 398161 ms debug. > >> >> > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_1 is 598166 ms debug. > >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > attempt_201112231511_0001_m_000002_1 is 798172 ms debug. > >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Launching > >> >> > task attempt_201112231511_0001_m_000002_1 timed out. > >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress: > >> >> > Error > >> >> > from attempt_201112231511_0001_m_000002_1: Error launching task > >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Lost > >> >> > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress: > >> >> > Error > >> >> > from attempt_201112231511_0001_m_000002_1: Lost task tracker: > >> >> > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 > >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: > >> >> > Removing > >> >> > task 'attempt_201112231511_0001_m_000002_1' > >> >> > > >> >> > > >> >> > > >> >> > Date: Fri, 23 Dec 2011 15:40:20 +0100 > >> >> > Subject: Re: Task process exit with nonzero status of 134 > >> >> > From: [email protected] > >> >> > To: [email protected] > >> >> > > >> >> > > >> >> > Hi, > >> >> > > >> >> > take a look into the logs for the failed attempt at your Tasktracker. > >> >> > Also check the system logs with dmesg or /var/log/kern*. Could be a > >> >> > syskill (segfault). > >> >> > > >> >> > - Alex > >> >> > > >> >> > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier > >> >> > <[email protected]> > >> >> > wrote: > >> >> >> > >> >> >> Hi folks, > >> >> >> > >> >> >> I've just done a fresh install of Hadoop, Namenode and datanode are > >> >> >> up, > >> >> >> Task/job Tracker also up, but when I run the Map reduce worcount > >> >> >> exemple I > >> >> >> got this error on Task tracker: > >> >> >> > >> >> >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: > >> >> >> JVM : > >> >> >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. > >> >> >> Number > >> >> >> of > >> >> >> tasks it ran: 0 > >> >> >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > >> >> >> attempt_201112231511_0001_m_000002_0 : Child Error > >> >> >> java.io.IOException: Task process exit with nonzero status of 134. > >> >> >> at > >> >> >> org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > >> >> >> > >> >> >> And job tracker stuck : > >> >> >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ > >> >> >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to > >> >> >> process > >> >> >> : 1 > >> >> >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: > >> >> >> job_201112231511_0001 > >> >> >> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% > >> >> >> > >> >> >> > >> >> >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25 > >> >> >> > >> >> >> I've done some googling, apparently the JVM crash hard (maybe Out of > >> >> >> memory), does someone have any hint ? > >> >> >> > >> >> >> Regards, > >> >> >> > >> >> >> Anthony Garnier > >> >> >> /DSIN/ASTI/ETSO > >> >> >> IT Center > >> >> >> PSA Peugeot Citroen > >> >> >> Bessoncourt 90160 > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Alexander Lorenz > >> >> > http://mapredit.blogspot.com > >> >> > > >> >> > P Think of the environment: please don't print this email unless you > >> >> > really need to. > >> >> > >> >> > >> >> > >> >> -- > >> >> Alexander Lorenz > >> >> http://mapredit.blogspot.com > >> >> > >> >> P Think of the environment: please don't print this email unless you > >> >> really need to. > >> > >> > >> > >> -- > >> Alexander Lorenz > >> http://mapredit.blogspot.com > >> > >> P Think of the environment: please don't print this email unless you > >> really need to. > > > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > P Think of the environment: please don't print this email unless you > really need to.
