Alex, Memory available on namenode / Jobtracker : Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 15360M total, 11609M used, 3750M free, 311M buffers Swap: 2047M total, 1M used, 2046M free, 8833M cached
On datanode / Tasktracker : top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 72373M total, 4321M used, 68051M free, 348M buffers Swap: 2047M total, 0M used, 2047M free, 2771M cached src/mapred/mapred-default.xml : <property> <name>mapred.child.java.opts</name> <value>-Xmx200m</value> </property> So there should be enough memory Anthony > Date: Tue, 27 Dec 2011 11:58:46 +0100 > Subject: Re: Task process exit with nonzero status of 134 > From: [email protected] > To: [email protected] > > Anthony, > > How much memory you have available? Did the system going into swap? > > - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). > - what says "top -Hc"? > > - Alex > > On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[email protected]> > wrote: > > Hi, > > > > I got Nothing in the dmesg > > I've checked the Tasktracker and this is what I got : > > > > /************************************************************ > > STARTUP_MSG: Starting TaskTracker > > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.203.0 > > STARTUP_MSG: build = > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > > ************************************************************/ > > 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: > > loaded properties from hadoop-metrics2.properties > > 2011-12-23 15:11:02,330 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > MetricsSystem,sub=Stats registered. > > 2011-12-23 15:11:02,331 INFO > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period > > at 10 second(s). > > 2011-12-23 15:11:02,331 INFO > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics > > system started > > 2011-12-23 15:11:02,597 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi > > registered. > > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global > > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 > > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting > > tasktracker with owner as root > > 2011-12-23 15:11:02,870 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm > > registered. > > 2011-12-23 15:11:02,871 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > TaskTrackerMetrics registered. > > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting > > SocketReader > > 2011-12-23 15:11:02,900 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > RpcDetailedActivityForPort58709 registered. > > 2011-12-23 15:11:02,900 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > RpcActivityForPort58709 registered. > > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server > > Responder: starting > > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > > listener on 58709: starting > > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 0 on 58709: starting > > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 58709: starting > > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 2 on 58709: starting > > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: > > TaskTracker up at: localhost/127.0.0.1:58709 > > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: Starting > > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 58709: starting > > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: Using > > MemoryCalculatorPlugin : > > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1 > > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: Starting > > thread: Map-events fetcher for all reduce tasks on > > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid > > exited with exit code 0 > > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker: > > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > > disabled. > > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: IndexCache > > created with max memory = 10485760 > > 2011-12-23 15:11:02,976 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > ShuffleServerMetrics registered. > > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port > > returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. > > Opening the listener on 50060 > > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: > > listener.getLocalPort() returned 50060 > > webServer.getConnectors()[0].getLocalPort() returned 50060 > > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty bound > > to port 50060 > > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26 > > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started > > [email protected]:50060 > > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker: > > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231432_0002 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231121_0001 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231312_0002 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231142_0002 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112230933_0002 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231357_0001 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231052_0002 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding > > job_201112231426_0001 for user-log deletion with > > retainTimeStamp:1324735862967 > > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0 task's > > state:UNASSIGNED > > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: Trying to > > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots > > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201112231511_0001_m_000002_0 which needs 1 slots > > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.data.dir; Ignoring. > > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.name.dir; Ignoring. > > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer: > > Initializing user root on this TT. > > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.data.dir; Ignoring. > > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.name.dir; Ignoring. > > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.data.dir; Ignoring. > > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: dfs.name.dir; Ignoring. > > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 > > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner > > jvm_201112231511_0001_m_-1653678851 spawned. > > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController: > > Writing commands to > > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh > > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM with > > ID: jvm_201112231511_0001_m_-1653678851 given task: > > attempt_201112231511_0001_m_000002_0 > > 2011-12-23 15:11:52,675 WARN org.apache.hadoop.mapred.DefaultTaskController: > > Exit code from task is : 134 > > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.DefaultTaskController: > > Output from DefaultTaskController's launchTask follows: > > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController: > > > > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of > > tasks it ran: 0 > > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > > attempt_201112231511_0001_m_000002_0 : Child Error > > java.io.IOException: Task process exit with nonzero status of 134. > > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > > > > > > > > I also checked the logs of the task : > > > > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout > > > > A fatal error has been detected by the Java Runtime Environment: > > # > > # SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472 > > # > > # JRE version: 6.0_30-b12 > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode > > linux-amd64 compressed oops) > > # Problematic frame: > > # C [ld-linux-x86-64.so.2+0x882f] double+0xcf > > # > > # An error report file with more information is saved as: > > # > > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log > > # > > # If you would like to submit a bug report, please visit: > > # http://java.sun.com/webapps/bugreport/crash.jsp > > # The crash happened outside the Java Virtual Machine in native code. > > # See problematic frame for where to report the bug. > > --------------- > > > > > > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog > > > > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > > > > > > On the Job tracker : > > STARTUP_MSG: Starting JobTracker > > STARTUP_MSG: host = yval4950.inetpsa.com/10.68.76.61 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.203.0 > > STARTUP_MSG: build = > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > > ************************************************************/ > > 2011-12-23 15:11:01,401 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: > > loaded properties from hadoop-metrics2.properties > > 2011-12-23 15:11:01,416 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > MetricsSystem,sub=Stats registered. > > 2011-12-23 15:11:01,417 INFO > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period > > at 10 second(s). > > 2011-12-23 15:11:01,417 INFO > > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system > > started > > 2011-12-23 15:11:01,727 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi > > registered. > > 2011-12-23 15:11:01,733 INFO > > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > > Updating the current master key for generating delegation tokens > > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker: Scheduler > > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, > > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) > > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader: > > Refreshing hosts (include/exclude) list > > 2011-12-23 15:11:01,745 INFO > > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > > Starting expired delegation token remover thread, > > tokenRemoverScanInterval=60 min(s) > > 2011-12-23 15:11:01,745 INFO > > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > > Updating the current master key for generating delegation tokens > > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker: Starting > > jobtracker with owner as root > > 2011-12-23 15:11:01,836 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > RpcDetailedActivityForPort9001 registered. > > 2011-12-23 15:11:01,837 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > RpcActivityForPort9001 registered. > > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting > > SocketReader > > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added global > > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port > > returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. > > Opening the listener on 50030 > > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: > > listener.getLocalPort() returned 50030 > > webServer.getConnectors()[0].getLocalPort() returned 50030 > > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty bound > > to port 50030 > > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26 > > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started > > [email protected]:50030 > > 2011-12-23 15:11:02,372 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm > > registered. > > 2011-12-23 15:11:02,373 INFO > > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > > JobTrackerMetrics registered. > > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: JobTracker > > up at: 9001 > > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: JobTracker > > webserver: 50030 > > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker: Cleaning > > up the system directory > > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker: History > > server being initialized in embedded mode > > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobHistoryServer: > > Started job history server at: localhost:50030 > > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job > > History Server web address: localhost:50030 > > 2011-12-23 15:11:02,585 INFO > > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is > > inactive > > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker: Refreshing > > hosts information > > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: Setting > > the includes file to > > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: Setting > > the excludes file to > > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > > Refreshing hosts (include/exclude) list > > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker: > > Decommissioning 0 nodes > > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server > > Responder: starting > > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server > > listener on 9001: starting > > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 0 on 9001: starting > > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 9001: starting > > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 2 on 9001: starting > > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 9001: starting > > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 4 on 9001: starting > > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 6 on 9001: starting > > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 7 on 9001: starting > > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 9 on 9001: starting > > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 5 on 9001: starting > > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker: Starting > > RUNNING > > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 9001: starting > > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology: Adding a > > new node: /default-rack/ylal2960.inetpsa.com > > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: Adding > > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host > > ylal2960.inetpsa.com > > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology: Adding a > > new node: /default-rack/ylal2950.inetpsa.com > > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: Adding > > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host > > ylal2950.inetpsa.com > > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to > > override final parameter: hadoop.tmp.dir; Ignoring. > > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to > > override final parameter: dfs.data.dir; Ignoring. > > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration: > > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to > > override final parameter: dfs.name.dir; Ignoring. > > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress: > > job_201112231511_0001: nMaps=1 nReduces=1 max=-1 > > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job > > job_201112231511_0001 added successfully for user 'root' to queue 'default' > > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger: > > USER=root IP=10.68.76.61 OPERATION=SUBMIT_JOB > > TARGET=job_201112231511_0001 RESULT=SUCCESS > > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker: > > Initializing job_201112231511_0001 > > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress: > > Initializing job_201112231511_0001 > > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress: > > jobToken generated and stored with users keys in > > /users/hadoop/mapred/system/job_201112231511_0001/jobToken > > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress: Input > > size for job job_201112231511_0001 = 661630. Number of splits = 1 > > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201112231511_0001_m_000000 has split on > > node:/default-rack/ylal2960.inetpsa.com > > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5 > > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: Job > > job_201112231511_0001 initialized successfully with 1 map tasks and 1 reduce > > tasks. > > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip > > task_201112231511_0001_m_000002, for tracker > > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_0 is 151247 ms debug. > > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_0 is 351649 ms debug. > > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_0 is 551655 ms debug. > > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: Lost > > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress: Error > > from attempt_201112231511_0001_m_000002_0: Lost task tracker: > > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: Removing > > task 'attempt_201112231511_0001_m_000002_0' > > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_0 is 752357 ms debug. > > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker: Launching > > task attempt_201112231511_0001_m_000002_0 timed out. > > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip > > task_201112231511_0001_m_000002, for tracker > > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_1 is 198155 ms debug. > > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_1 is 398161 ms debug. > > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_1 is 598166 ms debug. > > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: > > attempt_201112231511_0001_m_000002_1 is 798172 ms debug. > > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: Launching > > task attempt_201112231511_0001_m_000002_1 timed out. > > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress: Error > > from attempt_201112231511_0001_m_000002_1: Error launching task > > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Lost > > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress: Error > > from attempt_201112231511_0001_m_000002_1: Lost task tracker: > > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 > > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Removing > > task 'attempt_201112231511_0001_m_000002_1' > > > > > > > > Date: Fri, 23 Dec 2011 15:40:20 +0100 > > Subject: Re: Task process exit with nonzero status of 134 > > From: [email protected] > > To: [email protected] > > > > > > Hi, > > > > take a look into the logs for the failed attempt at your Tasktracker. > > Also check the system logs with dmesg or /var/log/kern*. Could be a > > syskill (segfault). > > > > - Alex > > > > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <[email protected]> > > wrote: > >> > >> Hi folks, > >> > >> I've just done a fresh install of Hadoop, Namenode and datanode are up, > >> Task/job Tracker also up, but when I run the Map reduce worcount exemple I > >> got this error on Task tracker: > >> > >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : > >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of > >> tasks it ran: 0 > >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > >> attempt_201112231511_0001_m_000002_0 : Child Error > >> java.io.IOException: Task process exit with nonzero status of 134. > >> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > >> > >> And job tracker stuck : > >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ > >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process > >> : 1 > >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: > >> job_201112231511_0001 > >> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% > >> > >> > >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25 > >> > >> I've done some googling, apparently the JVM crash hard (maybe Out of > >> memory), does someone have any hint ? > >> > >> Regards, > >> > >> Anthony Garnier > >> /DSIN/ASTI/ETSO > >> IT Center > >> PSA Peugeot Citroen > >> Bessoncourt 90160 > >> > >> > >> > >> > > > > > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > P Think of the environment: please don't print this email unless you > > really need to. > > > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > P Think of the environment: please don't print this email unless you > really need to.
