Alex -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) Anyway, I tested it but same result. Is it wise to test hadoop with the new jdk7_2 ?
Anthony > Date: Tue, 27 Dec 2011 13:47:03 +0100 > Subject: Re: Task process exit with nonzero status of 134 > From: [email protected] > To: [email protected] > > Anthony, > > 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest > update, could be a bug inside. Can you test SE 7u2? > Todd mentioned in a older post to use -XX:+UseCompressedOops > (hadoop-env.sh). Another option could be to take a closer look at > garbage collection with compressed option. > > - Alex > > On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[email protected]> > wrote: > > Alex, > > > > Memory available on namenode / Jobtracker : > > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 15360M total, 11609M used, 3750M free, 311M buffers > > Swap: 2047M total, 1M used, 2046M free, 8833M cached > > > > On datanode / Tasktracker : > > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 > > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 72373M total, 4321M used, 68051M free, 348M buffers > > Swap: 2047M total, 0M used, 2047M free, 2771M cached > > > > src/mapred/mapred-default.xml : > > <property> > > <name>mapred.child.java.opts</name> > > <value>-Xmx200m</value> > > </property> > > > > > > So there should be enough memory > > > > Anthony > > > > > >> Date: Tue, 27 Dec 2011 11:58:46 +0100 > > > >> Subject: Re: Task process exit with nonzero status of 134 > >> From: [email protected] > >> To: [email protected] > > > >> > >> Anthony, > >> > >> How much memory you have available? Did the system going into swap? > >> > >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). > >> - what says "top -Hc"? > >> > >> - Alex > >> > >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[email protected]> > >> wrote: > >> > Hi, > >> > > >> > I got Nothing in the dmesg > >> > I've checked the Tasktracker and this is what I got : > >> > > >> > /************************************************************ > >> > STARTUP_MSG: Starting TaskTracker > >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 > >> > STARTUP_MSG: args = [] > >> > STARTUP_MSG: version = 0.20.203.0 > >> > STARTUP_MSG: build = > >> > > >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > >> > ************************************************************/ > >> > 2011-12-23 15:11:02,275 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsConfig: > >> > loaded properties from hadoop-metrics2.properties > >> > 2011-12-23 15:11:02,330 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > MetricsSystem,sub=Stats registered. > >> > 2011-12-23 15:11:02,331 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > >> > period > >> > at 10 second(s). > >> > 2011-12-23 15:11:02,331 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics > >> > system started > >> > 2011-12-23 15:11:02,597 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > ugi > >> > registered. > >> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to > >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > >> > org.mortbay.log.Slf4jLog > >> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added > >> > global > >> > filtersafety > >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > >> > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: > >> > Initializing logs' truncater with mapRetainSize=-1 and > >> > reduceRetainSize=-1 > >> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: > >> > Starting > >> > tasktracker with owner as root > >> > 2011-12-23 15:11:02,870 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > jvm > >> > registered. > >> > 2011-12-23 15:11:02,871 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > TaskTrackerMetrics registered. > >> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting > >> > SocketReader > >> > 2011-12-23 15:11:02,900 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > RpcDetailedActivityForPort58709 registered. > >> > 2011-12-23 15:11:02,900 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > RpcActivityForPort58709 registered. > >> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > Responder: starting > >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > listener on 58709: starting > >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 0 on 58709: starting > >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 1 on 58709: starting > >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 2 on 58709: starting > >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: > >> > TaskTracker up at: localhost/127.0.0.1:58709 > >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: > >> > Starting > >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 3 on 58709: starting > >> > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: > >> > Using > >> > MemoryCalculatorPlugin : > >> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1 > >> > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: > >> > Starting > >> > thread: Map-events fetcher for all reduce tasks on > >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid > >> > exited with exit code 0 > >> > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker: > >> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > >> > disabled. > >> > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: > >> > IndexCache > >> > created with max memory = 10485760 > >> > 2011-12-23 15:11:02,976 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > ShuffleServerMetrics registered. > >> > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port > >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is > >> > -1. > >> > Opening the listener on 50060 > >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: > >> > listener.getLocalPort() returned 50060 > >> > webServer.getConnectors()[0].getLocalPort() returned 50060 > >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty > >> > bound > >> > to port 50060 > >> > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26 > >> > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started > >> > [email protected]:50060 > >> > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker: > >> > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231432_0002 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231121_0001 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231312_0002 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231142_0002 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112230933_0002 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231357_0001 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231052_0002 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: > >> > Adding > >> > job_201112231426_0001 for user-log deletion with > >> > retainTimeStamp:1324735862967 > >> > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: > >> > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0 > >> > task's > >> > state:UNASSIGNED > >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: > >> > Trying to > >> > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots > >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In > >> > TaskLauncher, current free slots : 2 and trying to launch > >> > attempt_201112231511_0001_m_000002_0 which needs 1 slots > >> > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer: > >> > Initializing user root on this TT. > >> > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.data.dir; Ignoring. > >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: dfs.name.dir; Ignoring. > >> > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In > >> > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 > >> > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM > >> > Runner > >> > jvm_201112231511_0001_m_-1653678851 spawned. > >> > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController: > >> > Writing commands to > >> > > >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh > >> > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM > >> > with > >> > ID: jvm_201112231511_0001_m_-1653678851 given task: > >> > attempt_201112231511_0001_m_000002_0 > >> > 2011-12-23 15:11:52,675 WARN > >> > org.apache.hadoop.mapred.DefaultTaskController: > >> > Exit code from task is : 134 > >> > 2011-12-23 15:11:52,675 INFO > >> > org.apache.hadoop.mapred.DefaultTaskController: > >> > Output from DefaultTaskController's launchTask follows: > >> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController: > >> > > >> > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : > >> > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of > >> > tasks it ran: 0 > >> > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > >> > attempt_201112231511_0001_m_000002_0 : Child Error > >> > java.io.IOException: Task process exit with nonzero status of 134. > >> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > >> > > >> > > >> > > >> > I also checked the logs of the task : > >> > > >> > > >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout > >> > > >> > A fatal error has been detected by the Java Runtime Environment: > >> > # > >> > # SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472 > >> > # > >> > # JRE version: 6.0_30-b12 > >> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode > >> > linux-amd64 compressed oops) > >> > # Problematic frame: > >> > # C [ld-linux-x86-64.so.2+0x882f] double+0xcf > >> > # > >> > # An error report file with more information is saved as: > >> > # > >> > > >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log > >> > # > >> > # If you would like to submit a bug report, please visit: > >> > # http://java.sun.com/webapps/bugreport/crash.jsp > >> > # The crash happened outside the Java Virtual Machine in native code. > >> > # See problematic frame for where to report the bug. > >> > --------------- > >> > > >> > > >> > > >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog > >> > > >> > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration: > >> > > >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a > >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. > >> > > >> > > >> > On the Job tracker : > >> > STARTUP_MSG: Starting JobTracker > >> > STARTUP_MSG: host = yval4950.inetpsa.com/10.68.76.61 > >> > STARTUP_MSG: args = [] > >> > STARTUP_MSG: version = 0.20.203.0 > >> > STARTUP_MSG: build = > >> > > >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 > >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 > >> > ************************************************************/ > >> > 2011-12-23 15:11:01,401 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsConfig: > >> > loaded properties from hadoop-metrics2.properties > >> > 2011-12-23 15:11:01,416 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > MetricsSystem,sub=Stats registered. > >> > 2011-12-23 15:11:01,417 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot > >> > period > >> > at 10 second(s). > >> > 2011-12-23 15:11:01,417 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics > >> > system > >> > started > >> > 2011-12-23 15:11:01,727 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > ugi > >> > registered. > >> > 2011-12-23 15:11:01,733 INFO > >> > > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> > Updating the current master key for generating delegation tokens > >> > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker: > >> > Scheduler > >> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, > >> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) > >> > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader: > >> > Refreshing hosts (include/exclude) list > >> > 2011-12-23 15:11:01,745 INFO > >> > > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> > Starting expired delegation token remover thread, > >> > tokenRemoverScanInterval=60 min(s) > >> > 2011-12-23 15:11:01,745 INFO > >> > > >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: > >> > Updating the current master key for generating delegation tokens > >> > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker: > >> > Starting > >> > jobtracker with owner as root > >> > 2011-12-23 15:11:01,836 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > RpcDetailedActivityForPort9001 registered. > >> > 2011-12-23 15:11:01,837 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > RpcActivityForPort9001 registered. > >> > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting > >> > SocketReader > >> > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to > >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > >> > org.mortbay.log.Slf4jLog > >> > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added > >> > global > >> > filtersafety > >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) > >> > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port > >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is > >> > -1. > >> > Opening the listener on 50030 > >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: > >> > listener.getLocalPort() returned 50030 > >> > webServer.getConnectors()[0].getLocalPort() returned 50030 > >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty > >> > bound > >> > to port 50030 > >> > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26 > >> > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started > >> > [email protected]:50030 > >> > 2011-12-23 15:11:02,372 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > jvm > >> > registered. > >> > 2011-12-23 15:11:02,373 INFO > >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > >> > JobTrackerMetrics registered. > >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: > >> > JobTracker > >> > up at: 9001 > >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: > >> > JobTracker > >> > webserver: 50030 > >> > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker: > >> > Cleaning > >> > up the system directory > >> > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker: > >> > History > >> > server being initialized in embedded mode > >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobHistoryServer: > >> > Started job history server at: localhost:50030 > >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job > >> > History Server web address: localhost:50030 > >> > 2011-12-23 15:11:02,585 INFO > >> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is > >> > inactive > >> > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker: > >> > Refreshing > >> > hosts information > >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> > Setting > >> > the includes file to > >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> > Setting > >> > the excludes file to > >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: > >> > Refreshing hosts (include/exclude) list > >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker: > >> > Decommissioning 0 nodes > >> > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > Responder: starting > >> > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > listener on 9001: starting > >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 0 on 9001: starting > >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 1 on 9001: starting > >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 2 on 9001: starting > >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 3 on 9001: starting > >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 4 on 9001: starting > >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 6 on 9001: starting > >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 7 on 9001: starting > >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 9 on 9001: starting > >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 5 on 9001: starting > >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker: > >> > Starting > >> > RUNNING > >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server > >> > handler 8 on 9001: starting > >> > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology: > >> > Adding a > >> > new node: /default-rack/ylal2960.inetpsa.com > >> > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: Adding > >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host > >> > ylal2960.inetpsa.com > >> > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology: > >> > Adding a > >> > new node: /default-rack/ylal2950.inetpsa.com > >> > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: Adding > >> > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host > >> > ylal2950.inetpsa.com > >> > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration: > >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> > attempt to > >> > override final parameter: hadoop.tmp.dir; Ignoring. > >> > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration: > >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> > attempt to > >> > override final parameter: dfs.data.dir; Ignoring. > >> > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration: > >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a > >> > attempt to > >> > override final parameter: dfs.name.dir; Ignoring. > >> > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress: > >> > job_201112231511_0001: nMaps=1 nReduces=1 max=-1 > >> > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job > >> > job_201112231511_0001 added successfully for user 'root' to queue > >> > 'default' > >> > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger: > >> > USER=root IP=10.68.76.61 OPERATION=SUBMIT_JOB > >> > TARGET=job_201112231511_0001 RESULT=SUCCESS > >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker: > >> > Initializing job_201112231511_0001 > >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress: > >> > Initializing job_201112231511_0001 > >> > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress: > >> > jobToken generated and stored with users keys in > >> > /users/hadoop/mapred/system/job_201112231511_0001/jobToken > >> > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress: > >> > Input > >> > size for job job_201112231511_0001 = 661630. Number of splits = 1 > >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > >> > tip:task_201112231511_0001_m_000000 has split on > >> > node:/default-rack/ylal2960.inetpsa.com > >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: > >> > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5 > >> > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: Job > >> > job_201112231511_0001 initialized successfully with 1 map tasks and 1 > >> > reduce > >> > tasks. > >> > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: Adding > >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip > >> > task_201112231511_0001_m_000002, for tracker > >> > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > >> > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_0 is 151247 ms debug. > >> > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_0 is 351649 ms debug. > >> > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_0 is 551655 ms debug. > >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: Lost > >> > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' > >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress: > >> > Error > >> > from attempt_201112231511_0001_m_000002_0: Lost task tracker: > >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 > >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: > >> > Removing > >> > task 'attempt_201112231511_0001_m_000002_0' > >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_0 is 752357 ms debug. > >> > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker: > >> > Launching > >> > task attempt_201112231511_0001_m_000002_0 timed out. > >> > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: Adding > >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip > >> > task_201112231511_0001_m_000002, for tracker > >> > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > >> > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_1 is 198155 ms debug. > >> > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_1 is 398161 ms debug. > >> > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_1 is 598166 ms debug. > >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: > >> > attempt_201112231511_0001_m_000002_1 is 798172 ms debug. > >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: > >> > Launching > >> > task attempt_201112231511_0001_m_000002_1 timed out. > >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress: > >> > Error > >> > from attempt_201112231511_0001_m_000002_1: Error launching task > >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Lost > >> > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' > >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress: > >> > Error > >> > from attempt_201112231511_0001_m_000002_1: Lost task tracker: > >> > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 > >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: > >> > Removing > >> > task 'attempt_201112231511_0001_m_000002_1' > >> > > >> > > >> > > >> > Date: Fri, 23 Dec 2011 15:40:20 +0100 > >> > Subject: Re: Task process exit with nonzero status of 134 > >> > From: [email protected] > >> > To: [email protected] > >> > > >> > > >> > Hi, > >> > > >> > take a look into the logs for the failed attempt at your Tasktracker. > >> > Also check the system logs with dmesg or /var/log/kern*. Could be a > >> > syskill (segfault). > >> > > >> > - Alex > >> > > >> > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <[email protected]> > >> > wrote: > >> >> > >> >> Hi folks, > >> >> > >> >> I've just done a fresh install of Hadoop, Namenode and datanode are up, > >> >> Task/job Tracker also up, but when I run the Map reduce worcount > >> >> exemple I > >> >> got this error on Task tracker: > >> >> > >> >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : > >> >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number > >> >> of > >> >> tasks it ran: 0 > >> >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: > >> >> attempt_201112231511_0001_m_000002_0 : Child Error > >> >> java.io.IOException: Task process exit with nonzero status of 134. > >> >> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > >> >> > >> >> And job tracker stuck : > >> >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ > >> >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to > >> >> process > >> >> : 1 > >> >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: > >> >> job_201112231511_0001 > >> >> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% > >> >> > >> >> > >> >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25 > >> >> > >> >> I've done some googling, apparently the JVM crash hard (maybe Out of > >> >> memory), does someone have any hint ? > >> >> > >> >> Regards, > >> >> > >> >> Anthony Garnier > >> >> /DSIN/ASTI/ETSO > >> >> IT Center > >> >> PSA Peugeot Citroen > >> >> Bessoncourt 90160 > >> >> > >> >> > >> >> > >> >> > >> > > >> > > >> > > >> > -- > >> > Alexander Lorenz > >> > http://mapredit.blogspot.com > >> > > >> > P Think of the environment: please don't print this email unless you > >> > really need to. > >> > >> > >> > >> -- > >> Alexander Lorenz > >> http://mapredit.blogspot.com > >> > >> P Think of the environment: please don't print this email unless you > >> really need to. > > > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > P Think of the environment: please don't print this email unless you > really need to.
