Anthony, personally I haven't tested yet, jdk7 have bugs already. Was only a hint to see if the error occurs. I would focus on memory issues, the installed RAM are okay? No errors? My next step would be to downgrade into one JDK earlier to check for a bug. Did you update the OS before?
- Alex On Tue, Dec 27, 2011 at 2:54 PM, anthony garnier <[email protected]> wrote: > > Alex > > -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit > JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions) > Anyway, I tested it but same result. > Is it wise to test hadoop with the new jdk7_2 ? > > Anthony > > >> Date: Tue, 27 Dec 2011 13:47:03 +0100 >> Subject: Re: Task process exit with nonzero status of 134 >> From: [email protected] > >> To: [email protected] >> >> Anthony, >> >> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest >> update, could be a bug inside. Can you test SE 7u2? >> Todd mentioned in a older post to use -XX:+UseCompressedOops >> (hadoop-env.sh). Another option could be to take a closer look at >> garbage collection with compressed option. >> >> - Alex >> >> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[email protected]> >> wrote: >> > Alex, >> > >> > Memory available on namenode / Jobtracker : >> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie >> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0%st >> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers >> > Swap: 2047M total, 1M used, 2046M free, 8833M cached >> > >> > On datanode / Tasktracker : >> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26 >> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie >> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, >> > 0.0%st >> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers >> > Swap: 2047M total, 0M used, 2047M free, 2771M cached >> > >> > src/mapred/mapred-default.xml : >> > <property> >> > <name>mapred.child.java.opts</name> >> > <value>-Xmx200m</value> >> > </property> >> > >> > >> > So there should be enough memory >> > >> > Anthony >> > >> > >> >> Date: Tue, 27 Dec 2011 11:58:46 +0100 >> > >> >> Subject: Re: Task process exit with nonzero status of 134 >> >> From: [email protected] >> >> To: [email protected] >> > >> >> >> >> Anthony, >> >> >> >> How much memory you have available? Did the system going into swap? >> >> >> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx). >> >> - what says "top -Hc"? >> >> >> >> - Alex >> >> >> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[email protected]> >> >> wrote: >> >> > Hi, >> >> > >> >> > I got Nothing in the dmesg >> >> > I've checked the Tasktracker and this is what I got : >> >> > >> >> > /************************************************************ >> >> > STARTUP_MSG: Starting TaskTracker >> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86 >> >> > STARTUP_MSG: args = [] >> >> > STARTUP_MSG: version = 0.20.203.0 >> >> > STARTUP_MSG: build = >> >> > >> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 >> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 >> >> > ************************************************************/ >> >> > 2011-12-23 15:11:02,275 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsConfig: >> >> > loaded properties from hadoop-metrics2.properties >> >> > 2011-12-23 15:11:02,330 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > MetricsSystem,sub=Stats registered. >> >> > 2011-12-23 15:11:02,331 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >> >> > period >> >> > at 10 second(s). >> >> > 2011-12-23 15:11:02,331 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics >> >> > system started >> >> > 2011-12-23 15:11:02,597 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > ugi >> >> > registered. >> >> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> >> > org.mortbay.log.Slf4jLog >> >> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added >> >> > global >> >> > filtersafety >> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >> >> > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: >> >> > Initializing logs' truncater with mapRetainSize=-1 and >> >> > reduceRetainSize=-1 >> >> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > Starting >> >> > tasktracker with owner as root >> >> > 2011-12-23 15:11:02,870 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > jvm >> >> > registered. >> >> > 2011-12-23 15:11:02,871 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > TaskTrackerMetrics registered. >> >> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting >> >> > SocketReader >> >> > 2011-12-23 15:11:02,900 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > RpcDetailedActivityForPort58709 registered. >> >> > 2011-12-23 15:11:02,900 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > RpcActivityForPort58709 registered. >> >> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > Responder: starting >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > listener on 58709: starting >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 0 on 58709: starting >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 1 on 58709: starting >> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 2 on 58709: starting >> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > TaskTracker up at: localhost/127.0.0.1:58709 >> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > Starting >> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 >> >> > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 3 on 58709: starting >> >> > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > Using >> >> > MemoryCalculatorPlugin : >> >> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1 >> >> > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > Starting >> >> > thread: Map-events fetcher for all reduce tasks on >> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 >> >> > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid >> >> > exited with exit code 0 >> >> > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker: >> >> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is >> >> > disabled. >> >> > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: >> >> > IndexCache >> >> > created with max memory = 10485760 >> >> > 2011-12-23 15:11:02,976 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > ShuffleServerMetrics registered. >> >> > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port >> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is >> >> > -1. >> >> > Opening the listener on 50060 >> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: >> >> > listener.getLocalPort() returned 50060 >> >> > webServer.getConnectors()[0].getLocalPort() returned 50060 >> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty >> >> > bound >> >> > to port 50060 >> >> > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26 >> >> > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started >> >> > [email protected]:50060 >> >> > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231432_0002 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231121_0001 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231312_0002 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231142_0002 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112230933_0002 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231357_0001 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231052_0002 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: >> >> > Adding >> >> > job_201112231426_0001 for user-log deletion with >> >> > retainTimeStamp:1324735862967 >> >> > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0 >> >> > task's >> >> > state:UNASSIGNED >> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: >> >> > Trying to >> >> > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots >> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In >> >> > TaskLauncher, current free slots : 2 and trying to launch >> >> > attempt_201112231511_0001_m_000002_0 which needs 1 slots >> >> > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. >> >> > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. >> >> > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. >> >> > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer: >> >> > Initializing user root on this TT. >> >> > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. >> >> > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. >> >> > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. >> >> > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. >> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.data.dir; Ignoring. >> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: dfs.name.dir; Ignoring. >> >> > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In >> >> > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 >> >> > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM >> >> > Runner >> >> > jvm_201112231511_0001_m_-1653678851 spawned. >> >> > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController: >> >> > Writing commands to >> >> > >> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh >> >> > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM >> >> > with >> >> > ID: jvm_201112231511_0001_m_-1653678851 given task: >> >> > attempt_201112231511_0001_m_000002_0 >> >> > 2011-12-23 15:11:52,675 WARN >> >> > org.apache.hadoop.mapred.DefaultTaskController: >> >> > Exit code from task is : 134 >> >> > 2011-12-23 15:11:52,675 INFO >> >> > org.apache.hadoop.mapred.DefaultTaskController: >> >> > Output from DefaultTaskController's launchTask follows: >> >> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController: >> >> > >> >> > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : >> >> > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of >> >> > tasks it ran: 0 >> >> > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: >> >> > attempt_201112231511_0001_m_000002_0 : Child Error >> >> > java.io.IOException: Task process exit with nonzero status of 134. >> >> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) >> >> > >> >> > >> >> > >> >> > I also checked the logs of the task : >> >> > >> >> > >> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout >> >> > >> >> > A fatal error has been detected by the Java Runtime Environment: >> >> > # >> >> > # SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472 >> >> > # >> >> > # JRE version: 6.0_30-b12 >> >> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode >> >> > linux-amd64 compressed oops) >> >> > # Problematic frame: >> >> > # C [ld-linux-x86-64.so.2+0x882f] double+0xcf >> >> > # >> >> > # An error report file with more information is saved as: >> >> > # >> >> > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log >> >> > # >> >> > # If you would like to submit a bug report, please visit: >> >> > # http://java.sun.com/webapps/bugreport/crash.jsp >> >> > # The crash happened outside the Java Virtual Machine in native code. >> >> > # See problematic frame for where to report the bug. >> >> > --------------- >> >> > >> >> > >> >> > >> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog >> >> > >> >> > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration: >> >> > >> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a >> >> > attempt to override final parameter: hadoop.tmp.dir; Ignoring. >> >> > >> >> > >> >> > On the Job tracker : >> >> > STARTUP_MSG: Starting JobTracker >> >> > STARTUP_MSG: host = yval4950.inetpsa.com/10.68.76.61 >> >> > STARTUP_MSG: args = [] >> >> > STARTUP_MSG: version = 0.20.203.0 >> >> > STARTUP_MSG: build = >> >> > >> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 >> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011 >> >> > ************************************************************/ >> >> > 2011-12-23 15:11:01,401 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsConfig: >> >> > loaded properties from hadoop-metrics2.properties >> >> > 2011-12-23 15:11:01,416 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > MetricsSystem,sub=Stats registered. >> >> > 2011-12-23 15:11:01,417 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot >> >> > period >> >> > at 10 second(s). >> >> > 2011-12-23 15:11:01,417 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics >> >> > system >> >> > started >> >> > 2011-12-23 15:11:01,727 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > ugi >> >> > registered. >> >> > 2011-12-23 15:11:01,733 INFO >> >> > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> >> > Updating the current master key for generating delegation tokens >> >> > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Scheduler >> >> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, >> >> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) >> >> > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader: >> >> > Refreshing hosts (include/exclude) list >> >> > 2011-12-23 15:11:01,745 INFO >> >> > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> >> > Starting expired delegation token remover thread, >> >> > tokenRemoverScanInterval=60 min(s) >> >> > 2011-12-23 15:11:01,745 INFO >> >> > >> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: >> >> > Updating the current master key for generating delegation tokens >> >> > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Starting >> >> > jobtracker with owner as root >> >> > 2011-12-23 15:11:01,836 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > RpcDetailedActivityForPort9001 registered. >> >> > 2011-12-23 15:11:01,837 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > RpcActivityForPort9001 registered. >> >> > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting >> >> > SocketReader >> >> > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to >> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> >> > org.mortbay.log.Slf4jLog >> >> > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added >> >> > global >> >> > filtersafety >> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >> >> > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port >> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is >> >> > -1. >> >> > Opening the listener on 50030 >> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: >> >> > listener.getLocalPort() returned 50030 >> >> > webServer.getConnectors()[0].getLocalPort() returned 50030 >> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty >> >> > bound >> >> > to port 50030 >> >> > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26 >> >> > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started >> >> > [email protected]:50030 >> >> > 2011-12-23 15:11:02,372 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > jvm >> >> > registered. >> >> > 2011-12-23 15:11:02,373 INFO >> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source >> >> > JobTrackerMetrics registered. >> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: >> >> > JobTracker >> >> > up at: 9001 >> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: >> >> > JobTracker >> >> > webserver: 50030 >> >> > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Cleaning >> >> > up the system directory >> >> > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker: >> >> > History >> >> > server being initialized in embedded mode >> >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobHistoryServer: >> >> > Started job history server at: localhost:50030 >> >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job >> >> > History Server web address: localhost:50030 >> >> > 2011-12-23 15:11:02,585 INFO >> >> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is >> >> > inactive >> >> > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Refreshing >> >> > hosts information >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: >> >> > Setting >> >> > the includes file to >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: >> >> > Setting >> >> > the excludes file to >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: >> >> > Refreshing hosts (include/exclude) list >> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Decommissioning 0 nodes >> >> > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > Responder: starting >> >> > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > listener on 9001: starting >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 0 on 9001: starting >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 1 on 9001: starting >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 2 on 9001: starting >> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 3 on 9001: starting >> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 4 on 9001: starting >> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 6 on 9001: starting >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 7 on 9001: starting >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 9 on 9001: starting >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 5 on 9001: starting >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Starting >> >> > RUNNING >> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server >> >> > handler 8 on 9001: starting >> >> > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology: >> >> > Adding a >> >> > new node: /default-rack/ylal2960.inetpsa.com >> >> > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: Adding >> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host >> >> > ylal2960.inetpsa.com >> >> > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology: >> >> > Adding a >> >> > new node: /default-rack/ylal2950.inetpsa.com >> >> > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: Adding >> >> > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host >> >> > ylal2950.inetpsa.com >> >> > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration: >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a >> >> > attempt to >> >> > override final parameter: hadoop.tmp.dir; Ignoring. >> >> > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration: >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a >> >> > attempt to >> >> > override final parameter: dfs.data.dir; Ignoring. >> >> > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration: >> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a >> >> > attempt to >> >> > override final parameter: dfs.name.dir; Ignoring. >> >> > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > job_201112231511_0001: nMaps=1 nReduces=1 max=-1 >> >> > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job >> >> > job_201112231511_0001 added successfully for user 'root' to queue >> >> > 'default' >> >> > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger: >> >> > USER=root IP=10.68.76.61 OPERATION=SUBMIT_JOB >> >> > TARGET=job_201112231511_0001 RESULT=SUCCESS >> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Initializing job_201112231511_0001 >> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > Initializing job_201112231511_0001 >> >> > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > jobToken generated and stored with users keys in >> >> > /users/hadoop/mapred/system/job_201112231511_0001/jobToken >> >> > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > Input >> >> > size for job job_201112231511_0001 = 661630. Number of splits = 1 >> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > tip:task_201112231511_0001_m_000000 has split on >> >> > node:/default-rack/ylal2960.inetpsa.com >> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress: >> >> > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5 >> >> > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: Job >> >> > job_201112231511_0001 initialized successfully with 1 map tasks and 1 >> >> > reduce >> >> > tasks. >> >> > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: Adding >> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip >> >> > task_201112231511_0001_m_000002, for tracker >> >> > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' >> >> > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_0 is 151247 ms debug. >> >> > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_0 is 351649 ms debug. >> >> > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_0 is 551655 ms debug. >> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: Lost >> >> > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' >> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress: >> >> > Error >> >> > from attempt_201112231511_0001_m_000002_0: Lost task tracker: >> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 >> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Removing >> >> > task 'attempt_201112231511_0001_m_000002_0' >> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_0 is 752357 ms debug. >> >> > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Launching >> >> > task attempt_201112231511_0001_m_000002_0 timed out. >> >> > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: Adding >> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip >> >> > task_201112231511_0001_m_000002, for tracker >> >> > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' >> >> > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_1 is 198155 ms debug. >> >> > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_1 is 398161 ms debug. >> >> > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_1 is 598166 ms debug. >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: >> >> > attempt_201112231511_0001_m_000002_1 is 798172 ms debug. >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Launching >> >> > task attempt_201112231511_0001_m_000002_1 timed out. >> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress: >> >> > Error >> >> > from attempt_201112231511_0001_m_000002_1: Error launching task >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Lost >> >> > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress: >> >> > Error >> >> > from attempt_201112231511_0001_m_000002_1: Lost task tracker: >> >> > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 >> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Removing >> >> > task 'attempt_201112231511_0001_m_000002_1' >> >> > >> >> > >> >> > >> >> > Date: Fri, 23 Dec 2011 15:40:20 +0100 >> >> > Subject: Re: Task process exit with nonzero status of 134 >> >> > From: [email protected] >> >> > To: [email protected] >> >> > >> >> > >> >> > Hi, >> >> > >> >> > take a look into the logs for the failed attempt at your Tasktracker. >> >> > Also check the system logs with dmesg or /var/log/kern*. Could be a >> >> > syskill (segfault). >> >> > >> >> > - Alex >> >> > >> >> > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <[email protected]> >> >> > wrote: >> >> >> >> >> >> Hi folks, >> >> >> >> >> >> I've just done a fresh install of Hadoop, Namenode and datanode are up, >> >> >> Task/job Tracker also up, but when I run the Map reduce worcount >> >> >> exemple I >> >> >> got this error on Task tracker: >> >> >> >> >> >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : >> >> >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number >> >> >> of >> >> >> tasks it ran: 0 >> >> >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: >> >> >> attempt_201112231511_0001_m_000002_0 : Child Error >> >> >> java.io.IOException: Task process exit with nonzero status of 134. >> >> >> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) >> >> >> >> >> >> And job tracker stuck : >> >> >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ >> >> >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to >> >> >> process >> >> >> : 1 >> >> >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: >> >> >> job_201112231511_0001 >> >> >> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% >> >> >> >> >> >> >> >> >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25 >> >> >> >> >> >> I've done some googling, apparently the JVM crash hard (maybe Out of >> >> >> memory), does someone have any hint ? >> >> >> >> >> >> Regards, >> >> >> >> >> >> Anthony Garnier >> >> >> /DSIN/ASTI/ETSO >> >> >> IT Center >> >> >> PSA Peugeot Citroen >> >> >> Bessoncourt 90160 >> >> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > >> >> > >> >> > -- >> >> > Alexander Lorenz >> >> > http://mapredit.blogspot.com >> >> > >> >> > P Think of the environment: please don't print this email unless you >> >> > really need to. >> >> >> >> >> >> >> >> -- >> >> Alexander Lorenz >> >> http://mapredit.blogspot.com >> >> >> >> P Think of the environment: please don't print this email unless you >> >> really need to. >> >> >> >> -- >> Alexander Lorenz >> http://mapredit.blogspot.com >> >> P Think of the environment: please don't print this email unless you >> really need to. > -- Alexander Lorenz http://mapredit.blogspot.com P Think of the environment: please don't print this email unless you really need to.
