do you have more log on tasktracker? Is this all the log you can get from the tasktracker? Can you check the log of mesos-slave? I guess something is wrong when slave try to start mesos-executor for tasktracker. But I am not sure where the problem is.
Guodong On Fri, Apr 19, 2013 at 4:39 PM, 王瑜 <[email protected]> wrote: > Hi, Guodong, > > I have read the task tracker logs, there is a warn in it, but I do not > know how to fix it, do you have any ideas? > > 2013-04-18 17:40:37,816 WARN org.apache.hadoop.mapred.TaskTracker: > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > disabled. > > The whole log is as follows: > 2013-04-18 17:40:37,714 INFO org.apache.hadoop.ipc.Server: IPC Server > Responder: starting > 2013-04-18 17:40:37,715 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 36036: starting > 2013-04-18 17:40:37,716 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 0 on 36036: starting > 2013-04-18 17:40:37,717 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 1 on 36036: starting > 2013-04-18 17:40:37,718 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 2 on 36036: starting > 2013-04-18 17:40:37,719 INFO org.apache.hadoop.mapred.TaskTracker: > TaskTracker up at: localhost/127.0.0.1:36036 > 2013-04-18 17:40:37,719 INFO org.apache.hadoop.mapred.TaskTracker: > Starting tracker tracker_master:localhost/127.0.0.1:36036 > 2013-04-18 17:40:37,719 INFO org.apache.hadoop.ipc.Server: IPC Server > handler 3 on 36036: starting > 2013-04-18 17:40:37,791 INFO org.apache.hadoop.mapred.TaskTracker: > Starting thread: Map-events fetcher for all reduce tasks on > tracker_master:localhost/127.0.0.1:36036 > 2013-04-18 17:40:37,807 INFO org.apache.hadoop.util.ProcessTree: setsid > exited with exit code 0 > 2013-04-18 17:40:37,812 INFO org.apache.hadoop.mapred.TaskTracker: Using > ResourceCalculatorPlugin : > org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3cf28f60 > 2013-04-18 17:40:37,816 WARN org.apache.hadoop.mapred.TaskTracker: > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > disabled. > 2013-04-18 17:40:37,822 INFO org.apache.hadoop.mapred.IndexCache: > IndexCache created with max memory = 10485760 > 2013-04-18 17:40:37,829 INFO > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source > ShuffleServerMetrics registered. > 2013-04-18 17:40:37,832 INFO org.apache.hadoop.http.HttpServer: Port > returned by webServer.getConnectors()[0].getLocalPort() before open() is > -1. Opening the listener on 50060 > 2013-04-18 17:40:37,832 INFO org.apache.hadoop.http.HttpServer: > listener.getLocalPort() returned 50060 > webServer.getConnectors()[0].getLocalPort() returned 50060 > 2013-04-18 17:40:37,833 INFO org.apache.hadoop.http.HttpServer: Jetty > bound to port 50060 > 2013-04-18 17:40:37,833 INFO org.mortbay.log: jetty-6.1.26 > 2013-04-18 17:40:38,639 INFO org.mortbay.log: Started > [email protected]:50060 > 2013-04-18 17:40:38,639 INFO org.apache.hadoop.mapred.TaskTracker: > FILE_CACHE_SIZE for mapOutputServlet set to : 2000 > > Do you know what's this mean? Thanks very much! > > From: 王国栋 > Date: 2013-04-18 17:10 > To: mesos-dev; wangyu > Subject: Re: org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: http://slave5:50060 > You can check the slave log and the mesos-executor log, which is normally > located in the dir like > > "/tmp/mesos/slaves/201304181115-16842879-5050-4680-13/frameworks/201304181115-16842879-5050-4680-0003/executors/executor_Task_Tracker_16/runs/latest/stderr". > The log is tasktracker log. > > I hope it will help. > > Guodong > > > On Thu, Apr 18, 2013 at 5:03 PM, 王瑜 <[email protected]> wrote: > > > ** > > Hi All, > > > > I have deployed mesos on three node: master, slave1, slave5. and it works > > well. > > Then I set hadoop over it, using master as namenode, and master, slave1, > > slave5 as datanode. When I using 'jps', it looks works well. > > [root@master logs]# jps > > 13896 RunJar > > 14123 Jps > > 12718 NameNode > > 12900 DataNode > > 13374 TaskTracker > > 13218 JobTracker > > > > Then I run test benchmark, it can not go on working... > > [root@master > > hadoop-0.20.205.0]# bin/hadoop jar hadoop-examples-0.20.205.0.jar > randomwriter -Dtest.randomwrite.bytes_per_map=6710886 > -Dtest.randomwriter.maps_per_host=10 rand > > Running 30 maps. > > Job started: Thu Apr 18 16:49:36 CST 2013 > > 13/04/18 16:49:36 INFO mapred.JobClient: Running job: > job_201304181646_0001 > > 13/04/18 16:49:37 INFO mapred.JobClient: map 0% reduce 0% > > It stopped here. > > > > Then I read the log file: hadoop-root-jobtracker-master.log, it shows: > > 2013-04-18 16 > > :46:51,724 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING > > 2013-04-18 16 > > :46:51,726 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on > 9001: starting > > 2013-04-18 16 > > :46:52,557 INFO org.apache.hadoop.net.NetworkTopology: Adding a new > node: /default-rack/master > > 2013-04-18 16 > > :46:52,560 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker > tracker_master:localhost/ > > 127.0.0.1:44997 to host master > > 2013-04-18 16 > > :46:52,568 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :46:55,581 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :46:58,590 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :47:01,600 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:04,609 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:07,618 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:10,625 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:13,632 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:13,686 INFO org.apache.hadoop.net.NetworkTopology: > Adding a new node: /default-rack/slave5 > > > > 2013-04-18 16:47:13,686 INFO org.apache.hadoop.mapred.JobTracker: Adding > tracker tracker_slave5: > > 127.0.0.1/127.0.0.1:60621 to host slave5 > > > > 2013-04-18 16:47:13,687 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:16,638 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:16,697 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:19,645 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:19,707 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:22,651 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:22,715 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:25,658 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:25,725 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:28,665 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > Does anybody can help me? Thanks very much! > > >
