Hi, Guodong, I start hadoop as you said, then I saw this error: 13/04/23 13:03:43 ERROR mapred.MesosScheduler: Error from scheduler driver: Cannot parse '@0.0.0.0:0'
What's this mean? where should I change MesosScheduler code to fix this? Thanks very much! I am so sorry for interrupt you once again... The whole log is as follows: [root@master hadoop-0.20.205.0]# hadoop jobtracker 13/04/23 13:21:04 INFO mapred.JobTracker: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting JobTracker STARTUP_MSG: host = master/192.168.0.2 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.205.0 STARTUP_MSG: build = -r ; compiled by 'root' on Sat Apr 13 11:19:33 CST 2013 ************************************************************/ 13/04/23 13:21:04 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 13/04/23 13:21:04 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 13/04/23 13:21:04 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 13/04/23 13:21:04 INFO impl.MetricsSystemImpl: JobTracker metrics system started 13/04/23 13:21:04 INFO impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered. 13/04/23 13:21:04 INFO impl.MetricsSourceAdapter: MBean for source ugi registered. 13/04/23 13:21:04 INFO delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 13/04/23 13:21:04 INFO delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 13/04/23 13:21:04 INFO mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) 13/04/23 13:21:04 INFO delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 13/04/23 13:21:04 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 13/04/23 13:21:04 INFO mapred.JobTracker: Starting jobtracker with owner as root 13/04/23 13:21:04 INFO ipc.Server: Starting SocketReader 13/04/23 13:21:04 INFO impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered. 13/04/23 13:21:04 INFO impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered. 13/04/23 13:21:04 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 13/04/23 13:21:05 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 13/04/23 13:21:05 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030 13/04/23 13:21:05 INFO http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030 13/04/23 13:21:05 INFO http.HttpServer: Jetty bound to port 50030 13/04/23 13:21:05 INFO mortbay.log: jetty-6.1.26 13/04/23 13:21:05 INFO mortbay.log: Started [email protected]:50030 13/04/23 13:21:05 INFO impl.MetricsSourceAdapter: MBean for source jvm registered. 13/04/23 13:21:05 INFO impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered. 13/04/23 13:21:05 INFO mapred.JobTracker: JobTracker up at: 9001 13/04/23 13:21:05 INFO mapred.JobTracker: JobTracker webserver: 50030 13/04/23 13:21:05 INFO mapred.JobTracker: Cleaning up the system directory 13/04/23 13:21:05 INFO mapred.JobTracker: History server being initialized in embedded mode 13/04/23 13:21:05 INFO mapred.JobHistoryServer: Started job history server at: localhost:50030 13/04/23 13:21:05 INFO mapred.JobTracker: Job History Server web address: localhost:50030 13/04/23 13:21:05 INFO mapred.CompletedJobStatusStore: Completed job store is inactive 13/04/23 13:21:05 INFO mapred.MesosScheduler: Starting MesosScheduler 13/04/23 13:21:05 INFO mapred.JobTracker: Refreshing hosts information 13/04/23 13:21:05 ERROR mapred.MesosScheduler: Error from scheduler driver: Cannot parse '@0.0.0.0:0' 13/04/23 13:21:05 INFO util.HostsFileReader: Setting the includes file to 13/04/23 13:21:05 INFO util.HostsFileReader: Setting the excludes file to 13/04/23 13:21:05 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 13/04/23 13:21:05 INFO mapred.JobTracker: Decommissioning 0 nodes 13/04/23 13:21:05 INFO ipc.Server: IPC Server Responder: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server listener on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 0 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 1 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 3 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 2 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 5 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 4 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 6 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 7 on 9001: starting 13/04/23 13:21:05 INFO mapred.JobTracker: Starting RUNNING 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 8 on 9001: starting 13/04/23 13:21:05 INFO ipc.Server: IPC Server handler 9 on 9001: starting 13/04/23 13:21:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/04/23 13:21:32 INFO mapred.JobInProgress: job_201304231321_0001: nMaps=0 nReduces=0 max=-1 13/04/23 13:21:32 INFO mapred.MesosScheduler: Added job job_201304231321_0001 13/04/23 13:21:32 INFO mapred.JobTracker: Job job_201304231321_0001 added successfully for user 'root' to queue 'default' 13/04/23 13:21:32 INFO mapred.AuditLogger: USER=root IP=192.168.0.2 OPERATION=SUBMIT_JOB TARGET=job_201304231321_0001 RESULT=SUCCESS 13/04/23 13:21:32 INFO mapred.JobTracker: Initializing job_201304231321_0001 13/04/23 13:21:32 INFO mapred.JobInProgress: Initializing job_201304231321_0001 13/04/23 13:21:32 INFO mapred.JobInProgress: jobToken generated and stored with users keys in /home/HadoopRun/tmp/mapred/system/job_201304231321_0001/jobToken 13/04/23 13:21:32 INFO mapred.JobInProgress: Input size for job job_201304231321_0001 = 0. Number of splits = 0 13/04/23 13:21:32 INFO mapred.JobInProgress: Job job_201304231321_0001 initialized successfully with 0 map tasks and 0 reduce tasks. Wang Yu From: 王国栋 Date: 2013-04-23 11:34 To: mesos-dev; wangyu Subject: Re: Re: org.apache.hadoop.mapred.MesosScheduler: Unknown/exited TaskTracker: http://slave5:50060 Hi Yu, Mesos will just launch tasktracker on each slave node as long as the required resource is enough for the tasktracker. So you have to run NameNode, Jobtracker and DataNode by your own. Basicly, starting the hadoop on mesos is like this. 1. start the dfs. use hadoop/bin/start-dfs.sh. (you should configure core-sites.xml and hdfs-site.xml). dfs is no different from the normal one. 2. start jobtracker, use hadoop/bin/hadoop jobtracker (you should configure mapred-site.xml, this jobtracker should contains the patch for mesos) Then, you can use mesos web UI and jobtracker web UI to check the status of Jobtracker. Guodong On Tue, Apr 23, 2013 at 11:06 AM, 王瑜 <[email protected]> wrote: Oh, yes, I start my hadoop using "start-all.sh". I know what's my problem. Thanks very much! ps: Besides TaskTracker, is there any other roles(like JobTracker, DataNode) I should stop it first? Wang Yu 发件人： Benjamin Mahler 发送时间： 2013-04-23 10:56 收件人： [email protected]; wangyu 主题： Re: Re: org.apache.hadoop.mapred.MesosScheduler: Unknown/exited TaskTracker: http://slave5:50060 The scheduler we wrote for Hadoop will start its own TaskTrackers, meaning you do not have to start any TaskTrackers yourself Are you starting your own TaskTrackers? Are there any TaskTrackers running in your cluster? Looking at your jps output, is there already a TaskTracker running? [root@master logs]# jps 13896 RunJar 14123 Jps 12718 NameNode 12900 DataNode 13374 TaskTracker <--- How was this started? 13218 JobTracker On Mon, Apr 22, 2013 at 7:47 PM, 王瑜 <[email protected]> wrote: > Hi, Ben and Guodong, > > What do you mean "managing your own TaskTrackers"? How should I know > whether I have manager my own TaskTrackers? Sorry, I do not familiar with > mesos very much. > Dies it mean I do not need configure hdfs-site.xml and core-site.xml in > hadoop? I do not want to run my own TaskTracker, I just want to set up > hadoop on mesos, and run my MR tasks. > > Thanks very much for your patient reply...Maybe I have a long way to go... > > > > The log messages you see: > 2013-04-18 16:47:19,645 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: http://master:50060. > > Are printed when mesos does not know about the TaskTracker. We currently > don't support running your own TaskTrackers, as the MesosScheduler will > launch them on your behalf when needed. > > Are you managing your own TaskTrackers? The purpose of using Hadoop with > mesos is that you no longer have to do that. We will detect that jobs have > pending map / reduce tasks and launch TaskTrackers accordingly. > > Guodong may be able to help further getting set up! > > > > > Wang Yu > > From: 王国栋 > Date: 2013-04-18 17:10 > To: mesos-dev; wangyu > Subject: Re: org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: http://slave5:50060 > You can check the slave log and the mesos-executor log, which is normally > located in the dir like > > "/tmp/mesos/slaves/201304181115-16842879-5050-4680-13/frameworks/201304181115-16842879-5050-4680-0003/executors/executor_Task_Tracker_16/runs/latest/stderr". > The log is tasktracker log. > > I hope it will help. > > Guodong > > > On Thu, Apr 18, 2013 at 5:03 PM, 王瑜 <[email protected]> wrote: > > > ** > > Hi All, > > > > I have deployed mesos on three node: master, slave1, slave5. and it works > > well. > > Then I set hadoop over it, using master as namenode, and master, slave1, > > slave5 as datanode. When I using 'jps', it looks works well. > > [root@master logs]# jps > > 13896 RunJar > > 14123 Jps > > 12718 NameNode > > 12900 DataNode > > 13374 TaskTracker > > 13218 JobTracker > > > > Then I run test benchmark, it can not go on working... > > [root@master > > hadoop-0.20.205.0]# bin/hadoop jar hadoop-examples-0.20.205.0.jar > randomwriter -Dtest.randomwrite.bytes_per_map=6710886 > -Dtest.randomwriter.maps_per_host=10 rand > > Running 30 maps. > > Job started: Thu Apr 18 16:49:36 CST 2013 > > 13/04/18 16:49:36 INFO mapred.JobClient: Running job: > job_201304181646_0001 > > 13/04/18 16:49:37 INFO mapred.JobClient: map 0% reduce 0% > > It stopped here. > > > > Then I read the log file: hadoop-root-jobtracker-master.log, it shows: > > 2013-04-18 16 > > :46:51,724 INFO org.apache.hadoop.mapred.JobTracker: Starting RUNNING > > 2013-04-18 16 > > :46:51,726 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on > 9001: starting > > 2013-04-18 16 > > :46:51,727 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on > 9001: starting > > 2013-04-18 16 > > :46:52,557 INFO org.apache.hadoop.net.NetworkTopology: Adding a new > node: /default-rack/master > > 2013-04-18 16 > > :46:52,560 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker > tracker_master:localhost/ > > 127.0.0.1:44997 to host master > > 2013-04-18 16 > > :46:52,568 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :46:55,581 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :46:58,590 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > 2013-04-18 16 > > :47:01,600 INFO org.apache.hadoop.mapred.MesosScheduler: Unknown/exited > TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:04,609 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:07,618 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:10,625 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:13,632 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:13,686 INFO org.apache.hadoop.net.NetworkTopology: > Adding a new node: /default-rack/slave5 > > > > 2013-04-18 16:47:13,686 INFO org.apache.hadoop.mapred.JobTracker: Adding > tracker tracker_slave5: > > 127.0.0.1/127.0.0.1:60621 to host slave5 > > > > 2013-04-18 16:47:13,687 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:16,638 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:16,697 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:19,645 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:19,707 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:22,651 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:22,715 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:25,658 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > 2013-04-18 16:47:25,725 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://slave5:50060. > > > > 2013-04-18 16:47:28,665 INFO org.apache.hadoop.mapred.MesosScheduler: > Unknown/exited TaskTracker: > > http://master:50060. > > > > Does anybody can help me? Thanks very much! > > >
