Progress: I found the very latest r9 patch on the Apache Incubation board. Unlike most of the other r1-r8 patches, r9 applies cleanly to trunk. With some work, I was able to build that and do "make hadoop-2.0.0-mr1-cdh4.1.2" in the "hadoop" subdirectory. The build process downloaded the needed tarball, built Hadoop 2 with the Mesos patch, and then successfully ran some Hadoop self tests.
Now I will install this Mesos and Hadoop 2 on a test machine and see if I can get it all to work in the Mesos "standalone" configuration. Craig On Fri, Mar 1, 2013 at 10:22 AM, Craig Vanderborgh < [email protected]> wrote: > Thanks for the reply, Vinod.. > > We are running 0.9.0-incubating downloaded from here: > > > http://www.apache.org/dyn/closer.cgi/incubator/mesos/mesos-0.9.0-incubating/ > > The Hadoop I'm trying to run is v0.20.205 that's included with this Mesos > distribution. But we WANT to run hadoop-2.0.0-mr1-cdh4.1.2. I've got a > patch for this, but not sure exactly what I should use as the starting > point. Can you give me a few pointers on how to proceed? We want the > hadoop-2.0.0-mr-cdh4.1.2 version, and we'll build/run anything required to > get it. > > Craig > > > On Fri, Mar 1, 2013 at 9:16 AM, Vinod Kone <[email protected]> wrote: > >> Hey Craig, >> >> Which version of Mesos are you running? We recently (~1 month) revamped >> our Hadoop port and unfortunately didn't update the wiki. >> >> @vinodkone >> Sent from my mobile >> >> On Mar 1, 2013, at 7:48 AM, Craig Vanderborgh <[email protected]> >> wrote: >> >> > I'm getting close. I've got the namenode formatted and HDFS running. I >> > started hdfs with $HADOOP_HOME/bin/start-dfs.sh. Basic HDFS operations >> > seem to be working. Now it's time to start the jobtracker - I have been >> > following the instructions here: >> > >> > https://github.com/mesos/mesos/wiki/Running-Hadoop-on-Mesos >> > >> > The instructions say "Launch a JobTracker with bin/hadoop jobtracker (do >> > not use bin/start-mapred.sh)... >> > >> > So here's what happens: >> > >> > [craigv@sleuth-mesos-01 bin]$ hadoop jobtracker >> > 13/03/01 00:23:02 INFO mapred.JobTracker: STARTUP_MSG: >> > /************************************************************ >> > STARTUP_MSG: Starting JobTracker >> > STARTUP_MSG: host = sleuth-mesos-01.lab.webapps.rr.com/71.74.48.186 >> > STARTUP_MSG: args = [] >> > STARTUP_MSG: version = 0.20.205.1 >> > STARTUP_MSG: build = -r ; compiled by 'craigv' on Thu Feb 28 22:59:28 >> > UTC 2013 >> > ************************************************************/ >> > 13/03/01 00:23:02 INFO impl.MetricsConfig: loaded properties from >> > hadoop-metrics2.properties >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source >> > MetricsSystem,sub=Stats registered. >> > 13/03/01 00:23:02 INFO impl.MetricsSystemImpl: Scheduled snapshot >> period at >> > 10 second(s). >> > 13/03/01 00:23:02 INFO impl.MetricsSystemImpl: JobTracker metrics system >> > started >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source >> > QueueMetrics,q=default registered. >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source ugi >> > registered. >> > 13/03/01 00:23:02 INFO delegation.AbstractDelegationTokenSecretManager: >> > Updating the current master key for generating delegation tokens >> > 13/03/01 00:23:02 INFO delegation.AbstractDelegationTokenSecretManager: >> > Starting expired delegation token remover thread, >> > tokenRemoverScanInterval=60 min(s) >> > 13/03/01 00:23:02 INFO delegation.AbstractDelegationTokenSecretManager: >> > Updating the current master key for generating delegation tokens >> > 13/03/01 00:23:02 INFO mapred.JobTracker: Scheduler configured with >> > (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, >> limitMaxMemForMapTasks, >> > limitMaxMemForReduceTasks) (-1, -1, -1, -1) >> > 13/03/01 00:23:02 INFO util.HostsFileReader: Refreshing hosts >> > (include/exclude) list >> > 13/03/01 00:23:02 INFO mapred.JobTracker: Starting jobtracker with >> owner as >> > craigv >> > 13/03/01 00:23:02 INFO ipc.Server: Starting SocketReader >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source >> > RpcDetailedActivityForPort54311 registered. >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source >> > RpcActivityForPort54311 registered. >> > 13/03/01 00:23:02 INFO mortbay.log: Logging to >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via >> > org.mortbay.log.Slf4jLog >> > 13/03/01 00:23:02 INFO http.HttpServer: Added global filtersafety >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) >> > 13/03/01 00:23:02 INFO http.HttpServer: Port returned by >> > webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening >> > the listener on 50030 >> > 13/03/01 00:23:02 INFO http.HttpServer: listener.getLocalPort() returned >> > 50030 webServer.getConnectors()[0].getLocalPort() returned 50030 >> > 13/03/01 00:23:02 INFO http.HttpServer: Jetty bound to port 50030 >> > 13/03/01 00:23:02 INFO mortbay.log: jetty-6.1.26 >> > 13/03/01 00:23:02 INFO mortbay.log: Started >> > [email protected]:50030 >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source jvm >> > registered. >> > 13/03/01 00:23:02 INFO impl.MetricsSourceAdapter: MBean for source >> > JobTrackerMetrics registered. >> > 13/03/01 00:23:02 INFO mapred.JobTracker: JobTracker up at: 54311 >> > 13/03/01 00:23:02 INFO mapred.JobTracker: JobTracker webserver: 50030 >> > 13/03/01 00:23:02 INFO mapred.JobTracker: Cleaning up the system >> directory >> > 13/03/01 00:23:02 INFO mapred.JobTracker: History server being >> initialized >> > in embedded mode >> > 13/03/01 00:23:02 INFO mapred.JobHistoryServer: Started job history >> server >> > at: localhost:50030 >> > 13/03/01 00:23:02 INFO mapred.JobTracker: Job History Server web >> address: >> > localhost:50030 >> > 13/03/01 00:23:02 INFO mapred.CompletedJobStatusStore: Completed job >> store >> > is inactive >> > 13/03/01 00:23:03 INFO mapred.MesosScheduler: Starting MesosScheduler >> > java: ./common/try.hpp:77: T Try<T>::get() const [with T = >> > mesos::internal::MasterDetector*]: Assertion `state == SOME' failed. >> > Aborted >> > >> > Close but no cigar. The Mesos code in this proximity: >> > >> > src/common/try.hpp >> > >> > Try<T>& operator = (const Try<T>& that) >> > { >> > if (this != &that) { >> > state = that.state; >> > if (that.t != NULL) { >> > t = new T(*that.t); >> > } else { >> > t = NULL; >> > } >> > message = that.message; >> > } >> > >> > return *this; >> > } >> > >> > bool isSome() const { return state == SOME; } >> > bool isError() const { return state == ERROR; } >> > >> > T get() const { assert(state == SOME); return *t; } >> > >> > std::string error() const { assert(state == ERROR); return message; } >> > >> > What's wrong here? Obviously it can't start MesosScheduler, but why? >> > >> > Here is my mapred-site.xml: >> > >> > <?xml version="1.0"?> >> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >> > >> > <!-- Put site-specific property overrides in this file. --> >> > >> > <configuration> >> > <property> >> > <name>mapred.job.tracker</name> >> > <value>localhost:54311</value> >> > </property> --> >> > <property> >> > <name>mapred.jobtracker.taskScheduler</name> >> > <value>org.apache.hadoop.mapred.MesosScheduler</value> >> > </property> >> > <property> >> > <name>mapred.mesos.master</name> >> > <value>mesos://sleuth-mesos-01:5050</value> >> > </property> >> > </configuration> >> > >> > THANKS! >> > Craig >> > >
