The reason for this is that Hadoop wasn't designed to run on top of a cluster manager -- it was designed to "own" a bunch of nodes and do its own scheduling within them. If you wanted, you could run Hadoop on top of Mesos in a coarse-grained way, where you give it one long-lived task on each node within which it can schedule its own work. However, the patch we wrote lets Hadoop use Mesos in a fine-grained way, acquiring nodes only for the duration of a task and then giving them back. This is useful if you want Hadoop to coexist with other frameworks and share resources dynamically. Maybe we should include a version for the coarse-grained sharing use case too; we actually had one working a while back.
By the way, I believe the patch file will work without major changes for Hadoop 1.0, but I'm going to ask the student who's worked with Hadoop the most recently, Charles Reiss, because he's been looking at making it work with newer Hadoop releases. Matei On Jan 2, 2012, at 8:59 AM, Daniel Wilkerson wrote: > Actually, I have to say it seems a bit sub-optimal for Hadoop to > require a 2K+ line patch file just to work with Mesos. I thought the > point of Mesos was that running another service on top of it would be > straightforward. For the customer to have to know this much about the > internals of both just to get them to work together can't be a good > end-state. It just seems that there must be an architectural > mis-match somewhere. > > Daniel > > On Sun, Jan 1, 2012 at 11:45 PM, Daniel Wilkerson > <[email protected]> wrote: >> We want to run Hadoop inside mesos, but we want to run the latest >> version, Hadoop 1.0.0. Therefore I was considering porting your >> changes to the new version of Hadoop. Looking for mesos-related files >> in frameworks/hadoop-0.20.2 I find this. >> >> find . -iname "*mesos*" | grep -v '\.svn\b' >> ./bin/mesos-executor >> ./bin/mesos-jobtracker-launcher >> ./bin/mesos-jobtracker-runner >> ./hadoop-mesos.patch >> ./src/contrib/mesos >> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosJobTrackerInstrumentation.java >> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosJobTrackerLauncher.java >> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosJobTrackerRunner.java >> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java >> ./src/contrib/mesos/src/java/org/apache/hadoop/mapred/MesosTaskTrackerInstrumentation.java >> >> This hadoop-mesos.patch file looks suspiciously useful, however at 2K >> lines, it is not short. I suppose I could take Hadoop 1.0.0, apply >> the patch, copy over the files, and start it up. However even if this >> seems to work, who knows what lurks in subtle invariants of which I >> know not. At least from the Mesos side, do you have any advice? >> >> Daniel
