We run both mesos and yarn in prod and it does not make sense to run yarn over mesos.
However it would be interesting to find a way to run MRv2 jobs on mesos with some custom layer to swap yarn with mesos. Not sure how to start though... MRv2 contains a yarn application master that needs to be rewritten as a mesos framework scheduler. This is probably doable. However with MRv2 every map reduce job would be mapped as a new framework in Mesos. Not sure how many frameworks mesos can run and scale up to. Especially short lived frameworks. On Jul 25, 2014 8:54 PM, "Tom Arnfeld" <[email protected]> wrote: > Hey Luyi, > > That's correct, the Hadoop framework currently only supports Hadoop 2 > MRv1. It also doesn't have great support for the HA jobtracker available in > newer versions of Hadoop, but I've been working on that the past few weeks. > > I'm not sure how Hadoop 2 would play with Mesos, but very interested to > find out more. Am I correct in thinking MRv2 will only run on top of YARN? > > I wonder if anyone else on the mailing list is running YARN on top of > Mesos... > > Tom. > > On Friday, 25 July 2014, Luyi Wang <[email protected]> wrote: > >> Checked the mesos github(https://github.com/mesos/hadoop). It listed >> support for MapReduce V1 >> >> How about the MR V2? >> >> Right now we are using cloudera to manage hadoop clusters where uses >> MRV2. We are planning to migrate all our services to mesos(still in the >> initial investigating stage). Good suggestions, advice and experiences are >> welcomed. >> >> Thanks a lot! >> >> >> -Luyi. >> >> >> >>
