Hi, So typically people run two sorts of workloads on hadoop - ad-hoc/scheduled batch work, and stream workloads (spark, flink, etc.).
Both spark and flink integrate natively with mesos, so no need for an intermediate yarn layer. For batch work, we're looking at the aurora project for job scheduling. hadoop brings some interesting things, but I've not found integration with mesos to ever be pain-free, so we're moving to other tools instead of continuing down the path of trying to get hadoop working with mesos. Good luck! On 27/07/17 08:50, Traiano Welcome wrote: > Hi Stephen > > > On Wed, Jul 26, 2017 at 5:18 PM, Stephen Gran <stephen.g...@piksel.com > <mailto:stephen.g...@piksel.com>> wrote: > > Hi, > > It is having discussions about whether to stop, as it's having trouble > getting enough contributors. > > I guess I'd ask what you need to run on hadoop, why you're looking at > mesos, and then see what else is in that space. > > > > I don't know what we'd need to run on hadoop at this point - it's open > ended, and for our developers to decide. However, should this make a > difference? > > We have mesos in place as a resource scheduler for a number of > frameworks and would like to resource manage it using the same > semantics, tools and mechanisms mesos provides. > > I've looked at two books so far that show how this is done, so it seems > this way of managing hadoop is in use in places (ref: "Apache Mesos > Essentials", "Mastering Mesos"), however these books are probably out of > date because the procedure they describe for integrating mesos and > hadoop is broken. > > > > > > > > > > > Cheers, > > On 26/07/17 14:13, Brandon Gulla wrote: > > Have you looked into Apache Myriad? > > > > http://myriad.apache.org/ > > > > On Wed, Jul 26, 2017 at 4:12 AM, Traiano Welcome <trai...@gmail.com > <mailto:trai...@gmail.com> > > <mailto:trai...@gmail.com <mailto:trai...@gmail.com>>> wrote: > > > > Hi > > > > Would anyone know of some reliable guides to deploying apache > > hadoop on top of the mesos scheduler? > > > > Thanks, > > Traiano > > > > > > > > > > -- > > Brandon > > -- > Stephen Gran > Senior Technical Architect > > picture the possibilities | piksel.com <http://piksel.com> > > -- Stephen Gran Senior Technical Architect picture the possibilities | piksel.com
