Hi Andre Myriad allows you to run YARN on Mesos. Since YARN is also a resource manager, imagine this as a kind of a delegation of resources from Mesos to YARN.
With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon startup reports back to YARN's Resource Manager and joins the cluster. Now, in your scenario you would like to spin up 'ephemeral' hadoop clusters. You can certainly do so in Mesos ecosystem, with a little bit of work (right now). My suggestion would be to use Marathon ( https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource Manager, and then trigger Myriad's flexUp API ( https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup) to spin N number of NodeManagers on mesos cluster. Regards Mohit On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe <[email protected]> wrote: > HI, > > yes, I looked at it, but from what I understand it is a project to run > YARN next to mesos, not on top of it. I am looking for a way to have > “ephemeral” hadoop clusters one some hardware, not a static YARN setup > sharing resources with mesos. Maybe mesos is not what I need here, I was > just wondering if somebody here does something similar. > > - Andre > > > On 14 Jan 2015, at 16:11, Joseph Jacks <[email protected]> wrote: > > > > Hey Andre, > > > > Have you seen: https://github.com/mesos/myriad > > > > JJ. > > > > On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <[email protected]> > wrote: > > Hi, > > > > we are currently looking for an easy way to spin up Hadoop YARN clusters > on some hardware that we have lying around. I came across mesos and was > wondering if somebody here is using it this way. What I am looking for is > something where I say: Give me a cluster with n nodes. Run some tests and > then destroy the Hadoop cluster. Similar to how EMR would work or how you > work with vagrant locally. Is somebody here doing this and if so, do you > have any starter docs for such a setup somewhere? > > > > Note that we def. need YARN and we want to use vanilla Apache Hadoop, so > no commercial distribution. > > > > Thanks for your help! > > > > - Andre > > > > -- > > André Kelpe > > [email protected] > > http://concurrentinc.com > > > > > > > > > > > > -- > André Kelpe > [email protected] > http://concurrentinc.com > > > > >
