Thanks for the pointers! I will try to get something up and running. - Andre
On Wed, Jan 14, 2015 at 5:36 PM, mohit soni <[email protected]> wrote: > Hi Andre > > Myriad allows you to run YARN on Mesos. Since YARN is also a resource > manager, imagine this as a kind of a delegation of resources from Mesos to > YARN. > > With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon > startup reports back to YARN's Resource Manager and joins the cluster. > > Now, in your scenario you would like to spin up 'ephemeral' hadoop > clusters. You can certainly do so in Mesos ecosystem, with a little bit of > work (right now). My suggestion would be to use Marathon ( > https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource > Manager, and then trigger Myriad's flexUp API ( > https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup) > to spin N number of NodeManagers on mesos cluster. > > Regards > Mohit > > On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe <[email protected]> > wrote: > >> HI, >> >> yes, I looked at it, but from what I understand it is a project to run >> YARN next to mesos, not on top of it. I am looking for a way to have >> “ephemeral” hadoop clusters one some hardware, not a static YARN setup >> sharing resources with mesos. Maybe mesos is not what I need here, I was >> just wondering if somebody here does something similar. >> >> - Andre >> >> > On 14 Jan 2015, at 16:11, Joseph Jacks <[email protected]> wrote: >> > >> > Hey Andre, >> > >> > Have you seen: https://github.com/mesos/myriad >> > >> > JJ. >> > >> > On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe <[email protected]> >> wrote: >> > Hi, >> > >> > we are currently looking for an easy way to spin up Hadoop YARN >> clusters on some hardware that we have lying around. I came across mesos >> and was wondering if somebody here is using it this way. What I am looking >> for is something where I say: Give me a cluster with n nodes. Run some >> tests and then destroy the Hadoop cluster. Similar to how EMR would work or >> how you work with vagrant locally. Is somebody here doing this and if so, >> do you have any starter docs for such a setup somewhere? >> > >> > Note that we def. need YARN and we want to use vanilla Apache Hadoop, >> so no commercial distribution. >> > >> > Thanks for your help! >> > >> > - Andre >> > >> > -- >> > André Kelpe >> > [email protected] >> > http://concurrentinc.com >> > >> > >> > >> > >> > >> >> -- >> André Kelpe >> [email protected] >> http://concurrentinc.com >> >> >> >> >> > -- André Kelpe [email protected] http://concurrentinc.com
