Porting YARN to run atop Mesos is quite reasonable. Some folks at eBay
have started some work on this (https://github.com/mesos/myriad). If
you're interested, you should check it out, and contribute to the project.
On Tue, Oct 28, 2014 at 5:21 AM, Yaneeve Shekel yaneeve.she...@sizmek.com
wrote:
/#!topic/docker-dev/6tt1y9FTWKg
Cheers,
Tim
- Original Message -
From: Jie Yu yujie@gmail.com
To: user@mesos.apache.org
Cc: mesos d...@mesos.apache.org
Sent: Saturday, July 26, 2014 5:19:28 PM
Subject: Re: Does Mesos support Hadoop MR V2
I am interested in investigating the idea
Thanks for the explanation John, that's very useful. I wasn't aware each
job in MRv2 was considered it's own entity to the scheduler, that's
interesting... I think Maxime's point about some kind of hadoop compatible
framework would work well, it sounds to me like the
FrameworkExecutorTask flow
When I said that running yarn over mesos did not make sense I meant that
running a resource manager in a resource manager was very sub-optimal. You
will eventually do static allocation of resources for the Yarn framework in
Mesos or have complex logic to determine how much resource should be given
So excuse my naivety in this space, but my ignorance has never really
stopped me from asking questions:
I see YARN (Yet another resource negotiator) as very similar to Mesos. I.e.
something to manage resources on a cluster of machines. So when I hear talk
of running YARN on Mesos it's seems very
I second john's opinion on the confusing part of different terminology of
hadoop v2. That's the reason I asked the question on if mesos support mr
v2. As maxime's concern, the decoupling part might be difficult. After
reading the mesos mrv1's implementation, I think possibly mrv2 migration
can
I haven't written yarn app masters myself and browsing the
hadoop-mapreduce-project directories is really not easy. I think it is
feasible to get a prototype to work but it would take time.
From what I know one difference is that the app master (which is equivalent
to the mesos framework) is run
I am interested in investigating the idea of YARN on top of Mesos. One of
the benefits I can think of is that we can get rid of the static resource
allocation between YARN and Mesos clusters. In that way, Mesos can allocate
those resources that are not used by YARN to other Mesos frameworks like
Hey Luyi,
That's correct, the Hadoop framework currently only supports Hadoop 2 MRv1.
It also doesn't have great support for the HA jobtracker available in newer
versions of Hadoop, but I've been working on that the past few weeks.
I'm not sure how Hadoop 2 would play with Mesos, but very
Hey Tom:
Really nice to get your reply. I am also looking forward that. If your
progress can be shared, that would be great. I will also look into this.
Will report back if I got any progress.
Thanks a lot!
-Luyi.
On Fri, Jul 25, 2014 at 11:54 AM, Tom Arnfeld t...@duedil.com wrote:
I've not seen any issues pertaining to running many short lived frameworks,
but that's not near the number of frameworks you'd see if each job was a
framework.
We've been pushing all our work on MRv1 High Availability JT upstream on
the github.com/mesos/hadoop repo, though there hasn't been much
11 matches
Mail list logo
