Re: (elastic) Hadoop YARN on mesos?

[Shivaji Dutta]

Andre

You can take a look at Apache Ambari. It allows you to spin up clusters very 
fast, add nodes and remove nodes.
If you have an existing cluster and you want to run long running jobs, you can 
definitely use Apache Slider which is like Marathon in some ways.

There is doc
https://cwiki.apache.org/confluence/display/AMBARI/Ambari

Shivaji

From: Joseph Jacks mailto:jacks@gmail.com>>
Reply-To: "user@mesos.apache.org" 
mailto:user@mesos.apache.org>>
Date: Wednesday, January 14, 2015 at 4:11 PM
To: "user@mesos.apache.org" 
mailto:user@mesos.apache.org>>
Subject: Re: (elastic) Hadoop YARN on mesos?

Hey Andre,

Have you seen: https://github.com/mesos/myriad

JJ.

On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe 
mailto:ake...@concurrentinc.com>> wrote:
Hi,

we are currently looking for an easy way to spin up Hadoop YARN clusters on 
some hardware that we have lying around. I came across mesos and was wondering 
if somebody here is using it this way. What I am looking for is something where 
I say: Give me a cluster with n nodes. Run some tests and then destroy the 
Hadoop cluster. Similar to how EMR would work or how you work with vagrant 
locally. Is somebody here doing this and if so, do you have any starter docs 
for such a setup somewhere?

Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no 
commercial distribution.

Thanks for your help!

- Andre

--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com






CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

Re: (elastic) Hadoop YARN on mesos?

[mohit soni]

Hi Andre

Myriad allows you to run YARN on Mesos. Since YARN is also a resource
manager, imagine this as a kind of a delegation of resources from Mesos to
YARN.

With Myriad one can spin up NodeManager(s) as Mesos tasks, which upon
startup reports back to YARN's Resource Manager and joins the cluster.

Now, in your scenario you would like to spin up 'ephemeral' hadoop
clusters. You can certainly do so in Mesos ecosystem, with a little bit of
work (right now). My suggestion would be to use Marathon (
https://mesosphere.github.io/marathon/) to launch Myriad/YARN Resource
Manager, and then trigger Myriad's flexUp API (
https://github.com/mesos/myriad/blob/phase1/docs/API.md#put-apiclusterflexup)
to spin N number of NodeManagers on mesos cluster.

Regards
Mohit

On Wed, Jan 14, 2015 at 4:37 PM, Andre Kelpe 
wrote:

> HI,
>
> yes, I looked at it, but from what I understand it is a project to run
> YARN next to mesos, not on top of it. I am looking for a way to have
> “ephemeral” hadoop clusters one some hardware, not a static YARN setup
> sharing resources with mesos. Maybe mesos is not what I need here, I was
> just wondering if somebody here does something similar.
>
> - Andre
>
> > On 14 Jan 2015, at 16:11, Joseph Jacks  wrote:
> >
> > Hey Andre,
> >
> > Have you seen: https://github.com/mesos/myriad
> >
> > JJ.
> >
> > On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe 
> wrote:
> > Hi,
> >
> > we are currently looking for an easy way to spin up Hadoop YARN clusters
> on some hardware that we have lying around. I came across mesos and was
> wondering if somebody here is using it this way. What I am looking for is
> something where I say: Give me a cluster with n nodes. Run some tests and
> then destroy the Hadoop cluster. Similar to how EMR would work or how you
> work with vagrant locally. Is somebody here doing this and if so, do you
> have any starter docs for such a setup somewhere?
> >
> > Note that we def. need YARN and we want to use vanilla Apache Hadoop, so
> no commercial distribution.
> >
> > Thanks for your help!
> >
> > - Andre
> >
> > --
> > André Kelpe
> > an...@concurrentinc.com
> > http://concurrentinc.com
> >
> >
> >
> >
> >
>
> --
> André Kelpe
> an...@concurrentinc.com
> http://concurrentinc.com
>
>
>
>
>

Re: (elastic) Hadoop YARN on mesos?

[Andre Kelpe]

HI,

yes, I looked at it, but from what I understand it is a project to run YARN 
next to mesos, not on top of it. I am looking for a way to have “ephemeral” 
hadoop clusters one some hardware, not a static YARN setup sharing resources 
with mesos. Maybe mesos is not what I need here, I was just wondering if 
somebody here does something similar.

- Andre

> On 14 Jan 2015, at 16:11, Joseph Jacks  wrote:
> 
> Hey Andre,
> 
> Have you seen: https://github.com/mesos/myriad
> 
> JJ.
> 
> On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe  wrote:
> Hi,
> 
> we are currently looking for an easy way to spin up Hadoop YARN clusters on 
> some hardware that we have lying around. I came across mesos and was 
> wondering if somebody here is using it this way. What I am looking for is 
> something where I say: Give me a cluster with n nodes. Run some tests and 
> then destroy the Hadoop cluster. Similar to how EMR would work or how you 
> work with vagrant locally. Is somebody here doing this and if so, do you have 
> any starter docs for such a setup somewhere?
> 
> Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no 
> commercial distribution.
> 
> Thanks for your help!
> 
> - Andre
> 
> --
> André Kelpe
> an...@concurrentinc.com
> http://concurrentinc.com
> 
> 
> 
> 
> 

--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

Re: (elastic) Hadoop YARN on mesos?

[Joseph Jacks]

Hey Andre,

Have you seen: https://github.com/mesos/myriad

JJ.

On Wed, Jan 14, 2015 at 4:07 PM, Andre Kelpe 
wrote:

> Hi,
>
> we are currently looking for an easy way to spin up Hadoop YARN clusters
> on some hardware that we have lying around. I came across mesos and was
> wondering if somebody here is using it this way. What I am looking for is
> something where I say: Give me a cluster with n nodes. Run some tests and
> then destroy the Hadoop cluster. Similar to how EMR would work or how you
> work with vagrant locally. Is somebody here doing this and if so, do you
> have any starter docs for such a setup somewhere?
>
> Note that we def. need YARN and we want to use vanilla Apache Hadoop, so
> no commercial distribution.
>
> Thanks for your help!
>
> - Andre
>
> --
> André Kelpe
> an...@concurrentinc.com
> http://concurrentinc.com
>
>
>
>
>

(elastic) Hadoop YARN on mesos?

[Andre Kelpe]

Hi,

we are currently looking for an easy way to spin up Hadoop YARN clusters on 
some hardware that we have lying around. I came across mesos and was wondering 
if somebody here is using it this way. What I am looking for is something where 
I say: Give me a cluster with n nodes. Run some tests and then destroy the 
Hadoop cluster. Similar to how EMR would work or how you work with vagrant 
locally. Is somebody here doing this and if so, do you have any starter docs 
for such a setup somewhere? 

Note that we def. need YARN and we want to use vanilla Apache Hadoop, so no 
commercial distribution.

Thanks for your help!

- Andre

--
André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

Re: hadoop job stuck.

[Brenden Matthews]

Would need the task logs from the slave which the TaskTracker was launched
on, to debug this further.

On Wed, Jan 14, 2015 at 1:28 PM, Dan Dong  wrote:

> Checked /etc/hosts is correct, master and slave can ssh login each other
> by hostname without password, and hadoop runs well without mesos, but it
> stucks when running on mesos.
>
> Cheers,
> Dan
>
> 2015-01-14 15:02 GMT-06:00 Brenden Matthews :
>
> At a first glance, it looks like `/etc/hosts` might be set incorrectly and
>> it cannot resolve the hostname of the worker.
>>
>> See here for more: https://wiki.apache.org/hadoop/UnknownHost
>>
>> On Wed, Jan 14, 2015 at 12:32 PM, Vinod Kone 
>> wrote:
>>
>>> What do the master logs say?
>>>
>>> On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong  wrote:
>>>
 Hi,
   When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to
 process : 8
 15/01/14 13:59:30 INFO mapred.JobClient: Running job:
 job_201501141358_0001
 15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%

 From jobtracker log I see:
 2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
 Launching task Task_Tracker_0 on http://centos-2.local:31911 with
 mapSlots=1 reduceSlots=0
 2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
 Tracker http://centos-2.local:31911 failed to launch within 300
 seconds, killing it

  I started manually namenode and jobtracker on master node and datanode
 on slave, but I could not see tasktracker started by mesos on slave. Note
 that if I ran hadoop directly without Mesos( of course the conf files are
 different and tasktracker will be started manually on slave), everything
 works fine. Any hints?

 Cheers,
 Dan

>>>
>>>
>>
>

Re: hadoop job stuck.

[Dan Dong]

Checked /etc/hosts is correct, master and slave can ssh login each other by
hostname without password, and hadoop runs well without mesos, but it
stucks when running on mesos.

Cheers,
Dan

2015-01-14 15:02 GMT-06:00 Brenden Matthews :

> At a first glance, it looks like `/etc/hosts` might be set incorrectly and
> it cannot resolve the hostname of the worker.
>
> See here for more: https://wiki.apache.org/hadoop/UnknownHost
>
> On Wed, Jan 14, 2015 at 12:32 PM, Vinod Kone  wrote:
>
>> What do the master logs say?
>>
>> On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong  wrote:
>>
>>> Hi,
>>>   When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
>>> 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to
>>> process : 8
>>> 15/01/14 13:59:30 INFO mapred.JobClient: Running job:
>>> job_201501141358_0001
>>> 15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%
>>>
>>> From jobtracker log I see:
>>> 2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
>>> Launching task Task_Tracker_0 on http://centos-2.local:31911 with
>>> mapSlots=1 reduceSlots=0
>>> 2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
>>> Tracker http://centos-2.local:31911 failed to launch within 300
>>> seconds, killing it
>>>
>>>  I started manually namenode and jobtracker on master node and datanode
>>> on slave, but I could not see tasktracker started by mesos on slave. Note
>>> that if I ran hadoop directly without Mesos( of course the conf files are
>>> different and tasktracker will be started manually on slave), everything
>>> works fine. Any hints?
>>>
>>> Cheers,
>>> Dan
>>>
>>
>>
>

Re: hadoop job stuck.

[Dan Dong]

Hi, All,
  I'm using hadoop-2.5.0-cdh5.2.0 on mesos 0.21.0, the following are from
master log of mesos:
I0114 14:47:04.855033 17756 master.cpp:1768] Deactivating framework
20150114-144506-3205108908-5050-17750- (Hadoop: (RPC port: 9001, WebUI
port: 50030)) at
scheduler-d37097ae-8b55-4f73-9936-1652e6726fe3@172.20.10.191:35511
I0114 14:47:04.857833 17755 hierarchical_allocator_process.hpp:405]
Deactivated framework 20150114-144506-3205108908-5050-17750-
I0114 14:47:04.858381 17755 hierarchical_allocator_process.hpp:563]
Recovered cpus(*):2; mem(*):2808; disk(*):45148; ports(*):[31000-32000]
(total allocatable: cpus(*):2; mem(*):2808; disk(*):45148;
ports(*):[31000-32000]) on slave 20150108-152507-3205108908-5050-27619-S0
from framework 20150114-144506-3205108908-5050-17750-
I0114 14:47:11.107666 17753 http.cpp:478] HTTP request for
'/master/state.json'
I0114 14:47:21.117486 17757 http.cpp:478] HTTP request for
'/master/state.json'
I0114 14:47:31.129355 17759 http.cpp:478] HTTP request for
'/master/state.json'

Cheers,
Dan

2015-01-14 14:32 GMT-06:00 Vinod Kone :

> What do the master logs say?
>
> On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong  wrote:
>
>> Hi,
>>   When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
>> 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to
>> process : 8
>> 15/01/14 13:59:30 INFO mapred.JobClient: Running job:
>> job_201501141358_0001
>> 15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%
>>
>> From jobtracker log I see:
>> 2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
>> Launching task Task_Tracker_0 on http://centos-2.local:31911 with
>> mapSlots=1 reduceSlots=0
>> 2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
>> Tracker http://centos-2.local:31911 failed to launch within 300 seconds,
>> killing it
>>
>>  I started manually namenode and jobtracker on master node and datanode
>> on slave, but I could not see tasktracker started by mesos on slave. Note
>> that if I ran hadoop directly without Mesos( of course the conf files are
>> different and tasktracker will be started manually on slave), everything
>> works fine. Any hints?
>>
>> Cheers,
>> Dan
>>
>
>

Re: hadoop job stuck.

[Brenden Matthews]

At a first glance, it looks like `/etc/hosts` might be set incorrectly and
it cannot resolve the hostname of the worker.

See here for more: https://wiki.apache.org/hadoop/UnknownHost

On Wed, Jan 14, 2015 at 12:32 PM, Vinod Kone  wrote:

> What do the master logs say?
>
> On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong  wrote:
>
>> Hi,
>>   When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
>> 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to
>> process : 8
>> 15/01/14 13:59:30 INFO mapred.JobClient: Running job:
>> job_201501141358_0001
>> 15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%
>>
>> From jobtracker log I see:
>> 2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
>> Launching task Task_Tracker_0 on http://centos-2.local:31911 with
>> mapSlots=1 reduceSlots=0
>> 2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
>> Tracker http://centos-2.local:31911 failed to launch within 300 seconds,
>> killing it
>>
>>  I started manually namenode and jobtracker on master node and datanode
>> on slave, but I could not see tasktracker started by mesos on slave. Note
>> that if I ran hadoop directly without Mesos( of course the conf files are
>> different and tasktracker will be started manually on slave), everything
>> works fine. Any hints?
>>
>> Cheers,
>> Dan
>>
>
>

Re: hadoop job stuck.

[Vinod Kone]

What do the master logs say?

On Wed, Jan 14, 2015 at 12:21 PM, Dan Dong  wrote:

> Hi,
>   When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
> 15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to
> process : 8
> 15/01/14 13:59:30 INFO mapred.JobClient: Running job: job_201501141358_0001
> 15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%
>
> From jobtracker log I see:
> 2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
> Launching task Task_Tracker_0 on http://centos-2.local:31911 with
> mapSlots=1 reduceSlots=0
> 2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
> Tracker http://centos-2.local:31911 failed to launch within 300 seconds,
> killing it
>
>  I started manually namenode and jobtracker on master node and datanode on
> slave, but I could not see tasktracker started by mesos on slave. Note that
> if I ran hadoop directly without Mesos( of course the conf files are
> different and tasktracker will be started manually on slave), everything
> works fine. Any hints?
>
> Cheers,
> Dan
>

Re: hadoop job stuck.

[Tom Arnfeld]

Hi Dan,




Can you look at the stdout/stderr logs in the task sandbox for me and share any 
errors here?




Also – What version of Hadoop are you using, and what version of the Hadoop on 
Mesos framework?




Thanks.



--


Tom Arnfeld

Developer // DueDil






On Wednesday, Jan 14, 2015 at 8:22 pm, Dan Dong , wrote:
Hi,
  When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to process : 8
15/01/14 13:59:30 INFO mapred.JobClient: Running job: job_201501141358_0001
15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%

>From jobtracker log I see:
2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy: Launching 
task Task_Tracker_0 on http://centos-2.local:31911 with mapSlots=1 reduceSlots=0
2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler: Tracker 
http://centos-2.local:31911 failed to launch within 300 seconds, killing it

 I started manually namenode and jobtracker on master node and datanode on 
slave, but I could not see tasktracker started by mesos on slave. Note that if 
I ran hadoop directly without Mesos( of course the conf files are different and 
tasktracker will be started manually on slave), everything works fine. Any 
hints?

Cheers,
Dan

hadoop job stuck.

[Dan Dong]

Hi,
  When I run hadoop jobs on Mesos(0.21.0), the jobs are stuck for ever:
15/01/14 13:59:30 INFO mapred.FileInputFormat: Total input paths to process
: 8
15/01/14 13:59:30 INFO mapred.JobClient: Running job: job_201501141358_0001
15/01/14 13:59:31 INFO mapred.JobClient:  map 0% reduce 0%

>From jobtracker log I see:
2015-01-14 13:59:35,542 INFO org.apache.hadoop.mapred.ResourcePolicy:
Launching task Task_Tracker_0 on http://centos-2.local:31911 with
mapSlots=1 reduceSlots=0
2015-01-14 14:04:35,552 WARN org.apache.hadoop.mapred.MesosScheduler:
Tracker http://centos-2.local:31911 failed to launch within 300 seconds,
killing it

 I started manually namenode and jobtracker on master node and datanode on
slave, but I could not see tasktracker started by mesos on slave. Note that
if I ran hadoop directly without Mesos( of course the conf files are
different and tasktracker will be started manually on slave), everything
works fine. Any hints?

Cheers,
Dan

MesosCon Early-Bird Registration Now Open

[Dave Lester]

A limited number of early-bird tickets are now available for #MesosCon 2015
, an annual conference organized by the Apache Mesos
community. MesosCon brings together users and developers to share and learn
about the project and its growing ecosystem, and will take place in
Seattle, WA.

Early-bird registration is open today
 to the
first 140 attendees / through February 14th, 2015.

MesosCon will take place August 20 - 21, 2015 at Sheraton Seattle in
Seattle, WA. The cost of early-bird registration is US$299. The standard
attendee registration fee will be US$399 (Through July 31, 2015), and late
registration US$499 beginning August 1, 2015. Students are eligible for a
special registration fee of US$99 (Note that a valid student id will be
required at onsite check-in).

As part of the registration process, you’ll have an opportunity to donate
to a Diversity Scholarship

which has been established for the conference. MesosCon’s scholarship
program provides support to women, people of color, and people with
disabilities who may not otherwise have the opportunity to attend MesosCon
for financial reasons. Equal access and diversity are important to
MesosCon, and we aim to remove this obstacle for underrepresented attendee
groups. We encourage you to donate what you can! If you have any questions
regarding the diversity scholarship, please contact
mesoscon-divers...@googlegroups.com.

On behalf of the #MesosCon Program Committee, we couldn’t be more excited
about this year’s conference and what this means for the open source
project’s community. We hope that you’ll join us!