Re: AM creation in yarn client mode

2016-02-10 Thread Steve Loughran
On 10 Feb 2016, at 13:20, Manoj Awasthi > wrote: On Wed, Feb 10, 2016 at 5:20 PM, Steve Loughran > wrote: On 10 Feb 2016, at 04:42, praveen S

Re: AM creation in yarn client mode

2016-02-10 Thread Manoj Awasthi
My pardon to writing that "there is no AM". I realize it! :-) :-) On Wed, Feb 10, 2016 at 7:14 PM, Steve Loughran wrote: > > On 10 Feb 2016, at 13:20, Manoj Awasthi wrote: > > > > On Wed, Feb 10, 2016 at 5:20 PM, Steve Loughran

Re: AM creation in yarn client mode

2016-02-10 Thread Steve Loughran
On 10 Feb 2016, at 14:18, Manoj Awasthi > wrote: My pardon to writing that "there is no AM". I realize it! :-) :-) There is the unmanaged AM option, which was originally written for debugging, but has been used in various apps. Spark

Re: AM creation in yarn client mode

2016-02-10 Thread Steve Loughran
On 10 Feb 2016, at 04:42, praveen S > wrote: Hi, I have 2 questions when running the spark jobs on yarn in client mode : 1) Where is the AM(application master) created : in the cluster A) is it created on the client where the job was

Re: AM creation in yarn client mode

2016-02-10 Thread Manoj Awasthi
On Wed, Feb 10, 2016 at 5:20 PM, Steve Loughran wrote: > > On 10 Feb 2016, at 04:42, praveen S wrote: > > Hi, > > I have 2 questions when running the spark jobs on yarn in client mode : > > 1) Where is the AM(application master) created : > > > in

AM creation in yarn client mode

2016-02-09 Thread praveen S
Hi, I have 2 questions when running the spark jobs on yarn in client mode : 1) Where is the AM(application master) created : A) is it created on the client where the job was submitted? i.e driver and AM on the same client? Or B) yarn decides where the the AM should be created? 2) Driver and AM

Re: AM creation in yarn client mode

2016-02-09 Thread ayan guha
It depends on yarn-cluster and yarn-client mode. On Wed, Feb 10, 2016 at 3:42 PM, praveen S wrote: > Hi, > > I have 2 questions when running the spark jobs on yarn in client mode : > > 1) Where is the AM(application master) created : > > A) is it created on the client where

Re: AM creation in yarn-client mode

2016-02-09 Thread Alexander Pivovarov
the pictures to illustrate it http://www.cloudera.com/documentation/enterprise/5-4-x/topics/cdh_ig_running_spark_on_yarn.html On Tue, Feb 9, 2016 at 10:18 PM, Jonathan Kelly wrote: > In yarn-client mode, the driver is separate from the AM. The AM is created > in YARN,

Re: AM creation in yarn client mode

2016-02-09 Thread Diwakar Dhanuskodi
gt; Cc: user <user@spark.apache.org> Subject: Re: AM creation in yarn client mode It depends on yarn-cluster and yarn-client mode. On Wed, Feb 10, 2016 at 3:42 PM, praveen S <mylogi...@gmail.com> wrote: Hi, I have 2 questions when running the spark jobs on yarn in client mo

Re: AM creation in yarn-client mode

2016-02-09 Thread praveen S
Can you explain what happens in yarn client mode? Regards, Praveen On 10 Feb 2016 10:55, "ayan guha" wrote: > It depends on yarn-cluster and yarn-client mode. > > On Wed, Feb 10, 2016 at 3:42 PM, praveen S wrote: > >> Hi, >> >> I have 2 questions when

Re: AM creation in yarn-client mode

2016-02-09 Thread Jonathan Kelly
In yarn-client mode, the driver is separate from the AM. The AM is created in YARN, and YARN controls where it goes (though you can somewhat control it using YARN node labels--I just learned earlier today in a different thread on this list that this can be controlled by