Re: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-12-07 Thread Nisrina Luthfiyati 
Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
TaskSetManager and it does looked like tasks are directly sent to
executors. Also would love to be corrected if mistaken as I have little
knowledge about Spark internals and very new at scala.

On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski  wrote:

> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
> nisrina.luthfiy...@gmail.com> wrote:
>
>> Hi all,
>> I'm trying to understand how yarn-client mode works and found these two
>> diagrams:
>>
>>
>>
>>
>> In the first diagram, it looks like the driver running in client directly
>> communicates with executors to issue application commands, while in the
>> second diagram it looks like application commands is sent to application
>> master first and then forwarded to executors.
>>
>
> My limited understanding tells me that regardless of deploy mode (local,
> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
> TaskSets to executors once they're launched. YARN and Mesos are only used
> until they offer resources (CPU and memory) and once executors start, these
> cluster managers are not engaged in the communication (driver and executors
> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>
> I'd love being corrected if mistaken. Thanks.
>
> Jacek
>



-- 
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina

Re: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-12-07 Thread Jacek Laskowski 
Hi,

That's my understanding, too. Just spent an entire morning today to check
it out and would be surprised to hear otherwise.

Pozdrawiam,
Jacek

--
Jacek Laskowski | https://medium.com/@jaceklaskowski/ |
http://blog.jaceklaskowski.pl
Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
Follow me at https://twitter.com/jaceklaskowski
Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski

On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati <
nisrina.luthfiy...@gmail.com> wrote:

> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and
> TaskSetManager and it does looked like tasks are directly sent to
> executors. Also would love to be corrected if mistaken as I have little
> knowledge about Spark internals and very new at scala.
>
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski  wrote:
>
>> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
>> nisrina.luthfiy...@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm trying to understand how yarn-client mode works and found these two
>>> diagrams:
>>>
>>>
>>>
>>>
>>> In the first diagram, it looks like the driver running in client
>>> directly communicates with executors to issue application commands, while
>>> in the second diagram it looks like application commands is sent to
>>> application master first and then forwarded to executors.
>>>
>>
>> My limited understanding tells me that regardless of deploy mode (local,
>> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
>> TaskSets to executors once they're launched. YARN and Mesos are only used
>> until they offer resources (CPU and memory) and once executors start, these
>> cluster managers are not engaged in the communication (driver and executors
>> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
>>
>> I'd love being corrected if mistaken. Thanks.
>>
>> Jacek
>>
>
>
>
> --
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
>
>

Re: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-12-07 Thread Ali Tajeldin EDU 
Checkout the Sameer Farooqui video on youtube for spark internals 
(https://www.youtube.com/watch?v=7ooZ4S7Ay6Y=PLIxzgeMkSrQ-2Uizm4l0HjNSSy2NxgqjX)
Starting at 2:15:00, he describes YARN mode.

btw, highly recommend the entire video.  Very detailed and concise.

--
Ali


On Dec 7, 2015, at 8:38 AM, Jacek Laskowski  wrote:

> Hi,
> 
> That's my understanding, too. Just spent an entire morning today to check it 
> out and would be surprised to hear otherwise.
> 
> Pozdrawiam,
> Jacek
> 
> --
> Jacek Laskowski | https://medium.com/@jaceklaskowski/ | 
> http://blog.jaceklaskowski.pl
> Mastering Spark https://jaceklaskowski.gitbooks.io/mastering-apache-spark/
> Follow me at https://twitter.com/jaceklaskowski
> Upvote at http://stackoverflow.com/users/1305344/jacek-laskowski
> 
> On Mon, Dec 7, 2015 at 4:01 PM, Nisrina Luthfiyati 
>  wrote:
> Hi Jacek, thank you for your answer. I looked at TaskSchedulerImpl and 
> TaskSetManager and it does looked like tasks are directly sent to executors. 
> Also would love to be corrected if mistaken as I have little knowledge about 
> Spark internals and very new at scala.
> 
> On Tue, Dec 1, 2015 at 1:16 AM, Jacek Laskowski  wrote:
> On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati 
>  wrote:
> Hi all,
> I'm trying to understand how yarn-client mode works and found these two 
> diagrams:
> 
> 
> 
> 
> In the first diagram, it looks like the driver running in client directly 
> communicates with executors to issue application commands, while in the 
> second diagram it looks like application commands is sent to application 
> master first and then forwarded to executors. 
> 
> My limited understanding tells me that regardless of deploy mode (local, 
> standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends TaskSets 
> to executors once they're launched. YARN and Mesos are only used until they 
> offer resources (CPU and memory) and once executors start, these cluster 
> managers are not engaged in the communication (driver and executors 
> communicate using RPC over netty since 1.6-SNAPSHOT or akka before).
> 
> I'd love being corrected if mistaken. Thanks.
> 
> Jacek
> 
> 
> 
> -- 
> Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
> http://www.facebook.com/nisrina.luthfiyati
> http://id.linkedin.com/in/nisrina
> 
>

Re: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-11-30 Thread Jacek Laskowski 
On Fri, Nov 27, 2015 at 12:12 PM, Nisrina Luthfiyati <
nisrina.luthfiy...@gmail.com> wrote:

> Hi all,
> I'm trying to understand how yarn-client mode works and found these two
> diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly
> communicates with executors to issue application commands, while in the
> second diagram it looks like application commands is sent to application
> master first and then forwarded to executors.
>

My limited understanding tells me that regardless of deploy mode (local,
standalone, YARN or mesos), drivers (using TaskSchedulerImpl) sends
TaskSets to executors once they're launched. YARN and Mesos are only used
until they offer resources (CPU and memory) and once executors start, these
cluster managers are not engaged in the communication (driver and executors
communicate using RPC over netty since 1.6-SNAPSHOT or akka before).

I'd love being corrected if mistaken. Thanks.

Jacek

RE: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-11-27 Thread Mich Talebzadeh 
Hi,

 

In general YARN is used as the resource scheduler regardless of the execution 
engine whether it is MapReduce or Spark.

 

Yarn will create a resource container for the submitted job (that is the Spark 
client) and will execute it in the default engine (in this case Spark). There 
will be a job scheduler and one or more Spark Executors depending on the 
cluster. So as far as I can see both diagrams are correct,

 

HTH

 

Mich Talebzadeh

 

Sybase ASE 15 Gold Medal Award 2008

A Winning Strategy: Running the most Critical Financial Data on ASE 15

 

 http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf

Author of the books "A Practitioner’s Guide to Upgrading to Sybase ASE 15", 
ISBN 978-0-9563693-0-7. 

co-author "Sybase Transact SQL Guidelines Best Practices", ISBN 
978-0-9759693-0-4

Publications due shortly:

Complex Event Processing in Heterogeneous Environments, ISBN: 978-0-9563693-3-8

Oracle and Sybase, Concepts and Contrasts, ISBN: 978-0-9563693-1-4, volume one 
out shortly

 

  http://talebzadehmich.wordpress.com

 

NOTE: The information in this email is proprietary and confidential. This 
message is for the designated recipient only, if you are not the intended 
recipient, you should destroy it immediately. Any information in this message 
shall not be understood as given or endorsed by Peridale Technology Ltd, its 
subsidiaries or their employees, unless expressly so stated. It is the 
responsibility of the recipient to ensure that this email is virus free, 
therefore neither Peridale Ltd, its subsidiaries nor their employees accept any 
responsibility.

 

From: Nisrina Luthfiyati [mailto:nisrina.luthfiy...@gmail.com] 
Sent: 27 November 2015 11:12
To: user@spark.apache.org
Subject: In yarn-client mode, is it the driver or application master that issue 
commands to executors?

 

Hi all,

I'm trying to understand how yarn-client mode works and found these two 
diagrams:


  

 

   

In the first diagram, it looks like the driver running in client directly 
communicates with executors to issue application commands, while in the second 
diagram it looks like application commands is sent to application master first 
and then forwarded to executors. 

Would anyone knows which case is true or is there any other interpretation to 
these diagrams?

Thanks!

Nisrina

In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-11-27 Thread Nisrina Luthfiyati 
Hi all,
I'm trying to understand how yarn-client mode works and found these two
diagrams:




In the first diagram, it looks like the driver running in client directly
communicates with executors to issue application commands, while in the
second diagram it looks like application commands is sent to application
master first and then forwarded to executors.

Would anyone knows which case is true or is there any other interpretation
to these diagrams?

Thanks!
Nisrina

Re: In yarn-client mode, is it the driver or application master that issue commands to executors?

2015-11-27 Thread Nisrina Luthfiyati 
Hi Mich, thank you for the answer. Regarding the diagrams, I'm specifically
referring to the direct line between spark yarn client to spark executor in
the first diagram which implies direct communication to executor when
issuing application commands. And the 'Application commands' & 'Issue
application commands' lines in the second diagram which implies that spark
driver in client communicates to executor via yarn application master
(Correct me if i'm wrong in these interpretations). Would you happen to
know how spark drivers communicates with executor in yarn-client mode or if
both can be true under different circumstances?

Thanks again,
Nisrina.

On Fri, Nov 27, 2015 at 6:22 PM, Mich Talebzadeh 
wrote:

> Hi,
>
>
>
> In general YARN is used as the resource scheduler regardless of the
> execution engine whether it is MapReduce or Spark.
>
>
>
> Yarn will create a resource container for the submitted job (that is the
> Spark client) and will execute it in the default engine (in this case
> Spark). There will be a job scheduler and one or more Spark Executors
> depending on the cluster. So as far as I can see both diagrams are correct,
>
>
>
> HTH
>
>
>
> Mich Talebzadeh
>
>
>
> *Sybase ASE 15 Gold Medal Award 2008*
>
> A Winning Strategy: Running the most Critical Financial Data on ASE 15
>
>
> http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf
>
> Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE
> 15", ISBN 978-0-9563693-0-7*.
>
> co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN
> 978-0-9759693-0-4*
>
> *Publications due shortly:*
>
> *Complex Event Processing in Heterogeneous Environments*, ISBN:
> 978-0-9563693-3-8
>
> *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume
> one out shortly
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> NOTE: The information in this email is proprietary and confidential. This
> message is for the designated recipient only, if you are not the intended
> recipient, you should destroy it immediately. Any information in this
> message shall not be understood as given or endorsed by Peridale Technology
> Ltd, its subsidiaries or their employees, unless expressly so stated. It is
> the responsibility of the recipient to ensure that this email is virus
> free, therefore neither Peridale Ltd, its subsidiaries nor their employees
> accept any responsibility.
>
>
>
> *From:* Nisrina Luthfiyati [mailto:nisrina.luthfiy...@gmail.com]
> *Sent:* 27 November 2015 11:12
> *To:* user@spark.apache.org
> *Subject:* In yarn-client mode, is it the driver or application master
> that issue commands to executors?
>
>
>
> Hi all,
>
> I'm trying to understand how yarn-client mode works and found these two
> diagrams:
>
>
>
>
> In the first diagram, it looks like the driver running in client directly
> communicates with executors to issue application commands, while in the
> second diagram it looks like application commands is sent to application
> master first and then forwarded to executors.
>
> Would anyone knows which case is true or is there any other interpretation
> to these diagrams?
>
> Thanks!
>
> Nisrina
>



-- 
Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010
http://www.facebook.com/nisrina.luthfiyati
http://id.linkedin.com/in/nisrina