Hi Mich, thank you for the answer. Regarding the diagrams, I'm specifically referring to the direct line between spark yarn client to spark executor in the first diagram which implies direct communication to executor when issuing application commands. And the 'Application commands' & 'Issue application commands' lines in the second diagram which implies that spark driver in client communicates to executor via yarn application master (Correct me if i'm wrong in these interpretations). Would you happen to know how spark drivers communicates with executor in yarn-client mode or if both can be true under different circumstances?
Thanks again, Nisrina. On Fri, Nov 27, 2015 at 6:22 PM, Mich Talebzadeh <[email protected]> wrote: > Hi, > > > > In general YARN is used as the resource scheduler regardless of the > execution engine whether it is MapReduce or Spark. > > > > Yarn will create a resource container for the submitted job (that is the > Spark client) and will execute it in the default engine (in this case > Spark). There will be a job scheduler and one or more Spark Executors > depending on the cluster. So as far as I can see both diagrams are correct, > > > > HTH > > > > Mich Talebzadeh > > > > *Sybase ASE 15 Gold Medal Award 2008* > > A Winning Strategy: Running the most Critical Financial Data on ASE 15 > > > http://login.sybase.com/files/Product_Overviews/ASE-Winning-Strategy-091908.pdf > > Author of the books* "A Practitioner’s Guide to Upgrading to Sybase ASE > 15", ISBN 978-0-9563693-0-7*. > > co-author *"Sybase Transact SQL Guidelines Best Practices", ISBN > 978-0-9759693-0-4* > > *Publications due shortly:* > > *Complex Event Processing in Heterogeneous Environments*, ISBN: > 978-0-9563693-3-8 > > *Oracle and Sybase, Concepts and Contrasts*, ISBN: 978-0-9563693-1-4, volume > one out shortly > > > > http://talebzadehmich.wordpress.com > > > > NOTE: The information in this email is proprietary and confidential. This > message is for the designated recipient only, if you are not the intended > recipient, you should destroy it immediately. Any information in this > message shall not be understood as given or endorsed by Peridale Technology > Ltd, its subsidiaries or their employees, unless expressly so stated. It is > the responsibility of the recipient to ensure that this email is virus > free, therefore neither Peridale Ltd, its subsidiaries nor their employees > accept any responsibility. > > > > *From:* Nisrina Luthfiyati [mailto:[email protected]] > *Sent:* 27 November 2015 11:12 > *To:* [email protected] > *Subject:* In yarn-client mode, is it the driver or application master > that issue commands to executors? > > > > Hi all, > > I'm trying to understand how yarn-client mode works and found these two > diagrams: > > > > > In the first diagram, it looks like the driver running in client directly > communicates with executors to issue application commands, while in the > second diagram it looks like application commands is sent to application > master first and then forwarded to executors. > > Would anyone knows which case is true or is there any other interpretation > to these diagrams? > > Thanks! > > Nisrina > -- Nisrina Luthfiyati - Ilmu Komputer Fasilkom UI 2010 http://www.facebook.com/nisrina.luthfiyati http://id.linkedin.com/in/nisrina
