On 10 Feb 2016, at 13:20, Manoj Awasthi <awasthi.ma...@gmail.com<mailto:awasthi.ma...@gmail.com>> wrote:
On Wed, Feb 10, 2016 at 5:20 PM, Steve Loughran <ste...@hortonworks.com<mailto:ste...@hortonworks.com>> wrote: On 10 Feb 2016, at 04:42, praveen S <mylogi...@gmail.com<mailto:mylogi...@gmail.com>> wrote: Hi, I have 2 questions when running the spark jobs on yarn in client mode : 1) Where is the AM(application master) created : in the cluster A) is it created on the client where the job was submitted? i.e driver and AM on the same client? no Or B) yarn decides where the the AM should be created? yes 2) Driver and AM run in different processes : is my assumption correct? yes. the driver runs on your local system, which had better be close to the cluster and stay up for the duration of the work This is not correct. In yarn-cluster mode driver is what runs inside the application master and the node on which application master gets allocated is decided by yarn. I agree In yarn-client mode, there is no application master and driver runs in the context of the same unix process as the spark-submit. I must beg to differ There is an AM. All YARN apps need an AM, as it is the only way you can get containers to run your work. And, except in the special case of an "Unmanaged AM", the AM runs in the cluster. In spark, an AM is always set up by the YARN client and submitted to the cluster https://github.com/apache/spark/blob/4f60651cbec1b4c9cc2e6d832ace77e89a233f3a/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L116 The big difference is where that driver lives. In --cluster, its in the AM. in --client, it's in the client here's the AM making its decision https://github.com/apache/spark/blob/4f60651cbec1b4c9cc2e6d832ace77e89a233f3a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L187 -Steve (why yes, I have spent too much time staring at AM logs)
