Re: Understanding code/closure shipment to Spark workers‏

ssing something. > > Thanks again, Arijit > > -- > Date: Thu, 8 Oct 2015 10:26:55 -0700 > Subject: Re: Understanding code/closure shipment to Spark workers‏ > From: gatorsm...@gmail.com > To: arij...@live.com > CC: dev@spark.apache.org > > > Hi, Arijit

RE: Understanding code/closure shipment to Spark workers‏

the closures are serialized and shipped. Maybe I am missing something. Thanks again, Arijit Date: Thu, 8 Oct 2015 10:26:55 -0700 Subject: Re: Understanding code/closure shipment to Spark workers‏ From: gatorsm...@gmail.com To: arij...@live.com CC: dev@spark.apache.org Hi, Arijit, The code flow

Re: Understanding code/closure shipment to Spark workers‏

Hi, Arijit, The code flow of spark-submit is simple. Enter the main function of SparkSubmit.scala --> case SparkSubmitAction.SUBMIT => submit(appArgs) --> doRunMain() in function submit() in the same file --> runMain(childArgs,...) in the same file --> mainMethod.invoke(null, chil

Understanding code/closure shipment to Spark workers‏

Hi, I want to understand the code flow starting from the Spark jar that I submit through spark-submit, how does Spark identify and extract the closures, clean and serialize them and ship them to workers to execute as tasks. Can someone point me to any documentation or a pointer to the source