Hi Preeze, > Is there any designed way that the client connects back to the driver (still running in YARN) for collecting results at a later stage?
No, there is not support built into Spark for this. For this to happen seamlessly the driver will have to start a server (pull model) or send the results to some other server once the jobs complete (push model), both of which add complexity to the driver. Alternatively, you can just poll on the output files that your application produces; e.g. you can have your driver write the results of a count to a file and poll on that file. Something like that. -Andrew 2015-01-19 5:59 GMT-08:00 Romi Kuntsman <r...@totango.com>: > "in yarn-client mode it only controls the environment of the executor > launcher" > > So you either use yarn-client mode, and then your app keeps running and > controlling the process > Or you use yarn-cluster mode, and then you send a jar to YARN, and that jar > should have code to report the result back to you > > *Romi Kuntsman*, *Big Data Engineer* > http://www.totango.com > > On Thu, Jan 15, 2015 at 1:52 PM, preeze <etan...@gmail.com> wrote: > > > From the official spark documentation > > (http://spark.apache.org/docs/1.2.0/running-on-yarn.html): > > > > "In yarn-cluster mode, the Spark driver runs inside an application master > > process which is managed by YARN on the cluster, and the client can go > away > > after initiating the application." > > > > Is there any designed way that the client connects back to the driver > > (still > > running in YARN) for collecting results at a later stage? > > > > > > > > -- > > View this message in context: > > > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-client-reconnect-to-driver-in-yarn-cluster-deployment-mode-tp10122.html > > Sent from the Apache Spark Developers List mailing list archive at > > Nabble.com. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > For additional commands, e-mail: dev-h...@spark.apache.org > > > > >
