Re: Launching multiple spark jobs within a main spark job.

2016-12-24 Thread Naveen
sage in context: http://apache-spark- > developers-list.1001551.n3.nabble.com/Launching-multiple- > spark-jobs-within-a-main-spark-job-tp20311p20327.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > - > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Liang-Chi Hsieh
.nabble.com/Launching-multiple-spark-jobs-within-a-main-spark-job-tp20311p20327.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

RE: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread David Hodeffi
...@spark.apache.org Subject: Launching multiple spark jobs within a main spark job. Hi Team, Is it ok to spawn multiple spark jobs within a main spark job, my main spark job's driver which was launched on yarn cluster, will do some preprocessing and based on it, it needs to launch multilple spark jobs

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Sebastian Piu
; > *From:* Naveen [mailto:hadoopst...@gmail.com] > *Sent:* Wednesday, December 21, 2016 9:18 AM > *To:* dev@spark.apache.org; u...@spark.apache.org > *Subject:* Launching multiple spark jobs within a main spark job. > > > > Hi Team, > > > > Is it ok to spawn multip

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Naveen
gt; Similarly, future2 to futureN. >>> >>> future1.onComplete{...} >>> } >>> >>> }// end of mainsparkjob >>> -- >>> >>> >>> [image: Inline image 1] >>> >>> On Wed, Dec 21, 2016 at 3:1

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Naveen
yway, If you run spark applicaction you would have multiple jobs, which >> makes sense that it is not a problem. >> >> >> >> Thanks David. >> >> >> >> *From:* Naveen [mailto:hadoopst...@gmail.com] >> *Sent:* Wednesday, December 21, 2016 9:18

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Liang-Chi Hsieh
/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Launching-multiple-spark-jobs-within-a-main-spark-job-tp20311p20315.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Naveen
rg; u...@spark.apache.org > *Subject:* Launching multiple spark jobs within a main spark job. > > > > Hi Team, > > > > Is it ok to spawn multiple spark jobs within a main spark job, my main > spark job's driver which was launched on yarn cluster, will do some > preproc

Re: Launching multiple spark jobs within a main spark job.

2016-12-21 Thread Liang-Chi Hsieh
this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Launching-multiple-spark-jobs-within-a-main-spark-job-tp20311p20312.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com

Launching multiple spark jobs within a main spark job.

2016-12-20 Thread Naveen
Hi Team, Is it ok to spawn multiple spark jobs within a main spark job, my main spark job's driver which was launched on yarn cluster, will do some preprocessing and based on it, it needs to launch multilple spark jobs on yarn cluster. Not sure if this right pattern. Please share your thoughts.