I am not familiar of any problem with that. Anyway, If you run spark applicaction you would have multiple jobs, which makes sense that it is not a problem.
Thanks David. From: Naveen [mailto:[email protected]] Sent: Wednesday, December 21, 2016 9:18 AM To: [email protected]; [email protected] Subject: Launching multiple spark jobs within a main spark job. Hi Team, Is it ok to spawn multiple spark jobs within a main spark job, my main spark job's driver which was launched on yarn cluster, will do some preprocessing and based on it, it needs to launch multilple spark jobs on yarn cluster. Not sure if this right pattern. Please share your thoughts. Sample code i ve is as below for better understanding.. --------------------- Object Mainsparkjob { main(...){ val sc=new SparkContext(..) Fetch from hive..using hivecontext Fetch from hbase //spawning multiple Futures.. Val future1=Future{ Val sparkjob= SparkLauncher(...).launch; spark.waitFor } Similarly, future2 to futureN. future1.onComplete{...} } }// end of mainsparkjob ---------------------- Confidentiality: This communication and any attachments are intended for the above-named persons only and may be confidential and/or legally privileged. Any opinions expressed in this communication are not necessarily those of NICE Actimize. If this communication has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender by e-mail immediately. Monitoring: NICE Actimize may monitor incoming and outgoing e-mails. Viruses: Although we have taken steps toward ensuring that this e-mail and attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
