Hi, As you launch multiple Spark jobs through `SparkLauncher`, I think it actually works like you run multiple Spark applications with `spark-submit`.
By default each application will try to use all available nodes. If your purpose is to share cluster resources across those Spark jobs/applications, you may need to set some configs properly. Please check out: http://spark.apache.org/docs/latest/job-scheduling.html#scheduling-across-applications As you said you launch the main Spark job on yarn cluster, if you are using cluster mode, actually you will submit those Spark jobs/applications on the node which the driver runs. It looks weird. Looks like you try to fetch some data first and do some jobs on the data. Can't you just do those jobs in the main driver as Spark actions with its API? ----- Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Launching-multiple-spark-jobs-within-a-main-spark-job-tp20311p20312.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
