Hi Marcelo, Thanks for your comments. I'd like to know your thoughts a different approach.
Livy can directly call org.apache.spark.deploy.SparkSubmit.main() with proper arguments, which is what spark-submit ends up doing. I have at least three problems with this approach: 1. It is a hack. 2. Now that you pointed out, I see it restricts Livy to a single version of Spark. 3. It becomes tricky to separate the output of different applications because all outputs will go to the stdout/stderr of Livy process. I am not sure how to fix 2. For fixing 3, I noticed that Scala has a Console.withOut method that can be used to set "the default output stream for the duration of execution of one thunk". I never used it though. Thanks, Meisam On Wed, Aug 16, 2017 at 12:39 PM Marcelo Vanzin <[email protected]> wrote: > On Wed, Aug 16, 2017 at 12:25 PM, Meisam Fathi <[email protected]> > wrote: > > Out of these two problems, calling one spark-submit per application is > the > > biggest problem, but it can be solved by adding more Livy servers. > > Something like SPARK-11035 could also help here. Although the > implementation of that particular solution would probably not be a > very good fit for Livy in its current state, since it would probably > mean restricting the Livy server to a single version of Spark. > > -- > Marcelo >
