I started looking into LIVY-414 [1] with the exact intention of multiple Spark versions with multiple Spark homes. Might see if that would help here?
[1] https://issues.apache.org/jira/browse/LIVY-414 Kevin Risden On Thu, Mar 8, 2018 at 5:41 PM, Meisam Fathi <[email protected]> wrote: > > 2) some users could have spark 1 and spark 2, and they have customized > with > sym link to switch between version easily and to be sure what versions they > was running, instead to inspect SPARK_HOME variable and PATH . > > I am under the impression that spark-submit needs SPARK_HOME to run.Can we > call spark-submit without setting SPARK_HOME? > > Thanks, > Meisam > > On Thu, Mar 8, 2018 at 3:26 AM Matteo Durighetto <[email protected]> > wrote: > > > Hello Livy Developers, > > I am new in livy community, and I start to understand how it > > works under the hood. I see in the source code of livy > > ( server/src/main/scala/org/apache/livy/LivyConf.scala ) > > that the command to submit to spark cluster a request is "spark-submit" > in > > function sparkSubmit : > > > > /** Return the path to the spark-submit executable. */ > > def sparkSubmit(): String = { > > sparkHome().map { _ + File.separator + "bin" + File.separator + > > "spark-submit" }.get > > } > > > > I think a good idea is to use a variable instead of a literal , for some > > reasons : > > 1) some distributions of hadoop/spark use sym link to spark-submit like > > spark2-submit and so on > > 2) some users could have spark 1 and spark 2, and they have customized > with > > sym link to switch between version easily and to be sure what versions > they > > was running, instead to inspect SPARK_HOME variable and PATH . > > 3) in the future it's possible they will change the name of the command > > > > What do you think about ? If you are agree, Could I open a feature > request > > about this on Jira > > > > https://issues.apache.org/jira/projects/LIVY/issues/ > LIVY-449?filter=allopenissues > > ? > > > > I try to wrote a little patch based on the branch on github of apache > livy, > > I read on https://livy.incubator.apache.org/community/ that to propose a > > patch I have to include tests. > > I try to find a document about hoe to do tests correctly, but I don't > find > > it. > > > > > > Could you help me? > > > > Kind Regards > > > > Matteo Durighetto > > >
