Sure. Should I create a Jira as well? I saw there's already a broader ticket regarding the ambiguous use of SPARK_HOME [1] (cc: Patrick as owner of that ticket)
I don't know if it would be more relevant to remove the use of SPARK_HOME when using mesos and have the assembly as the only way forward, or whether that's a too radical change that might break some existing systems. >From a real-world ops perspective, the assembly should be the way to go. I don't see installing and configuring Spark distros on a mesos master as a way to have the mesos executor in place. -kr, Gerard. [1] https://issues.apache.org/jira/browse/SPARK-1110 On Thu, May 22, 2014 at 6:19 AM, Andrew Ash <and...@andrewash.com> wrote: > Hi Gerard, > > I agree that your second option seems preferred. You shouldn't have to > specify a SPARK_HOME if the executor is going to use the spark.executor.uri > instead. Can you send in a pull request that includes your proposed > changes? > > Andrew > > > On Wed, May 21, 2014 at 10:19 AM, Gerard Maas <gerard.m...@gmail.com> > wrote: > > > Spark dev's, > > > > I was looking into a question asked on the user list where a > > ClassNotFoundException was thrown when running a job on Mesos. Curious > > issue with serialization on Mesos: more details here [1]: > > > > When trying to run that simple example on my Mesos installation, I faced > > another issue: I got an error that "SPARK_HOME" was not set. I found that > > curious b/c a local spark installation should not be required to run a > job > > on Mesos. All that's needed is the executor package, being the > > assembly.tar.gz on a reachable location (HDFS/S3/HTTP). > > > > I went looking into the code and indeed there's a check on SPARK_HOME [2] > > regardless of the presence of the assembly but it's actually only used if > > the assembly is not provided (which is a kind-of best-effort recovery > > strategy). > > > > Current flow: > > > > if (!SPARK_HOME) fail("No SPARK_HOME") > > else if (assembly) { use assembly) } > > else { try use SPARK_HOME to build spark_executor } > > > > Should be: > > sparkExecutor = if (assembly) {assembly} > > else if (SPARK_HOME) {try use SPARK_HOME to build > > spark_executor} > > else { fail("No executor found. Please provide > > spark.executor.uri (preferred) or spark.home") > > > > What do you think? > > > > -kr, Gerard. > > > > > > [1] > > > > > http://apache-spark-user-list.1001560.n3.nabble.com/ClassNotFoundException-with-Spark-Mesos-spark-shell-works-fine-td6165.html > > > > [2] > > > > > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala#L89 > > >
