[
https://issues.apache.org/jira/browse/SPARK-1807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14057511#comment-14057511
]
jay vyas commented on SPARK-1807:
---------------------------------
Question: Are you saying the {{SPARK_EXECUTOR_URI}} should be a URI that points
to a script? Or that we can replace the URI with a simple path to an
executable script.
Proposed Alternative: Take this extension to its logical extreme, and actually
deprecate the SPARK_EXECUTOR_URI parameter, and instead have a parameter
{{SPARK_EXECUTOR_CLASS}}, which is invoked with any number of args, one of
which could be SPARK_EXECUTOR_URI.
Something like this:
{noformat}
val c = Class.forName(conf.get("SPARK_EXECUTOR_CLASS"))
val m = c.getMethod("startSpark",classOf[SparkExecutor])
m.invoke(null,conf.get("SPARK_EXECUTOR_URI"))
{noformat}
The advantage of using a class, rather than a shell script, for this - is that
we are gauranteed off the bat that the class is providing a contract - to start
spark - rather than just a one off script which could do any number of things.
Also it means that various classes for this can be maintained and tested over
time as first class "spark environment" providers.
> Modify SPARK_EXECUTOR_URI to allow for script execution in Mesos.
> -----------------------------------------------------------------
>
> Key: SPARK-1807
> URL: https://issues.apache.org/jira/browse/SPARK-1807
> Project: Spark
> Issue Type: Improvement
> Components: Mesos
> Affects Versions: 0.9.0
> Reporter: Timothy St. Clair
>
> Modify Mesos Scheduler integration to allow SPARK_EXECUTOR_URI to be an
> executable script. This allows admins to launch spark in any fashion they
> desire, vs. just tarball fetching + implied context.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
