Bit of a race condition here it seems. Patrick made a few changes yesterday
around the same time as I did (in ClientBase.scala):
for ((k, v) <- sys.props.filterKeys(_.startsWith("spark")))
{ JAVA_OPTS += "-D" + k + "=" + "\\\"" + v + "\\\"" }
This would allow JAVA_OPTS to be passed on the command line to the
ApplicationMaster, and accomplishes the same things as creation of a new
command line flag --spark-java-opts.
Mridul, the use of SPARK_JAVA_OPTS has been intentionally suppressed.
On Wed, Apr 23, 2014 at 10:54 AM, Mridul Muralidharan <mri...@gmail.com>wrote:
> Sorry, I misread - I meant SPARK_JAVA_OPTS - not JAVA_OPTS.
> See here : https://issues.apache.org/jira/browse/SPARK-1588
>
> Regards,
> Mridul
>
> On Wed, Apr 23, 2014 at 6:37 PM, Mridul Muralidharan <mri...@gmail.com>
> wrote:
> > This breaks all existing jobs which are not using spark-submit.
> > The consensus was not to break compatibility unless there was an
> overriding
> > reason to do so
> >
> > On Apr 23, 2014 6:32 PM, "Thomas Graves (JIRA)" <j...@apache.org> wrote:
> >>
> >>
> >> [
> >>
> https://issues.apache.org/jira/browse/SPARK-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978164#comment-13978164
> >> ]
> >>
> >> Thomas Graves commented on SPARK-1576:
> >> --------------------------------------
> >>
> >> Is this meant for the driver or the executors? The spark-submit script
> >> has a command line option for the driver: --driver-java-options.
> >> I believe the intent of https://github.com/apache/spark/pull/299 was to
> >> not expose SPARK_JAVA_OPTS to the user anymore.
> >>
> >> > Passing of JAVA_OPTS to YARN on command line
> >> > --------------------------------------------
> >> >
> >> > Key: SPARK-1576
> >> > URL: https://issues.apache.org/jira/browse/SPARK-1576
> >> > Project: Spark
> >> > Issue Type: Improvement
> >> > Affects Versions: 0.9.0, 1.0.0, 0.9.1
> >> > Reporter: Nishkam Ravi
> >> > Fix For: 0.9.0, 1.0.0, 0.9.1
> >> >
> >> > Attachments: SPARK-1576.patch
> >> >
> >> >
> >> > JAVA_OPTS can be passed by using either env variables (i.e.,
> >> > SPARK_JAVA_OPTS) or as config vars (after Patrick's recent change).
> It would
> >> > be good to allow the user to pass them on command line as well to
> restrict
> >> > scope to single application invocation.
> >>
> >>
> >>
> >> --
> >> This message was sent by Atlassian JIRA
> >> (v6.2#6252)
>
