[
https://issues.apache.org/jira/browse/SPARK-4408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-4408.
------------------------------
Resolution: Not A Problem
Yes, I think you've accurately described how it works, but I think that's by
design. In client mode the launching JVM is already the driver JVM.
> Behavior difference between spark-submit conf vs cmd line args
> --------------------------------------------------------------
>
> Key: SPARK-4408
> URL: https://issues.apache.org/jira/browse/SPARK-4408
> Project: Spark
> Issue Type: Bug
> Components: Deploy, Documentation
> Affects Versions: 1.1.0, 1.2.0
> Reporter: Pedro Rodriguez
> Priority: Minor
>
> There seems to be a difference between the behavior of bin/spark-submit with
> using command line arguments vs configuration file. It looks like either a
> bug or at least where documentation could be clearer about this difference.
> Steps to Replicate:
> 1. Submit a job with a command similar to:
> bin/spark-submit --class nipslda.NipsLda --master local --conf
> spark.executor.memory=2g --conf spark.driver.memory=2g --verbose
> ~/Code/nips-lda/target/scala-2.10/nips-lda-assembly-0.1.jar
> 2. Navigate to SparkUI.
> 3. Environment tab lists driver and executor memory correctly.
> 4. Executor tab shows memory as ~260MB (my case) or default JVM limit
> 5. Write memory arguments to conf/spark-defaults.conf
> 6. Run same command without memory arguments
> 7. SparkUI executor tab correctly shows memory setting
> Looking at spark-submit, it includes this passage in the comments:
> # For client mode, the driver will be launched in the same JVM that launches
> # SparkSubmit, so we may need to read the properties file for any extra class
> # paths, library paths, java options and memory early on. Otherwise, it will
> # be too late by the time the driver JVM has started.
> Based on this, it seems that JVM parameters for spark-submit are used for the
> job itself when in client mode. Effectively, it makes it impossible to use
> the command line argument setting method to change JVM parameters since the
> JVM is already launched.
> This seems like unexpected/undesirable behavior which could be fixed or docs
> could be changed to better reflect how this works.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
