[
https://issues.apache.org/jira/browse/SPARK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094189#comment-15094189
]
John Vines edited comment on SPARK-12650 at 1/12/16 10:52 PM:
--------------------------------------------------------------
So I ran it and got this message
{code}SPARK_JAVA_OPTS was detected (set to '-Xmx512M').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with conf/spark-defaults.conf to set defaults for an
application
- ./spark-submit with --driver-java-options to set -X options for a driver
- spark.executor.extraJavaOptions to set -X options for executors
- SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or
worker)
{code}
but that was just a warning (so small complaint if this is the proper
solution), but it did properly cap the vmem use.
EDIT: it appears this value applies to ALL processes though. So when I did 512
it was fine, but when I did 256 exectutors failed since I have their mem set to
512 so I got the containers failing to start due to `Incompatible minimum and
maximum heap sizes specified`
was (Author: vines):
So I ran it and got this message
{code}SPARK_JAVA_OPTS was detected (set to '-Xmx512M').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with conf/spark-defaults.conf to set defaults for an
application
- ./spark-submit with --driver-java-options to set -X options for a driver
- spark.executor.extraJavaOptions to set -X options for executors
- SPARK_DAEMON_JAVA_OPTS to set java options for standalone daemons (master or
worker)
{code}
but that was just a warning (so small complaint if this is the proper
solution), but it did properly cap the vmem use.
> No means to specify Xmx settings for SparkSubmit in yarn-cluster mode
> ---------------------------------------------------------------------
>
> Key: SPARK-12650
> URL: https://issues.apache.org/jira/browse/SPARK-12650
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.5.2
> Environment: Hadoop 2.6.0
> Reporter: John Vines
>
> Background-
> I have an app master designed to do some work and then launch a spark job.
> Issue-
> If I use yarn-cluster, then the SparkSubmit does not Xmx itself at all,
> leading to the jvm taking a default heap which is relatively large. This
> causes a large amount of vmem to be taken, so that it is killed by yarn. This
> can be worked around by disabling Yarn's vmem check, but that is a hack.
> If I run it in yarn-client mode, it's fine as long as my container has enough
> space for the driver, which is manageable. But I feel that the utter lack of
> Xmx settings for what I believe is a very small jvm is a problem.
> I believe this was introduced with the fix for SPARK-3884
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
