Virgile Devaux created SPARK-4299:
-------------------------------------
Summary: In spark-submit, the driver-memory value is used for the
SPARK_SUBMIT_DRIVER_MEMORY value
Key: SPARK-4299
URL: https://issues.apache.org/jira/browse/SPARK-4299
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Reporter: Virgile Devaux
In the spark-submit script, the lines below:
elif [ "$1" = "--driver-memory" ]; then
export SPARK_SUBMIT_DRIVER_MEMORY=$2
are wrong: spark-submit is not the process that will handle the driver when
you're in yarn-cluster mode. So, when I lanch spark-submit on a light server
with only 2Gb of memory and want to allocate 4gb of memory to the driver (that
will run in the ressource manager on a big fat yarn server with, say, 64Gb of
RAM) spark submit fails with a OutOfMemory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
