Renato Falchi Brandão created SPARK-15067:
---------------------------------------------
Summary: YARN executors are launched with fixed perm gen size
Key: SPARK-15067
URL: https://issues.apache.org/jira/browse/SPARK-15067
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.6.1, 1.6.0
Reporter: Renato Falchi Brandão
It is impossible to change the executors max perm gen size using the property
"spark.executor.extraJavaOptions" when you are running on YARN.
When the JVM option "-XX:MaxPermSize" is set through the property
"spark.executor.extraJavaOptions", Spark put it properly in the shell command
that will start the JVM container but, in the ending of command, it sets again
this option using a fixed value of 256m, as you can see in the log I've
extracted:
2016-04-30 17:20:12 INFO ExecutorRunnable:58 -
===============================================================================
YARN executor launch context:
env:
CLASSPATH ->
{{PWD}}<CPS>{{PWD}}/__spark__.jar<CPS>$HADOOP_CONF_DIR<CPS>/usr/hdp/current/hadoop-client/*<CPS>/usr/hdp/current/hadoop-client/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/*:/usr/hdp/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/*:/usr/hdp/mr-framework/hadoop/share/hadoop/common/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/*:/usr/hdp/mr-framework/hadoop/share/hadoop/yarn/lib/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/*:/usr/hdp/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/current/hadoop/lib/hadoop-lzo-0.6.0.jar:/etc/hadoop/conf/secure
SPARK_LOG_URL_STDERR ->
http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stderr?start=-4096
SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1456962126505_329993
SPARK_YARN_CACHE_FILES_FILE_SIZES -> 191719054,166
SPARK_USER -> h_loadbd
SPARK_YARN_CACHE_FILES_VISIBILITIES -> PUBLIC,PUBLIC
SPARK_YARN_MODE -> true
SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1459806496093,1459808508343
SPARK_LOG_URL_STDOUT ->
http://xxxxx0668sl.xxxxx.br:8042/node/containerlogs/container_1456962126505_329993_01_000002/h_loadbd/stdout?start=-4096
SPARK_YARN_CACHE_FILES ->
hdfs://xxxxx/user/datalab/hdp/spark/lib/spark-assembly-1.6.0.2.3.4.1-10-hadoop2.7.1.2.3.4.1-10.jar#__spark__.jar,hdfs://tlvcluster/user/datalab/hdp/spark/conf/hive-site.xml#hive-site.xml
command:
{{JAVA_HOME}}/bin/java -server -XX:OnOutOfMemoryError='kill %p' -Xms6144m
-Xmx6144m '-XX:+PrintGCDetails' '-XX:MaxPermSize=1024M'
'-XX:+PrintGCTimeStamps' -Djava.io.tmpdir={{PWD}}/tmp
'-Dspark.akka.timeout=300000' '-Dspark.driver.port=62875'
'-Dspark.rpc.askTimeout=300000' '-Dspark.rpc.lookupTimeout=300000'
-Dspark.yarn.app.container.log.dir=<LOG_DIR> -XX:MaxPermSize=256m
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://[email protected]:62875 --executor-id 1 --hostname
xxxxx0668sl.xxxxx.br --cores 1 --app-id application_1456962126505_329993
--user-class-path file:$PWD/__app__.jar 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
Analyzing the code is possible to see that all the options set in the property
"spark.executor.extraJavaOptions" are enclosed, one by one, in single quotes
(ExecutorRunnable.scala:151) before the launcher take the decision if a default
value has to be provided or not for the option "-XX:MaxPermSize"
(ExecutorRunnable.scala:202).
This decision is taken examining all the options set and looking for a string
starting with the value "-XX:MaxPermSize" (CommandBuilderUtils.java:328). If
that value is not found, the default value is set.
A string option starting without single quote will never be found, then, a
default value will always be provided.
A possible solution is change the source code of CommandBuilderUtils.java in
the line 328:
From-> if (arg.startsWith("-XX:MaxPermSize="))
To-> if (arg.indexOf("-XX:MaxPermSize=") > -1)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
