Kalyana Chakravarthy Kadiyala created SPARK-4311:
----------------------------------------------------
Summary: ContainerLauncher setting up executor -- invalid Xms
settings (-Xms0m -Xmx0m)
Key: SPARK-4311
URL: https://issues.apache.org/jira/browse/SPARK-4311
Project: Spark
Issue Type: Question
Components: YARN
Affects Versions: 1.1.0
Reporter: Kalyana Chakravarthy Kadiyala
<spark_home>/conf/spark-defaults.conf entry for executor extra options:
spark.executor.extraJavaOptions -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode -XX:+UseCompressedOops
Sample from Container logs...(note - masked the nodes for privacy reasons;
driver runs on xxxx1 node and executor is being spawned on xxxx2 node in the 3
node YARN cluster)
- Setting up executor with commands: List($JAVA_HOME/bin/java, -server,
-XX:OnOutOfMemoryError='kill %p', -Xms0m -Xmx0m , -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseCompressedOops,
-Djava.io.tmpdir=$PWD/tmp, '-Dspark.authenticate=false',
'-Dspark.akka.timeout=100', '-Dspark.akka.frameSize=10',
'-Dspark.akka.heartbeat.pauses=600',
'-Dspark.akka.failure-detector.threshold=300',
'-Dspark.akka.heartbeat.interval=1000', '-Dspark.akka.threads=4',
-Dspark.yarn.app.container.log.dir=<LOG_DIR>,
org.apache.spark.executor.CoarseGrainedExecutorBackend,
akka.tcp://[email protected]:49760/user/CoarseGrainedScheduler, 6,
xxxx1.xxxxxxx.xxx, 1, application_1415440760385_0012, 1>, <LOG_DIR>/stdout, 2>,
<LOG_DIR>/stderr)
2014-11-08 17:19:07,201 INFO [ContainerLauncher #3] yarn.ExecutorRunnable
(Logging.scala:logInfo(59)) - Setting up executor with commands:
List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms0m
-Xmx0m , -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode
-XX:+UseCompressedOops, -Djava.io.tmpdir=$PWD/tmp,
'-Dspark.authenticate=false', '-Dspark.akka.timeout=100',
'-Dspark.akka.frameSize=10', '-Dspark.akka.heartbeat.pauses=600',
'-Dspark.akka.failure-detector.threshold=300',
'-Dspark.akka.heartbeat.interval=1000', '-Dspark.akka.threads=4',
-Dspark.yarn.app.container.log.dir=<LOG_DIR>,
org.apache.spark.executor.CoarseGrainedExecutorBackend,
akka.tcp://[email protected]:49760/user/CoarseGrainedScheduler, 4,
xxxx2.xxxxx.xxx, 1, application_1415440760385_0012, 1>, <LOG_DIR>/stdout, 2>,
<LOG_DIR>/stderr)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
