Simon Whitelaw created OOZIE-3122:
-------------------------------------
Summary: Oozie spark action unable to handle some options in
spark-defaults.conf
Key: OOZIE-3122
URL: https://issues.apache.org/jira/browse/OOZIE-3122
Project: Oozie
Issue Type: Bug
Components: action
Affects Versions: 4.3.0
Environment: Operating on EMR cluster (5.4.0) which includes Oozie
4.3.0
Reporter: Simon Whitelaw
When using the setting
oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf
to specify a spark-defaults.conf file for spark to use, a few options are not
handled properly by the oozie spark action and cause the job to fail. These
include the following:
* spark.driver.extraClassPath - causes a blank --conf tag to be sent to the
spark submit and the spark submit will fail
* spark.executor.extraClassPath - causes a blank --conf tag to be sent to the
spark submit and the spark submit will fail
* spark.executor.extraJavaOptions with multiple options set (e.g.
spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70
-XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes all
options except for the first to be passed as singletons rather than
extraJavaOptions
* spark.driver.extraJavaOptions with multiple option set - causes all options
except for the first to be passed as singletons rather than extraJavaOptions
* Within the spark.driver.extraJavaOptions and spark.executor.extraJavaOptions,
the option -XX:OnOutOfMemoryError='kill -9 %p' is not parsed correctly even
from the spark-opts tag and requires additional double quotations within the
single quote marks: '"kill -9 %p"'
These issues were noticed running the oozie spark action on an amazon EMR
cluster. Workaround involves sending the options as part of the spark-opts tag,
but is not ideal because in EMR, the spark-defaults.conf file is autogenerated
for the specifications of the cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
