[
https://issues.apache.org/jira/browse/OOZIE-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16251571#comment-16251571
]
Rohini Palaniswamy commented on OOZIE-3122:
-------------------------------------------
Just started a discussion on dev mailing list on doing a Apache 4.3.1 release.
Hopefully we will have a release in another month.
> Oozie spark action unable to handle some options in spark-defaults.conf
> -----------------------------------------------------------------------
>
> Key: OOZIE-3122
> URL: https://issues.apache.org/jira/browse/OOZIE-3122
> Project: Oozie
> Issue Type: Bug
> Components: action
> Affects Versions: 4.3.0
> Environment: Operating on EMR cluster (5.4.0) which includes Oozie
> 4.3.0
> Reporter: Simon Whitelaw
> Labels: usability
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> When using the setting
> oozie.service.SparkConfigurationService.spark.configurations *=etc/spark/conf
> to specify a spark-defaults.conf file for spark to use, a few options are not
> handled properly by the oozie spark action and cause the job to fail. These
> include the following:
> * spark.driver.extraClassPath - causes a blank --conf tag to be sent to the
> spark submit and the spark submit will fail
> * spark.executor.extraClassPath - causes a blank --conf tag to be sent to the
> spark submit and the spark submit will fail
> * spark.executor.extraJavaOptions with multiple options set (e.g.
> spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC
> -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70
> -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p') - causes
> all options except for the first to be passed as singletons rather than
> extraJavaOptions
> * spark.driver.extraJavaOptions with multiple option set - causes all options
> except for the first to be passed as singletons rather than extraJavaOptions
> * Within the spark.driver.extraJavaOptions and
> spark.executor.extraJavaOptions, the option -XX:OnOutOfMemoryError='kill -9
> %p' is not parsed correctly even from the spark-opts tag and requires
> additional double quotations within the single quote marks: '"kill -9 %p"'
> These issues were noticed running the oozie spark action on an amazon EMR
> cluster. Workaround involves sending the options as part of the spark-opts
> tag, but is not ideal because in EMR, the spark-defaults.conf file is
> autogenerated for the specifications of the cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
