nirav patel created OOZIE-2526:
----------------------------------
Summary: Spark action have no way to specify spark driver jvm
settings for yarn-client mode
Key: OOZIE-2526
URL: https://issues.apache.org/jira/browse/OOZIE-2526
Project: Oozie
Issue Type: Bug
Reporter: nirav patel
Currently oozie spark action has spark-opts elements which are basically passed
on to `org.apache.spark.deploy.SparkSubmit` as spark configuration. In
yarn-client mode this is too late and driver JVM is infact started when calling
`org.apache.oozie.action.hadoop.SparkMain` class. Because oozie bypasses
spark-submit.sh script and directly calls org.apache.spark.deploy.SparkSubmit.
Hence even user specify --driver-memory=3g it has no effect on running jvm as
it's already too late. I think oozie:launcher task which is a parent map-reduce
job itself should launch its map task (spark driver) with some user specified
JVM arguments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
