Andras Piros created OOZIE-2923:
-----------------------------------
Summary: Improve Spark options parsing
Key: OOZIE-2923
URL: https://issues.apache.org/jira/browse/OOZIE-2923
Project: Oozie
Issue Type: Improvement
Components: action
Affects Versions: 4.3.0
Reporter: Andras Piros
Assignee: Andras Piros
There are two issues w/ Spark action's argument parsing within {{SparkMain}}
h5. Driver and executor extra classpaths: equals sign used
When the user specifies {{\-\-conf spark.executor.extraClassPath=XYZ}} or
{{\-\-conf spark.driver.extraClassPath=ABC}}, the option {{\-\-conf}} will be
added to {{sparkArgs}}. Then when the code tries to evaluate
{{spark.executor.extraClassPath=XYZ}}, it uses special logic and set
{{addToSparkArgs = false}}. As a result there will be a extra {{\-\-conf}} in
the {{sparkArgs}} eventually.
For example: {{\-\-conf spark.executor.extraClassPath=XYZ \-\-conf
otherProperty=ABC}} will become {{\-\-conf \-\-conf otherProperty=ABC}}, which
will cause spark job submit failure later.
We might need to remove one prior {{\-\-conf}} in {{sparkArgs}} if the current
evaluated {{opt}} is {{EXECUTOR_CLASSPATH}} or {{DRIVER_CLASSPATH}}.
h5. User provided files and archives: equals sign used
For the following workflow XML snippet:
{code:xml}
<spark-opts>--files=${nameNode}/home/share/hive-site.xml --num-executors 4
--executor-memory 7g --driver-memory 7g</spark-opts>
{code}
the {{\-\-files=$\{nameNode\}/home/share/hive-site.xml}} {{opt}} will be placed
into {{sparkArgs}} in previous Oozie version without any modification, because
we don't have special handling for {{\-\-files}} {{opt}}.
If the user specifies {{\-\-files=$\{nameNode\}/home/share/hive-site.xml
--num-executor 4}}, then {{SparkMain}} code treats {{\-\-num-executor}} as a
file path / name. That caused the issue as I described in my previous comment.
We might need to change the handling logic for {{FILES_OPTION}} and
{{ARCHIVES_OPTION}} to be the same to {{DRIVER_CLASSPATH_OPTION}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
