[
https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15590968#comment-15590968
]
Peter Cseh commented on OOZIE-2712:
-----------------------------------
The user would probably define these in one way or the other and won't use both.
The problem emerges when Oozie uses some of these things to set log4j settings
or executor classpath and users can overwrite those internal things without
even knowing it. Even after the fix in OOZIE-2705 if the user passes in
--driver-java-options in <spark-opts> the the log4j settings will be
overwritten by Spark and child IDs will be lost.
Maybe we should separate the user-defined option list parsing from adding our
own stuff in there and add a post-process step that add Oozie-specific things.
I'll probably won't have the time to mess around with this for a couple weeks,
if anybody has some ideas, feel free to grab this!
> Merge Spark opts with two possible options
> -------------------------------------------
>
> Key: OOZIE-2712
> URL: https://issues.apache.org/jira/browse/OOZIE-2712
> Project: Oozie
> Issue Type: Bug
> Reporter: Peter Cseh
> Assignee: Peter Cseh
> Attachments: OOZIE-2712.001.patch
>
>
> Issues like OOZIE-2658 can happen with more properties.
> || --conf || overwritten by ||
> |spark.driver.memory | --driver-memory |
> |spark.driver.extraJavaOptions | --driver-java-options|
> |spark.driver.extraLibraryPath|--driver-library-path |
> |spark.executor.instances|--num-executors|
> |spark.driver.extraClassPath| --driver-class-path|
> We should make sure that one is converted to the other (the current pattern
> is to use --conf versions of these) so we don't run into issues where the
> log4j properties are overwritten. (OOZIE-2705 can be affected by this)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
