[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[
https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Cseh updated OOZIE-2712:
--
Priority: Critical (was: Major)
> Merge Spark opts with two possible options
> ---
>
> Key: OOZIE-2712
> URL: https://issues.apache.org/jira/browse/OOZIE-2712
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
>Priority: Critical
> Attachments: OOZIE-2712.001.patch
>
>
> Issues like OOZIE-2658 can happen with more properties.
> || {{--conf}} || overwritten by ||
> | {{spark.driver.memory}} | {{--driver-memory}} |
> | {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
> | {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
> | {{spark.executor.instances}} | {{--num-executors}} |
> | {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
> | {{spark.yarn.dist.files}} || {{--files}}|
> We should make sure that one is converted to the other (the current pattern
> is to use {{--conf}} versions of these) so we don't run into issues where the
> log4j properties are overwritten. (OOZIE-2705 can be affected by this)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[
https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Cseh updated OOZIE-2712:
--
Description:
Issues like OOZIE-2658 can happen with more properties.
|| {{--conf}} || overwritten by ||
| {{spark.driver.memory}} | {{--driver-memory}} |
| {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
| {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
| {{spark.executor.instances}} | {{--num-executors}} |
| {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
| {{spark.yarn.dist.files}} || {{--files}}|
We should make sure that one is converted to the other (the current pattern is
to use {{--conf}} versions of these) so we don't run into issues where the
log4j properties are overwritten. (OOZIE-2705 can be affected by this)
was:
Issues like OOZIE-2658 can happen with more properties.
|| {{--conf}} || overwritten by ||
| {{spark.driver.memory}} | {{--driver-memory}} |
| {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
| {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
| {{spark.executor.instances}} | {{--num-executors}} |
| {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
We should make sure that one is converted to the other (the current pattern is
to use {{--conf}} versions of these) so we don't run into issues where the
log4j properties are overwritten. (OOZIE-2705 can be affected by this)
> Merge Spark opts with two possible options
> ---
>
> Key: OOZIE-2712
> URL: https://issues.apache.org/jira/browse/OOZIE-2712
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2712.001.patch
>
>
> Issues like OOZIE-2658 can happen with more properties.
> || {{--conf}} || overwritten by ||
> | {{spark.driver.memory}} | {{--driver-memory}} |
> | {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
> | {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
> | {{spark.executor.instances}} | {{--num-executors}} |
> | {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
> | {{spark.yarn.dist.files}} || {{--files}}|
> We should make sure that one is converted to the other (the current pattern
> is to use {{--conf}} versions of these) so we don't run into issues where the
> log4j properties are overwritten. (OOZIE-2705 can be affected by this)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[
https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Piros updated OOZIE-2712:
Description:
Issues like OOZIE-2658 can happen with more properties.
|| {{--conf}} || overwritten by ||
| {{spark.driver.memory}} | {{--driver-memory}} |
| {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
| {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
| {{spark.executor.instances}} | {{--num-executors}} |
| {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
We should make sure that one is converted to the other (the current pattern is
to use {{--conf}} versions of these) so we don't run into issues where the
log4j properties are overwritten. (OOZIE-2705 can be affected by this)
was:
Issues like OOZIE-2658 can happen with more properties.
|| --conf || overwritten by ||
|spark.driver.memory | --driver-memory |
|spark.driver.extraJavaOptions | --driver-java-options|
|spark.driver.extraLibraryPath|--driver-library-path |
|spark.executor.instances|--num-executors|
|spark.driver.extraClassPath| --driver-class-path|
We should make sure that one is converted to the other (the current pattern is
to use --conf versions of these) so we don't run into issues where the log4j
properties are overwritten. (OOZIE-2705 can be affected by this)
> Merge Spark opts with two possible options
> ---
>
> Key: OOZIE-2712
> URL: https://issues.apache.org/jira/browse/OOZIE-2712
> Project: Oozie
> Issue Type: Bug
>Reporter: Peter Cseh
>Assignee: Andras Piros
> Attachments: OOZIE-2712.001.patch
>
>
> Issues like OOZIE-2658 can happen with more properties.
> || {{--conf}} || overwritten by ||
> | {{spark.driver.memory}} | {{--driver-memory}} |
> | {{spark.driver.extraJavaOptions}} | {{--driver-java-options}}|
> | {{spark.driver.extraLibraryPath}} | {{--driver-library-path}} |
> | {{spark.executor.instances}} | {{--num-executors}} |
> | {{spark.driver.extraClassPath}} | {{--driver-class-path}} |
> We should make sure that one is converted to the other (the current pattern
> is to use {{--conf}} versions of these) so we don't run into issues where the
> log4j properties are overwritten. (OOZIE-2705 can be affected by this)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[ https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh updated OOZIE-2712: -- Assignee: (was: Peter Cseh) > Merge Spark opts with two possible options > --- > > Key: OOZIE-2712 > URL: https://issues.apache.org/jira/browse/OOZIE-2712 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh > Attachments: OOZIE-2712.001.patch > > > Issues like OOZIE-2658 can happen with more properties. > || --conf || overwritten by || > |spark.driver.memory | --driver-memory | > |spark.driver.extraJavaOptions | --driver-java-options| > |spark.driver.extraLibraryPath|--driver-library-path | > |spark.executor.instances|--num-executors| > |spark.driver.extraClassPath| --driver-class-path| > We should make sure that one is converted to the other (the current pattern > is to use --conf versions of these) so we don't run into issues where the > log4j properties are overwritten. (OOZIE-2705 can be affected by this) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[ https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh updated OOZIE-2712: -- Description: Issues like OOZIE-2658 can happen with more properties. || --conf || overwritten by || |spark.driver.memory | --driver-memory | |spark.driver.extraJavaOptions | --driver-java-options| |spark.driver.extraLibraryPath|--driver-library-path | |spark.executor.instances|--num-executors| |spark.driver.extraClassPath| --driver-class-path| We should make sure that one is converted to the other (the current pattern is to use --conf versions of these) so we don't run into issues where the log4j properties are overwritten. (OOZIE-2705 can be affected by this) was: Issues like OOZIE-2658 can happen with more properties. || --conf || overwritten by || |spark.driver.memory | --driver-memory | |spark.driver.extraJavaOptions | --driver-java-options| |spark.driver.extraLibraryPath|--driver-library-path | |spark.executor.instances|--num-executors| We should make sure that one is converted to the other (the current pattern is to use --conf versions of these) so we don't run into issues where the log4j properties are overwritten. (OOZIE-2705 can be affected by this) > Merge Spark opts with two possible options > --- > > Key: OOZIE-2712 > URL: https://issues.apache.org/jira/browse/OOZIE-2712 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh >Assignee: Peter Cseh > Attachments: OOZIE-2712.001.patch > > > Issues like OOZIE-2658 can happen with more properties. > || --conf || overwritten by || > |spark.driver.memory | --driver-memory | > |spark.driver.extraJavaOptions | --driver-java-options| > |spark.driver.extraLibraryPath|--driver-library-path | > |spark.executor.instances|--num-executors| > |spark.driver.extraClassPath| --driver-class-path| > We should make sure that one is converted to the other (the current pattern > is to use --conf versions of these) so we don't run into issues where the > log4j properties are overwritten. (OOZIE-2705 can be affected by this) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2712) Merge Spark opts with two possible options
[ https://issues.apache.org/jira/browse/OOZIE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh updated OOZIE-2712: -- Attachment: OOZIE-2712.001.patch Attaching the first version to fix this. It simplifies the code at some places, but I really don't like the number of HashMaps involved. I will probably work refactor this a bit soon. Until then, [~satishsaley], [~abhishekbafna], [~andras.piros], [~pbacsko], [~asasvari], what do you think about the direction? > Merge Spark opts with two possible options > --- > > Key: OOZIE-2712 > URL: https://issues.apache.org/jira/browse/OOZIE-2712 > Project: Oozie > Issue Type: Bug >Reporter: Peter Cseh >Assignee: Peter Cseh > Attachments: OOZIE-2712.001.patch > > > Issues like OOZIE-2658 can happen with more properties. > || --conf || overwritten by || > |spark.driver.memory | --driver-memory | > |spark.driver.extraJavaOptions | --driver-java-options| > |spark.driver.extraLibraryPath|--driver-library-path | > |spark.executor.instances|--num-executors| > We should make sure that one is converted to the other (the current pattern > is to use --conf versions of these) so we don't run into issues where the > log4j properties are overwritten. (OOZIE-2705 can be affected by this) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
