[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2687: -- Attachment: OOZIE-2687-011.patch > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, > OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, > OOZIE-2687-010.patch, OOZIE-2687-011.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2687: -- Attachment: OOZIE-2687-010.patch > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, > OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch, > OOZIE-2687-010.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2687: -- Attachment: OOZIE-2687-009.patch Following [~rkanter]'s example, I managed to create and use a oozie common schema. > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, > OOZIE-2687-003.patch, OOZIE-2687-008.patch, OOZIE-2687-009.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2687: -- Attachment: OOZIE-2687-008.patch > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, > OOZIE-2687-003.patch, OOZIE-2687-008.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Sasvari updated OOZIE-2687: -- Attachment: OOZIE-2687-003.patch > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch, > OOZIE-2687-003.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-002.patch
Tested on pseudo hadoop that submitting a workflow like:
{code}
${jobTracker}
${nameNode}
Q=Question:A=42
...
{code}
will in fact result in a generated {{launch_container.sh}} that includes
environment variables:
{code}
head -3
nm-local-dir/nmPrivate/application_1502749591067_0005/container_1502749591067_0005_01_01/launch_container.sh
#!/bin/bash
export A="42"
{code}
> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
> Issue Type: Sub-task
>Affects Versions: 5.0.0
>Reporter: Peter Cseh
>Assignee: Attila Sasvari
>Priority: Critical
> Attachments: OOZIE-2687-001.patch, OOZIE-2687-002.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through
> properties like:
> # mapreduce.map.memory.mb
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts
> # mapred.job.queue.name - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper
> and make PySpark work.
> Fixing OOZIE-2596 added a hack. We should decide how we support or break
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in
> sub-workflows as well. It did not use to work before and was only fixed in
> OOZIE-2030. It would be good to have that testcase
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new
> launcher section as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Piros updated OOZIE-2687: Affects Version/s: 5.0.0 Priority: Critical (was: Major) > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Affects Versions: 5.0.0 >Reporter: Peter Cseh >Assignee: Attila Sasvari >Priority: Critical > Attachments: OOZIE-2687-001.patch > > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[
https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Sasvari updated OOZIE-2687:
--
Attachment: OOZIE-2687-001.patch
I had to make some modifications in [~pbacsko] wip
Tested
- TestJavaActionExecutor, TestLiteWorkflowAppParser tests passed
- java-main example on pseudo hadoop succeeded, modified workflow xml so that
verbose class loading information is printed out of stdout of the launcher job
{code}
${jobTracker}
${nameNode}
-verbose:class
mapred.job.queue.name
${queueName}
org.apache.oozie.example.DemoJavaMain
Hello
Oozie!
Java failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]
{code}
Logs:
{code}
head -3
/Users/asasvari/workspace/pseudo/hadoop-2.6.0/logs/userlogs/application_1502291380639_0001/container_1502291380639_0001_01_01/stdout
[Opened
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/rt.jar]
[Loaded java.lang.Object from
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/rt.jar]
[Loaded java.io.Serializable from
/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre/lib/rt.jar]
{code}
> Create XML schema for launcher configurations
> -
>
> Key: OOZIE-2687
> URL: https://issues.apache.org/jira/browse/OOZIE-2687
> Project: Oozie
> Issue Type: Sub-task
>Reporter: Peter Cseh
>Assignee: Peter Bacsko
> Attachments: OOZIE-2687-001.patch
>
>
> It was possible to manipulate the MapperLauncher's environment through
> properties like:
> # mapreduce.map.memory.mb
> # mapreduce.map.cpu.vcores
> # mapred.child.env
> # mapred.child.java.opts
> # mapred.job.queue.name - ability to set launcher queue
> E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper
> and make PySpark work.
> Fixing OOZIE-2596 added a hack. We should decide how we support or break
> compatibility and how we allow the manipulation of the Launcher environment.
> Verify if the new launcher section in global applies to actions in
> sub-workflows as well. It did not use to work before and was only fixed in
> OOZIE-2030. It would be good to have that testcase
> (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new
> launcher section as well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (OOZIE-2687) Create XML schema for launcher configurations
[ https://issues.apache.org/jira/browse/OOZIE-2687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Cseh updated OOZIE-2687: -- Summary: Create XML schema for launcher configurations (was: OYA: Figure out how to handle MR-specific properties) > Create XML schema for launcher configurations > - > > Key: OOZIE-2687 > URL: https://issues.apache.org/jira/browse/OOZIE-2687 > Project: Oozie > Issue Type: Sub-task >Reporter: Peter Cseh >Assignee: Peter Bacsko > > It was possible to manipulate the MapperLauncher's environment through > properties like: > # mapreduce.map.memory.mb > # mapreduce.map.cpu.vcores > # mapred.child.env > # mapred.child.java.opts > # mapred.job.queue.name - ability to set launcher queue > E.g. We were using mapred.child.env to pass SPARK_HOME to the LauncherMapper > and make PySpark work. > Fixing OOZIE-2596 added a hack. We should decide how we support or break > compatibility and how we allow the manipulation of the Launcher environment. > Verify if the new launcher section in global applies to actions in > sub-workflows as well. It did not use to work before and was only fixed in > OOZIE-2030. It would be good to have that testcase > (TestSubWorkflowActionExecutor. testParentGlobalConf) updated with the new > launcher section as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
