[
https://issues.apache.org/jira/browse/OOZIE-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobin Zheng updated OOZIE-2786:
---------------------------------
Description:
When using Oozie to launch Spark applications, the Oozie work flow ID and etc
do not show up in Spark Application's Environment settings. There is no Spark
application ID exposed from Oozie side as well about the Spark applications
Oozie launches.
When looking at applications in Spark History Server, it is hard to figure out
which Oozie workflow launched it. This makes debugging the workflow hard. We
want to be able to let Spark History server display the Oozie workflow ID in
the application's 'Environment' settings.
For reference, oozie properties that can be passed to Spark are:
oozie.job.id
oozie.HadoopAccessorService.created
oozie.action.spark.setup.hadoop.conf.dir
oozie.child.mapreduce.job.tags
oozie.action.id
oozie.action.rootlogger.log.level
was:
When using Oozie to launch Spark applications, the Oozie work flow ID and etc
do not show up in Spark Application's Environment settings. There is no Spark
application ID exposed from Oozie side as well about the Spark applications
Oozie launches.
When looking at applications in Spark History Server, it is hard to figure out
which Oozie workflow launched it. This makes debugging the workflow hard. We
want to be able to let Spark History server display the Oozie workflow ID in
the application's 'Environment' settings.
> Pass Oozie workflow ID and settings to Spark application configuration
> ----------------------------------------------------------------------
>
> Key: OOZIE-2786
> URL: https://issues.apache.org/jira/browse/OOZIE-2786
> Project: Oozie
> Issue Type: Improvement
> Components: action
> Reporter: Xiaobin Zheng
> Assignee: Xiaobin Zheng
> Priority: Critical
>
> When using Oozie to launch Spark applications, the Oozie work flow ID and etc
> do not show up in Spark Application's Environment settings. There is no Spark
> application ID exposed from Oozie side as well about the Spark applications
> Oozie launches.
> When looking at applications in Spark History Server, it is hard to figure
> out which Oozie workflow launched it. This makes debugging the workflow hard.
> We want to be able to let Spark History server display the Oozie workflow ID
> in the application's 'Environment' settings.
> For reference, oozie properties that can be passed to Spark are:
> oozie.job.id
> oozie.HadoopAccessorService.created
> oozie.action.spark.setup.hadoop.conf.dir
> oozie.child.mapreduce.job.tags
> oozie.action.id
> oozie.action.rootlogger.log.level
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
