[
https://issues.apache.org/jira/browse/OOZIE-2786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobin Zheng updated OOZIE-2786:
---------------------------------
Attachment: OOZIE-2786-1.patch
This change passes several oozie.xxx properties from Spark action conf as Spark
conf properties explicitly. Properties passed to Spark conf is prefixed with
'spark.' so that they get picked up by Spark as environment properties which
eventually appears in Spark history server.
This has been tested end to end, verified that those properties show up in
Spark History Server for Spark applications launched by Oozie workflow.
> Pass Oozie workflow ID and settings to Spark application configuration
> ----------------------------------------------------------------------
>
> Key: OOZIE-2786
> URL: https://issues.apache.org/jira/browse/OOZIE-2786
> Project: Oozie
> Issue Type: Improvement
> Components: action
> Reporter: Xiaobin Zheng
> Assignee: Xiaobin Zheng
> Priority: Critical
> Attachments: OOZIE-2786-1.patch
>
>
> When using Oozie to launch Spark applications, the Oozie work flow ID and etc
> do not show up in Spark Application's Environment settings. There is no Spark
> application ID exposed from Oozie side as well about the Spark applications
> Oozie launches.
> When looking at applications in Spark History Server, it is hard to figure
> out which Oozie workflow launched it. This makes debugging the workflow hard.
> We want to be able to let Spark History server display the Oozie workflow ID
> in the application's 'Environment' settings.
> For reference, oozie properties that can be passed to Spark are:
> oozie.job.id
> oozie.HadoopAccessorService.created
> oozie.action.spark.setup.hadoop.conf.dir
> oozie.child.mapreduce.job.tags
> oozie.action.id
> oozie.action.rootlogger.log.level
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
