[
https://issues.apache.org/jira/browse/OOZIE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2983:
---------------------------------
Attachment: OOZIE-2983.001.patch
I've been tinkering around with this for a while, and am going to hand it off
to [~andras.piros] to finish, who will have more time to work on it faster.
There's a number of TODOs describing what else needs to be done, but the basic
code is there. It uses the same logs command as workflows, coordinators, and
bundles, but you instead supply a workflow action id. For example, {{oozie job
-log 0000023-170627053314956-oozie-oozi-W@hive-node}}. This will give you the
stdout, stderr, and syslog for that launcher job. It would be nice to be able
to allow for requesting only a specific log type, however, the API for that
[{{ContainerLogsReader}}|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java#L1009]
is marked {{\@Private}}, so we can only safely use the API that prints out all
of the log types. Using an {{\@Private}} API is fine for some random
unsupported tool like the Aggregated Log Files Exploder I wrote, but probably
not for Oozie.
> Stream the Launcher AM Logs
> ---------------------------
>
> Key: OOZIE-2983
> URL: https://issues.apache.org/jira/browse/OOZIE-2983
> Project: Oozie
> Issue Type: Sub-task
> Affects Versions: 5.0.0
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Attachments: OOZIE-2983.001.patch
>
>
> Because OYA will get rid of the MR-based Launcher Job and have a custom AM
> instead, the Launcher Job info and logs will no longer show up in the JHS. It
> will show up in the ATSv2, however, the ATSv2 isn't quite ready yet and that
> doesn't help users on older versions of Hadoop. Launcher Jobs will stil show
> up in the RM, but the RM doesn't remember things forever (and can be quite
> short depending on cluster busyness and configs).
> We should have Oozie serve those logs itself (which would also be a nice
> thing to have anyway, as the launcher logs are typically the most useful).
> The code for reading those files is pretty easy to call. The yarn CLI, JHS,
> and this [Aggregated Log Files
> Exploder|https://github.com/rkanter/ALFExploder] tool I wrote all use it; the
> latter proves that you can do it outside of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
