[ https://issues.apache.org/jira/browse/OOZIE-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110783#comment-16110783 ]
Andras Piros commented on OOZIE-2983: ------------------------------------- [~rkanter] +1 on the original approach, your patch seems pretty advanced (apart from TODOs). I like the idea of Oozie being able to stream the launcher AM logs on its own, rather than relying on a running JobHistoryServer and the client's capability of following redirects in the first place. I'd be curious how Spark does that, though. [~wilfreds] can you point to the Spark sources where Spark does the redirect? I [might be missing|https://github.com/apache/spark/blob/6735433cde44b3430fb44edfff58ef8149c66c13/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala#L376-L380] the place. > Stream the Launcher AM Logs > --------------------------- > > Key: OOZIE-2983 > URL: https://issues.apache.org/jira/browse/OOZIE-2983 > Project: Oozie > Issue Type: Sub-task > Affects Versions: 5.0.0 > Reporter: Robert Kanter > Assignee: Andras Piros > Attachments: OOZIE-2983.001.patch > > > Because OYA will get rid of the MR-based Launcher Job and have a custom AM > instead, the Launcher Job info and logs will no longer show up in the JHS. It > will show up in the ATSv2, however, the ATSv2 isn't quite ready yet and that > doesn't help users on older versions of Hadoop. Launcher Jobs will stil show > up in the RM, but the RM doesn't remember things forever (and can be quite > short depending on cluster busyness and configs). > We should have Oozie serve those logs itself (which would also be a nice > thing to have anyway, as the launcher logs are typically the most useful). > The code for reading those files is pretty easy to call. The yarn CLI, JHS, > and this [Aggregated Log Files > Exploder|https://github.com/rkanter/ALFExploder] tool I wrote all use it; the > latter proves that you can do it outside of Hadoop. -- This message was sent by Atlassian JIRA (v6.4.14#64029)