HeartSaVioR edited a comment on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-455328233 @squito Yes, exactly. Nice summarization. > this includes when the SHS is showing "inprogress" applications (this could be changed I think) IMHO there're some cases we still want to get custom log URL while the status of application is "in progress", because according to the logic on determining whether the application is finished, app could be shown as "inprogress" when the app terminated unexpectedly. https://github.com/apache/spark/blob/0b3abef1950f486001160ec578e4f628c199eeb4/core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala#L98-L100 In this case, showing origin log URLs may not work but showing custom log URLs would work if the external log service gathers executor logs continuously rather than gathering log when app is finished - I expect the behavior of external log service as former not latter. Even if app is still running, app UI will still provide origin log URLs so we can get it from there instead of SHS. What I agreed to make change on only SHS was my prev. patch was making log urls being static again which is fragile if there's a change on external log service. Now we just require end users to modify the configuration and restart SHS (unless there's a cache mechanism to avoid re-reading events when restarting SHS.) IMHO, showing custom log URLs even for apps which are shown as "inprogress" still makes sense to me. Btw, I didn't add variables for other resource managers as they don't provide log URLs for now. (If possible we may be better to address it as well via separate PRs.)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
