squito commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-446668589 I'm sorry, I still don't totally follow, partially from my ignorance of some details in yarn here which I'd like to better understand. Marcelo's comment sounds like a good use of this: > My understanding is that this allows pointing the Spark UI directly at the history server (old JHS or new ATS) instead of hardcoding the NM URL and relying on the NM redirecting you, since the NM may not exist later on. So this change would allow the user to do that, if they change these parameters when they *submit* their application? It sounds like with this patch, if you configure things correctly, you can get the JHS / ATS to display those logs (I guess this works using yarn's log aggregation?). If so, it might make sense to include a description of how you'd configure things that way in the docs (it isn't obvious to me how to do it, anyway). Or maybe I'm still not quite following, and there is some 3rd party piece here, outside of spark & yarn, which collects the logs and can serve them later on, whether or not the NM can serve the logs?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
