HeartSaVioR commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-458410796 I just fixed a bug: the reason is we cannot retrieve NM_HOST / NM_PORT / NM_HTTP_PORT outside of container process for given container. In YARN cluster mode we can get them for driver, but not for executors in both cluster and client mode. In order to roll back these attributes to NM_HTTP_ADDRESS, I had to remove explanation on setting up custom log URL to JHS URL because it refers to NM_PORT (IPC) which cannot be retrieved. We might try moving out extracting log urls and attributes to YARN executor side and see whether it works, but I'd hope addressing that in other issue. I would like to get it done with stable shape first, and try out more.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
