HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-451871129 I'm afraid I'm over-thinking about this, but still don't feel easy to tackle it: not just moving logic to SHS, but also defining some interfaces and caring to run the implementation. > the application to save some information in the event log so that the SHS can use it I think you mean `exposing interface` for end users to `define a set of parameters for the log URL and let them be stored to event log` when launching executor in resource manager. Do I understand correctly? For specific to YARN, `the application` should be run in ExecutorRunnable (in AM process) which has all the information: this means we may need to care about classpath for the implementation. I also would like to be clear about `application`: are you suggesting it as a part of spark application (so end users should have it in their app jar) or an individual (global) plugin (so the jar should be placed or linked in spark classpath)? I guess a Spark application should run with any resource managers, and this may open the possibility to tie the application to specific resource manager. Btw, `SPARK_LOG_URL_` prefix is only used for standalone and YARN resource manager: not sure how mesos and k8s provide executors' log URLs. May need to spend some times to get familiar with other resource managers as well.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
