HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-448029536 OK. I guess I'm seeing what we concern about. Looks like we don't want to set specific URL to env which could be a permanently dead link at any chance (like webserver is moving, we changed pattern of URL, etc.) I feel this concern has been sitting in Spark for years (imagine the format of YARN url had to be changed in prior version) so I feel we have time to address it incrementally (doing better than current), but I also think we can investigate what to do to achieve that and just do it at once if it doesn't require huge efforts. > The parameters don't need to be pre-defined, each RM can have its own set, and it's up to the admin to set things up so that the URLs in the SHS make sense for their deployment. I'm not sure I follow. While some parameters can be retrieved from YARN config but there're other parameters we get from allocated container (and maybe other things we can't get from YARN config). Could you please elaborate?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
