HeartSaVioR commented on issue #23260: [SPARK-26311][YARN] New feature: custom log URL for stdout/stderr URL: https://github.com/apache/spark/pull/23260#issuecomment-447525509 @squito If we don't touch the default value it is just running as it is for now because the default URL is set to static URL of NM Spark has been using. So this doesn't break anything Spark currently has. This patch provides flexibility of executor log URL, not only our case but also all of the cases if log URL can be represented via provided patterns. If it turns out the set of patterns are not enough in future, we can simply add missing patterns when YARN runner can provide them. (We can't predict and enumerate everything.) If there's a case YARN runner cannot provide the pattern some services need to have to provide logs, there's no way for Spark to determine and set such URL to log URL. Regarding running application vs finished application, even NM provides unique URL and redirects URL when the application is finished. I don't think other services cannot do it, but if we have concrete case which running application and finished application should have different log URL (this would mean we may need to change log URL only for finished application) I can take a look and address it. Does it make sense to you?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
