gerashegalov commented on issue #23260: [SPARK-26311][CORE] New feature: apply custom log URL pattern for executor log URLs in SHS URL: https://github.com/apache/spark/pull/23260#issuecomment-456904716 Hi Tom, +1: singling out stdout and stderr is definitely an annoyance. We typically configure Spark jobs to write the GC log and dump heap on OOM using <LOG_DIR>, and/or we use the rolling file appender to deal with large logs during debugging. So linking the YARN container log overview page would make much more sense for us. We work it around with a custom submit process that logs all important URLs on the submit side log. On Wed, Jan 23, 2019 at 6:47 AM Tom Graves <[email protected]> wrote: > Sorry I haven't had time to look through all the code so this might be a > separate jira, but one thing I thought of here is it would be really nice > not to have specifically stderr/stdout. users can specify any > log4j.properties and some tools like oozie by default end up using hadoop > log4j rather then spark log4j, so files aren't necessarily the same. Also > users can put in other logs files so it would be nice to have links to > those from the UI. It seems simpler if we just had a link to the directory > and it read the files within there. Other things in Hadoop do it this way, > but I'm not sure if that works well for other resource managers, any > thoughts on that? As long as this doesn't prevent the above I can file a > separate jira for it. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/spark/pull/23260#issuecomment-456827963>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ADCk4gVe-JFIB6XkfoYqax8qEPzZVuF8ks5vGHX_gaJpZM4ZJfwb> > . >
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
