Jungtaek Lim created SPARK-26792:
------------------------------------
Summary: Apply custom log URL to Spark UI
Key: SPARK-26792
URL: https://issues.apache.org/jira/browse/SPARK-26792
Project: Spark
Issue Type: Improvement
Components: Web UI
Affects Versions: 3.0.0
Reporter: Jungtaek Lim
SPARK-23155 enables SHS to set up custom log URLs for incompleted / completed
apps.
While getting reviews from SPARK-23155, I've got two comments which applying
custom log URLs to UI would help achieving it. Quoting these comments here:
https://github.com/apache/spark/pull/23260#issuecomment-456827963
{quote}
Sorry I haven't had time to look through all the code so this might be a
separate jira, but one thing I thought of here is it would be really nice not
to have specifically stderr/stdout. users can specify any log4j.properties and
some tools like oozie by default end up using hadoop log4j rather then spark
log4j, so files aren't necessarily the same. Also users can put in other logs
files so it would be nice to have links to those from the UI. It seems simpler
if we just had a link to the directory and it read the files within there.
Other things in Hadoop do it this way, but I'm not sure if that works well for
other resource managers, any thoughts on that? As long as this doesn't prevent
the above I can file a separate jira for it.
{quote}
https://github.com/apache/spark/pull/23260#issuecomment-456904716
{quote}
Hi Tom, +1: singling out stdout and stderr is definitely an annoyance. We
typically configure Spark jobs to write the GC log and dump heap on OOM
using <LOG_DIR>, and/or we use the rolling file appender to deal with
large logs during debugging. So linking the YARN container log overview
page would make much more sense for us. We work it around with a custom
submit process that logs all important URLs on the submit side log.
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
