I'm using log aggregation on YARN with Spark and I am not able to see the
logs through the YARN web UI after the application completes:
Failed redirect for container_1425390894284_0066_01_000001
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configured
java.lang.Exception: Unknown container. Container either has not started or
has already completed or doesn't belong to this node at all.
I tried setting
log4j.appender.file_appender.File=${spark.yarn.app.container.log.dir}/spark.log
in log4j.properties as suggested in the documentation, but didn't seem to
change anything.
Note that I can see the logs fine using "yarn logs" from the command line,
so aggregation is working properly. For regular mapreduce jobs, the YARN UI
displays the logs correctly as well. Is there a spark configuration option
that needs to be set if aggregation is used?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/failure-to-display-logs-on-YARN-UI-with-log-aggregation-on-tp21974.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
