Hey Guys,
I am hitting the below issue when trying to access the STDOUT/STDERR logs in
Spark History Server for the executors of a Spark application executed in
Yarn mode. I have enabled Yarn log aggregation.
Repro Steps:
1) Run the spark-shell in yarn client mode. Or run Pi job in Yarn mode.
2) Once the job is completed, (in the case of spark shell, exit after doing
some simple operations), try to access the STDOUT or STDERR logs of the
application from the Executors tab in the Spark History Server UI.
3) If yarn log aggregation is enabled, then logs won't be available in node
manager's log location. But history Server is trying to access the logs from
the nodemanager's log
location({yarn.nodemanager.log-dirs}/application_${appid) giving below error
in the UI:
Failed redirect for container_e31_1496881617682_0003_01_000002
ResourceManager
RM Home
NodeManager
Tools
Failed while trying to construct the redirect url to the log server. Log
Server url may not be configured
java.lang.Exception: Unknown container. Container either has not started or
has already completed or doesn't belong to this node at all.
Either Spark History Server should be able to read from the aggregated logs
and display the logs in the UI or it should give a graceful message. As of
now its redirecting to the NM webpage and trying to fetch the logs from the
node managers local location.
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-History-Server-does-not-redirect-to-Yarn-aggregated-logs-for-container-logs-tp21706.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
