Carson Wang created SPARK-8405:
----------------------------------
Summary: Show executor logs on Web UI when Yarn log aggregation is
enabled
Key: SPARK-8405
URL: https://issues.apache.org/jira/browse/SPARK-8405
Project: Spark
Issue Type: Bug
Components: Web UI
Affects Versions: 1.4.0
Reporter: Carson Wang
When running Spark application in Yarn mode and Yarn log aggregation is
enabled, customer is not able to view executor logs on the history server Web
UI. The only way for customer to view the logs is through the Yarn command
"yarn logs -applicationId <appId>".
An screenshot of the error is attached. When you click an executor’s log link
on the Spark history server, you’ll see the error if Yarn log aggregation is
enabled. The log URL redirects user to the node manager’s UI. This works if the
logs are located on that node. But since log aggregation is enabled, the local
logs are deleted once log aggregation is completed.
The logs should be available through the web UIs just like other Hadoop
components like MapReduce. For security reasons, end users may not be able to
log into the nodes and run the yarn logs -applicationId command. The web UIs
can be viewable and exposed through the firewall if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
