[
https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14589475#comment-14589475
]
Carson Wang commented on SPARK-8405:
------------------------------------
I had some in progress works and here is the approach I was using.
1. If Yarn log aggreation is enabled, we update each executor's log URL on the
history server. The new URL link is a new added log page hosted on the history
server. These URLs are passed the same as how other URLs are passed. So we have
enough information like the container Id, appOwner, etc.
2. The log page reads the aggregated logs from HDFS by using Yarn APIs.
This is transparent to the end users. If Yarn log aggreation is not enabled,
nothing is changed. If it is eanbled, the end user will be albe to click the
executor's log link and view the logs on Web UI.
Is there any concerns regarding reading the aggregated logs from HDFS? The Map
Reduce history server reads the aggregated logs from HDFS as well to show the
logs so I suppose it is ok for Spark history server to read it.
> Show executor logs on Web UI when Yarn log aggregation is enabled
> -----------------------------------------------------------------
>
> Key: SPARK-8405
> URL: https://issues.apache.org/jira/browse/SPARK-8405
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.0
> Reporter: Carson Wang
> Attachments: SparkLogError.png
>
>
> When running Spark application in Yarn mode and Yarn log aggregation is
> enabled, customer is not able to view executor logs on the history server Web
> UI. The only way for customer to view the logs is through the Yarn command
> "yarn logs -applicationId <appId>".
> An screenshot of the error is attached. When you click an executor’s log link
> on the Spark history server, you’ll see the error if Yarn log aggregation is
> enabled. The log URL redirects user to the node manager’s UI. This works if
> the logs are located on that node. But since log aggregation is enabled, the
> local logs are deleted once log aggregation is completed.
> The logs should be available through the web UIs just like other Hadoop
> components like MapReduce. For security reasons, end users may not be able to
> log into the nodes and run the yarn logs -applicationId command. The web UIs
> can be viewable and exposed through the firewall if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
