[
https://issues.apache.org/jira/browse/SPARK-8405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605126#comment-14605126
]
Carson Wang commented on SPARK-8405:
------------------------------------
Thank you very much, [~hshreedharan]
After I configured yarn.log.server.url in yarn-site.xml, the log url did
redirect to MR job history server's UI and was able to show the logs. But this
seems a little strange because we need view Spark's log on MR history server.
Would it make more sense that Spark history server reads and shows the
aggregated logs itself?
> Show executor logs on Web UI when Yarn log aggregation is enabled
> -----------------------------------------------------------------
>
> Key: SPARK-8405
> URL: https://issues.apache.org/jira/browse/SPARK-8405
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 1.4.0
> Reporter: Carson Wang
> Attachments: SparkLogError.png
>
>
> When running Spark application in Yarn mode and Yarn log aggregation is
> enabled, customer is not able to view executor logs on the history server Web
> UI. The only way for customer to view the logs is through the Yarn command
> "yarn logs -applicationId <appId>".
> An screenshot of the error is attached. When you click an executor’s log link
> on the Spark history server, you’ll see the error if Yarn log aggregation is
> enabled. The log URL redirects user to the node manager’s UI. This works if
> the logs are located on that node. But since log aggregation is enabled, the
> local logs are deleted once log aggregation is completed.
> The logs should be available through the web UIs just like other Hadoop
> components like MapReduce. For security reasons, end users may not be able to
> log into the nodes and run the yarn logs -applicationId command. The web UIs
> can be viewable and exposed through the firewall if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
