GitHub user carsonwang opened a pull request:
https://github.com/apache/spark/pull/7033
[SPARK-8405][WebUI] Show executor logs on Web UI when Yarn log aggregation
is enabled
https://issues.apache.org/jira/browse/SPARK-8405
When running Spark application in Yarn mode and Yarn log aggregation is
enabled, customer is not able to view executor logs on the history server Web
UI. The only way for customer to view the logs is through the Yarn command
"yarn logs -applicationId <appId>".
An screenshot of the error is attached in the Jira. When you click an
executorâs log link on the Spark history server, youâll see the error if
Yarn log aggregation is enabled. The log URL redirects user to the node
managerâs UI. This works if the logs are located on that node. But since log
aggregation is enabled, the local logs are deleted once log aggregation is
completed.
The logs should be available through the web UIs just like other Hadoop
components like MapReduce. For security reasons, end users may not be able to
log into the nodes and run the yarn logs -applicationId command. The web UIs
can be viewable and exposed through the firewall if necessary.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/carsonwang/spark yarnlog2
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7033.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7033
----
commit ccfd37756d83d69d7bc5a2aaadba060c90e788b9
Author: Carson Wang <[email protected]>
Date: 2015-06-04T08:40:30Z
Show aggregated log for history app running in Yarn mode
commit 43493ce2bdd3fff45c487155a67405b79d81909a
Author: Carson Wang <[email protected]>
Date: 2015-06-11T01:40:40Z
Show log links for every executors
commit 313b7920cd282484b9f81f41ec9150689fa85900
Author: Carson Wang <[email protected]>
Date: 2015-06-11T09:04:56Z
Use Yarn Api to read the log file
commit 54f2cdbd7a60740e6ddfeb9a3632d5b5f592873e
Author: Carson Wang <[email protected]>
Date: 2015-06-12T03:00:56Z
Fix code style
commit d6ece2cade5dac4466e8889c5d4dc82500978777
Author: Carson Wang <[email protected]>
Date: 2015-06-26T05:48:34Z
Move the log page to yarn module.
commit 0871b35be098ac152f2ef449f1b080c35a7be7a3
Author: Carson Wang <[email protected]>
Date: 2015-06-26T06:57:32Z
Fix code style
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
