GitHub user ksakellis opened a pull request:
https://github.com/apache/spark/pull/3486
[WIP] SPARK-2450 Adds exeuctor log links to Web UI
Creates a general way to add log links to the executor
page in the web UI.
This is achieved by:
1) Adding two new spark listener events: onExecutorAdded and
onExecutorRemoved(this is only for symmetry). The events carry some executor
information including log url locations.
2) Utilizing environment variables to pass the log urls to the executor
backend. The prefix is: SPARK_LOG_URL_
Since each cluster manager exposes logs in different ways, we use the
environment variable prefix as described above to prevent us from subclassing
the CoarseGrainedExecutorBacked.
This is a work in progress and as such, there are minimal unit tests. In
addition log urls are only propagated for yarn-client and yarn-cluster mode.
With minimal work, adding log url propagation for standalone mode is also
possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ksakellis/spark kostas-spark-2450
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3486.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3486
----
commit e78308e23c719d8dcd27f9b9d615a54c5fef13b8
Author: Kostas Sakellis <[email protected]>
Date: 2014-11-26T19:14:11Z
[WIP] SPARK-2450 Adds exeuctor log links to Web UI
Creates a general way to add log links to the executor
page in the web UI.
This is achieved by:
1) Adding two new spark listener events: onExecutorAdded
and onExecutorRemoved(this is only for symmetry). The events
carry some executor information including log url locations.
2) Utilizing environment variables to pass the log urls to the
executor backend. The prefix is: SPARK_LOG_URL_
Since each cluster manager exposes logs in different ways,
we use the environment variable prefix as described above
to prevent us from subclassing the CoarseGrainedExecutorBacked.
This is a work in progress and as such, there are minimal unit
tests. In addition log urls are only propagated for yarn-client
and yarn-cluster mode. With minimal work, adding log url
propagation for standalone mode is also possible.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
