GitHub user woggle opened a pull request:
https://github.com/apache/spark/pull/3218
[WIP][SPARK-4351] RDD cached read logging + display on web UI
This set of patches adds cached RDD block attempted read/write trace
logging to TaskMetrics and therefore the event log, and uses this information
to display/compute RDD miss rates on the Web UI.
This patch removes the elegance of the contents of the storage web UI being
entirely from StorageStatuses (as RDD access records shouldn't go away when a
block manager does for the purpose of tracking hit/miss rates); I'd appreciate
comments on cleaner solutions.
This still needs some additions to the UI tests, and I'd like to make sure
it doesn't do anything bad for perf.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/woggle/spark read-logging
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3218.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3218
----
commit 5c52aade67f9af2d778878fac51f9513dd37dc78
Author: Charles Reiss <[email protected]>
Date: 2014-11-09T18:51:00Z
Add event logging of block accesses via CacheManager.
commit 5cefcf8fb89266b9caac25833d9b25e98ebeef08
Author: Charles Reiss <[email protected]>
Date: 2014-11-09T18:52:42Z
Create SparkListener that uses block access logs to compute RDD miss rates
commit 33fbd8b5b43339b9f8043776be1aaace408d603b
Author: Charles Reiss <[email protected]>
Date: 2014-11-09T18:54:40Z
Record miss rates in RDDINfo and display on the web UI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
