GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/10115
[SPARK-12108] Make event logs smaller
Event logs in 1.6 were much bigger than 1.5. I ran page rank and the event
log size in 1.6 was almost 5x that in 1.5. I did a bisect to find that the RDD
callsite added in #9398 is largely responsible for this.
This patch removes the long form of the callsite (which is not used!) from
the event log. This reduces the size of the event log significantly.
Note on compatibility: if this patch is to be merged into 1.6.0, then it
won't break any compatibility. Otherwise, if it is merged into 1.6.1, then we
might need to add more backward compatibility handling logic (currently does
not exist yet).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark smaller-event-logs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10115.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10115
----
commit 5a0ebdfbd56bc289900ddd26579c3800149f4062
Author: Andrew Or <[email protected]>
Date: 2015-12-02T23:52:50Z
Store only callsite short forms in event log
The long form is not currently used and inflates the size of the
event log significantly.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
