GitHub user superbobry opened a pull request:
https://github.com/apache/spark/pull/19992
[SPARK-22805][CORE] Use StorageLevel aliases in event logs
The format of event logs uses redundant representation for storage
levels, for instance StorageLevel.DISK_ONLY is represented as
{"Use Disk":true,"Use
Memory":false,"Deserialized":false,"Replication":1}
which is 64 bytes more. This commit changes the event log representation
of the StorageLevel to predefined constants: NONE, DISK_ONLY, etc. The
change is fully backward compatibly, because
* StorageLevel constructor is private, meaning that existing event
logs can only contain these predefined levels;
* The JsonProtocol supports reading both the old format and the new one.
## What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/criteo-forks/spark compact-storage-level
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19992.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19992
----
commit 7869e63a569a6fb6725996084f0c5c55fc130ac8
Author: Sergei Lebedev <[email protected]>
Date: 2017-12-15T17:29:32Z
[SPARK-22805][CORE] Use StorageLevel aliases in event logs
The format of event logs uses redundant representation for storage
levels, for instance StorageLevel.DISK_ONLY is represented as
{"Use Disk":true,"Use
Memory":false,"Deserialized":false,"Replication":1}
which is 64 bytes more. This commit changes the event log representation
of the StorageLevel to predefined constants: NONE, DISK_ONLY, etc. The
change is fully backward compatibly, because
* StorageLevel constructor is private, meaning that existing event
logs can only contain these predefined levels;
* The JsonProtocol supports reading both the old format and the new one.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
