GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/11791
[SPARK-13980][WIP] Incrementally serialize blocks while unrolling them in
MemoryStore
When a block is persisted in the MemoryStore at a serialized storage level,
the current MemoryStore.putIterator() code will unroll the entire iterator as
Java objects in memory, then will turn around and serialize an iterator
obtained from the unrolled array. This is inefficient and doubles our peak
memory requirements.
Instead, I think that we should incrementally serialize blocks while
unrolling them.
A downside to incremental serialization is the fact that we will need to
deserialize the partially-unrolled data in case there is not enough space to
unroll the block and the block cannot be dropped to disk. However, I'm hoping
that the memory efficiency improvements will outweigh any performance losses as
a result of extra serialization in that hopefully-rare case.
Diff containing only this patch's changes:
https://github.com/JoshRosen/spark/compare/chunked-block-serialization...JoshRosen:serialize-incrementally?expand=1
This patch is marked as WIP because it's rebased on top of / blocked by
#11748.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark serialize-incrementally
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11791.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11791
----
commit 735eca68d8efcd150d47631644cf848b4d98603e
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T04:57:16Z
Split MemoryEntry into two separate classes (serialized and deserialized)
commit 8f0828986b72ce722cfe0360ae863971547fc58b
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T18:53:54Z
Add ChunkedByteBuffer and use it in storage layer.
commit 79b1a6a31236b81c444dda1e8ee1cfdf2f3c36ae
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T20:53:27Z
Add test cases and fix bug in ChunkedByteBuffer.toInputStream()
commit 7dbcd5a9ef0c669f5db97990af944d8b63300e97
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:05:23Z
WIP towards understanding destruction.
commit 3fbec212d9f714386121b4aed791d6c9fb1359a2
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:39:27Z
Small fixes to dispose behavior.
commit e5e663f22094333dac6e184c78176ee658e3441e
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:49:24Z
Modify BlockManager.dataSerialize to write ChunkedByteBuffers.
commit de62f0d0a5f128dd91173e73b214a3297dd203d4
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:47:21Z
Merge remote-tracking branch 'origin/master' into
chunked-block-serialization
commit 0a347fdd9ec0e94eab17eb0f33c93acd1afbdcfb
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:56:02Z
Fix test compilation in streaming.
commit 6852c482a4935b992c199810f1156952f1e93a8c
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T20:47:45Z
Merge remote-tracking branch 'origin/master' into
chunked-block-serialization
commit 43f8fa6ae5ba093655cdbd55ca56959a7652de56
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T20:54:55Z
Allow ChunkedByteBuffer to contain no chunks.
commit 25e68841541b45d7eedc0447cc8154d746ee8db2
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:00:21Z
Document toByteBuffer() and toArray() size limitations.
commit 325c83d8909472428ae65620033fff4887c36e06
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:07:42Z
Move dispose() from BlockManager to StorageUtils.
It was a static method before, but its location was confusing.
commit 4f5074ece49030a6e7134f7ece706ed441c02ee4
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:11:14Z
Better documentation for dispose() methods.
commit b6ddf3ed40cc90ec94b7e4917808f8a726b597ee
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:12:39Z
Rename limit to size.
commit 719ad3c4e9e942ce62cbcf288788aca785690a7e
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:20:08Z
Implement missing InputStream methods.
commit 23006076dcb73095a9eaa7e2524a10c048bae646
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T22:00:10Z
More comments.
commit 3fc0b66981aa2d45be129986f0dc5bd595e08b22
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T22:02:42Z
Fix confusing getChunks().head
commit c747c8546ff248b8b08285e92afad2fe71875acd
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T18:08:34Z
Merge remote-tracking branch 'origin/master' into
chunked-block-serialization
commit cb9311b30636ced3854fd035340092e497750b47
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T18:09:29Z
Fix logging import.
commit 8515cab13a7dfe08919b9500ee9194f5b7b7aa24
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T00:48:26Z
Duplicate the code to separate putIterator() by serialization level.
commit 7dc362331f3f549670ecd9488db456b4136a3ad7
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T23:48:21Z
Incrementally serialize blocks while unrolling them.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
