GitHub user JoshRosen opened a pull request:
https://github.com/apache/spark/pull/11805
[SPARK-13992][WIP] Add support for off-heap caching
This patch adds support for caching blocks in the executor processes using
direct / off-heap memory. I'll expand the description later; this is WIP and
blocked behind a large series of smaller patches (opening now to test + be able
to use GH PRs to track tasks).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/JoshRosen/spark off-heap-caching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11805.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11805
----
commit 735eca68d8efcd150d47631644cf848b4d98603e
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T04:57:16Z
Split MemoryEntry into two separate classes (serialized and deserialized)
commit 8f0828986b72ce722cfe0360ae863971547fc58b
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T18:53:54Z
Add ChunkedByteBuffer and use it in storage layer.
commit 79b1a6a31236b81c444dda1e8ee1cfdf2f3c36ae
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T20:53:27Z
Add test cases and fix bug in ChunkedByteBuffer.toInputStream()
commit 7dbcd5a9ef0c669f5db97990af944d8b63300e97
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:05:23Z
WIP towards understanding destruction.
commit 3fbec212d9f714386121b4aed791d6c9fb1359a2
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:39:27Z
Small fixes to dispose behavior.
commit e5e663f22094333dac6e184c78176ee658e3441e
Author: Josh Rosen <[email protected]>
Date: 2016-03-15T22:49:24Z
Modify BlockManager.dataSerialize to write ChunkedByteBuffers.
commit 035f227a93c2d69f03d0abdd5701245e1962a8f4
Author: Josh Rosen <[email protected]>
Date: 2016-03-08T07:36:33Z
Remove Serializer.getSerializer()
commit 35b32b3150327fc4cf50123211abfcef4d9bcedb
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T05:55:29Z
Wire up automatic serializer selection.
commit 876f038b56688357e90b0c5edbfaf6553587f1fd
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:15:27Z
Remove print statements.
commit f36e816283bd7a2c34e7b5193f5d161514e3e405
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:34:50Z
Merge remote-tracking branch 'origin/master' into
automatically-pick-best-serializer
commit ca923b5c368168c14581ccd055eb18e101b1dbdd
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:45:47Z
Add MiMa excludes.
commit de62f0d0a5f128dd91173e73b214a3297dd203d4
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:47:21Z
Merge remote-tracking branch 'origin/master' into
chunked-block-serialization
commit 0a347fdd9ec0e94eab17eb0f33c93acd1afbdcfb
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T06:56:02Z
Fix test compilation in streaming.
commit 51205ee3957c657aa19d06a5ed0c8d4afe6f2c39
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T08:00:33Z
More test compilation fixes.
commit 6852c482a4935b992c199810f1156952f1e93a8c
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T20:47:45Z
Merge remote-tracking branch 'origin/master' into
chunked-block-serialization
commit 43f8fa6ae5ba093655cdbd55ca56959a7652de56
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T20:54:55Z
Allow ChunkedByteBuffer to contain no chunks.
commit 25e68841541b45d7eedc0447cc8154d746ee8db2
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:00:21Z
Document toByteBuffer() and toArray() size limitations.
commit 325c83d8909472428ae65620033fff4887c36e06
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:07:42Z
Move dispose() from BlockManager to StorageUtils.
It was a static method before, but its location was confusing.
commit 4f5074ece49030a6e7134f7ece706ed441c02ee4
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:11:14Z
Better documentation for dispose() methods.
commit b6ddf3ed40cc90ec94b7e4917808f8a726b597ee
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:12:39Z
Rename limit to size.
commit 719ad3c4e9e942ce62cbcf288788aca785690a7e
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:20:08Z
Implement missing InputStream methods.
commit 09f533988aa223640b13debc2cb1701960e0ae3b
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:24:21Z
Update TaskResult.scala
commit 45b0c0be3791e518f0a8783951ad9b9e53196a1e
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T21:25:39Z
Merge remote-tracking branch 'origin/master' into
automatically-pick-best-serializer
commit 23006076dcb73095a9eaa7e2524a10c048bae646
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T22:00:10Z
More comments.
commit 3fc0b66981aa2d45be129986f0dc5bd595e08b22
Author: Josh Rosen <[email protected]>
Date: 2016-03-16T22:02:42Z
Fix confusing getChunks().head
commit ab7a81f069a15838512974eaf2d40e71ade27737
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T00:28:46Z
Fix incorrect commends in StorageLevle constructors.
commit 9c89779c7eebfac7763c9311c1f5c50f87d2f9e6
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T00:48:57Z
Update StorageLevel to support off-heap.
commit d4ffe399c4945c236b03d6192d85203ce965af58
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T01:44:20Z
WIP towards separate on- and off-heap storage memory pools
commit c4b94b7d9c7d37a76c72fe5b39155a3b1a3434c6
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T02:28:12Z
Allow chunked byte buffers to be allocated off-heap.
commit 69c9459e5994c6f3a7f11907978a89f0545dd4dc
Author: Josh Rosen <[email protected]>
Date: 2016-03-17T02:30:15Z
Fix some imports.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
