GitHub user yzotov opened a pull request:
https://github.com/apache/spark/pull/11494
[SPARK-10399][CORE][SQL] Introduce OffHeapMemoryBlock to hold
DirectByteBuffer, refactor spark.unsafe.Platform
## What changes were proposed in this pull request?
+ OffHeapMemoryBlock is introduced which holds reference to
java.nio.DirectByteBuffer.
This allows to have offheap memory which is automatically deallocated by
JVM.
+ spark.unsafe.Platform interface refactored from indefinite Objects, to
MemoryBlocks and arrays of primitives. This has led to large code changes in
core, catalyst, sql.
+ org.apache.spark.sql.catalyst.expressions.codege.BufferHolder is replaced
by MemoryBlockHolder which tests spark.memory.offHeap.enadled configuration
property and allocates ByteArrayMemoryBlock or OffHeapMemoryBlock.
## How was this patch tested?
mvn test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yzotov/spark use-offheap-memory
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11494.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11494
----
commit 7c7caa5daf24b3f5013f4b704d0af1f126123abd
Author: Zotov Yuriy <[email protected]>
Date: 2016-03-03T12:11:51Z
new classes
commit 5b96eebb56d333d25749a858a9ccbe876b9717e4
Author: Zotov Yuriy <[email protected]>
Date: 2016-03-03T12:18:37Z
changes to unsafe subsystem
commit d777b80a66934fb85952aa1abc19c345d6e208f1
Author: Zotov Yuriy <[email protected]>
Date: 2016-03-03T12:25:11Z
changes to spark-core
commit aeec194d0cd244fbb59b90eae43a0c64b6cd73ed
Author: Zotov Yuriy <[email protected]>
Date: 2016-03-03T12:26:04Z
spark.unsafe.Platform interface changed, BufferHolder is replaced by
MemoryBlockHolder, OffHeapMemoryBlock uses DirectByteBuffer for off-heap memory
allocation, UnsafeRow and others hold memory in MemoryBlocks instead of
indefinite Objects
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
