[
https://issues.apache.org/jira/browse/FLINK-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16929262#comment-16929262
]
Andrey Zagrebin edited comment on FLINK-13985 at 9/13/19 2:53 PM:
------------------------------------------------------------------
Thanks for sharing the thoughts [~StephanEwen].
I looked into the issue and rough idea atm is that _MemorySegmentFactory_ gets
one more method to allocate _MemorySegment_ with _unsafe_.
We need one more concrete class _HybridUnsafeMemorySegment_:
MemorySegment << _HybridUnsafeMemorySegment_ << HybridMemorySegment
_HybridUnsafeMemorySegment_ is pretty much copy of _HybridMemorySegment_,
except it does not have _ByteBuffer offHeapBuffer_ and returns _ByteBuffer_
from _getOffHeapBuffer_ and wrap methods in a different way.
There is no straightforward way to wrap unsafe memory with _ByteBuffer_. I
found [netty does this|#L469] by hacking into the private constructor of
_DirectByteBuffer_. This is possible if it is available in JVM (not sure why it
might be not).
For the mentioned aspects:
- {color:#57d9a3}Add {{long allocate()}} and {{void release(long)}} to a core
util, like {{MemoryUtils}} to abstract over Unsafe, so that it is easier to
deal with the Unsafe removal later{color}: as I see we can extend
_MemorySegmentFactory._
- {color:#57d9a3}Release the memory in the {{free()}} method of the memory
segments.{color} Good point, that was also my plan for
_HybridUnsafeMemorySegment_.
- {color:#57d9a3}Add a safety net that frees the memory segments upon garbage
collection. Have a look at how the DirectByteBuffer does it with the cleaners
and reference queues.{color} Good point I will have a look into the details
was (Author: azagrebin):
Thanks for sharing the thoughts [~StephanEwen].
I looked into the issue and rough idea atm is that _MemorySegmentFactory_ gets
one more method to allocate _MemorySegment_ with _unsafe_.
We need one more concrete class _HybridUnsafeMemorySegment_:
MemorySegment << _HybridUnsafeMemorySegment_ << HybridMemorySegment
_HybridUnsafeMemorySegment_ is pretty much copy of _HybridMemorySegment_,
except it does not have _ByteBuffer offHeapBuffer_ and returns _ByteBuffer_
from _getOffHeapBuffer_ and wrap methods in a different way.
There is no straightforward way to wrap unsafe memory with _ByteBuffer_. I
found [netty does this|#L469] by hacking into the private constructor of
_DirectByteBuffer_. This is possible if it is available in JVM (not sure why it
might be not).
For the mentioned aspects:
- {color:#57d9a3}Add {{long allocate()}} and {{void release(long)}} to a core
util, like {{MemoryUtils}} to abstract over Unsafe, so that it is easier to
deal with the Unsafe removal later{color}: as I see we can extend
_MemorySegmentFactory._
- {color:#57d9a3}Release the memory in the {{free()}} method of the memory
segments. {color:#172b4d}Good point, that was also my plan for
_HybridUnsafeMemorySegment_{color}.{color}
- {color:#57d9a3}Add a safety net that frees the memory segments upon garbage
collection. Have a look at how the DirectByteBuffer does it with the cleaners
and reference queues.{color} Good point I will have a look into the details
> Use native memory for managed memory.
> -------------------------------------
>
> Key: FLINK-13985
> URL: https://issues.apache.org/jira/browse/FLINK-13985
> Project: Flink
> Issue Type: Sub-task
> Reporter: Xintong Song
> Assignee: Andrey Zagrebin
> Priority: Major
>
> * Allocate memory with {{Unsafe.allocateMemory}}
> ** {{MemoryManager}}
> Implement this issue in common code paths for the legacy / new mode. This
> should only affect the GC behavior.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
