[
https://issues.apache.org/jira/browse/FLINK-18695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17184963#comment-17184963
]
Zhijiang commented on FLINK-18695:
----------------------------------
If I understood correctly, there seems three options to unblock netty version
upgrade to support allocate heap memory:
* 1. NettyBufferPool#heapBuffer() to just take the internal implementation of
`#directBuffer()` instead, as [~chesnay] mentioned above.
* 2. `#heapBuffer()` takes the original internal implementation and we need to
assign the same amount of heap arenas as did for direct arenas.
* 3. `#heapBuffer()` allocates the temporary ByteBuf to bypass the netty
internal memory management without heap arenas.
The second option might enlarge the total heap memory overhead because of
introducing heap arenas, and the ssl engine seems only need less heap memory in
practice.
The third option might reduce the total heap memory overhead compared with
second option, but it might be not friendly for GC. And it might bring
potential risks in future if the heap memory allocation is frequent in other
parts.
Therefore I am a bit preferring the first option since it can make use of
exiting direct arenas to avoid extra heap memory overhead considered in
framework. And after upgrading the netty version, the effect should be tiny
even ignored because the previous direct memory usage caused by ssl has no
changes and the new involved direct memory overhead(heap -> direct) seems
really small by [~gaoyunhaii]'s testing results.
> Allow NettyBufferPool to allocate heap buffers
> ----------------------------------------------
>
> Key: FLINK-18695
> URL: https://issues.apache.org/jira/browse/FLINK-18695
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Chesnay Schepler
> Assignee: Yun Gao
> Priority: Major
> Fix For: 1.12.0
>
>
> in 4.1.43 netty made a change to their SslHandler to always use heap buffers
> for JDK SSLEngine implementations, to avoid an additional memory copy.
> However, our {{NettyBufferPool}} forbids heap buffer allocations.
> We will either have to allow heap buffer allocations, or create a custom
> SslHandler implementation that does not use heap buffers (although this seems
> ill-adviced?).
> /cc [~sewen] [~uce] [~NicoK] [~zjwang] [~pnowojski]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
