[
https://issues.apache.org/jira/browse/RATIS-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17899928#comment-17899928
]
Ivan Andika commented on RATIS-2189:
------------------------------------
[~szetszwo] The USED_DIRECT_MEM issue only happens when using Streaming.
Previously we were using the Ratis async API (write pipeline v1), but due to a
large volume of write chunks, we hit the bottleneck of the Ratis pipeline that
causes a lot of NotReplicatedException on the client. We decided to change to
write pipeline v2 since the size of Kafka segments are quite large (we set to
256 MB / 1 Ozone block) and to reduce the number of Ratis logs created.
cc: [~Symious]
> Use ByteBufAllocator#ioBuffer in NettyDataStreamUtils
> -----------------------------------------------------
>
> Key: RATIS-2189
> URL: https://issues.apache.org/jira/browse/RATIS-2189
> Project: Ratis
> Issue Type: Improvement
> Components: Streaming
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Minor
> Fix For: 3.2.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, NettyDataStreamUtils uses ByteBufAllocator#directBuffer which
> forces all ByteBufAllocator to allocate direct buffer even for
> PreferHeapByteBufAllocator (e.g. when we set
> -Dorg.apache.ratis.thirdparty.io.netty.noPreferDirect=true).
> It's better to use ioBuffer and delegates to the actual ByteBufAllocator to
> the type of memory it will use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
