[
https://issues.apache.org/jira/browse/HDDS-9032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duong reassigned HDDS-9032:
---------------------------
Assignee: Duong
> CodecBuffer results in Ozone Datanode using 2 separate Netty memory pool
> instances
> ----------------------------------------------------------------------------------
>
> Key: HDDS-9032
> URL: https://issues.apache.org/jira/browse/HDDS-9032
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Duong
> Assignee: Duong
> Priority: Major
>
> Ozone datanode today uses 2 separated Netty memory pool (or
> PooledByteBufAllocator) instances.
> {code:java}
> [root@myserver ~]# /usr/java/jdk-11.0.17/bin/jmap -histo 33437 | grep
> PooledByteBufAllocator
> 2249: 2 128
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator
> 3564: 2 48
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator$PoolThreadLocalCache
> 4181: 2 32
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator$1
> 4182: 2 32
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocatorMetric
> {code}
> First pool instance is created with NettyServer (used by Ratis server and
> Replication server). All NettyServer instances share the same
> PooledByteBufAllocator instances (or the same direct memory pool) which is
> created and cached by ByteBufAllocatorPreferDirectHolder.allocator. This
> resolves to the usage of "io.grpc.netty.Utils#getByteBufAllocator".
> {code:java}
> public static ByteBufAllocator getByteBufAllocator(boolean forceHeapBuffer) {
> if (Boolean.parseBoolean(
>
> System.getProperty("org.apache.ratis.thirdparty.io.grpc.netty.useCustomAllocator",
> "true"))) {
> boolean defaultPreferDirect =
> PooledByteBufAllocator.defaultPreferDirect();
> logger.log(
> Level.FINE,
> String.format(
> "Using custom allocator: forceHeapBuffer=%s,
> defaultPreferDirect=%s",
> forceHeapBuffer,
> defaultPreferDirect));
> if (forceHeapBuffer || !defaultPreferDirect) {
> return ByteBufAllocatorPreferHeapHolder.allocator;
> } else {
> return ByteBufAllocatorPreferDirectHolder.allocator;
> } {code}
> The second instance is created from the usage of
> [CodecBuffer|https://github.com/apache/ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/db/CodecBuffer.java#L85-86].
> This CodecBuffer uses the default Netty memory pool, created and cached by
> PooledByteBufAllocator.DEFAULT, to create temporary caches to decode/encode
> data (from/to storage like RocksDb or network).
> {code:java}
> private static final ByteBufAllocator POOL
> = PooledByteBufAllocator.DEFAULT; {code}
>
> Netty has a decent set of config to ensure its memory pool usage doesn't
> exceed the JVM limits, aka, maxMemory and maxDirectMemory. However, as
> there're multiple pool instances exists, Ozone service instances are prone to
> OutOfMemoryError.
> Below are some examples of OOME we've seen from Netty.
> E1: from sending container for re-replication.
> {code:java}
> Exception in thread "ContainerReplicationThread-2"
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at
> java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.java:139)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:230)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:169)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:142)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:65)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ForwardingClientStream.writeMessage(ForwardingClientStream.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.DelayedStream.writeMessage(DelayedStream.java:278)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.RetriableStream.sendMessage(RetriableStream.java:545)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.sendMessageInternal(ClientCallImpl.java:521)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.sendMessage(ClientCallImpl.java:507)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall.sendMessage(DelayedClientCall.java:324)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$CallToStreamObserverAdapter.onNext(ClientCalls.java:374)
> at
> org.apache.hadoop.ozone.container.replication.SendContainerOutputStream.sendPart(SendContainerOutputStream.java:46)
> at
> org.apache.hadoop.ozone.container.replication.GrpcOutputStream.flushBuffer(GrpcOutputStream.java:136)
> at
> org.apache.hadoop.ozone.container.replication.GrpcOutputStream.write(GrpcOutputStream.java:96)
> at
> org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:92)
> at
> org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
> at
> org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:244)
> at
> org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:92)
> at
> org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:86)
> at
> org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:122)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:462)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
> at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.includeFile(TarContainerPacker.java:265)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.includePath(TarContainerPacker.java:255)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.pack(TarContainerPacker.java:167)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.packContainerToDestination(KeyValueContainer.java:940)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.exportContainerData(KeyValueContainer.java:650)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.exportContainer(KeyValueHandler.java:1046)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.ContainerController.exportContainer(ContainerController.java:167)
> at
> org.apache.hadoop.ozone.container.replication.OnDemandContainerReplicationSource.copyData(OnDemandContainerReplicationSource.java:62)
> at
> org.apache.hadoop.ozone.container.replication.PushReplicator.replicate(PushReplicator.java:67)
> at
> org.apache.hadoop.ozone.container.replication.MeasuredReplicator.replicate(MeasuredReplicator.java:83)
> at
> org.apache.hadoop.ozone.container.replication.ReplicationTask.runTask(ReplicationTask.java:122)
> at
> org.apache.hadoop.ozone.container.replication.ReplicationSupervisor$TaskRunner.run(ReplicationSupervisor.java:357)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Suppressed: java.io.IOException: This archive contains unclosed
> entries.
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:291)
> at
> org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:309)
> at
> org.apache.hadoop.ozone.container.keyvalue.TarContainerPacker.pack(TarContainerPacker.java:169)
> {code}
> E2: From processing readChunk command (Ratis)
> {code:java}
> 2023-07-16 23:07:11,132 [ChunkReader-14] ERROR
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService:
> Got exception when processing ContainerCommandRequestProto cmdType: ReadChunk
> traceID: ""
> containerID: 10047
> datanodeUuid: "eb5580f7-03cf-4b14-a19e-8c350fb90a6e"
> readChunk {
> blockID {
> containerID: 10047
> localID: 111677748019225467
> blockCommitSequenceId: 0
> }
> chunkData {
> chunkName: "111677748019225467_chunk_13"
> offset: 12582912
> len: 1048576
> checksumData {
> type: CRC32
> bytesPerChecksum: 1048576
> checksums: "\334}w\374"
> }
> }
> readChunkVersion: V1
> }
> encodedToken:
> "VgoCZG4SJmNvbklEOiAxMDA0NyBsb2NJRDogMTExNjc3NzQ4MDE5MjI1NDY3GJjz7byWMSgBKAIoBDCc2Kp_OhYIg4_B_obNkNywARDIgtLMovHGp4IBIHmmC2gLI4CkpaO54_a4aoIAqmeczb5AmGk9RWEurGBHEEhERFNfQkxPQ0tfVE9LRU4mY29uSUQ6IDEwMDQ3IGxvY0lEOiAxMTE2Nzc3NDgwMTkyMjU0Njc"
> java.lang.OutOfMemoryError: Direct buffer memory
> at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
> at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
> at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:701)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:676)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:215)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.tcacheAllocateNormal(PoolArena.java:197)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.java:139)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PoolArena.allocate(PoolArena.java:129)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:396)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
> at
> org.apache.ratis.thirdparty.io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:230)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:169)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:142)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:65)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl.sendMessageInternal(ServerCallImpl.java:172)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:154)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:380)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:58)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
> at
> org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834){code}
>
> {*}Solution{*}: CodecBuffer should use the same memory pool from
> io.grpc.netty.Utils#getByteBufAllocator.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
