eolivelli opened a new issue #9721:
URL: https://github.com/apache/pulsar/issues/9721
**Describe the bug**
```
14:34:25.058 [pulsar-client-io-10-1] INFO
org.apache.pulsar.common.allocator.PulsarByteBufAllocator - Exiting JVM process
for OOM error: Direct buffer memory
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[?:?]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
at
io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:758)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:734)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:245)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:227)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:147)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:356)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
~[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at
org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:164)
[org.apache.bookkeeper-bookkeeper-common-allocator-4.11.1.jar:4.11.1]
at
org.apache.bookkeeper.common.allocator.impl.ByteBufAllocatorImpl.newDirectBuffer(ByteBufAllocatorImpl.java:158)
[org.apache.bookkeeper-bookkeeper-common-allocator-4.11.1.jar:4.11.1]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
[io.netty-netty-buffer-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.unix.PreferredDirectByteBufAllocator.ioBuffer(PreferredDirectByteBufAllocator.java:53)
[io.netty-netty-transport-native-unix-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:75)
[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:777)
[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
```
**To Reproduce**
I am running pulsar-perf produce (on Pulsar 2.6.2, actually Luna Streaming
2.6.2_1.0.0) with very simple configuration, after 3 minutes it breaks.
```
[/pulsar/conf/client.conf] Applying config brokerServiceUrl =
pulsar://pulsar-proxy:6650/
[/pulsar/conf/client.conf] Applying config webServiceUrl =
http://pulsar-proxy:8080/
Warning: Nashorn engine is planned to be removed from a future JDK release
14:31:51.660 [main] INFO org.apache.pulsar.testclient.PerformanceProducer -
Starting Pulsar perf producer with config: {
"confFile" : "/pulsar/conf/client.conf",
"topics" : [ "persistent://public/default/test" ],
"numTestThreads" : 10,
"msgRate" : 10000,
"msgSize" : 1024,
"numTopics" : 10,
"numProducers" : 1,
"serviceURL" : "http://pulsar-proxy:8080";,
"authPluginClassName" : "",
"authParams" : "",
"maxOutstanding" : 1000,
"maxPendingMessagesAcrossPartitions" : 50000,
"maxConnections" : 100,
"numMessages" : 0,
"statsIntervalSeconds" : 2,
"compression" : "NONE",
"payloadFilename" : null,
"payloadDelimiter" : "\\n",
"batchTimeMillis" : 1.0,
"batchMaxMessages" : 1000,
"batchMaxBytes" : 4194304,
"testTime" : 3600,
"warmupTimeSeconds" : 1.0,
"tlsTrustCertsFilePath" : "",
"tlsAllowInsecureConnection" : false,
"encKeyName" : null,
"encKeyFile" : null,
"delay" : 0,
"exitOnFailure" : true,
"messageKeyGenerationMode" : null,
"ioThreads" : 1
}
```
Relevant logs on broker
```
14:34:25.236 [pulsar-io-23-5] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49994] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.236 [pulsar-io-23-6] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49964] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.236 [pulsar-io-23-7] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49966] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.236 [pulsar-io-23-1] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49976] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.236 [pulsar-io-23-8] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49972] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.237 [pulsar-io-23-3] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:50050] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.237 [pulsar-io-23-4] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:49956] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
14:34:25.237 [pulsar-io-23-2] WARN
org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.16:50044] Got exception
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed:
Connection reset by peer
```**Expected behavior**
No OOM error
It may sound like some memory leak in pulsar-perf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
