[jira] [Updated] (KAFKA-3990) Kafka New Producer may raise an OutOfMemoryError
[ https://issues.apache.org/jira/browse/KAFKA-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brice Dutheil updated KAFKA-3990: - Environment: Docker, Base image : CentOS Java 8u77 Marathon was: Docker, Base image : CentOS Java 8u77 > Kafka New Producer may raise an OutOfMemoryError > > > Key: KAFKA-3990 > URL: https://issues.apache.org/jira/browse/KAFKA-3990 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.1 > Environment: Docker, Base image : CentOS > Java 8u77 > Marathon >Reporter: Brice Dutheil > Attachments: app-producer-config.log, kafka-broker-logs.zip > > > We are regularly seeing OOME errors on a kafka producer, we first saw : > {code} > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] > at > org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.Selector.poll(Selector.java:286) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > ~[kafka-clients-0.9.0.1.jar:na] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] > {code} > This line refer to a buffer allocation {{ByteBuffer.allocate(receiveSize)}} > (see > https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L93) > Usually the app runs fine within 200/400 MB heap and a 64 MB Metaspace. And > we are producing small messages 500B at most. > Also the error don't appear on the devlopment environment, in order to > identify the issue we tweaked the code to give us actual data of the > allocation size, we got this stack : > {code} > 09:55:49.484 [auth] [kafka-producer-network-thread | producer-1] WARN > o.a.k.c.n.NetworkReceive HEAP-ISSUE: constructor : Integer='-1', String='-1' > 09:55:49.485 [auth] [kafka-producer-network-thread | producer-1] WARN > o.a.k.c.n.NetworkReceive HEAP-ISSUE: method : > NetworkReceive.readFromReadableChannel.receiveSize=1213486160 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to /tmp/tomcat.hprof ... > Heap dump file created [69583827 bytes in 0.365 secs] > 09:55:50.324 [auth] [kafka-producer-network-thread | producer-1] ERROR > o.a.k.c.utils.KafkaThread Uncaught exception in kafka-producer-network-thread > | producer-1: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] > at > org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.Selector.poll(Selector.java:286) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > ~[kafka-clients-0.9.0.1.jar:na] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] > {code} > Notice the size to allocate {{1213486160}} ~1.2 GB. I'm not yet sure how this > size is initialised. > Notice as well that every time this OOME appear the {{NetworkReceive}} > constructor at > https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L49 > receive the parameters : {{maxSize=-1}}, {{source="-1"}} > We may have missed configuration in our setup but kafka clients shouldn't > raise
[jira] [Updated] (KAFKA-3990) Kafka New Producer may raise an OutOfMemoryError
[ https://issues.apache.org/jira/browse/KAFKA-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brice Dutheil updated KAFKA-3990: - Attachment: app-producer-config.log kafka-broker-logs.zip > Kafka New Producer may raise an OutOfMemoryError > > > Key: KAFKA-3990 > URL: https://issues.apache.org/jira/browse/KAFKA-3990 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.1 > Environment: Docker, Base image : CentOS > Java 8u77 >Reporter: Brice Dutheil > Attachments: app-producer-config.log, kafka-broker-logs.zip > > > We are regularly seeing OOME errors on a kafka producer, we first saw : > {code} > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] > at > org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.Selector.poll(Selector.java:286) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > ~[kafka-clients-0.9.0.1.jar:na] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] > {code} > This line refer to a buffer allocation {{ByteBuffer.allocate(receiveSize)}} > (see > https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L93) > Usually the app runs fine within 200/400 MB heap and a 64 MB Metaspace. And > we are producing small messages 500B at most. > Also the error don't appear on the devlopment environment, in order to > identify the issue we tweaked the code to give us actual data of the > allocation size, we got this stack : > {code} > 09:55:49.484 [auth] [kafka-producer-network-thread | producer-1] WARN > o.a.k.c.n.NetworkReceive HEAP-ISSUE: constructor : Integer='-1', String='-1' > 09:55:49.485 [auth] [kafka-producer-network-thread | producer-1] WARN > o.a.k.c.n.NetworkReceive HEAP-ISSUE: method : > NetworkReceive.readFromReadableChannel.receiveSize=1213486160 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to /tmp/tomcat.hprof ... > Heap dump file created [69583827 bytes in 0.365 secs] > 09:55:50.324 [auth] [kafka-producer-network-thread | producer-1] ERROR > o.a.k.c.utils.KafkaThread Uncaught exception in kafka-producer-network-thread > | producer-1: > java.lang.OutOfMemoryError: Java heap space > at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] > at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] > at > org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) > ~[kafka-clients-0.9.0.1.jar:na] > at > org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.common.network.Selector.poll(Selector.java:286) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) > ~[kafka-clients-0.9.0.1.jar:na] > at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) > ~[kafka-clients-0.9.0.1.jar:na] > at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] > {code} > Notice the size to allocate {{1213486160}} ~1.2 GB. I'm not yet sure how this > size is initialised. > Notice as well that every time this OOME appear the {{NetworkReceive}} > constructor at > https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L49 > receive the parameters : {{maxSize=-1}}, {{source="-1"}} > We may have missed configuration in our setup but kafka clients shouldn't > raise an OOME. For reference the producer is
[jira] [Updated] (KAFKA-3990) Kafka New Producer may raise an OutOfMemoryError
[ https://issues.apache.org/jira/browse/KAFKA-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brice Dutheil updated KAFKA-3990: - Description: We are regularly seeing OOME errors on a kafka producer, we first saw : {code} java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.Selector.poll(Selector.java:286) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) ~[kafka-clients-0.9.0.1.jar:na] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] {code} This line refer to a buffer allocation {{ByteBuffer.allocate(receiveSize)}} (see https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L93) Usually the app runs fine within 200/400 MB heap and a 64 MB Metaspace. And we are producing small messages 500B at most. Also the error don't appear on the devlopment environment, in order to identify the issue we tweaked the code to give us actual data of the allocation size, we got this stack : {code} 09:55:49.484 [auth] [kafka-producer-network-thread | producer-1] WARN o.a.k.c.n.NetworkReceive HEAP-ISSUE: constructor : Integer='-1', String='-1' 09:55:49.485 [auth] [kafka-producer-network-thread | producer-1] WARN o.a.k.c.n.NetworkReceive HEAP-ISSUE: method : NetworkReceive.readFromReadableChannel.receiveSize=1213486160 java.lang.OutOfMemoryError: Java heap space Dumping heap to /tmp/tomcat.hprof ... Heap dump file created [69583827 bytes in 0.365 secs] 09:55:50.324 [auth] [kafka-producer-network-thread | producer-1] ERROR o.a.k.c.utils.KafkaThread Uncaught exception in kafka-producer-network-thread | producer-1: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57) ~[na:1.8.0_77] at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[na:1.8.0_77] at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.Selector.poll(Selector.java:286) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128) ~[kafka-clients-0.9.0.1.jar:na] at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_77] {code} Notice the size to allocate {{1213486160}} ~1.2 GB. I'm not yet sure how this size is initialised. Notice as well that every time this OOME appear the {{NetworkReceive}} constructor at https://github.com/apache/kafka/blob/0.9.0.1/clients/src/main/java/org/apache/kafka/common/network/NetworkReceive.java#L49 receive the parameters : {{maxSize=-1}}, {{source="-1"}} We may have missed configuration in our setup but kafka clients shouldn't raise an OOME. For reference the producer is initialised with : {code} Properties props = new Properties(); props.put(BOOTSTRAP_SERVERS_CONFIG, properties.bootstrapServers); props.put(ACKS_CONFIG, "ONE"); props.put(RETRIES_CONFIG, 0); props.put(BATCH_SIZE_CONFIG, 16384); props.put(LINGER_MS_CONFIG, 0); props.put(BUFFER_MEMORY_CONFIG, 33554432); props.put(REQUEST_TIMEOUT_MS_CONFIG, 1000); props.put(MAX_BLOCK_MS_CONFIG, 1000); props.put(KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put(VALUE_SERIALIZER_CLASS_CONFIG, JSONSerializer.class.getName()); {code} For reference while googling for the issue we found a similar stack trace with the new