Ludo created KAFKA-15947:
----------------------------
Summary: Null pointer on LZ4 compression since Kafka 3.6
Key: KAFKA-15947
URL: https://issues.apache.org/jira/browse/KAFKA-15947
Project: Kafka
Issue Type: Bug
Components: compression
Affects Versions: 3.6.0
Reporter: Ludo
I have a Kafka Stream application running well since month using client version
{{3.5.1 }}with 3.5.1 (bitnami image: {{bitnami/3.5.1-debian-11-r44)}} using{{
compression.type: "lz4"}}
I've recently updated a my kafka server to kafka 3.6 (bitnami image:
{{{}bitnami/kafka:3.6.0-debian-11-r0){}}}.
The startup is working well for days, and after some time, Kafka Stream crash
and Kafka output a lot of NullPointerException on the console:
{code:java}
org.apache.kafka.common.KafkaException: java.lang.NullPointerException: Cannot
invoke "java.nio.ByteBuffer.hasArray()" because "this.intermediateBufRef" is
null
at
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:134)
at
org.apache.kafka.common.record.DefaultRecordBatch.recordInputStream(DefaultRecordBatch.java:273)
at
org.apache.kafka.common.record.DefaultRecordBatch.compressedIterator(DefaultRecordBatch.java:277)
at
org.apache.kafka.common.record.DefaultRecordBatch.skipKeyValueIterator(DefaultRecordBatch.java:352)
at
org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsetsCompressed(LogValidator.java:358)
at
org.apache.kafka.storage.internals.log.LogValidator.validateMessagesAndAssignOffsets(LogValidator.java:165)
at kafka.log.UnifiedLog.$anonfun$append$2(UnifiedLog.scala:805)
at kafka.log.UnifiedLog.append(UnifiedLog.scala:1845)
at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:719)
at
kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1313)
at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1301)
at
kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1210)
at
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at
scala.collection.mutable.HashMap.$anonfun$foreach$1(HashMap.scala:149)
at scala.collection.mutable.HashTable.foreachEntry(HashTable.scala:237)
at scala.collection.mutable.HashTable.foreachEntry$(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:44)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:149)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
at scala.collection.AbstractTraversable.map(Traversable.scala:108)
at
kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1198)
at
kafka.server.ReplicaManager.$anonfun$appendRecords$18$adapted(ReplicaManager.scala:754)
at
kafka.server.KafkaRequestHandler$.$anonfun$wrap$3(KafkaRequestHandler.scala:73)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:130)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke
"java.nio.ByteBuffer.hasArray()" because "this.intermediateBufRef" is null
at
org.apache.kafka.common.utils.ChunkedBytesStream.<init>(ChunkedBytesStream.java:89)
at
org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:132)
... 25 more {code}
At the same time the Kafka Stream raise this error:
{code:java}
org.apache.kafka.streams.errors.StreamsException: Error encountered sending
record to topic kestra_workertaskresult for task 3_6 due
to:org.apache.kafka.common.errors.UnknownServerException: The server
experienced an unexpected error when processing the request.Written offsets
would not be recorded and no more records would be sent since this is a fatal
error.at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:297)at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:284)at
org.apache.kafka.clients.producer.KafkaProducer$AppendCallbacks.onCompletion(KafkaProducer.java:1505)at
org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)at
org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)at
org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)at
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:772)at
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:757)at
org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:709)at
org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:648)at
org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:589)at
java.base/java.util.ArrayList.forEach(Unknown Source)at
org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:576)at
java.base/java.lang.Iterable.forEach(Unknown Source)at
org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:576)at
org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:850)at
org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:154)at
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:594)at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:586)at
org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)at
java.base/java.lang.Thread.run(Unknown Source)Caused by:
org.apache.kafka.common.errors.UnknownServerException: The server experienced
an unexpected error when processing the request.
{code}
The error will loop on the Kafka Stream (pod are restarted, I supposed that
topic are consumed on the same offset, and crash with the same error on both
server and client).
Always, when I restart Kafka server, every goes well and I don't have any more
issues.
The error is mostly transitive and happen every few days without any others
solution than restart the server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
