e-marchand-exensa opened a new issue #13712:
URL: https://github.com/apache/pulsar/issues/13712
**Describe the bug**
Just trying to upgrade from Pulsar 2.6.4 to 2.8.2, running into this
exception very quickly on a standalone test (from scratch, so no backlog or
anything).
```
WAR|11/145138.238 o.a.p.c.i.ProducerImpl@-client-io-68-10
[persistent://<tenant>/<namespace>/<topic>] [<producerName>] error while create
opSendMsg by batch message container
java.lang.IllegalStateException: Overflow detected
at
org.apache.pulsar.shade.io.airlift.compress.zstd.Util.checkState(Util.java:59)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.BitOutputStream.close(BitOutputStream.java:85)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compressSingleStream(HuffmanCompressor.java:130)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.HuffmanCompressor.compress4streams(HuffmanCompressor.java:75)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.encodeLiterals(ZstdFrameCompressor.java:333)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressBlock(ZstdFrameCompressor.java:224)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compressFrame(ZstdFrameCompressor.java:172)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.ZstdFrameCompressor.compress(ZstdFrameCompressor.java:145)
at
org.apache.pulsar.shade.io.airlift.compress.zstd.ZStdRawCompressor.compress(ZStdRawCompressor.java:27)
at
org.apache.pulsar.common.compression.CompressionCodecZstd.encode(CompressionCodecZstd.java:65)
at
org.apache.pulsar.client.impl.BatchMessageContainerImpl.getCompressedBatchMetadataAndPayload(BatchMessageContainerImpl.java:135)
at
org.apache.pulsar.client.impl.BatchMessageContainerImpl.createOpSendMsg(BatchMessageContainerImpl.java:189)
at
org.apache.pulsar.client.impl.ProducerImpl.batchMessageAndSend(ProducerImpl.java:1796)
at
org.apache.pulsar.client.impl.ProducerImpl.lambda$connectionOpened$13(ProducerImpl.java:1457)
at
org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:53)
at
org.apache.pulsar.shade.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at
org.apache.pulsar.shade.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:176)
at
org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at
org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
```
and consumer received a erroneous message.
I changed nothing to the code and/or the configuration, so I was using
batching and ZSTD compression with version 2.6.4.
**To Reproduce**
I'm not sure how to reproduce, here is the configuration for the producer:
```
builder
.producerName( producerName )
.topic( topic )
.enableBatching( true )
.batchingMaxMessages( 256 )
.batchingMaxPublishDelay( 150, TimeUnit.MILLISECONDS )
.blockIfQueueFull( true )
.maxPendingMessages( 1024 )
.maxPendingMessagesAcrossPartitions( 2 * 1024 )
.sendTimeout( 60000, TimeUnit.MILLISECONDS )
.compressionType( CompressionType.ZSTD );
```
The producer may be used by 16 threads concurrently. Individual messages are
small.
I will try with a different compression if possible.
**Expected behavior**
Should not fail to compress the message.
**Desktop (please complete the following information):**
- OS: Ubuntu 20.04 / Java 11.0.13 2021-10-19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
