keyboardbobo opened a new issue #14943:
URL: https://github.com/apache/pulsar/issues/14943
**Is your enhancement request related to a problem? Please describe.**
Guys, during the stress test, we found that the client's memory is very
large, and even Full gc appeared. After analyzing the dump file, we found that
the memory space occupied is much larger than the actual message, and the
serialization of 1KB of messages takes up 1MB.
```
batchedMessageMetadataAndPayload = PulsarByteBufAllocator.DEFAULT
.buffer(Math.min(maxBatchSize,
ClientCnx.getMaxMessageSize()));
```
`maxBatchSize = Math.max(maxBatchSize, uncompressedSize);`
Debugging found that the maxbatchsize that controls the pre-allocated
`ByteBufPair.b2` memory size is stateful. As the size of the largest batch or
the largest single message grows, this may cause the pre-allocated
`ByteBufPair.b2` memory to grow larger and larger. , which is thousands of
times larger than the payload of MessageImpl
**Describe the solution you'd like**
It is best to loop through the messages to be packed to accurately calculate
the memory size to be allocated
**Describe alternatives you've considered**
Let the user choose whether to precisely allocate or pre-allocate
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
