MrThreepwood opened a new issue #12262:
URL: https://github.com/apache/pulsar/issues/12262
**Describe the bug**
When attempting to batch process messages from a topic, no messages are
processed when a single message is larger than the max payload size.
**To Reproduce**
Steps to reproduce the behavior:
Create a batch consumer with a batch policy that has a non zero max byte
amount.
**Expected behavior**
Either a single messages is delivered or an error is thrown.
**Desktop (please complete the following information):**
- OS: linux server
**Additional context**
We're trying to find a way to safely consume multiple messages from a queue
of items with highly variable size. Because the pulsar client does not allow us
to determine information about the next message (or even if it exists), the
best we can do is ask for the next X messages. While the batch receive looked
ideal, as we could limit on number of messages and message size, we ran into an
issue. If all workers attempted to grab 1MB of data across our 1kish
subscriptions, we would run OOM. If we set the size smaller than that, we
simply stop processing messages on some queues alltogether, with absolutely no
errors/observability on why.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
