anvinjain opened a new issue #11044:
URL: https://github.com/apache/pulsar/issues/11044
**Describe the bug**
When using byte rates, we see that dispatch rates are often not respected
(regardless of being a namespace or topic policy)
**To Reproduce**
This happens when dispatch rates are set on the lower side, say, 100KB/s
while dispatcherMaxReadSizeBytes=5MB and dispatcherMaxReadBatchSize=100 which
are defaults
On debugging, we discovered that dispatcherMaxReadSizeBytes and
dispatcherMaxReadBatchSize have an impact on the minimum dispatch rate which we
see. Here although the settings have word "max" in them, when it comes to
reading from bookkeeper (lagging consumers), these control the minimum data
read (in terms of entries or bytes) when byte rates are used instead of message
based limits in dispatch.
**Expected behavior**
Expected that dispatch rate = 100KB/s for a subscription, but saw that it
was 5MB/s.
**Desktop (please complete the following information):**
- OS: Debian 10
- Pulsar 2.7.1
**Additional context**
In the code, I see that messagesToRead is initialized with readBatchSize and
only reduced if message based rate limiting has been used:
https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L397
When byte rate limiting is used, the first read can be much larger than the
configured limit (say 5MB because of dispatcherMaxReadBatchSize vs limit of
1MB as configured in dispatch rate of namespace). Post the first read for that
second, rate limiter permits gets exhausted rightly so but the user effectively
sees 5MB/s dispatch. On the start of the next second again this happens where
the first read issued to bookkeeper is larger than the overall configured limit
and it exhausts the permits but effective rate/s remains to be 5MB
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
