rdhabalia opened a new pull request #12294: URL: https://github.com/apache/pulsar/pull/12294
### Motivation We introduced dispatch throttling as part of [PIP-3](https://github.com/apache/pulsar/pull/634) which helps brokers to control fan-out and prevent broker's instability due to large message dispatch. However, current throttling applies rate-limiting based on a number of messages within batch-message. Because of that overall, broker dispatches less number of messages even though the broker has enough bandwidth if the given batch message has a large number of messages eg: 100 or 1000 msgs per batch. Also, various topics under a namespace have messages with various batch sizes and it makes unpredictable throttling for all topics under a namespace and it causes backlog building for some of the topics. Therefore, the broker should have a configuration to count a number of batch messages rather than total messages within each batch message during rate limiting. ### Modification Add configuration `dispatchThrottlingOnBatchMessageEnabled` to support rate-limiting dispatching on the batch messages rather than individual messages within batch messages. By default it will be disabled so, it won't change the existing behavior of the broker. ### Result Avoid backlog building for topics in a namespace with different sizes of batch messages during dispatch rate-limiting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
