nodece commented on PR #25061: URL: https://github.com/apache/pulsar/pull/25061#issuecomment-3644832786
> The optimal solution would be to acquire tokens based on what consumers would be available to read and do an adjustment after sending the entries. In the adjustment phase, it would be possible to consume or return tokens. The previous implementation already followed your approach: https://github.com/apache/pulsar/blob/v4.1.2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java#L228-L231 This PR changes that behavior by consuming tokens before sending entries, so that multiple dispatchers cannot use the same quota concurrently. > This would address the concerns that I have about performance impacts when entries are read and then dropped before sending out to consumers. Some performance impact is expected once a dispatch rate limit is configured. Similar to the producer rate limiter, the broker may need to pause processing message. From the broker's perspective, various settings such as read batch sizes can mask the overshoot, so Prometheus may still show values that appear aligned with the configured rate. However, this alignment is influenced by several layers of constraints, not strictly by the limiter. My goal is to ensure the code path itself behaves rigorously so that the limiter remains accurate even under concurrency. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
