nodece commented on PR #25061: URL: https://github.com/apache/pulsar/pull/25061#issuecomment-3646322547
I understand that pre-estimating tokens and adjusting afterwards is an ideal model to reduce extra work. However, the issue addressed here is a concurrency problem: multiple dispatchers sharing a global limiter can observe the same available tokens and cause deterministic overshoot. This PR consumes tokens before sending entries, preventing multiple dispatchers from using the same quota concurrently. While this may introduce minor delays, enforcing the configured rate is necessary, and this approach ensures correctness under concurrency. Pre-estimating tokens could be explored as a future optimization. > Dispatchers have a slow start. They start by dispatching one entry (`dispatcherMinReadBatchSize=1`) at a time. Regarding the dispatch batch size: it should be `dispatcherMaxReadBatchSize = 100`, not `dispatcherMinReadBatchSize = 1`. > Does the problem really exist that you are describing? What kind of impact is really observed? I am using the ResourceGroup limiter, which is a global limiter. When multiple topics share this limiter, deterministic overshoot is observable. The traffic may return to normal in the next second, but I don't want to cause too much fluctuation. I will discuss with my team further, as there is indeed a trade-off between strict rate enforcement and performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
