lhotari opened a new pull request, #23901: URL: https://github.com/apache/pulsar/pull/23901
Fixes #23504 Fixes #23506 ### Motivation In the current implementation before this PR there are multiple problems: * redelivery (replay) queue reads don't get deduplicated or limited by `managedLedgerMaxReadsInFlightSizeInMB`, reported as #23504 * The `managedLedgerMaxReadsInFlightSizeInMB` limit doesn't work properly if `managedLedgerReadEntryTimeoutSeconds` isn't set, reported as #23506. In addition to this, even if `managedLedgerReadEntryTimeoutSeconds` is set, when the limit is reached, there's no queuing and all threads will be retrying in a tight loop to acquire permits. This tight loop causes unnecessary CPU consumption and contention. ### Modifications - When the `managedLedgerMaxReadsInFlightSizeInMB` limit is reached, queue pending requests up to a configurable limit (`managedLedgerMaxReadsInFlightPermitsAcquireQueueSize`) and complete the waiting requests asynchronously to overcome the previous problem of acquiring permits in a tight loop. - When the limit isn't reached, there will be no queuing. This ensures that there will be minimal overhead of the limiting. - Acquire permits also for replay queue reads - Improve the estimated size calculation by using an average entry size (instead of size of a single latest result, as it is in the current implementation) ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
