berg223 commented on code in PR #24400: URL: https://github.com/apache/pulsar/pull/24400#discussion_r2149027867
########## pip/pip-426.md: ########## @@ -0,0 +1,198 @@ + +# PIP-426: Enhanced Consumer Throttling and Unacknowledged Message Tracking for Exclusive and Failover Subscriptions + +# Background knowledge + +In [13618](https://github.com/apache/pulsar/pull/48), we have added a configuration named maxUnackedMessagesPerConsumer to restrict consumer for receiving messages without acknowledging-msg up to the threshold. + +# Motivation + +Apache Pulsar currently lacks full support for enforcing unacknowledged message limits and consumer-side flow control in exclusive and failover subscriptions. While these mechanisms function correctly for shared subscriptions, their absence in exclusive/failover modes causes critical limitations: + +1. **Throttling Inaccuracies** + Consumers can exceed configured `maxUnackedMessagesOnConsumer` limits, risking resource exhaustion. + +2. **Tracking Deficiencies** + `pendingAcks` tracking is disabled or partially functional, compromising visibility into message processing. + +3. **Inaccurate unacked message count** + Current implementation lacks support for: + - Cumulative acknowledgements + - Message batching + - Transactional operations + + +This proposal addresses these gaps by extending the `pendingAcks` system to support all subscription types. + +# Goals + +## In Scope + +1. **Strict Throttling Enforcement** + Apply `maxUnackedMessagesOnConsumer` limits to exclusive and failover subscriptions. + +2. **Unified Tracking Mechanism** + Enable `pendingAcks` for accurate unacknowledged message tracking across all subscription types. + +3. **Accurate unacked message count** + Support cumulative acknowledgements, message batching, and transactional operations. + +4. **Feature Flag** + Add a feature flag to enable/disable the new feature. + +## Out of Scope +- No guarantee that messages with the same key are delivered and allowed to be in unacknowledged state to only one consumer at a time in exclusive/failover subscriptions. +- No changes to the existing public API or wire protocol. +- No change in behavior for shared subscriptions. + +# High Level Design + +1. Remove the `Subscription.isIndividualAckMode()` restriction that limits `pendingAcks` usage to shared subscriptions. +2. Extend `PersistentAcknowledgmentsGroupingTracker` for exclusive/failover consumers. +3. Enhance flow control in `PersistentDispatcherSingleActiveConsumer` + +# Detailed Design + +## Design & Implementation Details Review Comment: Thanks for more background knowledge! It's really helpful to solve origin issue #24159 . Since there is no need to add flow permits to exlusive or failover subscription (details discussion see: https://lists.apache.org/thread/glvd8lrvyll9mdtp62d34x1k38swrls1), there is no motivation except fix the origin issue. IMO, we can close this pip to keep the pip from too much useless information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
