eolivelli opened a new pull request, #16545: URL: https://github.com/apache/pulsar/pull/16545
Fixes #7623 ### Motivation Currently there is no way to have a accurate backlog for a subscription, the main problem reported in #7623 but PIP-105 Server side filters adds more weight to that request. It is nearly impossible to predict the number of messages that will be dispatched by a Subscription with server side filtering. The idea of this patch is to provide a dedicate API (REST, pulsar-admin, and Java PulsarAdmin) to "analise" a subscription and provide detailed information about that is expected to be delivered to Consumers. The operation is expensive because we have to load the messages from storage and pass them to the filters, but due to the dynamic nature of Pulsar subscriptions there is no other way to have this value. One good strategy to do monitoring/alerting is to setup alerts on the usual "stats" and use this new API to inspect the subscription deeper, typically my issuing a manual command. ### Modifications - Add a ManagedCursor.scan() API to allow scanning a Cursor from the lastMark position up the end of the Cursor - Add a REST API to perform the scan - the Scan considers Server Side filtering, and it is able to report very accurate numbers Follow up works: - add a limit on the number of entries processed or a time bound ### Verifying this change This change added tests ### Does this pull request potentially affect one of the following parts: It is only a addition of new feature, existing features are not touched. ### Documentation The new command is automatically documented for the REST API and for pulsar-admin -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
