lhotari commented on PR #25239: URL: https://github.com/apache/pulsar/pull/25239#issuecomment-3890574906
Sharing some thoughts around publish throttling / backpressure / flow control: In Pulsar, there's already a solution to limit publishing with memory backpressure. `maxMessagePublishBufferSizeInMB` is set to 50% of direct memory by default, which might be a too high of a default. The downside of the current solution is that the limit is a fixed limit and it doesn't take existing other direct memory consumption into account since that might vary. Another property is that it's not possible to prioritize specific tenants/namespaces with the existing solution. One of the problems with existing publish throttling is that there's no explicit flow control in the Pulsar protocol and publish throttling happens by pausing reads on the TCP/IP connection. This solution impacts all producers and consumers sharing the same TCP/IP connection. One of the improvement directions would be to introduce explicit producer flow control. https://pulsar.apache.org/blog/2024/10/24/announcing-apache-pulsar-4-0/#enhanced-quality-of-service-qos-controls What is currently missing from Pulsar is the ability to explicitly prioritize tenants/namespaces/topics and have dynamic limits instead of fixed rate limits. This would be useful in cases where an overall broker limit is hit. Pulsar's "Resource Groups" ([PIP-82](https://github.com/apache/pulsar/blob/master/pip/pip-82.md)) could be further developed to the direction what Confluent Kora has with the [dynamic tenant-level quotas](https://jack-vanlightly.com/analyses/2023/11/14/kora-serverless-kafka-asds-chapter-2#:~:text=Dynamic%20tenant%2Dlevel%20quotas) feature. Regarding this PR, I don't currently understand what essential benefit it brings over the existing `maxMessagePublishBufferSizeInMB` solution and existing solution for limiting the per-topic backlog. Explaining the benefit and impact of this PR would be useful in justifying the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
