lhotari opened a new pull request, #24727: URL: https://github.com/apache/pulsar/pull/24727
### Motivation **Problem**: The current topic listing implementation lacks flow control, creating unbounded memory allocation scenarios: - OutOfMemoryError when multiple clients request large topic lists simultaneously - Proxy cascading failures due to unbuffered response forwarding - Unpredictable resource usage making capacity planning difficult - Performance degradation from GC pressure affecting all broker operations A namespace with 10K topics can consume ~1MB per response. With 1K concurrent requests, this creates ~1GB memory pressure that can crash brokers. **Solution**: PIP-442 introduces MaxTopicListInFlightLimiter - a memory-aware semaphore system: - Dual memory tracking for heap (topic assembly) and direct memory (network buffers) - Asynchronous flow control with configurable timeouts and queue limits - Permit-based system ensuring memory is released after response transmission - Graceful degradation instead of broker crashes **Benefits**: - Prevents broker crashes from topic listing memory exhaustion - Predictable resource usage for capacity planning - Maintains full backward compatibility (no client changes required) - Comprehensive monitoring with detailed metrics - Fair resource sharing through queueing mechanisms The implementation adds flow control at two key points: after topic retrieval from metadata store (heap) and before response serialization (direct memory). ### Modifications See pip/pip-442.md in this PR ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [x] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [ ] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
