lhotari commented on issue #25046: URL: https://github.com/apache/pulsar/issues/25046#issuecomment-3631976371
> [@lhotari](https://github.com/lhotari) Thank you so much for your help! When I ran pressure tests with 1KB small messages, the performance was solid and stable—only partition imbalance occurred, with a TPS of around 250K. However, when consuming 500KB large messages, performance degrades after a few minutes, paired with severe partition consumption imbalance. The most critical issue is that the consumption rate metrics for partitions vanish entirely: neither pulsar-manager API queries nor the pulsar_rate_out metric returns any data (this affects all brokers). Other metrics appear normal, and no thread blockages were detected in the broker thread dumps. Additionally, no priority distinctions were configured for the consumers. Have you configured `managedLedgerMaxReadsInFlightSizeInMB` to a reasonable value? The feature isn't enabled by default. It was added in 2.11 with #18245 and improved later in #23901 . There's some details about broker memory limits in PIP-442 https://github.com/apache/pulsar/blob/master/pip/pip-442.md#existing-broker-memory-management . If you haven't configured `managedLedgerMaxReadsInFlightSizeInMB`, the broker can get overloaded with a large amount of consumers. You can try setting it to a value like 500 MB when you have at least 4GB of direct memory available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
