soumya-ghosh commented on issue #23920: URL: https://github.com/apache/pulsar/issues/23920#issuecomment-2631436117
Some observations from heap which was captured on the broker of Pulsar cluster exhibiting this issue. `org.apache.pulsar.broker.service.ServerCnx$PendingBytesPerThreadTracker` - `pendingBytes` and `limitExceeded` field values  `org.apache.pulsar.broker.service.ServerCnxThrottleTracker` instance fields  `org.apache.pulsar.broker.service.PublishRateLimiterImpl` --> `unthrottlingQueue` `MpscUnboundedArrayQueue` <img width="975" alt="Image" src="https://github.com/user-attachments/assets/5f8f8773-de99-4f8a-b875-3fd8b539c33e"; /> We observed high negative value for tokens for the PublishRateLimiterImpl object which have non-zero value for throttledProducersCount `SELECT * FROM org.apache.pulsar.broker.service.PublishRateLimiterImpl WHERE (throttledProducersCount.value > 0) and tokenBucketOnByte.tokens < 0` <img width="1296" alt="Image" src="https://github.com/user-attachments/assets/90f0cf74-1d43-4531-8474-d448c90031b6"; /> We will try to capture some more heap dumps once issue occurs again and confirm this above observations. Thanks @lhotari for assisting in heap dump analysis. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
