danielsinai opened a new issue #9562: URL: https://github.com/apache/pulsar/issues/9562
After some load testing on my pulsar cluster, thr maxMessageBufferSizeInMB configuration does not protect the broker from being OOMED, im publishing to a single topic about 150 mb/s (5mb message size) and my direct memory is going up until the broker crashed because of bookie disk latencies, I would expect some throttling on the producers side using this configuration released in 2.5.1 https://github.com/apache/pulsar/pull/6178 I allocate 8 gb of direct memory to the broker and I configured it to a range of 1-4096 Steps to reproduce the behavior: 1. Use the default maxMessageBufferSizeInMB (1/2 of direct memory) 2. Load the cluster with 5mb messages untill bookies disk have high latencies 3. wait for broker to crash I would expect producers to get throttled untill the broker does succeefully get acks from the qa of bookies.  The holes in the graphs describes when the broker failed and also when I tried to tune this configuration - OS: Centos 7.8 You can see more details about it in this slack thread https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1612886327337200?thread_ts=1612886327.337200&cid=C5Z4T36F7 Btw, it can also happen with small messages ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
