codelipenghui opened a new issue #5751: [DISCUSS] Avoid broker OOM URL: https://github.com/apache/pulsar/issues/5751 **Is your feature request related to a problem? Please describe.** Currently, we have do some performance tests, if broker with a smaller direct memory(e.g. 2G) and run pulsar-perf to write messages, the broker becomes unstable, because broker read messages from the channel automatically and the ByteBuf released until the entry write success or timeout. Since #3986 and #5710 introduce the topic level and broker level publish rate limiter which can allow users to limit the publish rate, this can effectively avoid broker OOM. If only think about the broker OOM(the publish rate limiter is more useful in terms of resource constraints), use publish rate limiter to avoid broker OOM will limit the performance of the broker. For example, if the bookie write timeout is 10sec, and the direct memory is 2G, if we use publish rate limiter to avoid broker OOM, we need to limit the publish rate in bytes < 200MB. It's better to add a publish buffer size limit(default is 1/4 total direct memory) , when the message incoming, the publish buffer size increase and when the message is write to bookie successful or timeout the publish buffer size decrease. If the buffer size is over the limitation, stop auto read from the producer channel.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
