danielsinai commented on issue #9562: URL: https://github.com/apache/pulsar/issues/9562#issuecomment-798965007
> @danielsinai Have you tried enabling backpressure? > > there is a discussion in the bookkeeper dev mailing list (subject: "Unbounded memory usage for WQ > AQ ?") on this topic, I'll copy my comment from there with some ideas for you to experiment with: > > ``` > > I remember issues with bookies OOMing/slowing down due to memory pressure > > under load. > > https://github.com/apache/bookkeeper/issues/1409 > > https://github.com/apache/bookkeeper/pull/1410 > > > > IIRC, there were a couple of problems: > > > > - Slow bookie kept on accepting data hat it could not process (netty kept > > on reading it and throwing it into the queue) > > AQ < WQ means that the client does not wait after AQ acks received and > > keeps on throwing data to the slow bookie and ensemble change did not > > happen (or did not happen fast enough?) > > > > - client submitted a lot of requests but was too slow to process responses > > (network capacity, NIC bandwidth, something else), and the bookie kept to > > the data > > > > It's been a while and I don't recall all the details but the PR is merged. > > Have you played with these settings: > > https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java > > // backpressure control > > protected static final String MAX_ADDS_IN_PROGRESS_LIMIT = > > "maxAddsInProgressLimit"; > > protected static final String MAX_READS_IN_PROGRESS_LIMIT = > > "maxReadsInProgressLimit"; > > protected static final String CLOSE_CHANNEL_ON_RESPONSE_TIMEOUT = > > "closeChannelOnResponseTimeout"; > > protected static final String WAIT_TIMEOUT_ON_RESPONSE_BACKPRESSURE = "waitTimeoutOnResponseBackpressureMs"; > > https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java > > // backpressure configuration > > protected static final String WAIT_TIMEOUT_ON_BACKPRESSURE ="waitTimeoutOnBackpressureMs"; > > ``` > Thanks for your answer and I am really sorry for the delay. I will try to tune this stuff later on but isn't it strange that the broker getting OOMed? Is back pressuring to the broker will solve the problem? I mean I would expect pulsar broker to throttle the producer before getting his memory filled ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
