dlg99 commented on issue #9562: URL: https://github.com/apache/pulsar/issues/9562#issuecomment-789101618
@danielsinai Have you tried enabling backpressure? there is a discussion in dev mailing list (subject: "Unbounded memory usage for WQ > AQ ?") on this topic, I'll copy my comment from there with some ideas for you to experiment with: ``` I remember issues with bookies OOMing/slowing down due to memory pressure under load. https://github.com/apache/bookkeeper/issues/1409 https://github.com/apache/bookkeeper/pull/1410 IIRC, there were a couple of problems: - Slow bookie kept on accepting data hat it could not process (netty kept on reading it and throwing it into the queue) AQ < WQ means that the client does not wait after AQ acks received and keeps on throwing data to the slow bookie and ensemble change did not happen (or did not happen fast enough?) - client submitted a lot of requests but was too slow to process responses (network capacity, NIC bandwidth, something else), and the bookie kept to the data It's been a while and I don't recall all the details but the PR is merged. Have you played with these settings: https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java // backpressure control protected static final String MAX_ADDS_IN_PROGRESS_LIMIT = "maxAddsInProgressLimit"; protected static final String MAX_READS_IN_PROGRESS_LIMIT = "maxReadsInProgressLimit"; protected static final String CLOSE_CHANNEL_ON_RESPONSE_TIMEOUT = "closeChannelOnResponseTimeout"; protected static final String WAIT_TIMEOUT_ON_RESPONSE_BACKPRESSURE = "waitTimeoutOnResponseBackpressureMs"; https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ClientConfiguration.java // backpressure configuration protected static final String WAIT_TIMEOUT_ON_BACKPRESSURE ="waitTimeoutOnBackpressureMs"; ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
