oneby-wang commented on PR #24867: URL: https://github.com/apache/pulsar/pull/24867#issuecomment-3417842279
Not tried autoscaling yet. I noticed that we can auto scale receiverQueueSize using memory limiter by turning on `autoScaledReceiverQueueSizeEnabled` switch, but after some code analysis, I think autoscaling may probably not work in multi-topics situation. Every single partition consumer starts with currentReceiverQueueSize flow action, meaning that the consumer will receive currentReceiverQueueSize messages. https://github.com/apache/pulsar/blob/e6560657e20d30103f2f01c3a24600dad1ba9ab6/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java#L236-L249 `MultiTopicsConsumerImpl` creates single partition consumer using there own receiverQueueSize and set batch receive maxNumMessages to (receiverQueueSize / 2). https://github.com/apache/pulsar/blob/678db6b34587b0042235f0d31e4c406bb0263bcd/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java#L1216-L1231 If `autoScaledReceiverQueueSizeEnabled` switch is on, we set `currentReceiverQueueSize` to `minReceiverQueueSize()`. https://github.com/apache/pulsar/blob/678db6b34587b0042235f0d31e4c406bb0263bcd/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java#L220-L226 `minReceiverQueueSize()` is (2 * batchReceivePolicy.getMaxNumMessages() - 2≈ receiverQueueSize) according to above analysis. So, the application boots with about (n * receiverQueueSize) messages, n is the topic nums. https://github.com/apache/pulsar/blob/678db6b34587b0042235f0d31e4c406bb0263bcd/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L502-L509 Every single partition consumer's `minReceiverQueueSize()` is about receiverQueueSize due to batch receive maxNumMessages is (receiverQueueSize / 2), so it can't be reduced below (2 * batchReceivePolicy.getMaxNumMessages() - 2≈ receiverQueueSize). https://github.com/apache/pulsar/blob/678db6b34587b0042235f0d31e4c406bb0263bcd/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java#L250-L259 This PR can tune single partition consumer's receiverQueueSize and batch receive maxNumMessages using multiTopicsSinglePartitionReceiverQueueSize config. Above is just pure code analysis, not tested by myself yet. And more, I think multi-topics consumer and it's inner single partition consumers just like parent container and child containers relation, and their receiverQueueSize should be tuned independently in someway. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
