315157973 commented on a change in pull request #8418:
URL: https://github.com/apache/pulsar/pull/8418#discussion_r515787156
##########
File path:
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherSingleActiveConsumer.java
##########
@@ -393,6 +397,16 @@ protected void readMoreEntries(Consumer consumer) {
// active-cursor reads message from cache rather from bookkeeper
(2) if topic has reached message-rate
// threshold: then schedule the read after MESSAGE_RATE_BACKOFF_MS
if
(serviceConfig.isDispatchThrottlingOnNonBacklogConsumerEnabled() ||
!cursor.isActive()) {
+ //If it reaches the threshold of the entire Broker rate limit,
try to read again later.
+ if (topic.getBrokerService().getBrokerDispatchRateLimiter() !=
null
+ &&
topic.getBrokerService().getBrokerDispatchRateLimiter().isDispatchRateLimitingEnabled()
+ &&
topic.getBrokerService().getBrokerDispatchRateLimiter().isConsumeRateExceeded())
{
Review comment:
All current limiting in Pulsar will have this problem:
We judge whether the threshold is exceeded before reading the entry; but
increasing the count of msg and byte is after reading data, so there will be a
time difference.
Even if we use the precise Limiter, the count will not increase until the
data is read. If you read concurrently in an instant, the threshold will still
be exceeded.
MultiConsumer will create multiple Consumers to consume each partition at
the same time, which is considered as a concurrent read.
If we want to be absolutely accurate, we may need to make Cursor judge
whether it exceeds the threshold every time it reads a message. I think this is
a huge loss in performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
