rdhabalia commented on issue #402: Rate limiting in the dispatcher URL: https://github.com/apache/incubator-pulsar/issues/402#issuecomment-317104964 >Can you add information on the implementation approach? Sure, I will broker changes details. > Another thought, should we differentiate between cached vs uncached reads? we could bet set up for a much bigger amount of work later when we have to fetch it from bookies. Actually, there are two things here: if we are trying to protect broker's resources against a topic which is draining with much higher rate then it make sense to throttle overall msg-out-rate and consider both cached and uncached. However, it can put higher load on bookie later on because by the time next read comes, bookie might have discarded entries from the cache. But then to address this issue, broker needs visibility of bookie's caching. Broker also cache entry at managed-ledger but broker doesn't do caching if consumer has not caught up or consumer is draining backlog, and one of the need of throttling is for namespace which are draining backlog. so, I think we can throttle both kind of entries.? > though it's not clear to me whether the configured limit is per-topic (eg: shared across all subscriptions) or applied to each individual subscription Configuration per topic which will be shared across all the subscriptions and only reason is to put cap at the topic so, topic with many subscribers can't misuse the throttling and if it is needed then we can configure higher number of message-rate for the namespace. However, there is very low probability to have many subscribers. So, do you think individual subscribers can have separate quota would be better option? > I would say both should be allowed to apply. Actually, both CPU and n/w bandwidth are concerns while draining the backlog. Don't you think allowing both the policies may create unnecessary complexity because then it will be hard to decide which one should be kept as resource consumption is at runtime and can be varied broker to broker? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
With regards, Apache Git Services
