Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Just created an official CEP-41 incorporating the feedback from this discussion. Feel free to let me know if I may have missed some important feedback in this thread that is not

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Thanks, Josh. I will file an official CEP with all the details in a few days and update this thread with that CEP number. Thanks a lot everyone for providing valuable insights! Jaydeep On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie wrote: > Do folks think we should file an official CEP and take

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

> Do folks think we should file an official CEP and take it there? +1 here. Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a draft seems like a solid next step. On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote: > I see a lot of great ideas being discussed

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

I see a lot of great ideas being discussed or proposed in the past to cover the most common rate limiter candidate use cases. Do folks think we should file an official CEP and take it there? Jaydeep On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe wrote: > I just remembered the other day that I

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

I just remembered the other day that I had done a quick writeup on the state of compaction stress-related throttling in the project: https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing I'm sure most of it is old news to the people on this thread, but

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

> 2.) We should make sure the links between the "known" root causes of > cascading failures and the mechanisms we introduce to avoid them remain very > strong. Seems to me that our historical strategy was to address individual known cases one-by-one rather than looking for a more holistic

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

I almost forgot CASSANDRA-15817, which introduced reject_repair_compaction_threshold, which provides a mechanism to stop repairs while compaction is underwater.On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe wrote:Hey all,I'm a bit late to the discussion. I see that we've already discussed 

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Hey all, I'm a bit late to the discussion. I see that we've already discussed CASSANDRA-15013 and CASSANDRA-16663 at least in passing. Having written the latter, I'd be the first to

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

. If it turns out that >>> there is no widespread interest in the idea then we can do 1:1 and see how >>> we can help each other on a private fork, etc. >>> >>> Jaydeep >>> >>> On Wed, Jan 17, 2024 at 7:57 AM German Eichberger via dev < >&g

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

assandra.apache.org> wrote: >> >>> Jaydeep, >>> >>> I concur with Stefan that extensibility of this should be a design goal: >>> >>>- It should be easy to add additional metrics (e.g. write queue >>>depth) and decision logic &

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

have been thinking about siimilar things internally  Thanks, German From: Jaydeep Chovatia <chovatia.jayd...@gmail.com> Sent: Tuesday, January 16, 2024 1:16 PM To: dev@cassandra.apache.org <dev@cassandra.apache.org> Subject: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

t with other systems to signal a >>resource need which then could kick off things like scaling >> >> >> Super interested in this and we have been thinking about siimilar things >> internally  >> >> Thanks, >> German >>

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

imilar things > internally  > > Thanks, > German > -- > *From:* Jaydeep Chovatia > *Sent:* Tuesday, January 16, 2024 1:16 PM > *To:* dev@cassandra.apache.org > *Subject:* [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in > Cassandra >

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

things like scaling Super interested in this and we have been thinking about siimilar things internally  Thanks, German From: Jaydeep Chovatia Sent: Tuesday, January 16, 2024 1:16 PM To: dev@cassandra.apache.org Subject: [EXTERNAL] Re: [Discuss] Generic Purpose Rate

Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Hi Stefan, Please find my response below: 1) Currently, I am keeping the signals as interface, so one can override with a different implementation, but a point noted that even the interface APIs could be also made dynamic so one can define APIs and its implementation, if they wish to override. 2)

Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Jaydeep, thanks for reaching out and for sharing this proposal. I like the direction. Please also take a look at https://issues.apache.org/jira/browse/CASSANDRA-16663 , which adds coordinator-level rate limiting on request rate. This ticket introduced a lockless rate limiter patterned on an

Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Server side rate limiting can be useful, but imo if we were to focus effort into a single place, time would be much better spent adding adaptive rate limiting to the drivers. Rate limiting at the driver level can be done based on 2 simple feedback mechanisms - error rate and latency. When a node

Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

Hi Jaydeep, That seems quite interesting. Couple points though: 1) It would be nice if there is a way to "subscribe" to decisions your detection framework comes up with. Integration with e.g. diagnostics subsystem would be beneficial. This should be pluggable - just coding up an interface to

[Discuss] Generic Purpose Rate Limiter in Cassandra

Hi, Happy New Year! I would like to discuss the following idea: Open-source Cassandra (CASSANDRA-15013 ) has an elementary built-in memory rate limiter based on the incoming payload from user requests. This rate limiter activates if any