Resolved comment discussions in the design document that are closed. If there is no further feedback, I will start a voting thread tomorrow.
Thanks, Sumanth On Thu, Sep 2, 2021 at 6:54 AM Joshua McKenzie <jmcken...@apache.org> wrote: > I'm +1 on where it currently stands after the revisions. Consider resolving > out comment threads on the design doc that are closed so we can see if > there's any outstanding discussions from a high level? > > ~Josh > > On Mon, Aug 30, 2021 at 1:14 AM Sumanth Pasupuleti < > sumanth.pasupuleti...@gmail.com> wrote: > > > +1. Made changes to the design document linked against the CEP to reflect > > this feedback. Specifically, the following sections have been updated > > * Operations to blacklist > > * Blacklist information store > > > > Thanks, > > Sumanth > > > > > > On Fri, Aug 27, 2021 at 7:57 AM Joshua McKenzie <jmcken...@apache.org> > > wrote: > > > > > I can see the case for all three: > > > * Deny both reads and writes to a partition (wide, heavily tombstones, > > too > > > many stables, etc) causing disruption to a replica set; don't want > > further > > > growth nor reads until operator intervention > > > * Deny reads but allow writing to rectify problems on a partition > > > (intervention window; see above) > > > * Deny writes to a partition but allow reads (prevent partitions > growing > > > unbounded, or potentially evolving into a future feature creating a > > ceiling > > > on partition sizes that kicks in and demands application intervention > to > > > reduce partition size at a guardrail limit) > > > > > > So yeah, at least to me at face value it seems like it'd be worth it > not > > > only to allow denylisting both reads and writes, but to be able to > choose > > > from the set of reads|writes|both on a per-partition basis. > > > > > > ~Josh > > > > > > > > > On Thu, Aug 26, 2021 at 2:16 PM Sumanth Pasupuleti < > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > > > Thank you, Josh for the elaborate explanation of a potential scenario > > > where > > > > denylisting writes would make sense. > > > > I, 100% agree that could benefit in a situation where we would want > to > > > deny > > > > writes to a partition that we do not have much control on (which is > > true > > > in > > > > most situations) and such behavior can eventually lead to > > unavailability > > > of > > > > other partitions too, as you indicate. > > > > > > > > Do you think it makes sense to make it configurable per partition > > though? > > > > As in, maybe by default, we would want to deny both reads and writes > > to a > > > > partition, but for certain partitions, we may still want to allow > > writes > > > > just so we can issue a delete against that partition as an example. > > > > Ofcourse this would make the feature and the interface more heavy, > and > > we > > > > need to think through if its worth it. I personally feel it could be > > > worth > > > > it, especially if we agree on the default behavior that makes the > > > interface > > > > simple in most cases. Thoughts? > > > > > > > > And yes, so good to see CEP process reaping benefits in multiple > ways - > > > > especially around collaboration and documentation. > > > > > > > > > > > > On Thu, Aug 26, 2021 at 8:31 AM Joshua McKenzie < > jmcken...@apache.org> > > > > wrote: > > > > > > > > > The design doc and CEP currently pass on blocklisting / denylisting > > > > writes > > > > > at this time. In the proposed new patch it states: > > > > > "Note: We do not want to blacklist writes since it is the reads > that > > > > > primarily impact the performance when reading a bad partition, and > we > > > may > > > > > want writes to be allowed to “fix” a bad partition. We could > revisit > > > this > > > > > in the future" > > > > > > > > > > In situations where you have an air gap between database ops and > > > > > application access (ops <> application teams, or more autonomous > > > > > application access patterns, self-service, etc), you can easily get > > > into > > > > a > > > > > situation where you have either a pathological client hammering > > writes > > > > to a > > > > > specific partition causing impact to other clients or in the worst > > > case, > > > > > the replica set, or unbounded partition growth that again leads to > > > > > performance degradation or replica set unavailability. The tradeoff > > > there > > > > > becomes "do we interrupt the application's ability to write to this > > > > > partition now, or do we instead defer and risk losing access to > *all* > > > > > partitions on this replica set and still interrupt their access > > > > eventually > > > > > anyway?" > > > > > > > > > > Given this, I strongly advocate for support of denylisting both > reads > > > > *and* > > > > > writes on these grounds; operators need another tool in their > toolbox > > > to > > > > > deal with situations where specific partition writing has wider > > > negative > > > > > impacts on the replicas. > > > > > > > > > > Acknowledging of course that there was extensive discussion on this > > > back > > > > in > > > > > 2018, and that would have been a *great* time to engage in the > > > > discussion. > > > > > =/ Good thing we have this new CEP process! :) > > > > > > > > > > Curious what you think about this perspective Sumanth. > > > > > > > > > > ~Josh > > > > > > > > > > > > > > > On Tue, Aug 17, 2021 at 2:04 PM Joshua McKenzie < > > jmcken...@apache.org> > > > > > wrote: > > > > > > > > > > > Certainly. I'll take on distilling a high level view of the > feature > > > > from > > > > > > what Jon and I are working on to bring to this discussion. > > > > > > > > > > > > > > > > > > On Tue, Aug 17, 2021 at 1:40 PM Sumanth Pasupuleti < > > > > > > sumanth.pasupuleti...@gmail.com> wrote: > > > > > > > > > > > >> +1 to collaborating on the patch, Josh. My 2 cents would be to > > > > continue > > > > > to > > > > > >> pursue this CEP in the community through Discuss and Vote phases > > and > > > > > then > > > > > >> invest further on the patch (based on the Vote phase outcome), > so > > we > > > > can > > > > > >> reflect any additional feedback we may gather from the community > > > > through > > > > > >> this CEP. > > > > > >> > > > > > >> Thanks, > > > > > >> Sumanth > > > > > >> > > > > > >> On Tue, Aug 17, 2021 at 10:13 AM Joshua McKenzie < > > > > jmcken...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > >> > Sumanth, > > > > > >> > > > > > > >> > Jon Meredith and I are recently working on an OSS patch of one > > of > > > > > those > > > > > >> "ad > > > > > >> > hoc" implementations of this feature that's been running at > > scale > > > > for > > > > > >> > awhile like Jirsa mentioned; sorry for not catching the > > discussion > > > > on > > > > > >> > https://issues.apache.org/jira/browse/CASSANDRA-12106 and > > > engaging > > > > > >> > earlier! > > > > > >> > > > > > > >> > I believe I can have a tidied up patch on 4.0 of this by early > > > next > > > > > >> week - > > > > > >> > then we can look at the two implementations and take best of > > both. > > > > > What > > > > > >> do > > > > > >> > you think? > > > > > >> > > > > > > >> > ~Josh > > > > > >> > > > > > > >> > On Mon, Aug 16, 2021 at 2:14 PM Sumanth Pasupuleti < > > > > > >> > sumanth.pasupuleti...@gmail.com> wrote: > > > > > >> > > > > > > >> > > Starting a discussion thread for CEP-13 > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-13%3A+Denylisting+partitions > > > > > >> > > > > > > > >> > > This CEP proposes adding a new feature to Cassandra to be > able > > > to > > > > > >> > denylist > > > > > >> > > partitions. > > > > > >> > > > > > > > >> > > Looking forward to any feedback/ thoughts. > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > Sumanth > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > >