Re: [Discuss] CASSANDRA-17666, disable write path for cdc

[Josh McKenzie]

While it's a fairly small and simple change, 4.1 is an LTS branch right now and 
it's getting bugfixes only. If you absolutely need it for a deployment 
internally, it should be relatively easy to rejigger the patch to a checkout of 
4.1 and build a distribution of the software for yourself (see pr 
)

On Mon, Sep 23, 2024, at 6:44 AM, Nikolai Loginov wrote:
> Is it possible to  backport changes from the CASSANDRA-17666 to the 4.1 
> branch?
> 
> Regards
> 
> Nikolai Loginov

Re: [EXTERNAL] [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie]

 
>> be a huge benefit. I would go as far as saying the CEP should focus more on 
>> a framework for pluggable implementations that has low to zero cost when 
>> disabled than a specific set of metrics to use or specific approach. 
>> 
>> Jordan 
>> 
>> On Thu, Sep 19, 2024 at 14:38 Benedict Elliott Smith  
>> wrote:
>>> I just want to flag here that this is a topic I have strong opinions on, 
>>> but the CEP is not really specific or detailed enough to understand 
>>> precisely how it will be implemented. So, if a patch is already being 
>>> produced, most of my feedback is likely to be provided some time after a 
>>> patch appears, through the normal review process. I want to flag this now 
>>> to avoid any surprise.
>>> 
>>> I will say that upfront that, ideally, this system should be designed to 
>>> have ~zero overhead when disabled, and with minimal coupling (between its 
>>> own components and C* itself), so that entirely orthogonal approaches can 
>>> be integrated in future without polluting the codebase.
>>> 
>>> 
>>>> On 19 Sep 2024, at 19:14, Patrick McFadin  wrote:
>>>> 
>>>> The work has begun but we don't have a VOTE thread for this CEP. Can one 
>>>> get started?
>>>> 
>>>> On Mon, May 6, 2024 at 9:24 PM Jaydeep Chovatia 
>>>>  wrote:
>>>>> Sure, Caleb. I will include the work as part of CASSANDRA-19534 
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-19534> in the CEP-41.
>>>>> 
>>>>> Jaydeep
>>>>> 
>>>>> On Fri, May 3, 2024 at 7:48 AM Caleb Rackliffe  
>>>>> wrote:
>>>>>> FYI, there is some ongoing sort-of-related work going on in 
>>>>>> CASSANDRA-19534 <https://issues.apache.org/jira/browse/CASSANDRA-19534>
>>>>>> 
>>>>>> On Wed, Apr 10, 2024 at 6:35 PM Jaydeep Chovatia 
>>>>>>  wrote:
>>>>>>> Just created an official CEP-41 
>>>>>>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-41+%28DRAFT%29+Apache+Cassandra+Unified+Rate+Limiter>
>>>>>>>  incorporating the feedback from this discussion. Feel free to let me 
>>>>>>> know if I may have missed some important feedback in this thread that 
>>>>>>> is not captured in the CEP-41.
>>>>>>> 
>>>>>>> Jaydeep
>>>>>>> 
>>>>>>> On Thu, Feb 22, 2024 at 11:36 AM Jaydeep Chovatia 
>>>>>>>  wrote:
>>>>>>>> Thanks, Josh. I will file an official CEP with all the details in a 
>>>>>>>> few days and update this thread with that CEP number.
>>>>>>>> Thanks a lot everyone for providing valuable insights!
>>>>>>>> 
>>>>>>>> Jaydeep
>>>>>>>> 
>>>>>>>> On Thu, Feb 22, 2024 at 9:24 AM Josh McKenzie  
>>>>>>>> wrote:
>>>>>>>>> __
>>>>>>>>>> Do folks think we should file an official CEP and take it there?
>>>>>>>>> +1 here.
>>>>>>>>> 
>>>>>>>>> Synthesizing your gdoc, Caleb's work, and the feedback from this 
>>>>>>>>> thread into a draft seems like a solid next step.
>>>>>>>>> 
>>>>>>>>> On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
>>>>>>>>>> I see a lot of great ideas being discussed or proposed in the past 
>>>>>>>>>> to cover the most common rate limiter candidate use cases. Do folks 
>>>>>>>>>> think we should file an official CEP and take it there?
>>>>>>>>>> 
>>>>>>>>>> Jaydeep
>>>>>>>>>> 
>>>>>>>>>> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe 
>>>>>>>>>>  wrote:
>>>>>>>>>>> I just remembered the other day that I had done a quick writeup on 
>>>>>>>>>>> the state of compaction stress-related throttling in the project:
>>>>>>>>>>> 
>>>>>>>>>>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>>>>>>>>>>> 
>

Re: CEP-15: Accord status

[Josh McKenzie]

> This presents an opportune moment for those interested to review the code.
> ...
> +88,341 −7,341
> 1003 Files changed

O.o 
This is... **very large**. If we use CASSANDRA-8099 as our "banana for scale":
> 645 files changed, 49381 insertions(+), 42227 deletions(-)

To be clear - I don't think we collectively should be worried about disruption 
from this patch since:
 1. Each commit (or the vast majority?) has already been reviewed by >= 1 other 
committer
 2. 7.3k deletions is a lot less than 42k
 3. We now have fuzzing, property based testing, and the simulator
 4. Most of this code is additive
How would you recommend interested parties engage with reviewing this behemoth? 
Or perhaps subsections of it or key areas to familiarize themselves with the 
structure?

On Fri, Sep 20, 2024, at 12:17 PM, David Capwell wrote:
> Recently, we rebased against the trunk branch, ensuring that the accord 
> branch is now in sync with the latest trunk version. This presents an 
> opportune moment for those interested to review the code.
> 
> 
> 
> We have a pending pull request 
> (_https://github.com/apache/cassandra/pull/3552_) that we do not intend to 
> merge.
> 
> 
> 
> Our current focus is on addressing several bug fixes and ensuring the safety 
> of topology changes (as evidenced by the number of issues filed against the 
> trunk). Once we wrap up bug fixes and safety features, we will likely discuss 
> the merge to trunk, so now is a great time to start engaging.
> 
> 
> 
> Thank you everyone for your patience!
> 

Re: [Discuss] Repair inside C*

[Josh McKenzie]

Not quite; finishing touches on the CEP and design doc are in flight (as of 
last / this week).

Soon(tm).

On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:
> Is this CEP ready for a VOTE thread? 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+Solution
> 
> On Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia 
>  wrote:
>> Thanks, Josh. I've just updated the CEP 
>> <https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Official+Repair+Solution>
>>  and included all the solutions you mentioned below.  
>> 
>> Jaydeep
>> 
>> On Thu, Feb 22, 2024 at 9:33 AM Josh McKenzie  wrote:
>>> __
>>> Very late response from me here (basically necro'ing this thread).
>>> 
>>> I think it'd be useful to get this condensed into a CEP that we can then 
>>> discuss in that format. It's clearly something we all agree we need and 
>>> having an implementation that works, even if it's not in your preferred 
>>> execution domain, is vastly better than nothing IMO.
>>> 
>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you 
>>> do Jaydeep given the implementation you have on a private fork + design.
>>> 
>>> A non-exhaustive list of things that might be useful incorporating into or 
>>> referencing from a CEP:
>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>> Even older automatic repair scheduling: 
>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>> Your design gdoc: 
>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>> PR with automated repair: 
>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>> 
>>> My intuition is that we're all basically in agreement that this is 
>>> something the DB needs, we're all willing to bikeshed for our personal 
>>> preference on where it lives and how it's implemented, and at the end of 
>>> the day, code talks. I don't think anyone's said they'll die on the hill of 
>>> implementation details, so that feels like CEP time to me.
>>> 
>>> If you were willing and able to get a CEP together for automated repair 
>>> based on the above material, given you've done the work and have the proof 
>>> points it's working at scale, I think this would be a *huge contribution* 
>>> to the community.
>>> 
>>> On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
>>>> Is anyone going to file an official CEP for this?
>>>> As mentioned in this email thread, here is one of the solution's design 
>>>> doc 
>>>> <https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0>
>>>>  and source code on a private Apache Cassandra patch. Could you go through 
>>>> it and let me know what you think?
>>>> 
>>>> Jaydeep
>>>> 
>>>> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad  
>>>> wrote:
>>>>> > That said I would happily support an effort to bring repair scheduling 
>>>>> > to the sidecar immediately. This has nothing blocking it, and would 
>>>>> > potentially enable the sidecar to provide an official repair scheduling 
>>>>> > solution that is compatible with current or even previous versions of 
>>>>> > the database.
>>>>> 
>>>>> This is something I hadn't thought much about, and is a pretty good 
>>>>> argument for using the sidecar initially.  There's a lot of deployments 
>>>>> out there and having an official repair option would be a big win. 
>>>>> 
>>>>> 
>>>>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>>>>> > I agree that it would be ideal for Cassandra to have a repair scheduler 
>>>>> > in-DB.
>>>>> >
>>>>> > That said I would happily support an effort to bring repair scheduling 
>>>>> > to the sidecar immediately. This has nothing blocking it, and would 
>>>>> > potentially enable the sidecar to provide an official repair scheduling 
>>>>> > solution that is compatible with current or ev

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

> there is another perfectly sensible option
My apologies; I wasn't clear. *If we choose to continue to use chronicle 
queue*, what I enumerated was the only logical option I saw for us.

Altogether I think we should just move away from the library as you've laid out 
here Benedict.

On Thu, Sep 19, 2024, at 11:34 AM, Benedict wrote:
> 
> No, there is another perfectly sensible option: just implement a simple 
> serialisation format ourselves.
> 
> I am against forking their code; that is a much higher maintenance burden 
> than just writing something simple ourselves. We’ve spent longer collectively 
> discussing and maintaining this dependency than it would take to implement 
> the features we use.
> 
> I still have not heard a compelling reason we adopted it as a dependency in 
> the first place.
> 
>> On 19 Sep 2024, at 16:26, Josh McKenzie  wrote:
>> 
>>> a jerk move, but they started it with this weird release model
>> I think that's the only option given their release model and lack of 
>> backporting bugfixes to the latest ea. Either you run tip of the spear, pay 
>> them for bugfixes, or run what's effectively an unsupported LTS in the form 
>> of ea.
>> 
>> So doesn't seem like a jerk move to me as much as it seems like an 
>> eventuality of their release model.
>> 
>> On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote:
>>> I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas ~ 
>>> 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining 
>>> the string operations overhead in the JVM of log concatenation vs slapping 
>>> binary to CQ’s off heap-and-append operation was substantial. 
>>> 
>>> We could hostile fork and bring the bits we use in tree (a jerk move, but 
>>> they started it with this weird release model). I’d rather avoid this, but 
>>> it’s an option seeing as how it’s ASFv2. 
>>> 
>>> On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan  
>>> wrote:
>>>> 
>>>>> When it comes to alternatives, what about logback + slf4j? It has 
>>>>> appenders where we want, it is sync / async, we can code some nio 
>>>>> appender too I guess, it logs it as text into a file so we do not need 
>>>>> any special tooling to review that. For tailing which Chronicle also 
>>>>> offers, I guess "tail -f that.log" just does the job? logback even rolls 
>>>>> the files after they are big enough so it rolls the files the same way 
>>>>> after some configured period / size as Chronicle does (It even compresses 
>>>>> the logs).
>>>> 
>>>> Yes it was considered.  The whole point was to have a binary log because 
>>>> serialization to/from (remember replay is part off this) text explodes the 
>>>> size on disk and in memory as well as the processing time required and 
>>>> does not meet the timing requirements of fqltool.
>>>> 
>>>> -Jeremiah
>> 

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

> a jerk move, but they started it with this weird release model
I think that's the only option given their release model and lack of 
backporting bugfixes to the latest ea. Either you run tip of the spear, pay 
them for bugfixes, or run what's effectively an unsupported LTS in the form of 
ea.

So doesn't seem like a jerk move to me as much as it seems like an eventuality 
of their release model.

On Wed, Sep 18, 2024, at 7:02 PM, Nate McCall wrote:
> I feel like a group of us discussed this IRL a bit at ApacheCon in Vegas ~ 
> 2019 maybe? Anyhoo, the tidbit sticking in my mind was someone explaining the 
> string operations overhead in the JVM of log concatenation vs slapping binary 
> to CQ’s off heap-and-append operation was substantial. 
> 
> We could hostile fork and bring the bits we use in tree (a jerk move, but 
> they started it with this weird release model). I’d rather avoid this, but 
> it’s an option seeing as how it’s ASFv2. 
> 
> On Thu, 19 Sep 2024 at 5:08 AM, Jeremiah Jordan  
> wrote:
>> 
>>> When it comes to alternatives, what about logback + slf4j? It has appenders 
>>> where we want, it is sync / async, we can code some nio appender too I 
>>> guess, it logs it as text into a file so we do not need any special tooling 
>>> to review that. For tailing which Chronicle also offers, I guess "tail -f 
>>> that.log" just does the job? logback even rolls the files after they are 
>>> big enough so it rolls the files the same way after some configured period 
>>> / size as Chronicle does (It even compresses the logs).
>> 
>> Yes it was considered.  The whole point was to have a binary log because 
>> serialization to/from (remember replay is part off this) text explodes the 
>> size on disk and in memory as well as the processing time required and does 
>> not meet the timing requirements of fqltool.
>> 
>> -Jeremiah

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[Josh McKenzie]

I think it's FQLTool only right now; I bumped into it recently doing the JDK21 
compat work.

I'm not concerned about current usage / dependency, but if our usage expands 
this could start to become a problem and that's going to be a hard thing to 
track and mange.

So reading through those issues Stefan, I think it boils down to:
 • The latest ea is code identical to the stable release
 • Subsequent bugfixes get applied to the customer-only stable branch *and one 
release forward*
 • Projects running ea releases would need to cherry-pick those bugfixes back 
or run on the next branch's ea, which could introduce the project to API 
changes or other risks
Assuming that's the case... blech. Our exposure is low, but that seems like a 
real pain.

On Mon, Sep 16, 2024, at 5:16 PM, Benedict wrote:
> 
> Don’t we essentially just use it as a file format for storing a couple of 
> kinds of append-only data?
> 
> I was never entirely clear on the value it brought to the project.
> 
> 
>> On 16 Sep 2024, at 22:11, Jordan West  wrote:
>> 
>> Thanks for the sleuthing Stefan! This definitely is a bit unfortunate. It 
>> sounds like a replacement is not really practical so I'll ignore that option 
>> for now, until a viable alternative is proposed. I am -1 on us writing our 
>> own without strong, strong justification -- primarily because I think the 
>> likelihood is we introduce more bugs before getting to something stable. 
>> 
>> Regarding the remaining options, mostly some thoughts:
>> 
>> - it would be nice to have some specific evidence of other projects using 
>> the EA versions and what their developers have said about it.
>> - it sounds like if we go with the EA route, the onus to test for 
>> correctness / compatibility increases. They do test but anything marked 
>> "early access" I think deserves more scrutiny from the C* community before 
>> release. That could come in the form of more tests (or showing that we 
>> already have good coverage of where its used).
>> - i assume each time we upgrade we would pick the most recently released EA 
>> version
>> 
>> Jordan
>> 
>> 
>> On Mon, Sep 16, 2024 at 1:46 PM Štefan Miklošovič  
>> wrote:
>>> We are using a library called Chronicle Queue (1) and its dependencies and 
>>> we ship them in the distribution tarball.
>>> 
>>> The version we use in 5.0 / trunk as I write this is 2.23.36. If you look 
>>> closely here (2), there is one more release like this, 2.23.37 and after 
>>> that all these releases have "ea" in their name.
>>> 
>>> "ea" stands for "early access". The project has changed the versioning / 
>>> development model in such a way that "ea" releases act, more or less, as 
>>> glorified snapshots which are indeed released to Maven Central but the 
>>> "regular" releases are not there. The reason behind this is that "regular" 
>>> releases are published only for customers who pay to the company behind 
>>> this project and they offer commercial support for that.
>>> 
>>> "regular" releases are meant to get all the bug fixes after "ea" is 
>>> published and they are official stable releases. On the other hand "ea" 
>>> releases are the ones where the development happens and every now and then, 
>>> once the developers think that it is time to cut new 2.x, they just publish 
>>> that privately.
>>> 
>>> I was investigating how this all works here (3) and while they said that, I 
>>> quote (4):
>>> 
>>> "In my experience this is consumed by a large number of open source 
>>> projects reliably (for our other artifacts too). This development/ea branch 
>>> still goes through an extensive test suite prior to release. Releases from 
>>> this branch will contain the latest features and bug fixes."
>>> 
>>> I am not completely sure if we are OK with this. For the record, Mick is 
>>> not overly comfortable with that and Brandon would prefer to just replace 
>>> it / get rid of this dependency (comments / reasons / discussion from (5) 
>>> to the end)
>>> 
>>> The question is if we are OK with how things are and if we are then what 
>>> are the rules when upgrading the version of this project in Cassandra in 
>>> the context of "ea" versions they publish.
>>> 
>>> If we are not OK with this, then the question is what we are going to 
>>> replace it with.
>>> 
>>> If we are going to replace it, I very briefly took a look and there is 
>>> practically nothing out there which would hit all the buttons for us. 
>>> Chronicle is just perfect for this job and I am not a fan of rewriting this 
>>> at all. 
>>> 
>>> I would like to have this resolved because there is CEP-12 I plan to 
>>> deliver and I hit this and I do not want to base that work on something we 
>>> might eventually abandon. There are some ideas for CEP-12 how to bypass 
>>> this without using Chronicle but I would like to firstly hear your opinion.
>>> 
>>> Regards
>>> 
>>> (1) https://github.com/OpenHFT/Chronicle-Queue
>>> (2) https://repo1.maven.org/maven2/net/openhft/chronicle-core/
>>> (3) https:

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Josh McKenzie]

I think it's worth exploring where the disconnect is just a *bit* more, though 
I agree with you Benedict in that there appears to be a clear consensus so the 
thread has evolved more into talking about our principles than what to do in 
this specific scenario.

Mick - this patch doesn't fix things 100%. It can't. BUT - it does take us from 
"In all cases where this occurs you will silently lose data" to "in some cases 
where this occurs you will have a rejected write, in others you'll have 
coordinator level logging, and in the worst split-brain case you can still have 
data loss (split-brain ownership where the quorum agrees on incorrect state)".

Whether we're 100% fixing data loss, or 10%, at the end of the day my position 
is that we should fix all the data loss we can (this statement is unique to 
gossip's architectural limitations) *even if that means loss of availability in 
edge-cases where it is not strictly required*. I think where we might be 
misaligned:
> It can prevent data mislocation in some scenarios but it offers no guarantees 
> about that, and can also degrade availability unnecessarily. 
It *can't* prevent mis-location in all scenarios. That's a shortcoming of 
Gossip's architecture.

If the availability degradation is part and parcel to the mitigation, and the 
mitigation approach is the *best worst option* in an eventually consistent 
topology sync environment, *it's not unnecessary*. It's the price we pay for 
mitigating a flaw in the architecture as best we can.

A comparable: if we have 1% data loss, and the only way we can mitigate that is 
to have a 1% hit in availability instead, the former is clearly worse for our 
users than the latter. Recovery from data loss is multiple orders of magnitude 
harder than retrying a query on availability degradation.

On Fri, Sep 13, 2024, at 8:15 AM, Benedict wrote:
> 
> I think everyone has made their case, the costs and benefits are fairly well 
> understood, and there appears to be a strong quorum that favours an approach 
> that prioritises avoiding data loss. 
> 
> So, I propose we either recognise that this is the clear position of the 
> community, or move to a vote to formalise it one way or the other.
> 
> 
>> On 13 Sep 2024, at 13:02, Alex Petrov  wrote:
>> 
>> I agree with folks saying that we absolutely need to reject misplaced 
>> writes. It may not preclude coordinator making a local write, or making a 
>> write to a local replica, but even reducing probability of a misplaced write 
>> shown as success to the client is a substantial win. 
>> 
>> On Fri, Sep 13, 2024, at 10:41 AM, Mick Semb Wever wrote:
>>> replies below (to Scott, Josh and Jeremiah).
>>> 
>>> tl;dr all my four points remain undisputed, when the patch is applied.   
>>> This is a messy situation, but no denying the value of rejection writes to 
>>> various known popular scenarios.  Point (1) remains important to highlight 
>>> IMHO.
>>> 
>>> 
>>> On Fri, 13 Sept 2024 at 03:03, C. Scott Andreas  
>>> wrote:
 Since that time, I’ve received several urgent messages from major users of 
 Apache Cassandra and even customers of Cassandra ecosystem vendors asking 
 about this bug. Some were able to verify the presence of lost data in 
 SSTables on nodes where it didn’t belong, demonstrate empty read responses 
 for data that is known proof-positive to exist (think content-addressable 
 stores), or reproduce this behavior in a local cluster after forcing 
 disagreement.
 
>>> 
>>> 
>>> 
>>> Having been privy to the background of those "urgent messages" I can say 
>>> the information you received wasn't correct (or complete). 
>>> 
>>> My challenge on this thread is about understanding where this might 
>>> unexpectedly bite users, which should be part of our due diligence when 
>>> applying such patches to stable branches.   I ask you to run through my 
>>> four points, which AFAIK still stand true.  
>>> 
>>> 
 But I **don't** think it's healthy for us to repeatedly re-litigate 
 whether data loss is acceptable based on how long it's been around, or how 
 frequently some of us on the project have observed some given phenomenon. 
>>> 
>>> 
>>> Josh, that's true, but talking to these things helps open up the 
>>> discussion, see my point above wrt being aware of second-hand evidence that 
>>> was inaccurate.
>>> 
>>> 
 The severity and frequency of this issue combined with the business risk 
 to Apache Cassandra users changed my mind about fixing it in earlier 
 branches despite TCM having been merged to fix it for good on trunk.
 
>>> 
>>> 
>>> That shouldn't prevent us from investigating known edge-cases, collateral 
>>> damage, and unexpected behavioural changes in patch versions.   
>>> 
>>>  
>>> 
>>>  
> On Sep 12, 2024, at 3:40 PM, Jeremiah Jordan  
> wrote:
>> 1. Rejecting writes does not prevent data loss in this situation.  It 
>> only reduces it.  The investigation and remediation of p

Re: Welcome Chris Bannister, James Hartig, Jackson Flemming and João Reis, as cassandra-gocql-driver committers

[Josh McKenzie]

Congratulations and welcome! It's great to have you all on board!

On Thu, Sep 12, 2024, at 11:16 PM, guo Maxwell wrote:
> Congratulations!
> 
> James Hartig  于2024年9月13日周五 08:11写道：
>> __
>> Thanks everyone! Excited to contribute.
>> 
>> On Thu, Sep 12, 2024, at 4:59 PM, Francisco Guerrero wrote:
>>> Congratulations!
>>> 
>>> On 2024/09/12 11:39:40 Mick Semb Wever wrote:
>>> > The PMC's members are pleased to announce that Chris Bannister, James
>>> > Hartig, Jackson Flemming and João Reis have accepted invitations to become
>>> > committers on the Drivers subproject.
>>> > 
>>> > Thanks a lot for everything you have done with the gocql driver all these
>>> > years.  We are very excited to see the driver now inside the Apache
>>> > Cassandra project.
>>> > 
>>> > Congratulations and welcome!!
>>> > 
>>> > The Apache Cassandra PMC
>>> > 
>>> 

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Josh McKenzie]

> Even when the fix is only partial, so really it's more about more forcefully 
> alerting the operator to the problem via over-eager unavailability …? 
> 
> Sometimes a principled stance can take us away from the important details in 
> the discussions.
My understanding of the ticket (having not dug deeply into the code, just 
reviewed the JIRA and this thread), is that this is as effective of a solution 
as we can have in a non-deterministic, non-epoch based, non-transactional 
metadata system. i.e. Gossip. I don't see it as a partial fix but I might be 
misunderstanding.

I'm not advocating for us having a rigid principled stance where we reject all 
nuance and don't discuss things. I'm advocating for us coalescing on a shared 
**default** stance of correctness unless otherwise excepted. We know we're a 
diverse group, we're all different people with different histories / values / 
opinions / cultures, and I think that's what makes this community as effective 
as it is.

But I **don't** think it's healthy for us to repeatedly re-litigate whether 
data loss is acceptable based on how long it's been around, or how frequently 
some of us on the project have observed some given phenomenon. My gut tells me 
we'd all be in a better place if we all started from 0 on a discussion like 
this as "Ok, data loss is unacceptable. Unless otherwise warranted, we should 
do all we can to fix this on all supported branches as our default response".

On Thu, Sep 12, 2024, at 9:02 PM, C. Scott Andreas wrote:
> Thanks all for discussion on this.
> 
> 
> 
> It’s hard to describe the sinking feeling that hit me when it became clear to 
> me how common this problem is - and how horribly difficult it is to prove one 
> has encountered this bug.
> 
> 
> 
> Two years ago, my understanding was that this is an exceptionally rare and 
> transient issue unlikely to occur after all the work we put into gossip. My 
> view was that gossip had basically been shorn up and that Transactional 
> Metadata is the proper fix for this with its epoch design (which is true).
> 
> 
> 
> Since that time, I’ve received several urgent messages from major users of 
> Apache Cassandra and even customers of Cassandra ecosystem vendors asking 
> about this bug. Some were able to verify the presence of lost data in 
> SSTables on nodes where it didn’t belong, demonstrate empty read responses 
> for data that is known proof-positive to exist (think content-addressable 
> stores), or reproduce this behavior in a local cluster after forcing 
> disagreement.
> 
> 
> 
> The severity and frequency of this issue combined with the business risk to 
> Apache Cassandra users changed my mind about fixing it in earlier branches 
> despite TCM having been merged to fix it for good on trunk.
> 
> 
> 
> The guards in this patch are extensive: point reads, range reads, mutations, 
> repair, incoming / outgoing streams, hints, merkle tree requests, and others 
> I’m forgetting. They’re simple guards, and while they touch many subsystems, 
> they’re not invasive changes.
> 
> 
> 
> There is no reasonable scenario that’s common enough that would justify 
> disabling a guard preventing silent data loss by default. I appreciate that a 
> prop exists to permit or warn in the presence of data loss for anyone who may 
> want that, in the spirit of users being in control of their clusters’ 
> behavior.
> 
> 
> 
> Very large operators may only see indications the guard took effect for a 
> handful of queries per day — but in instances where ownership disagreement is 
> prolonged, the patch is an essential guard against large-scale unrecoverable 
> data loss and incorrect responses to queries. I’ll further take the position 
> that those few queries in transient disagreement scenarios would be 
> justification by themselves.
> 
> 
> 
> I support merging the patch to all proposed branches and enabling the guard 
> by default.
> 
> 
> 
> – Scott
> 
> 
>> On Sep 12, 2024, at 3:40 PM, Jeremiah Jordan  
>> wrote:
>> 
>>> 1. Rejecting writes does not prevent data loss in this situation.  It only 
>>> reduces it.  The investigation and remediation of possible mislocated data 
>>> is still required.
>> 
>> All nodes which reject a write prevent mislocated data.  There is still the 
>> possibility of some node having the same wrong view of the ring as the 
>> coordinator (including if they are the same node) accepting data.  Unless 
>> there are multiple nodes with the same wrong view of the ring, data loss is 
>> prevented for CL > ONE.
>> 
>>> 2. Rejecting writes is a louder form of alerting for users unaware of the 
>>> scenario, those not already monitoring logs or metrics.
>> 
>> Without this patch no one is aware of any issues at all.  Maybe you are 
>> referring to a situation where the patch is applied, but the default 
>> behavior is to still accept the “bad” data?  In that case yes, turning on 
>> rejection makes it “louder” in that your queries can fail if too many nodes 
>> ar

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Josh McKenzie]

I'd like to propose we treat all data loss bugs as "fix by default on all 
supported branches even if that might introduce user-facing changes".

Even if only N of M people on a thread have experienced it.
Even if we only uncover it through testing (looking at you Harry).

My gut tells me this is something we should have a clear cultural value system 
around as a project, and that value system should be "Above all else, we don't 
lose data". Just because users aren't aware it might be happening doesn't mean 
it's not a *massive* problem.

I would bet good money that there are *a lot* of user-felt pains using this 
project that we're all unfortunately insulated from. 

On Thu, Sep 12, 2024, at 3:35 PM, Mick Semb Wever wrote:
> Great that the discussion explores the issue as well.
> 
> So far we've heard three* companies being impacted, and four times in total…? 
>  Info is helpful here.  
> 
> *) Jordan, you say you've been hit by _other_ bugs _like_ it.  Jon i'm 
> assuming the company you refer to doesn't overlap. JD we know it had nothing 
> to do with range movements and could/should have been prevented far simpler 
> with operational correctness/checks.
> 
> In the extreme, when no writes have gone to any of the replicas, what 
> happened ? Either this was CL.*ONE, or it was an operational failure (not C* 
> at fault).  If it's an operational fault, both the coordinator and the node 
> can be wrong.  With CL.ONE, just the coordinator can be wrong and the problem 
> still exists (and with rejection enabled the operator is now more likely to 
> ignore it).
> 
> WRT to the remedy, is it not to either run repair (when 1+ replica has it), 
> or to load flushed and recompacted sstables (from the period in question) to 
> their correct nodes.  This is not difficult, but understandably lost-sleep 
> and time-intensive.
> 
> Neither of the above two points I feel are that material to the outcome, but 
> I think it helps keep the discussion on track and informative.   We also know 
> there are many competent operators out there that do detect data loss.
> 
> 
> 
> On Thu, 12 Sept 2024 at 20:07, Caleb Rackliffe  
> wrote:
>> If we don’t reject by default, but log by default, my fear is that we’ll 
>> simply be alerting the operator to something that has already gone very 
>> wrong that they may not be in any position to ever address.
>> 
>>> On Sep 12, 2024, at 12:44 PM, Jordan West  wrote:
>>> 
>>> I’m +1 on enabling rejection by default on all branches. We have been bit 
>>> by silent data loss (due to other bugs like the schema issues in 4.1) from 
>>> lack of rejection on several occasions and short of writing extremely 
>>> specialized tooling its unrecoverable. While both lack of availability and 
>>> data loss are critical, I will always pick lack of availability over data 
>>> loss. Its better to fail a write that will be lost than silently lose it. 
>>> 
>>> Of course, a change like this requires very good communication in NEWS.txt 
>>> and elsewhere but I think its well worth it. While it may surprise some 
>>> users I think they would be more surprised that they were silently losing 
>>> data. 
>>> 
>>> Jordan 
>>> 
>>> On Thu, Sep 12, 2024 at 10:22 Mick Semb Wever  wrote:
 Thanks for starting the thread Caleb, it is a big and impacting patch.
 
 Appreciate the criticality, in a new major release rejection by default is 
 obvious.   Otherwise the logging and metrics is an important addition to 
 help users validate the existence and degree of any problem.  
 
 Also worth mentioning that rejecting writes can cause degraded 
 availability in situations that pose no problem.  This is a coordination 
 problem on a probabilistic design, it's choose your evil: unnecessary 
 degraded availability or mislocated data (eventual data loss).   Logging 
 and metrics makes alerting on and handling the data mislocation possible, 
 i.e. avoids data loss with manual intervention.  (Logging and metrics also 
 face the same problem with false positives.)
 
 I'm +0 for rejection default in 5.0.1, and +1 for only logging default in 
 4.x
 
 
 On Thu, 12 Sept 2024 at 18:56, Jeff Jirsa  wrote:
> This patch is so hard for me. 
> 
> The safety it adds is critical and should have been added a decade ago.
> Also it’s a huge patch, and touches “everything”. 
> 
> It definitely belongs in 5.0. I’d probably reject by default in 5.0.1.  
> 
> 4.0 / 4.1 - if we treat this like a fix for latent opportunity for data 
> loss (which it implicitly is), I guess?
> 
> 
> 
> > On Sep 12, 2024, at 9:46 AM, Brandon Williams  wrote:
> > 
> > On Thu, Sep 12, 2024 at 11:41 AM Caleb Rackliffe
> >  wrote:
> >> 
> >> Are you opposed to the patch in its entirety, or just rejecting unsafe 
> >> operations by default?
> > 
> > I had the latter in mind.  Changing any default in a patch release i

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Josh McKenzie]

> 4.0 / 4.1 - if we treat this like a fix for latent opportunity for data loss 
> (which it implicitly is), I guess?
If we have known data loss scenarios we should fix them in all supported 
branches even if that fix can potentially modify user-facing behavior. We 
should definitely try and prioritize fixing issues w/out changing behavior of 
course, but if it's zero sum between risk of data loss and risk of user-facing 
disruptive behavior, I'll choose the 2nd every time.

Size of the patch shouldn't really be a make or break on that position, though 
something to consider in terms of risk it introduces, other things we may need 
to touch / test / qualify, extent of fuzzing and potential additional 
property-based testing for subsystems impacted, etc.

On Thu, Sep 12, 2024, at 12:55 PM, Jeff Jirsa wrote:
> This patch is so hard for me. 
> 
> The safety it adds is critical and should have been added a decade ago.
> Also it’s a huge patch, and touches “everything”. 
> 
> It definitely belongs in 5.0. I’d probably reject by default in 5.0.1.  
> 
> 4.0 / 4.1 - if we treat this like a fix for latent opportunity for data loss 
> (which it implicitly is), I guess?
> 
> 
> 
> > On Sep 12, 2024, at 9:46 AM, Brandon Williams  wrote:
> > 
> > On Thu, Sep 12, 2024 at 11:41 AM Caleb Rackliffe
> >  wrote:
> >> 
> >> Are you opposed to the patch in its entirety, or just rejecting unsafe 
> >> operations by default?
> > 
> > I had the latter in mind.  Changing any default in a patch release is
> > a potential surprise for operators and one of this nature especially
> > so.
> > 
> > Kind Regards,
> > Brandon
> 
> 

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[Josh McKenzie]

More or less surprising than learning that they've been at risk of or actively 
losing data for years?

On Thu, Sep 12, 2024, at 12:46 PM, Brandon Williams wrote:
> On Thu, Sep 12, 2024 at 11:41 AM Caleb Rackliffe
>  wrote:
> >
> > Are you opposed to the patch in its entirety, or just rejecting unsafe 
> > operations by default?
> 
> I had the latter in mind.  Changing any default in a patch release is
> a potential surprise for operators and one of this nature especially
> so.
> 
> Kind Regards,
> Brandon
> 

Re: [VOTE] Release test-api 0.0.17

[Josh McKenzie]

+1

On Tue, Sep 10, 2024, at 11:07 AM, Michael Shuler wrote:
> +1
> 
> On 9/10/24 09:34, Doug Rohrer wrote:
> > Proposing the test build of in-jvm dtest API 0.0.17 for release
> > 
> > Repository:
> > https://gitbox.apache.org/repos/asf?p=cassandra-in-jvm-dtest-api.git 
> > 
> > 
> > Candidate SHA:
> > https://github.com/apache/cassandra-in-jvm-dtest-api/commit/85b538ca8259dedc2aded8a633cf3174f551f664
> >  
> > 
> > Tagged with 0.0.17
> > 
> > Artifacts:
> > https://repository.apache.org/content/repositories/orgapachecassandra-1343/org/apache/cassandra/dtest-api/0.0.17/
> >  
> > 
> > 
> > Key signature: 9A648E3DEDA36EE374C4277B602ED2C52277
> > 
> > Changes since last release:
> > * CASSANDRA-19783 - In-jvm dtest to detect InstanceClassLoaderLeaks
> > * CASSANDRA-19239 - jvm-dtests crash on java 17
> > 
> > The vote will be open for 24 hours. Everyone who has tested the build
> > is invited to vote. Votes by PMC members are considered binding. A
> > vote passes if there are at least three binding +1s.
> 

Re: Welcome Jordan West and Stefan Miklosovic as Cassandra PMC members!

[Josh McKenzie]

Congratulations to both of you!

On Fri, Aug 30, 2024, at 6:11 PM, Leo Toff wrote:
> Congratulations you guys!
> 
> 
> On Fri, Aug 30, 2024, 2:30 PM Ekaterina Dimitrova  
> wrote:
>> Great news indeed! 👏🏻 Congrats!! 
>> 
>> On Fri, 30 Aug 2024 at 16:35, Bernardo Botella 
>>  wrote:
>>> Congrats you both!
>>> 
>>> 
 On Aug 30, 2024, at 1:32 PM, Melissa Logan  wrote:
 
 Great news - congrats Jordan and Stefan!
 
 On Fri, Aug 30, 2024 at 1:31 PM Sumanth Pasupuleti 
  wrote:
> Congratulations Jordan and Stefan!!!
> 
> On Fri, Aug 30, 2024 at 1:21 PM Jon Haddad  wrote:
>> The PMC's members are pleased to announce that Jordan West and Stefan 
>> Miklosovic have accepted invitations to become PMC members.
>> 
>> Thanks a lot, Jordan and Stefan, for everything you have done for the 
>> project all these years.
>> 
>> Congratulations and welcome!!
>> 
>> The Apache Cassandra PMC

Re: Welcome Doug Rohrer as Cassandra Committer

[Josh McKenzie]

Congratulations Doug! Great to have you on board.

On Fri, Aug 23, 2024, at 4:50 PM, Arjun Ashok wrote:
> Congratulations Doug!
> 
> > On Aug 23, 2024, at 11:55 AM, Dinesh Joshi  wrote:
> > 
> > The Apache Cassandra PMC is thrilled to announce that Doug Rohrer has
> > accepted the invitation to become a committer!
> > 
> > Doug has worked on several aspects of Cassandra, Sidecar, and
> > Analytics. Congratulations and welcome!
> > 
> > The Apache Cassandra PMC members
> 
> 

Re: CASSANDRA-19835 should block 5.0.0

[Josh McKenzie]

This looks like a "merge it to 5.0.0 and re-rev rc" to me.

On Wed, Aug 14, 2024, at 2:21 PM, David Capwell wrote:
> CASSANDRA-19835 fixes a very small bug but one that has big impact: if you 
> use trie Memtable and unslabbed_heap_buffers_logged Cassandra will crash on 
> boot!  If you use any other Memtable type and SAI you will break every 
> SSTable write….
> 
> The patch is up now, its a 1 line code change just adding the enum to the 
> switch statement.
> 
> This was found in CASSANDRA-19833 while making some tests randomize their 
> configs.

Re: [DISCUSS] inotify for detection of manually removed snapshots

[Josh McKenzie]

> I lean towards the documentation approach vs complicating the implementation. 
+1.

My question around the need for the optimization was purely driven by the 
motivation of exploring whether the complexity of caching this data was 
warranted. From talking with Jeremiah a bit offline, sounds like there's been 
some really egregious degradation cases and the minimal memory footprint and 
status quo complexity of implementation is a good enough solution.

On Fri, Aug 9, 2024, at 1:38 PM, Jordan West wrote:
> I lean towards the documentation approach vs complicating the implementation. 
> 
> For me personally: I regularly use shell commands to operate on snapshots. 
> That includes listing them. I probably should use nodetool for it all instead 
> though. 
> 
> Jordan 
> 
> On Fri, Aug 9, 2024 at 08:09 Štefan Miklošovič  wrote:
>> I understand and agree. It is just that it would be cool if we avoided the 
>> situation when there is a figurative ABC company which has these "bash 
>> scripts removing snapshots from cron by rm -rf every second Sunday at 3:00 
>> am" because "that was their workflow for ages".
>> 
>> I am particularly sensitive to this as Cassandra is very cautious when it 
>> comes to not disrupting the workflows already out there.
>> 
>> I do not know how frequent this would be and if somebody started to 
>> complain. I mean ... they could still remove it by hand, right? It is just 
>> listsnapshots would not be relevant anymore without refreshing it. I think 
>> that might be acceptable. It would be something else if we flat out made 
>> manual deletion forbidden, which it is not.
>> 
>> On Fri, Aug 9, 2024 at 4:50 PM Bowen Song via dev  
>> wrote:
>>> __
>>> If we have the documentation in place, we can then consider the cache to be 
>>> the master copy of metadata, and rely on it to be always accurate and up to 
>>> date. If someone deletes the snapshot files from filesystem, they can't 
>>> complain about Cassandra stopped working correctly - which is the same if 
>>> they had manually deleted some SSTable files (they shouldn't).
>>> 
>>> On 09/08/2024 11:16, Štefan Miklošovič wrote:
>>>> We could indeed do that. Does your suggestion mean that there should not 
>>>> be a problem with caching it all once explicitly stated like that?
>>>> 
>>>> On Fri, Aug 9, 2024 at 12:01 PM Bowen Song via dev 
>>>>  wrote:
>>>>> Has anyone considered simply updating the documentation saying this?
>>>>> 
>>>>> "Removing the snapshot files directly from the filesystem may break 
>>>>> things. Always use the `nodetool` command or JMX to remove snapshots."
>>>>> 
>>>>> On 09/08/2024 09:18, Štefan Miklošovič wrote:
>>>>>> If we consider caching it all to be too much, we might probably make 
>>>>>> caching an option an admin would need to opt-in into? There might be a 
>>>>>> flag in cassandra.yaml, once enabled, it would be in memory, otherwise 
>>>>>> it would just load it as it was so people can decide if caching is 
>>>>>> enough for them or they want to have it as it was before (would be by 
>>>>>> default set to as it was). This puts additional complexity into 
>>>>>> SnapshotManager but it should be in general doable. 
>>>>>> 
>>>>>> Let me know what you think, I would really like to have this resolved, 
>>>>>> 18111 brings a lot of code cleanup and simplifies stuff a lot.
>>>>>> 
>>>>>> On Wed, Aug 7, 2024 at 11:30 PM Josh McKenzie  
>>>>>> wrote:
>>>>>>>> If you have a lot of snapshots and have for example a metric 
>>>>>>>> monitoring them and their sizes, if you don’t cache it, creating the 
>>>>>>>> metric can cause performance degradation. We added the cache because 
>>>>>>>> we saw this happen to databases more than once.
>>>>>>> I mean, I believe you, I'm just surprised querying out metadata for 
>>>>>>> files and basic computation is leading to hundreds of ms pause times 
>>>>>>> even on systems with a lot of files. Aren't most / all of these values 
>>>>>>> cached at the filesystem layer so we're basically just tomato / tomahto 
>>>>>>> caching systems, either one we maintain or one the OS maintains?
>>>>>>> 
>>>>>>> Or is 

Re: [DISCUSS] inotify for detection of manually removed snapshots

[Josh McKenzie]

> If you have a lot of snapshots and have for example a metric monitoring them 
> and their sizes, if you don’t cache it, creating the metric can cause 
> performance degradation. We added the cache because we saw this happen to 
> databases more than once.
I mean, I believe you, I'm just surprised querying out metadata for files and 
basic computation is leading to hundreds of ms pause times even on systems with 
a lot of files. Aren't most / all of these values cached at the filesystem 
layer so we're basically just tomato / tomahto caching systems, either one we 
maintain or one the OS maintains?

Or is there really just a count of files well outside what I'm thinking here?

Anyway, not trying to cause a ruckus and make needless noise, trying to learn. 
;)


On Wed, Aug 7, 2024, at 3:20 PM, Štefan Miklošovič wrote:
> 
> 
> On Wed, Aug 7, 2024 at 6:39 PM Yifan Cai  wrote:
>> With WatcherService, when events are missed (which is to be expected), you 
>> will still need to list the files. It seems to me that WatcherService 
>> doesn't offer significant benefits in this case.
>  
> Yeah I think we leave it out eventually.
> 
>> 
>> Regarding listing directory with a refresh flag, my concern is the potential 
>> for abuse. End-users might/could always refresh before listing, which could 
>> undermine the purpose of caching. Perhaps Jeremiah can provide more insight 
>> on this.
> 
> Well, by default, it would not be refreshed every single time. You would need 
> to opt-in into that. If there is a shop which has users with a direct access 
> to the disk of Cassandra nodes and they are removing data manually, I do not 
> know what to say, what is nodetool clearsnapshot and jmx methods good for 
> then? I do not think we can prevent people from shooting into their feet if 
> they are absolutely willing to do that.
> 
> If they want to refresh that every time, that would be equal to the current 
> behavior. It would be at most as "bad" as it is now.
>  
>> IMO, caching is best handled internally. I have a few UX-related questions:
>> - Is it valid or acceptable to return stale data? If so, end-users have to 
>> do some form of validation before consuming each snapshot to account for 
>> potential deletions. 
> 
> answer below
> 
>> - Even if listsnapshot returns the most recent data, is it possible that 
>> some of the directories get deleted when end-users are accessing them? I 
>> think it is true. It, then, enforces end-users to do some validation first, 
>> similar to handling stale data. 
> 
> I think that what you were trying to say is that when at time T0 somebody 
> lists snapshots and at T1 somebody removes a snapshot manually then the list 
> of snapshots is not actual anymore? Sure. That is a thing. This is how it 
> currently works. 
> 
> Now, we want to cache them, so if you clear a snapshot which is not 
> physically there because somebody removed it manually, that should be a 
> no-op, it will be just removed from the internal tracker. So, if it is at 
> disk and in cache and you clear it, then all is fine. It is fine too if it is 
> not on disk anymore and you clear it, then it is just removed internally. It 
> would fail only in case you want to remove a snapshot which is not cached, 
> regardless whether it is on disk or not. The deletion of non-existing 
> snapshot ends up with a failure, nothing should be changed in that regard, 
> this is the current behavior too.
> 
> I want to say that I did not write it completely correctly at the very 
> beginning of this thread. Currently, we are caching only _expiring_ 
> snapshots, because we need to know what is their time of removal so we act on 
> it later. _normal_ snapshots are _not_ cached _yet_. I spent so much time 
> with 18111 that I live in a reality where it is already in, I forgot this is 
> not actually in place yet, we are very close to that.
>  
> OK thank you all for your insights, I will NOT use inotify.
> 
>> 
>> Just my 2 cents. 
>> 
>> - Yifan
>> 
>> On Wed, Aug 7, 2024 at 6:03 AM Štefan Miklošovič  
>> wrote:
>>> Yes, for example as reported here
>>> 
>>> https://issues.apache.org/jira/browse/CASSANDRA-13338
>>> 
>>> People who are charting this in monitoring dashboards might also hit this.
>>> 
>>> On Wed, Aug 7, 2024 at 2:59 PM J. D. Jordan  
>>> wrote:
>>>> If you have a lot of snapshots and have for example a metric monitoring 
>>>> them and their sizes, if you don’t cache it, creating the metric can cause 
>>>> performance degradation. We added the cache because we saw this happen to 
>>>> 

Re: [VOTE] Backport CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0

[Josh McKenzie]

+1.

I'm ultimately in favor of setting the precedent of allowing low-risk ecosystem 
interop changes in non-trunk.

On Wed, Aug 7, 2024, at 2:10 PM, Dinesh Joshi wrote:
> +1
> 
> I think this is a low risk change which is going to avoid duplication of 
> effort so it is worth it.
> 
> On Sat, Aug 3, 2024 at 11:19 PM Yifan Cai  wrote:
>> Hi,
>> 
>> I am proposing backporting CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0. 
>> 
>> There is a discussion thread 
>>  on the 
>> topic. In summary, the backport would benefit Cassandra Analytics by 
>> providing a unified solution, and the patch is considered low-risk. While 
>> there are concerns about adding features to 4.0 and 4.1, there is generally 
>> support for 5.0.
>> 
>> The vote will be open for 72 hours (longer if needed). Votes by PMC members 
>> are considered binding. A vote passes if there are at least three binding 
>> +1s and no -1's.
>> 
>> Kind regards,
>> Yifan Cai

Re: [DISCUSS] inotify for detection of manually removed snapshots

[Josh McKenzie]

> Snapshot metadata are currently stored in memory / they are cached so we do 
> not need to go to disk every single time we want to list them, the more 
> snapshots we have, the worse it is.
Are we enumerating our snapshots somewhere on the hot path, or is this 
performance concern misplaced?

On Wed, Aug 7, 2024, at 7:44 AM, Štefan Miklošovič wrote:
> Snapshot metadata are currently stored in memory / they are cached so we do 
> not need to go to disk every single time we want to list them, the more 
> snapshots we have, the worse it is.
> 
> When a snapshot is _manually_ removed from disk, not from nodetool 
> clearsnapshot, just by rm -rf on a respective snapshot directory, then such 
> snapshot will be still visible in nodetool listsnapshots. Manual removal of a 
> snapshot might be done e.g. by accident or by some "impatient" operator who 
> just goes to disk and removes it there instead of using nodetool or 
> respective JMX method.
> 
> To improve UX here, what I came up with is that we might use Java's 
> WatchService where each snapshot dir would be registered. WatchService is 
> part of Java, it uses inotify subsystem which is what Linux kernel offers. 
> The result of doing it is that once a snapshot dir is registered to be 
> watched and when it is removed then we are notified about that via inotify / 
> WatchService so we can react on it and remove the in-memory representation of 
> that so it will not be visible in the output anymore.
> 
> While this works, there are some questions / concerns
> 
> 1) What do people think about inotify in general? I tested this on 10k 
> snapshots and it seems to work just fine, nevertheless there is in general no 
> strong guarantee that every single event will come through, there is also a 
> family of kernel parameters around this where more tuning can be done etc. It 
> is also questionable how this will behave on other systems from Linux (Mac 
> etc). While JRE running on different platforms also implements this, I am not 
> completely sure these implementations are quality-wise the same as for Linux 
> etc. There is a history of not-so-quality implementations for other systems 
> (events not coming through on Macs etc) and while I think we are safe on 
> Linux, I am not sure we want to go with this elsewhere.
> 
> 2) inotify brings more entropy into the codebase, it is another thing we need 
> to take care of etc (however, it is all concentrated in one class and pretty 
> much "isolated" from everything else)
> 
> I made this feature optional and it is turned off by default so people need 
> to explicitly opt-in into this so we are not forcing it on anybody.
> 
> If we do not want to go with inotify, another option would be to have a 
> background thread which would periodically check if a manifest exists on a 
> disk, if it does not, then a snapshot does not either. While this works, what 
> I do not like about this is that the primary reason we moved it to memory was 
> to bypass IO as much as possible yet here we would introduce another check 
> which would go to disk, and this would be done periodically, which beats the 
> whole purpose. If an operator lists snapshots once a week and there is a 
> background check running every 10 minutes (for example), then the cummulative 
> number of IO operations migth be bigger than us just doing nothing at all. 
> For this reason, if we do not want to go with inotify, I would also not 
> implement any automatic background check. Instead of that, there would be 
> SnapshotManagerMbean#refresh() method which would just forcibly reload all 
> snapshots from scratch. I think that manual deletion of snapshots is not 
> something a user would do regularly, snapshots are meant to be removed via 
> nodetool or JMX. If manual removal ever happens then in order to make it 
> synchronized again, the refreshing of these snapshots would be required. 
> There might be an additional flag in nodetool listsnapshots, once specified, 
> refreshing would be done, otherwise not.
> 
> How does this all sound to people?
> 
> Regards

Re: [VOTE] Release Apache Cassandra 4.1.6

[Josh McKenzie]

> Unfortunately, I can not immediately see a good way to provide the critical 
> bugfix of CASSANDRA-19534 
> , affecting all 
> Cassandra users, without making at least some change in this API. 
> 
> I personally think that this method is very tightly coupled to the 
> implementation to expose it via -D. If anyone using it could provide some 
> context about why it is an important part of API, it would be give some 
> useful context.
Nobody stepping up to engage on the technical piece of this? Unless / until 
somebody does, Alex' argument holds the most weight as the expert with what's 
going on IMO.

The question we're facing is - when we find a defect that requires a change in 
a public facing API, which of the following 2 is more important:
 1. Keeping the API stable
 2. Having the defect resolved
Obviously this will be case-by-case. What CASSANDRA-19534 addresses:
> When a node is under pressure, hundreds of thousands of requests can show up 
> in the native transport queue, and it looks like it can take way longer to 
> timeout than is configured. 
> ...
> After stopping the load test altogether, it took nearly a minute before the 
> requests were no longer queued.
I believe our priority here should be having this defect resolved.

On Tue, Jul 30, 2024, at 1:43 PM, Jordan West wrote:
> I would make the case that loss of availability / significant performance 
> issue, regardless of the amount of time it has existed for, is worth fixing 
> on the branches that are widely deployed by the community. Especially when 
> weighed against a loosely defined public interface issue. 
> 
> The queuing issue has been a persistent problem (like you said 10 years) and 
> I regularly (approx once every 1-2 weeks) have to tell my customers “we 
> either have to wait for Cassandra to clear the queues or do a rolling restart 
> to fix it” both which come at a cost during an incident where a client 
> overloaded the DB and the impact is severe or business impacting. Especially 
> for customers doing LWTs or using non-standard RFs which are also more 
> prevalent in my experience than an external implementation of QueryHandler. 
> 
> While not data loss, I would argue this is a critical bug and if we did find 
> a data loss issue dormant for 10 years (which has happened in the past) we 
> would fix it as soon as it was found and a patch was made available on all 
> actively maintained versions. 
> 
> Jordan 
> 
> On Tue, Jul 30, 2024 at 10:30 Jeff Jirsa  wrote:
>> 
>> It’s a 10 year old flaw in an 18 month old branch. Why does it need to go 
>> into 4.1, it’s not a regression and it clearly breaks compatibility? 
>> 
>> 
>> 
>> 
>>> On Jul 30, 2024, at 8:52 AM, Jon Haddad  wrote:
>>> 
>>> This patch fixes a long standing issue that's the root cause of 
>>> availability failures.  Even though folks can specify a custom query 
>>> handler with the -D flag, the number of users impacted by this is going to 
>>> be incredibly small.  On the other hand, the fix helps every single user of 
>>> 4.1 that puts too much pressure on the cluster, which happens fairly 
>>> regularly.  
>>> 
>>> My POV is that it's a fairly weak argument that this is a public interface, 
>>> but I don't consider it worth debating whether it is or not, because even 
>>> if it is, this improves stability of the database for all users, so it's 
>>> worth going in.  Let's not be dogmatic about fixes that help 99% of users 
>>> because an incredibly small number that actually implement a custom query 
>>> handler will need to make a trivial update in order to use the latest 4.1.6 
>>> dependency.
>>> 
>>> Jon
>>> 
>>> 
>>> 
>>> On Tue, Jul 30, 2024 at 8:09 AM J. D. Jordan  
>>> wrote:
 
 Given we allow a pluggable query handler implementation to be specified 
 for the server with a -D during startup. So I would consider the query 
 handler one of our public interfaces.
 
> On Jul 30, 2024, at 9:35 AM, Alex Petrov  wrote:
> 
> Hi Tommy,
> 
> Thank you for spotting this and bringing this to community's attention.
> 
> I believe our primary interfaces are native and internode protocol, and 
> CLI tools. Most interfaces are used to to abstract implementations 
> internally. Few interfaces, such as DataType, Partitioner, and Triggers 
> can be depended upon by external tools using Cassandra as a library. 
> There is no official way to plug in a QueryHandler, so I did not consider 
> it to be a part of our public API. 
> 
> From [1]: 
> 
> > These considerations are especially important for public APIs, 
> > including CQL, virtual tables, JMX, yaml, system properties, etc. Any 
> > planned additions must be carefully considered in the context of any 
> > existing APIs. Where possible the approach of any existing API should 
> > be followed. 
> 
> Maybe we should have an exhaustive list of public APIs, 

Re: [DISCUSS] Backport CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0

[Josh McKenzie]

Some thoughts:
 1. Most of our PMC votes are majority-based, not binding -1. So Jeff being -1 
doesn't mean the whole PMC being -1. So don't take his -1 as being a show 
stopper or indicative of everyone on the PMC (and don't take me saying this as 
the converse ;))
 2. I expect we have a lot of debt when it comes to our ecosystem integrations 
on older branches. Bringing those projects into the ASF umbrella and into the 
project ecosystem is at odds with a hard policy of "we don't add improvements 
or new features to old branches", *specifically* in cases like this where the 
desire is to get uniform support for ecosystem projects across all supported 
branches of C*
 3. We're moving into a world where we will likely more frequently modify the 
mainline branch with new functionality to integrate with ecosystem changes 
(sidecar, analytics, drivers?). It's probably at least worth a conversation as 
to whether our current policy (improvements and new features main branch only) 
is optimal across everything equally or if there should be nuance for ecosystem 
integrations.
 4. To Jeff's point: everyone is always going to have some minor improvement 
they'd like to back-port to older branches.
I haven't thought deeply enough about this specific situation to have a well 
formed opinion, but figured calling out the above things is worth doing. This 
probably won't be the last time we look at our supported branches and have some 
pain we'd like to address based on the inconsistent ecosystem support and API 
piece across them.

On Mon, Jul 29, 2024, at 1:32 PM, Yifan Cai wrote:
> It sounds like we are all good with backporting to 5.0. 
> 
> Thank you all for the feedback.
> 
> - Yifan
> 
> On Fri, Jul 26, 2024 at 12:21 PM Jeff Jirsa  wrote:
>> 
>> 
>> 
>>> On Jul 26, 2024, at 11:09 AM, Yifan Cai  wrote:
>>> 
>>> Thanks Jeff for restating the policy.
>>> 
>>> According to the release lifecycle doc
  • Missing features from newer generation releases are back-ported on per 
 - PMC vote basis.
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle 
>>> 
>>> We do not have a policy to prevent new features strictly for the branches 
>>> in maintenance state. 
>>> 
>>> IMO, the patch qualifies as the missing feature. (As said, it is useful for 
>>> Cassandra Analytics, and it is good to have the same bridge implementation 
>>> amongst different cassandra versions)
>>> 
>>> Therefore, I would like to call for a vote. 
>> 
>> Sure
>> 
>> I’m -1 on 4.0 and 4.1
>> 
>> - Jeff
>> 
>>> 
>>> On Fri, Jul 26, 2024 at 10:25 AM Jeff Jirsa  wrote:
 Everyone has a low risk change they want to backport to every branch, 4.0 
 and 4.1 in particular are way past the point we should be adding features 
 
 The policy exists and it’s a pure feature not a regression
 
 
 
 
 
 > On Jul 26, 2024, at 9:59 AM, Brandon Williams  wrote:
 > 
 > Given how low risk this is, I don't see an issue with backporting it
 > and I'm sure the usefulness outweighs what risk there is. +1 (5.0.1
 > though, not 5.0.0)
 > 
 > Kind Regards,
 > Brandon
 > 
 >> On Fri, Jul 26, 2024 at 11:52 AM Yifan Cai  wrote:
 >> 
 >> Hi everyone,
 >> 
 >> CASSANDRA-19800 is currently in the state of ready to be committed. 
 >> Before that, I want to propose backporting it to 4.0, 4.1 and 5.0.
 >> 
 >> The ability to notify CQLSSTableWriter user when new sstables are 
 >> produced is especially useful for Cassandra Analytics and other 
 >> consumers. The API is more reliable than monitoring the file directory.
 >> 
 >> That being said, I am aware that the patch is an improvement and trunk 
 >> only. I want to ask for an exemption on backporting the patch for two 
 >> reasons. It is useful for Cassandra Analytics. The patch is low risk 
 >> for Cassandra server as it only touches CQLSSTableWriter, which is only 
 >> used by toolings.
 >> 
 >> - Yifan

Re: CASSANDRA-19779 in 5.0-rc

[Josh McKenzie]

+1 to should block as well.

On Fri, Jul 26, 2024, at 12:15 PM, Patrick McFadin wrote:
> This is a weird regression but I'm borderline on this being a blocker. On a 
> brand new node starting, this comes up but goes away on the first reboot of 
> the node. What I don't get is why directIO is disabled if there is no commit 
> log. What's specific about that?
> 
> Patrick
> 
> On Fri, Jul 26, 2024 at 7:41 AM Jeff Jirsa  wrote:
>> 
>> +1 Should block 
>> 
>>> On Jul 26, 2024, at 8:25 AM, Štefan Miklošovič  
>>> wrote:
>>> 
>>> I am formally asking for acknowledging (1) to be a blocker for 5.0-rc. I 
>>> think it was introduced in (2) and it behaves like this (3).
>>> 
>>> Regards
>>> 
>>> (1) https://issues.apache.org/jira/browse/CASSANDRA-19779
>>> 
>>> (2) 
>>> https://issues.apache.org/jira/browse/CASSANDRA-18464?focusedCommentId=17868946&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17868946
>>> 
>>> (3) 
>>> https://issues.apache.org/jira/browse/CASSANDRA-19779?focusedCommentId=17868944&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17868944
>>> 

Re: Welcome Joey Lynch as Cassandra PMC member

[Josh McKenzie]

Congrats Joey! 

On Wed, Jul 24, 2024, at 10:55 AM, Abe Ratnofsky wrote:
> Congratulations!
> 

Re: [DISCUSS] The way we log

[Josh McKenzie]

I argued this point 6 years ago and I'll continue to argue it today. Relying on 
assertions in production code by default, having debug live in production code 
by default, these are code and culture smells to me. Yes we work on complex 
distributed systems. No we should not be relying on crutches like this in 
production environments to make up for the fact that our property and 
fuzz-based testing has historically been inadequate. And don't get me started 
on it being 2024 and us not having nightly perf regression testing.

Let me quote Benedict and strongly +1 his words here:

> Debug logging *can* be enabled for production clusters, but it should not be 
> on by default.

> DEBUG should *not* be taken to mean “background activities” and we should 
> stamp this out wherever this has been occurring

> DEBUG means verbose logging for use debugging system behaviour. It should be 
> safe to enable in production, but it should not be on by default.

> We should anyway not be codifying something just because a handful of people 
> have been doing it.

Doing our own thing w/regards to logging as a project is a pretty big unforced 
error IMO.

A brief glance at cockroach's logging channels 
 
and sinks or postgres' various severity levels 

 etc. just reinforces for me how our approach to logging and debug feels 
ad-hoc, reactive, and organic to me. There's a lot of prior art we could learn 
from here.

So I'm in my glass house here chucking stones, because of course I don't have 
the bandwidth to help *address* any of this. I'm with you on this Mick:
> What's your proposal to change it to *how it should be*? And who will do that 
> work ? Along with better naming of log files, I wholeheartedly support a more 
> intuitive approach to using separate loggers in the code, e.g. with markers.
> 
> My proposal is only in lieu of such an agreement and change.

On Wed, Jul 24, 2024, at 4:59 AM, Berenguer Blasi wrote:
> I wouldn't forbid isDebugEnabled.
> 
> Guarding heavy debug params with it makes perfect sense per se, you avoid a 
> perf penalty with 1 loc. If in C* production clusters are expected/preferred 
> to be run with debug logging on so be it but being an OSS project we're 
> unaware of all tools, side-projects and any other consumers of our code. They 
> could be benefiting from not running in debug.
> 
> Parsing the use of isDebugEnabled as making debug not meant for production is 
> sthg I am not totally following. Debug is, imo, is perfectly fine in 
> production for it's intended uses and with it's know impact. If we want to 
> signal we want C* to be ran in debug in production then let's add a big 
> warning in the logback xml files on even log that as a WARN or ERROR. But I 
> wouldn't push down that concern into the code style.
> 
> Besides, imo, if we remove/forbid it getting it back if we changed our minds 
> wouldn't be an easy task. Whereas if it's there it doesn't hurt.
> 
> My 2cts
> 
> On 23/7/24 16:12, Mick Semb Wever wrote:
>> 
>> 
>> On Tue, 23 Jul 2024 at 15:27, J. D. Jordan  wrote:
>>> I don’t know that I agree we should remove use if isDebugEnabled, but we 
>>> should be assuming debug is enabled and not doing anything too crazy there.
>> 
>> 
>> The problem is that the use of isDebugEnabled, as trivial as it is, leads to 
>> the typical assumption that debug is not meant for production.  This has 
>> tripped us up in the past.  It's important that debug logging does not 
>> impact production performance.  Any use of isDebugEnabled is 
>> counter-intuitive to this.   Maybe restoring the logging guidelines is 
>> enough, but I'd argue that we can further simplify things (at least against 
>> our current state of affairs) by just disallowing the use of isDebugEnabled 
>> altogether.  
>> 
>> And yes, debug equalling background tasks is not entirely accurate, my bad 
>> (the doc provides a far more accurate description to what might be found in 
>> system vs debug log files). And there's for example read/write tracing, that 
>> if enabled, might only be found in the debug.log  . This has been brought up 
>> previously, and in the ticket and old ml thread there's mention in keeping 
>> system.log clean to read/writes/hotpaths.  The logging guideline was written 
>> (by Paulo I believe) as a result of 10241 and that old ml thread.
>> We have an existing logging guideline doc, we need debug to be performant in 
>> production – it is enabled by default,  and is designed and recommended to 
>> be used in production.
>> 
>> 
>>  

Re: [DISCUSS] Replace airlift/airline library with Picocli

[Josh McKenzie]

> If you have downstream operations frequently parsing unstructured CLI output 
> using sed or awk and the output has no defined contract (ideally openAPI) 
> then this seems backwards to me and is going to be a constant source of 
> breakage for users and friction when evolving our product.
Strong +1 to this sentiment. Having an OpenAPI description and having a default 
implementation that outputs to .json for instance would help decouple us from 
the current state of viewing our text-based CLI output as being an implicit 
API. Which is just not great for anyone.

Also - I for one love me some colored output. We do it with our IDE's for a 
reason; why not with our CLI tools or logs? ;)

On Sat, Jul 20, 2024, at 3:26 PM, Miles Garnsey wrote:
> I don’t usually chime in but I’m a +1 on PicoCLI. I think it’s a great 
> project and the autocompletion features are a game changer for people like me 
> who prefer the CLI and use many CLI tools infrequently. The coloured output 
> is… nice, I guess.
> 
> Whatever the decision, I think at a strategic level the project should 
> consider a more principled approach to CLI output. If you have downstream 
> operations frequently parsing unstructured CLI output using sed or awk and 
> the output has no defined contract (ideally openAPI) then this seems 
> backwards to me and is going to be a constant source of breakage for users 
> and friction when evolving our product.
> 
> I don’t deny that a change of that magnitude may be a big lift, but to the 
> extent that we’re concerned about moving off a deprecated library based on 
> the fact that we have what amounts to an undocumented, uncontracted, and 
> unversioned API, this seems poor practice bluntly.
> 
>> On 21 Jul 2024, at 1:44 am, Dinesh Joshi  wrote:
>> 
>> On Tue, Jul 16, 2024 at 8:48 AM Jeff Jirsa  wrote:
>>> if it’s unmaintained, let’s remove it before we’re doing it on fire.
>> 
>> +1
>> 
>> Fire drills are never pleasant.
>> 
>> CLI parsing isn't a huge area of personal interest to me. However, it 
>> presents a non-trivial attack surface as input processing is a ripe target 
>> for vulnerabilities. I don't know if there are vulnerabilities lying around 
>> in hiding but if / when they are reported we will need to address them 
>> outside of the library or migrate to a maintained library at that time. 
>> Neither option is very appealing at that point. So I am of the opinion we 
>> should transition to a maintained library with healthy community support. 
>> Both picocli and commons-cli have good adoption and community around them.

Re: [DISCUSS] Replace airlift/airline library with Picocli

[Josh McKenzie]

> if someone proposed adding this library right now, we’d all say no because 
> it’s unmaintained.
Interesting. This perspective resonates with me, but it also raises the 
immediate next thought of "we should make it a point to proactively move off 
unmaintained dependencies or take them in-tree". In terms of general project 
health this seems like a healthy posture for us to take.

Would the blast radius of that thought exercise extend far beyond the CLI 
parsing case here, or is it pretty isolated?

I'm sympathetic to the "it's small, it's stable, it's a well solved problem, 
don't fix what isn't broken" argument, as well as the fiddly edge-case breakage 
that can occur in terms of output formatting and stderr vs. stdout etc. That 
said, for me all the implications surrounding the argument of "don't depend on 
something external that's dead" marginally outweigh that.

On Wed, Jul 17, 2024, at 10:04 AM, Tibor Répási wrote:
> 
> 
>> On 17. Jul 2024, at 01:28, David Capwell  wrote:
>> 
>> We also must weigh the risk of stability… Changing something for the sake of 
>> changing it may get past all our CI then could break a user as they used 
>> something we were not aware was being done (return codes, stdout/stderr, 
>> etc.); replacing something public facing comes with massive risk.
>> 
> 
> That’s completely true, which augments the importance of proper tests. 
> Currently, many tests lack coverage to this, which in my opinion is an issue 
> that needs to be addressed in the migration work. The outcome should be to 
> have lots of tests more and higher coverage than we have today.
> 
>> 
>> Again, must weigh the risk vs the reward.  Sure, we could have colored 
>> output, but if that breaks users was it worth it?
>> 
> 
> Colored output might be very useful and nice, but honestly, it won’t justify 
> that risk. Let me try to put some more weight to the reward pan of the scale:
> 
> - shell command autocompletion: the current completion script is limited to 
> nodetool only, is barely up to date with the CLI and pretty clumsy and 
> tedious to maintain. PicoCLI can auto-generate the completion script enabling 
> to be part of the release process which will assure to have an always up to 
> date completion script shipped within the distribution packages without 
> additional maintenance efforts.
> 
> - man pages (feel free to call me old): Cassandra completely lacks of 
> distributing man pages now, but do not underestimate the power of them. 
> Documentation depends on version, and probably the easiest way to get proper 
> documentation that aligns with the currently installed version is to type 
> man. While the help subcommand might give you a short summary on usage, the 
> man page can provide deeper insight of what it does, explanation of arguments 
> and options, examples, return codes, etc. PicoCLI anticipates man page 
> content and allows to generate man pages, e.g. during the release process to 
> be distributed along with the software. Autogenerated man pages could also be 
> used to update the documentation on the website. All this information would 
> be single sourced from at the actual implementation.
> 
> Though, these improvements won’t come out of the box, but PicoCLI allow us to 
> go for it, while airline won’t.
> 
> 
>> On 16. Jul 2024, at 18:01, Caleb Rackliffe  wrote:
>> 
>> With concerns around licensing all but resolved, I'd support Pico as our 
>> airline replacement. It looks like it would entail the least risky 
>> migration, is being actively maintained, will make the development of a 
>> number of planned enhancements easier, etc.
> 
> Are these licensing/legal concerns bigger than those about stay on airline?
> 
>> On 16. Jul 2024, at 17:39, Jeff Jirsa  wrote:
>> 
>> (Answering this as a cassandra person, don’t confuse this reply as 
>> board/foundation guidance)
>> 
>> The legal landscape is rapidly evolving with the CRA. The answer may change 
>> in the future, but I think “we have a dependency we ship that’s 
>> user-accessible and known to be abandoned” is an unhappy state that’s 
>> indefensible if there’s ever a transitive dependency issue (e.g. log4j style 
>> “oh who would have guessed that was possible” ). 
>> 
>> I’d look at this slightly differently - if someone proposed adding this 
>> library right now, we’d all say no because it’s unmaintained. That alone 
>> should be enough justification to fix the obvious problem - if it’s 
>> unmaintained, let’s remove it before we’re doing it on fire.
> 
> I really like this PoV and the size of the work is big enough to not want it 
> to be done in an emergency. What I’d like to augment here is, someday PicoCLI 
> might be obsolete as well and the CLI would need to be migrated to the next 
> library. Maxim already tried to separate things as good as possible to limit 
> the impact of such a change in the future, however, we should have an eye on 
> that aspect too.
> 
>> On 16. Jul 2024, at 14:49, Aleksey Yeshchenko  wrote:

Re: [VOTE] Release Apache Cassandra 5.0-rc1

[Josh McKenzie]

I see the argument. In reality though we can never truly guarantee that 
something is API stable, only our intent to keep it so if at all possible, 
right?

On Wed, Jul 3, 2024, at 8:58 AM, Claude Warren, Jr wrote:
> Because you do not know if the issues stopping the release will require an 
> API change -- so is the API stable?
> 
> On Mon, Jul 1, 2024 at 3:02 PM Josh McKenzie  wrote:
>> __
>>> Perhaps we should consider a Milestone release.  At least in some projects 
>>> this is a way to provide a test bed with known issues that will be 
>>> corrected before an RC.
>> How does that differ from beta in our lifecycle? API stable but a test bed 
>> to suss out issues like this.
>> 
>> 
>> On Mon, Jul 1, 2024, at 9:30 AM, Claude Warren, Jr via dev wrote:
>>> Perhaps we should consider a Milestone release.  At least in some projects 
>>> this is a way to provide a test bed with known issues that will be 
>>> corrected before an RC.
>>> 
>>> On Sun, Jun 30, 2024 at 9:50 PM Jon Haddad  wrote:
>>>> This came in after our vote, but we might also have a problem with 
>>>> performing schema changes after a full restart.  Appears to only be if the 
>>>> entire cluster was shut down, according to the report.  If it's true, this 
>>>> might affect anyone trying to restore from a backup.  This would also be a 
>>>> blocker for me, if that's the case.
>>>> 
>>>> https://issues.apache.org/jira/browse/CASSANDRA-19735
>>>> 
>>>> Jon
>>>> 
>>>> 
>>>> On Thu, Jun 27, 2024 at 9:49 PM Jon Haddad  wrote:
>>>>> Thanks for confirming this, Blake. I agree that we should not knowingly 
>>>>> ship new versions with severe bugs that cause the DB to crash, regression 
>>>>> or not. 
>>>>> 
>>>>> -1 from me as well
>>>>> 
>>>>> 
>>>>> On Fri, Jun 28, 2024 at 1:39 AM Blake Eggleston  
>>>>> wrote:
>>>>>> Looking at the ticket, I’d say Jon’s concern is legitimate. The 
>>>>>> segfaults Jon is seeing are probably caused by paxos V2 when combined 
>>>>>> with off heap memtables for the reason Benedict suggests in the JIRA. 
>>>>>> This problem will continue to exist in 5.0. Unfortunately, it looks like 
>>>>>> the patch posted is not enough to address the issue and will need to be 
>>>>>> a bit more involved to properly fix the problem.
>>>>>> 
>>>>>> While this is not a regression, I think Jon’s point about trie memtables 
>>>>>> increasing usage of off heap memtables is a good one, and anyway we 
>>>>>> shouldn’t be doing major releases with known process crashing bugs.
>>>>>> 
>>>>>> So I’m voting -1 on this release and will work with Jon and Benedict to 
>>>>>> get this fixed.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Blake
>>>>>> 
>>>>>> 
>>>>>>> On Jun 26, 2024, at 6:47 AM, Josh McKenzie  wrote:
>>>>>>> 
>>>>>>> Blake or Benedict - can either of you speak to Jon's concerns around 
>>>>>>> CASSANDRA-19668?
>>>>>>> 
>>>>>>> On Wed, Jun 26, 2024, at 12:18 AM, Jeff Jirsa wrote:
>>>>>>>> 
>>>>>>>> +1
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Jun 25, 2024, at 5:04 AM, Mick Semb Wever  wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Proposing the test build of Cassandra 5.0-rc1 for release.
>>>>>>>>> 
>>>>>>>>> sha1: b43f0b2e9f4cb5105764ef9cf4ece404a740539a
>>>>>>>>> Git: https://github.com/apache/cassandra/tree/5.0-rc1-tentative
>>>>>>>>> Maven Artifacts: 
>>>>>>>>> https://repository.apache.org/content/repositories/orgapachecassandra-1336/org/apache/cassandra/cassandra-all/5.0-rc1/
>>>>>>>>> 
>>>>>>>>> The Source and Build Artifacts, and the Debian and RPM packages and 
>>>>>>>>> repositories, are available here: 
>>>>>>>>> https://dist.apache.org/repos/dist/dev/cassandra/5.0-rc1/
>>>>>>>>> 
>>>>>>>>> The vote will be open for 72 hours (longer if needed). Everyone who 
>>>>>>>>> has tested the build is invited to vote. Votes by PMC members are 
>>>>>>>>> considered binding. A vote passes if there are at least three binding 
>>>>>>>>> +1s and no -1's.
>>>>>>>>> 
>>>>>>>>> [1]: CHANGES.txt: 
>>>>>>>>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/CHANGES.txt
>>>>>>>>> [2]: NEWS.txt: 
>>>>>>>>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/NEWS.txt
>> 

Re: [VOTE] CEP-42: Constraints Framework

[Josh McKenzie]

+1

On Tue, Jul 2, 2024, at 1:18 PM, Abe Ratnofsky wrote:
> +1 (nb)
> 
>> On Jul 2, 2024, at 12:15 PM, Yifan Cai  wrote:
>> 
>> +1 on CEP-42.
>> 
>> - Yifan
>> 
>> On Tue, Jul 2, 2024 at 5:17 AM Jon Haddad  wrote:
>>> +1
>>> 
>>> On Tue, Jul 2, 2024 at 5:06 AM  wrote:
 +1
 
 
> On Jul 1, 2024, at 8:34 PM, Doug Rohrer  wrote:
> 
> +1 (nb) - Thanks for all of the suggestions and Bernardo for wrangling 
> the CEP into shape!
> 
> Doug
> 
>> On Jul 1, 2024, at 3:06 PM, Dinesh Joshi  wrote:
>> 
>> +1
>> 
>> On Mon, Jul 1, 2024 at 11:58 AM Ariel Weisberg  wrote:
>>> __
>>> Hi,
>>> 
>>> I am +1 on CEP-42 with the latest updates to the CEP to clarify syntax, 
>>> error messages, constraint naming and generated naming, alter/drop, 
>>> describe etc.
>>> 
>>> I think this now tracks very closely to how other SQL databases define 
>>> constraints and the syntax is easily extensible to multi-column and 
>>> multi-table constraints.
>>> 
>>> Ariel
>>> 
>>> On Mon, Jul 1, 2024, at 9:48 AM, Bernardo Botella wrote:
 With all the feedback that came in the discussion thread after the 
 call for votes, I’d like to extend the period another 72 hours 
 starting today.
 
 As before, a vote passes if there are at least 3 binding +1s and no 
 binding vetoes.
 
 Thanks,
 Bernardo Botella
 
> On Jun 24, 2024, at 7:17 AM, Bernardo Botella 
>  wrote:
> 
> Hi everyone,
> 
> I would like to start the voting for CEP-42.
> 
> Proposal: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-42%3A+Constraints+Framework
> Discussion: 
> https://lists.apache.org/thread/xc2phmxgsc7t3y9b23079vbflrhyyywj
> 
> The vote will be open for 72 hours. A vote passes if there are at 
> least 3 binding +1s and no binding vetoes.
> 
> Thanks,
> Bernardo Botella
>>> 

Re: [VOTE] Release Apache Cassandra 5.0-rc1

[Josh McKenzie]

> Perhaps we should consider a Milestone release.  At least in some projects 
> this is a way to provide a test bed with known issues that will be corrected 
> before an RC.
How does that differ from beta in our lifecycle? API stable but a test bed to 
suss out issues like this.


On Mon, Jul 1, 2024, at 9:30 AM, Claude Warren, Jr via dev wrote:
> Perhaps we should consider a Milestone release.  At least in some projects 
> this is a way to provide a test bed with known issues that will be corrected 
> before an RC.
> 
> On Sun, Jun 30, 2024 at 9:50 PM Jon Haddad  wrote:
>> This came in after our vote, but we might also have a problem with 
>> performing schema changes after a full restart.  Appears to only be if the 
>> entire cluster was shut down, according to the report.  If it's true, this 
>> might affect anyone trying to restore from a backup.  This would also be a 
>> blocker for me, if that's the case.
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-19735
>> 
>> Jon
>> 
>> 
>> On Thu, Jun 27, 2024 at 9:49 PM Jon Haddad  wrote:
>>> Thanks for confirming this, Blake. I agree that we should not knowingly 
>>> ship new versions with severe bugs that cause the DB to crash, regression 
>>> or not. 
>>> 
>>> -1 from me as well
>>> 
>>> 
>>> On Fri, Jun 28, 2024 at 1:39 AM Blake Eggleston  
>>> wrote:
>>>> Looking at the ticket, I’d say Jon’s concern is legitimate. The segfaults 
>>>> Jon is seeing are probably caused by paxos V2 when combined with off heap 
>>>> memtables for the reason Benedict suggests in the JIRA. This problem will 
>>>> continue to exist in 5.0. Unfortunately, it looks like the patch posted is 
>>>> not enough to address the issue and will need to be a bit more involved to 
>>>> properly fix the problem.
>>>> 
>>>> While this is not a regression, I think Jon’s point about trie memtables 
>>>> increasing usage of off heap memtables is a good one, and anyway we 
>>>> shouldn’t be doing major releases with known process crashing bugs.
>>>> 
>>>> So I’m voting -1 on this release and will work with Jon and Benedict to 
>>>> get this fixed.
>>>> 
>>>> Thanks,
>>>> 
>>>> Blake
>>>> 
>>>> 
>>>>> On Jun 26, 2024, at 6:47 AM, Josh McKenzie  wrote:
>>>>> 
>>>>> Blake or Benedict - can either of you speak to Jon's concerns around 
>>>>> CASSANDRA-19668?
>>>>> 
>>>>> On Wed, Jun 26, 2024, at 12:18 AM, Jeff Jirsa wrote:
>>>>>> 
>>>>>> +1
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Jun 25, 2024, at 5:04 AM, Mick Semb Wever  wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> Proposing the test build of Cassandra 5.0-rc1 for release.
>>>>>>> 
>>>>>>> sha1: b43f0b2e9f4cb5105764ef9cf4ece404a740539a
>>>>>>> Git: https://github.com/apache/cassandra/tree/5.0-rc1-tentative
>>>>>>> Maven Artifacts: 
>>>>>>> https://repository.apache.org/content/repositories/orgapachecassandra-1336/org/apache/cassandra/cassandra-all/5.0-rc1/
>>>>>>> 
>>>>>>> The Source and Build Artifacts, and the Debian and RPM packages and 
>>>>>>> repositories, are available here: 
>>>>>>> https://dist.apache.org/repos/dist/dev/cassandra/5.0-rc1/
>>>>>>> 
>>>>>>> The vote will be open for 72 hours (longer if needed). Everyone who has 
>>>>>>> tested the build is invited to vote. Votes by PMC members are 
>>>>>>> considered binding. A vote passes if there are at least three binding 
>>>>>>> +1s and no -1's.
>>>>>>> 
>>>>>>> [1]: CHANGES.txt: 
>>>>>>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/CHANGES.txt
>>>>>>> [2]: NEWS.txt: 
>>>>>>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/NEWS.txt

Re: [VOTE] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances

[Josh McKenzie]

+1

On Thu, Jun 27, 2024, at 12:40 AM, Abhijeet Dubey wrote:
> +1
> 
> On Thu, Jun 27, 2024 at 1:47 AM Francisco Guerrero  wrote:
>> +1
>> 
>> On 2024/06/21 15:13:31 Venkata Hari Krishna Nukala wrote:
>> > Hi everyone,
>> > 
>> > I would like to start the voting for CEP-40 as all the feedback in the
>> > discussion thread seems to be addressed.
>> > 
>> > Proposal:
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-40%3A+Data+Transfer+Using+Cassandra+Sidecar+for+Live+Migrating+Instances
>> > Discussion thread:
>> > https://lists.apache.org/thread/g397668tp0zybf29g8hgbllv7t3j493f
>> > 
>> > As per the CEP process documentation, this vote will be open for 72 hours
>> > (longer if needed).
>> > 
>> > Thanks!
>> > Hari
>> >
> 
> 
> --
> *Abhijeet Dubey*
> Software Engineer @ Apple Inc.
> IIT Bombay Computer Science & Engineering Class of 2019
> Contact : +91-9900190105
> Apple Inc. | IIT Bombay
> 

Re: [VOTE] Release Apache Cassandra 5.0-rc1

[Josh McKenzie]

Blake or Benedict - can either of you speak to Jon's concerns around 
CASSANDRA-19668?

On Wed, Jun 26, 2024, at 12:18 AM, Jeff Jirsa wrote:
> 
> +1
> 
> 
> 
>> On Jun 25, 2024, at 5:04 AM, Mick Semb Wever  wrote:
>> 
>> 
>> Proposing the test build of Cassandra 5.0-rc1 for release.
>> 
>> sha1: b43f0b2e9f4cb5105764ef9cf4ece404a740539a
>> Git: https://github.com/apache/cassandra/tree/5.0-rc1-tentative
>> Maven Artifacts: 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1336/org/apache/cassandra/cassandra-all/5.0-rc1/
>> 
>> The Source and Build Artifacts, and the Debian and RPM packages and 
>> repositories, are available here: 
>> https://dist.apache.org/repos/dist/dev/cassandra/5.0-rc1/
>> 
>> The vote will be open for 72 hours (longer if needed). Everyone who has 
>> tested the build is invited to vote. Votes by PMC members are considered 
>> binding. A vote passes if there are at least three binding +1s and no -1's.
>> 
>> [1]: CHANGES.txt: 
>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/CHANGES.txt
>> [2]: NEWS.txt: 
>> https://github.com/apache/cassandra/blob/5.0-rc1-tentative/NEWS.txt

Re: [VOTE][IP CLEARANCE] GoCQL driver

[Josh McKenzie]

+1; appreciate you driving this Mick and everyone rallying.

On Tue, Jun 25, 2024, at 1:32 PM, Mick Semb Wever wrote:
>   .
>  
>> The vote will be open for 72 hours (or longer). Votes by PMC members are 
>> considered binding. A vote passes if there are at least three binding +1s 
>> and no -1's.
> 
> 
> +1
> 
> 

Re: [DISCUSS] CEP-42: Constraints Framework

[Josh McKenzie]

> I was referring to the name guardrail, using the same infra as guardrails
Curious if there's a subtle distinction implicit in this (or just in my 
brain...). A guardrail is something one person puts in place for someone else - 
in our case operators to users. Constraints are something inherent to the 
use-case or the abstractions, and something users are putting on themselves. 
Maybe... either that or we just named guardrails in a NIH way. /sad

That aside, if other DB's are using "constraint" I think it's the most 
user-friendly approach to use terminology they're already familiar with from 
other contexts. And we need more user-friendly. ;)

As for the infra - if we're talking plumbing inside the guts I agree. I think 
the UX definitely should be via DDL rather than the .yaml file; I don't think 
that's what you were alluding to but just figured it's worth pointing it out.

My intuition is the vote got called a *smidge* early but that things are very 
much moving in the right direction and are very close.

On Tue, Jun 25, 2024, at 1:41 PM, Bernardo Botella wrote:
> Hi Ariel,
> 
> Your suggestions make sense, and I’ll be updating the CEP with the details. 
> Basically:
> - We have an optional name for the constraints. If the name is not provided, 
> a random name is generated for a constraint:
> CREATE TABLE keyspace.table ( p1 int, p2 int, ..., CONSTRAINT [name] CHECK p1 
> != p2 );
> 
> - Alter and Drop constraints are as follows
> ALTER CONSTRAINT [name] CHECK new_condition DROP CONSTRAINT [name]
> 
> - Describe table returns the list of constraints for a table.
> - The condition of the CONSTRAINT (after the CHECK keyword) can be surrounded 
> by optional parentheses to keep consistency with other databases syntax.
> 
> I will update the CEP with those details.
> 
> To Dinesh’s point, I agree that a NOT NULL constraint will be really useful. 
> I can add it to the list on the CEP
> 
> Regards,
> Bernardo
> 
> 
>> On Jun 25, 2024, at 9:22 AM, Ariel Weisberg  wrote:
>> 
>> Hi,
>> 
>> I am also +1 on Doug's distinction between things that can be managed by 
>> operators and things that can be managed by applications.
>> 
>> Some things to note about the syntax is that there are parens around the 
>> condition in SQL. In your example there are multiple anonymous constraints 
>> on the same column, how are anonymous constraints handled? Does the database 
>> automatically generate a named constraint for them so they can be referenced 
>> later? Do we allow multiple constraints on the same column and AND them 
>> together?
>> 
>> Ariel
>> 
>> 
>> 
>> On Mon, Jun 24, 2024, at 6:43 PM, Bernardo Botella wrote:
>>> Hi Ariel and Jon,
>>> 
>>> Let me address your question first. Yes, AND is supported in the proposal. 
>>> Below you can find some examples of different constraints applied to the 
>>> same column.
>>> 
>>> As per the LENGTH name instead of sizeOf as in the proposal, I am also not 
>>> opposed to it if it is more consistent with terminology in the databases 
>>> universe.
>>> 
>>> So, to recap, there seems to be general agreement on the usefulness of the 
>>> Constraints Framework.
>>> Now, from the feedback that has arrived after the voting has been called, I 
>>> see there are three different proposals for syntax:
>>> 
>>> 1.-
>>> The syntax currently described in the CEP. Example:
>>> CREATE TYPE keyspace.cidr_address_ipv4 (
>>>   ip_adress inet,
>>>   subnet_mask int,
>>>   CONSTRAINT subnet_mask > 0,
>>>   CONSTRAINT subnet_mask < 32
>>> )
>>> 
>>> 2.-
>>> As Jon suggested, leaving this definitions to more specific Guardrails at 
>>> table level. Example, something like:
>>> column_min_int_value_size_threshold_keyspace_address_ipv4_ip_adress = 0
>>> column_max_int_value_size_threshold_keyspace_address_ipv4_ip_adress = 32
>>> 
>>> 3.-
>>> As Ariel suggested, having the CHECK keyword added to align consistency 
>>> with SQL. Example:
>>> CREATE TYPE keyspace.cidr_address_ipv4 (
>>>   ip_adress inet,
>>>   subnet_mask int,
>>>   CONSTRAINT CHECK subnet_mask > 0,
>>>   CONSTRAINT CHECK subnet_mask < 32
>>> )
>>> 
>>> For the guardrails vs cql syntax, I think that keeping the conceptual 
>>> separation that has been explored in this thread, and perfectly recapped by 
>>> Doug, is closer to what we are trying to achieve with this framework. In my 
>>> opinion, having them in the CQL schema definition provides those 
>>> application level constraints that Doug mentions in an more accesible way 
>>> than having to configure such specific guardrais.
>>> 
>>> For the addition of the CHECK keyword, I'm definitely not opposed to it if 
>>> it helps Cassandra users coming from other databases understand concepts 
>>> that were already familiar to them.
>>> 
>>> I hope this helps move the conversation forward,
>>> Bernardo
>>> 
>>> 
>>> 
 On Jun 24, 2024, at 12:17 PM, Ariel Weisberg  wrote:
 
 Hi,
 
 I see a vote for this has been called. I should have provided more prompt 
 feedback 

Re: [DISCUSS] Increments on non-existent rows in Accord

[Josh McKenzie]

What about aborting the transaction / raising an exception if you try and do an 
incremental operator against a non-existent PK?

The reasonable inferred intent of the user is to change a value that's expected 
to be there, so if it's not there it's an error case right? Otherwise you'd 
append "IF EXISTS".

On Fri, Jun 21, 2024, at 1:56 AM, Caleb Rackliffe wrote:
> It does, but the primary reason it does is that it is setting a value, not 
> incrementing one. When we’re setting a value, we don’t care what was there 
> before. Incrementing a value is not possible in a non-transitional update, 
> hence this thread…
> 
>> On Jun 20, 2024, at 5:17 PM, Bernardo Botella  
>> wrote:
>> Doesn’t an UPDATE statement creates a row if the partition key does not 
>> exist? That’s also confirmed by the official Cassandra documentation here 
>> :
>> 
>> ”Unlike in SQL, `UPDATE` does not check the prior existence of the row by 
>> default. The row is created if none existed before, and updated otherwise. 
>> Furthermore, there is no means of knowing which action occurred.”
>> 
>> That being the case, I think the second option you mention is what keeps 
>> consistency with the UPDATEs out of the transaction.
>> 
>> Kind regards,
>> Bernardo
>> 
>>> On Jun 20, 2024, at 1:54 PM, Caleb Rackliffe  
>>> wrote:
>>> 
>>> We had a bug report a while back from Luis E Fernandez and team in 
>>> CASSANDRA-18988  
>>> around the behavior of increments/decrements on numeric fields for 
>>> non-existent rows. Consider the following, wich can be run on the 
>>> cep-15-accord branch:
>>> 
>>> CREATE KEYSPACE accord WITH replication = {'class': 'SimpleStrategy', 
>>> 'replication_factor': '1'} AND durable_writes = true
>>> 
>>> CREATE TABLE accord.accounts (
>>> partition text,
>>> account_id int,
>>> balance int,
>>> PRIMARY KEY (partition, account_id)
>>> ) WITH CLUSTERING ORDER BY (account_id ASC) AND transactional_mode='full'
>>> 
>>> BEGIN TRANSACTION
>>> INSERT INTO accord.accounts (partition, account_id, balance) VALUES 
>>> ('default', 0, 100);
>>> INSERT INTO accord.accounts (partition, account_id, balance) VALUES 
>>> ('default', 1, 100);
>>> COMMIT TRANSACTION
>>> 
>>> BEGIN TRANSACTION
>>> UPDATE accord.accounts SET balance -= 10 WHERE partition = 'default' 
>>> AND account_id = 1;
>>> UPDATE accord.accounts SET balance += 10 WHERE partition = 'default' 
>>> AND account_id = 3;
>>> COMMIT TRANSACTION
>>> 
>>> Reading the 'default' partition will produce the following result.
>>> 
>>>  partition | account_id | balance
>>> ---++-
>>>default |  0 | 100
>>>default |  1 |  90
>>> 
>>> As you will notice, we have not implicitly inserted a row for account_id 3, 
>>> which does not exist when we request that its balance be incremented by 10. 
>>> This is by design, as null + 10 == null.
>>> 
>>> Before I close CASSANDRA-18988 
>>> , *I'd like to 
>>> confirm with everyone reading this that the behavior above is reasonable*. 
>>> The only other option I've seen proposed that would make sense is perhaps 
>>> producing a result like:
>>> 
>>>  partition | account_id | balance
>>> ---++-
>>>default |  0 | 100
>>>default |  1 |  90
>>>default |  3 |null
>>> 
>>> Note however that this is exactly what we would produce if we had first 
>>> inserted a row w/ no value for balance:
>>> 
>>> INSERT INTO accord.accounts (partition, account_id) VALUES ('default', 3);

Re: [EXTERNAL] Re: [DISCUSS] Adding experimental vtables and rules around them

[Josh McKenzie]

> I like it as it makes this apart of its name, but is sad that we must break 
> compatibility once we mark it stable (as we move the keyspace).  
+1 to this exact set of sentiments. The next logical question to me is "should 
we alias the old system_experimental* table name to the new stable table name 
for vtables once they graduate to support backwards compatibility", but I think 
I fall on the "no" side there.

On Fri, May 31, 2024, at 5:15 PM, J. D. Jordan wrote:
> 
> We have already agreed in the past that having experimental features, behind 
> feature flags, in stable releases is a good thing for keeping those features 
> up to date, for getting feedback from end users, and many others.
> The question here is about how we ensure that end users are aware something 
> is experimental, because the person who installed C* and enabled the feature 
> flag is likely not the same person who is using the feature.
> 
>> On May 31, 2024, at 3:23 PM, German Eichberger via dev 
>>  wrote:
>> 
>> Hi,
>> 
>> To sum where everyone is coming from: We would like to have features in a 
>> stable version of Cassandra which are experimental and are subject to 
>> non-backward compatible change. This indicates to me that the feature is not 
>> finished and should likely not be included in a stable release.  What 
>> benefit are we looking for by including it into a stable release as opposed 
>> to rolling it to the next release.
>> 
>> Thanks,
>> German
>> 
>> 
>> *From:* Maxim Muzafarov 
>> *Sent:* Wednesday, May 29, 2024 1:09 PM
>> *To:* dev@cassandra.apache.org 
>> *Subject:* [EXTERNAL] Re: [DISCUSS] Adding experimental vtables and rules 
>> around them
>>  
>> Hello everyone,
>> 
>> I like the idea of highlighting some of the experimental virtual
>> tables whose model might be changed in future releases.
>> 
>> As another option, we could add an @Experimetal annotation (or another
>> name) and a configuration parameter
>> experimental_virtula_tables_enabled (default is false). This, in turn,
>> means if a virtual table is experimental, it won't be registered in a
>> virtual keyspace unless the corresponding configuration parameter is
>> enabled. This also means that a user must implicitly enable an
>> experimental API, and prevent us from spamming the log with warnings.
>> All of this does not preclude us from specifying the experimental
>> state of some virtual tables in the documentation.
>> 
>> On Wed, 29 May 2024 at 21:18, Abe Ratnofsky  wrote:
>> >
>> > I agree that ClientWarning is the best way to indicate the risk of using 
>> > an experimental feature directly to the user. Presenting information in 
>> > the client application's logs directly means that the person who wrote the 
>> > query is most likely to see the warning, rather than an operator who sees 
>> > cluster logs.
>> >
>> > I don't think it's necessary to attach a ClientWarning to every single 
>> > client response; a ClientWarning analog to NoSpamLogger would be useful 
>> > for this ("warn a client at most once per day").
>> >
>> > This would also be useful for warning on usage of deprecated features.
>> >
>> > > On May 29, 2024, at 3:01 PM, David Capwell  wrote:
>> > >
>> > > We agreed a long time ago that all new features are disabled by default, 
>> > > but I wanted to try to flesh out what we “should” do with something that 
>> > > might still be experimental and subject to breaking changes; I would 
>> > > prefer we keep this thread specific to vtables as the UX is different 
>> > > for different types of things…
>> > >
>> > > So, lets say we are adding a set of vtables but we are not 100% sure 
>> > > what the schema should be and we learn after the release that changes 
>> > > should be made, but that would end up breaking the table… we currently 
>> > > define everything as “don’t break this” so if we publish a table that 
>> > > isn’t 100% baked we are kinda stuck with it for a very very long time… I 
>> > > would like to define a way to expose vtables that are subject to change 
>> > > (breaking schema changes) across different release and rules around them 
>> > > (only in minor?  Maybe even in patch?).
>> > >
>> > > Lets try to use a concrete example so everyone is on the same page.
>> > >
>> > > Accord is disabled by default (it is a new feature), so the vtables to 
>> > > expose internals would be expected to be undefined and not present on 
>> > > the instance.
>> > >
>> > > When accord is enabled (accord.enabled = true) we add a set of vtables:
>> > >
>> > > Epochs - shows what epochs are known to accord
>> > > Cache - shows how the internal caches are performing
>> > > Etc.
>> > >
>> > > Using epochs as an example it currently only shows a single column: the 
>> > > long epoch
>> > >
>> > > CREATE VIRTUAL TABLE system_accord.epochs (epoch bigint PRIMARY KEY);
>> > >
>> > > Lets say we find that this table isn’t enough and we really need to 
>> > > scope it to each of the “stores” (threads for processing accord tasks)
>> > >
>>

Cassandra PMC Chair Rotation, 2024 Edition

[Josh McKenzie]

Another PMC Chair baton pass incoming! On behalf of the Apache Cassandra 
Project Management Committee (PMC) I would like to welcome and congratulate our 
next PMC Chair Dinesh Joshi (djoshi).

Dinesh has been a member of the PMC for a few years now and many of you likely 
know him from his thoughtful, measured presence on many of our collective 
discussions as we've grown and evolved over the past few years.

I appreciate the project trusting me as liaison with the board over the past 
year and look forward to supporting Dinesh in the role in the future.

Repeating Mick (repeating Paulo's) words from last year: The chair is an 
administrative position that interfaces with the Apache Software Foundation 
Board, by submitting regular reports about project status and health. Read more 
about the PMC chair role on Apache projects:
- https://www.apache.org/foundation/how-it-works.html#pmc
- https://www.apache.org/foundation/how-it-works.html#pmc-chair
- https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers

The PMC as a whole is the entity that oversees and leads the project and any 
PMC member can be approached as a representative of the committee. A list of 
Apache Cassandra PMC members can be found on: 
https://cassandra.apache.org/_/community.html

Re: CCM and CASSANDRA_USE_JDK11

[Josh McKenzie]

> The scripts that are in cassandra-builds seem like a starting point for 
> converging different CI systems so that they run the same set of tests in as 
> similar environments as possible
Yeah, I took a superset of circle and ASF tests to try and run :allthethings:. 
Part of how the checkstyle dependency check got in the way too, since we 
weren't running that on ASF CI. :)

Strong +1 to more convergence on what we're running in CI for sure.

On Fri, May 24, 2024, at 11:59 AM, Ariel Weisberg wrote:
> Hi,
> 
> There is definitely a mismatch between how the full range of dtests work and 
> the direction CCM is going in and we have some difficulty getting those to 
> match. I fully empathize with several of those CI systems not being publicly 
> visible/accessible, and the behavior of upgrade paths being absolutely 
> inscrutable relative to the environment variables that are set.
> 
> I am happy to volunteer to test things in advance on Apple's CI. I'll also 
> try to get on top of responding faster :-)
> 
> The window where reverting is useful is slightly past now that all the issues 
> I am aware of have been fixed, but in the future I think the burden for 
> revert might need to be lower. It's tough those because putting the burden on 
> ASF for non-ASF CI is not necessarily a given.
> 
> There is a big gap between CI systems where how they invoke the dtests 
> determines the exact set of tests they run and how they invoke CCM (and which 
> CCM bugs they expose). I really don't like this approach including relying on 
> environment variables to dictate dtests execution behavior. I hope to have 
> some time to spend on this once my live migration work is in a better place.
> 
> Right now ASF CI is not running the upgrade paths that trigger JDK version 
> switching which is at the root of our recent problems. Once we close that gap 
> we should be in a much better place in terms of divergence.
> 
> The scripts that are in cassandra-builds seem like a starting point for 
> converging different CI systems so that they run the same set of tests in as 
> similar environments as possible and harness specific quirks are pushed into 
> specific integration points where things like pointing to private mirrors is 
> supported.
> 
> Additionally what I would like to see is that CI harnesses specify the 
> location of all JDKs, and then provide flags (not environment variables) to 
> the dtests that dictate what should be run. What is currently in Java path or 
> Java home shouldn't be relevant for any dtests IMO, I would like the dtests 
> (themselves or delegating to CCM) to juggle that themselves.
> 
> Those flags should also be as declarative as possible and require specifying 
> C* versions and JDK versions so if you want to run the set of tests we 
> required to commit you don't need keep changing how the dtests are invoked. 
> 
> Ariel
> 
> On Thu, May 23, 2024, at 6:22 AM, Mick Semb Wever wrote:
>>> When starting Cassandra nodes, CCM uses the current env Java distribution 
>>> (defined by the JAVA_HOME env variable). This behavior is overridden in 
>>> three cases:
>>> 
>>> - Java version is not supported by the selected Cassandra distribution - in 
>>> which case, CCM looks for supported Java distribution across JAVAx_HOME env 
>>> variables
>>> 
>>> - Java version is specified explicitly (--jvm-version arg or jvm_version 
>>> param if used in Python)
>>> 
>>> - CASSANDRA_USE_JDK11 is defined in env, in which case, for Cassandra 4.x 
>>> CCM forces to use only JDK11
>>> 
>>> 
>>> 
>>> I want to ask you guys whether you are okay with removing the third 
>>> exception. If we remove it, Cassandra 4.x will not be treated in any 
>>> special way—CCM will use the current Java version, so if it is Java 11, it 
>>> will use Java 11 (and automatically set CASSANDRA_USE_JDK11), and if it is 
>>> Java 8, it will use Java 8 (and automatically unset CASSANDRA_USE_JDK11). 
>>> 
>>> 
>>> 
>>> I think there is no need for CCM to use CASSANDRA_USE_JDK11 to make a 
>>> decision about which Java version to use as it adds more complexity, makes 
>>> it work differently for Cassandra 4.x than for other Cassandra versions, 
>>> and actually provides no value at all because if we work with Cassandra 
>>> having our env configured for Java 11, we have to have CASSANDRA_USE_JDK11 
>>> and if not, we cannot have it. Therefore, CCM can be based solely on the 
>>> current Java version and not include the existence of CASSANDRA_USE_JDK11 
>>> in the Java version selection process.
>>> 
>>> 
>>> WDYT? 
>> 
>>  
>> With the recent commits to ccm we have now broken three different CI 
>> systems, in numerous different ways.  All remain broken.
>> 
>> At this point in time, the default behaviour should be to revert those 
>> commits.  Not to discuss whether we can further remove existing 
>> functionality on the assumption we know all consumers, or that they are all 
>> reading this thread and agreeing.
>> 
>> In ccm, the jdk selection and

Re: CCM and CASSANDRA_USE_JDK11

[Josh McKenzie]

Agree with Mick on the revert, and agree w/you Jacek on this area needing to be 
cleaned up.

I'm just as irritated as anyone about how archaic and hard to trace our CI 
system is (trust me. The last 9 months has been... challenging), but with 
things like this that are effectively API's, the blast radius is wide and deep. 
It deserves a cleanup, but it needs conversations like this (I say as one of 
those people w/a downstream CI system that got busted by this change =/)

The fact that we have CASSANDRA_USE_JDK11 in a world where we're supporting 17 
and looking towards 21 is just... ugh.


On Thu, May 23, 2024, at 6:22 AM, Mick Semb Wever wrote:
>> When starting Cassandra nodes, CCM uses the current env Java distribution 
>> (defined by the JAVA_HOME env variable). This behavior is overridden in 
>> three cases:
>> 
>> - Java version is not supported by the selected Cassandra distribution - in 
>> which case, CCM looks for supported Java distribution across JAVAx_HOME env 
>> variables
>> 
>> - Java version is specified explicitly (--jvm-version arg or jvm_version 
>> param if used in Python)
>> 
>> - CASSANDRA_USE_JDK11 is defined in env, in which case, for Cassandra 4.x 
>> CCM forces to use only JDK11
>> 
>> 
>> 
>> I want to ask you guys whether you are okay with removing the third 
>> exception. If we remove it, Cassandra 4.x will not be treated in any special 
>> way—CCM will use the current Java version, so if it is Java 11, it will use 
>> Java 11 (and automatically set CASSANDRA_USE_JDK11), and if it is Java 8, it 
>> will use Java 8 (and automatically unset CASSANDRA_USE_JDK11). 
>> 
>> 
>> 
>> I think there is no need for CCM to use CASSANDRA_USE_JDK11 to make a 
>> decision about which Java version to use as it adds more complexity, makes 
>> it work differently for Cassandra 4.x than for other Cassandra versions, and 
>> actually provides no value at all because if we work with Cassandra having 
>> our env configured for Java 11, we have to have CASSANDRA_USE_JDK11 and if 
>> not, we cannot have it. Therefore, CCM can be based solely on the current 
>> Java version and not include the existence of CASSANDRA_USE_JDK11 in the 
>> Java version selection process.
>> 
>> 
>> WDYT? 
> 
>  
> With the recent commits to ccm we have now broken three different CI systems, 
> in numerous different ways.  All remain broken.
> 
> At this point in time, the default behaviour should be to revert those 
> commits.  Not to discuss whether we can further remove existing functionality 
> on the assumption we know all consumers, or that they are all reading this 
> thread and agreeing.
> 
> In ccm, the jdk selection and switching does indeed deserve a clean up.  We 
> have found a number of superfluous ways of achieving the same thing that is 
> leading to unnecessary code complexity.  But we should not be hard breaking 
> things for downstream users and our CI.
> 
> The initial commit to ccm that broke things was to fix ccm running a binary 
> 5.0-beta1 with a particular jdk.  This patch and subsequent fixes has 
> included additional refactoring/cleaning changes that have broken a number of 
> things, like jdk-switching and upgrade_through_versions tests.  We keep 
> trying to fix each breakage, but are also including additional adjustments 
> "to do the right thing" that only ends up breaking yet another thing.   This 
> shouldn't be how we apply changes to a library that has many (unknown) 
> consumers, nor that we don't have full test coverage on.
> 
> Given the broken CI systems and the troubles we have already caused 
> consumers, my recommendation is that these commits are reverted, and we live 
> with the binary 5.0-beta1 breakage for now, while we more patiently work on a 
> more complete and thorough fix.  Furthermore to the specific question in the 
> post, I don't believe we should be removing working functionality without 
> first a deprecation cycle, given that ccm has many unknown consumers.  This 
> depreciation period can be time-based, since ccm doesn't have versions.
> 
> 
> 
> 
> 
> 

Re: [DISCUSS] ccm as a subproject

[Josh McKenzie]

Sounds like we have a general consensus from the project on being willing to 
accept the donation, should the current rights owners be interested in said 
donation.

> We've been working on this along with the python-driver (just haven't raised 
> it yet).
Which they indicate they are. :)

I'll follow up on this topic offline w/Mick. Thanks everyone for the good 
conversation and feedback on it.

~Josh

On Mon, May 20, 2024, at 2:36 PM, Jordan West wrote:
> I would also love to see CCM as an official side project. It is important to 
> the project and I personally use it regularly. 
> 
> Jordan
> 
> On Thu, May 16, 2024 at 7:55 AM Josh McKenzie  wrote:
>> __
>>> We do still have the issues of DSE-supporting code in it, as we do with the 
>>> drivers.  I doubt any of us strongly object to it: there's no trickery 
>>> happening here on the user; but we should be aware of it and have a rough 
>>> direction sketched out for when someone else comes along wanting to add 
>>> support for their proprietary product.
>> IMO as long as it's documented well at the outset and we have plans to 
>> slowly refactor to move it to clean boundaries (epic in JIRA anyone <3) so 
>> it can be extracted into a separately maintained module by folks that need 
>> it, I think we'd be in great shape. That'd also pave a path for others 
>> wanting to add support for their proprietary products as well. Win-win.
>> 
>> There's always this chicken or egg problem w/things like ccm. Do people not 
>> contribute to it because it's out of the umbrella, or is it out of the 
>> umbrella because people don't need to contribute to it?
>> 
>> I hadn't thought about other subprojects relying on it. That's a very good 
>> point.
>> 
>> On Thu, May 16, 2024, at 4:48 AM, Jacek Lewandowski wrote:
>>> +1 (my personal opinion)
>>> 
>>> How to deal with the DSE-supporting code is a separate discussion IMO
>>> 
>>> - - -- --- - ---- -
>>> Jacek Lewandowski
>>> 
>>> 
>>> czw., 16 maj 2024 o 10:21 Berenguer Blasi  
>>> napisał(a):
>>>> __
>>>> +1 ccm is super useful
>>>> 
>>>> On 16/5/24 10:09, Mick Semb Wever wrote:
>>>>> 
>>>>> 
>>>>> On Wed, 15 May 2024 at 16:24, Josh McKenzie  wrote:
>>>>>> Right now ccm isn't formally a subproject of Cassandra or under 
>>>>>> governance of the ASF. Given it's an integral components of our CI as 
>>>>>> well as for local testing for many devs, and we now have more experience 
>>>>>> w/our muscle on IP clearance and ingesting / absorbing subprojects where 
>>>>>> we can't track down every single contributor to get an ICLA, seems like 
>>>>>> it might be worth revisiting the topic of donation of ccm to Apache.
>>>>>> 
>>>>>> For what it's worth, Sylvain originally and then DataStax after transfer 
>>>>>> have both been incredible and receptive stewards of the projects and 
>>>>>> repos, so this isn't about any response to any behavior on their part. 
>>>>>> Structurally, however, it'd be better for the health of the project(s) 
>>>>>> long-term to have ccm promoted in. As far as I know there was strong 
>>>>>> receptivity to that donation in the past but the IP clearance was the 
>>>>>> primary hurdle.
>>>>>> 
>>>>>> Anyone have any thoughts for or against?
>>>>>> 
>>>>>> https://github.com/riptano/ccm
>>>>> 
>>>>> 
>>>>> 
>>>>> We've been working on this along with the python-driver (just haven't 
>>>>> raised it yet).  It is recognised, like the python-driver, as a key 
>>>>> dependency that would best be in the project.
>>>>> 
>>>>> Obtaining the CLAs should be much easier, the contributors to ccm are 
>>>>> less diverse, being more the people we know already.
>>>>> 
>>>>> We do still have the issues of DSE-supporting code in it, as we do with 
>>>>> the drivers.  I doubt any of us strongly object to it: there's no 
>>>>> trickery happening here on the user; but we should be aware of it and 
>>>>> have a rough direction sketched out for when someone else comes along 
>>>>> wanting to add support for their proprietary product.  We also don't want 
>>>>> to be pushing downstream users to be having to create their own forks 
>>>>> either.
>>>>> 
>>>>> Great to see general consensus (so far) in receiving it :) 
>>>>>  
>> 

Re: [DISCUSS] Gossip Protocol Change

[Josh McKenzie]

I'm +1 to continuing work on CASSANDRA-18917 for all the reasons Jordan listed.

Sounds like the request was to hit the pause button until TCM merged rather 
than skipping the work entirely so that's promising.

On Thu, May 16, 2024, at 1:43 PM, Jon Haddad wrote:
> I have also recently worked with a teams who lost critical data as a result 
> of gossip issues combined with collision in our token allocation.  I haven’t 
> filed a jira yet as it slipped my mind but I’ve seen it in my own testing as 
> well. I’ll get a JIRA in describing it in detail. 
>  
> It’s severe enough that it should probably block 5.0. 
> 
> Jon
> 
> On Thu, May 16, 2024 at 10:37 AM Jordan West  wrote:
>> I’m a big +1 on 18917 or more testing of gossip. While I appreciate that it 
>> makes TCM more complicated, gossip and schema propagation bugs have been the 
>> source of our two worst data loss events in the last 3 years. Data loss 
>> should immediately cause us to evaluate what we can do better. 
>> 
>> We will likely live with gossip for at least 1, maybe 2, more years. 
>> Otherwise outside of bug fixes (and to some degree even still) I think the 
>> only other solution is to not touch gossip *at all* until we are all 
>> TCM-only which I don’t think is practical or realistic. recent changes to 
>> gossip in 4.1 introduced several subtle bugs that had serious impact (from 
>> data loss to loss of ability to safely replace nodes in the cluster). 
>> 
>> I am happy to contribute some time to this if lack of folks is the issue. 
>> 
>> Jordan 
>> 
>> On Mon, May 13, 2024 at 17:05 David Capwell  wrote:
>>> So, I created https://issues.apache.org/jira/browse/CASSANDRA-18917 which 
>>> lets you do deterministic gossip simulation testing cross large clusters 
>>> within seconds… I stopped this work as it conflicted with TCM (they were 
>>> trying to merge that week) and it hit issues where some nodes never 
>>> converged… I didn’t have time to debug so I had to drop the patch…
>>> 
>>> This type of change would be a good reason to resurrect that patch as 
>>> testing gossip is super dangerous right now… its behavior is only in a few 
>>> peoples heads and even then its just bits and pieces scattered cross 
>>> multiple people (and likely missing pieces)… 
>>> 
>>> My brain is far too fried right now to say your idea is safe or not, but 
>>> honestly feel that we would need to improve our tests (we have 0) before 
>>> making such a change… 
>>> 
>>> I do welcome the patch though...
>>> 
>>> 
 On May 12, 2024, at 8:05 PM, Zemek, Cameron via dev 
  wrote:
 
 In looking into CASSANDRA-19580 I noticed something that raises a 
 question. With Gossip SYN it doesn't check for missing digests. If its 
 empty for shadow round it will add everything from endpointStateMap to the 
 reply. But why not included missing entries in normal replies? The 
 branching for reply handling of SYN requests could then be merged into 
 single code path (though shadow round handles empty state different with 
 CASSANDRA-16213). Potential is performance impact as this requires doing a 
 set difference.
 
 For example, something along the lines of:
 
 ```
 Set missing = new 
 HashSet<>(endpointStateMap.keySet());
 
 missing.removeAll(gDigestList.stream().map(GossipDigest::getEndpoint).collect(Collectors.toSet()));
 for ( InetAddressAndPort endpoint : missing)
 {
 gDigestList.add(new GossipDigest(endpoint, 0, 0));
 }
 ```
 
 It seems odd to me that after shadow round for a new node we have 
 endpointStateMap with only itself as an entry. Then the only way it gets 
 the gossip state is by another node choosing to send the new node a gossip 
 SYN. The choosing of this is random. Yeah this happens every second so 
 eventually its going to receive one (outside the issue of CASSANDRA-19580 
 were it doesn't if its in a dead state like hibernate) , but doesn't this 
 open up bootstrapping to failures on very large clusters as it can take 
 longer before its sent a SYN (as the odds of being chosen for SYN get 
 lower)? For years been seeing bootstrap failures with 'Unable to contact 
 any seeds' but they are infrequent and never been able to figure out how 
 to reproduce in order to open a ticket, but I wonder if some of them have 
 been due to not receiving a SYN message before it does the seenAnySeed 
 check.

Re: [DISCUSS] Adding support for BETWEEN operator

[Josh McKenzie]

> More of a "how could we technically reach mars?" discussion than a "how we 
> get congress to authorize a budget to reach mars?"
Wow - that is genuinely a great simile. Really good point.

To Jeff's point - want to kick off a [DISCUSS] thread referencing this thread 
Jon so we can take the conversation there? Definitely think it's worth 
continuing from a technical perspective.

On Wed, May 15, 2024, at 2:49 PM, Jeff Jirsa wrote:
> You can remove the shadowed values at compaction time, but you can’t ever 
> fully propagate the range update to point updates, so you’d be propagating 
> all of the range-update structures throughout everything forever. It’s JUST 
> like a range tombstone - you don’t know what it’s shadowing (and can’t, in 
> many cases, because the width of the range is uncountable for some types). 
> 
> Setting aside whether or not this construct is worth adding (I suspect a lot 
> of binding votes would say it’s not), the thread focuses on BETWEEN operator, 
> and there’s no reason we should pollute the conversation of “add a missing 
> SQL operator that basically maps to existing functionality” with creation of 
> a brand new form of update that definitely doesn’t map to any existing 
> concepts. 
> 
> 
> 
> 
> 
>> On May 14, 2024, at 10:05 AM, Jon Haddad  wrote:
>> 
>> Personally, I don't think that something being scary at first glance is a 
>> good reason not to explore an idea.  The scenario you've described here is 
>> tricky but I'm not expecting it to be any worse than say, SAI, which (the 
>> last I checked) has O(N) complexity on returning result sets with regard to 
>> rows returned.  We've also merged in Vector search which has O(N) overhead 
>> with the number of SSTables.  We're still fundamentally looking at, in most 
>> cases, a limited number of SSTables and some merging of values.
>> 
>> Write updates are essentially a timestamped mask, potentially overlapping, 
>> and I suspect potentially resolvable during compaction by propagating the 
>> values.  They could be eliminated or narrowed based on how they've 
>> propagated by using the timestamp metadata on the SSTable.
>> 
>> It would be a lot more constructive to apply our brains towards solving an 
>> interesting problem than pointing out all its potential flaws based on gut 
>> feelings.  We haven't even moved this past an idea.  
>> 
>> I think it would solve a massive problem for a lot of people and is 100% 
>> worth considering.  Thanks Patrick and David for raising this.
>> 
>> Jon
>> 
>> 
>> 
>> On Tue, May 14, 2024 at 9:48 AM Bowen Song via dev 
>>  wrote:
>>> __
>>> Ranged update sounds like a disaster for compaction and read performance.
>>> 
>>> Imagine compacting or reading some SSTables in which a large number of 
>>> overlapping but non-identical ranges were updated with different values. It 
>>> gives me a headache by just thinking about it.
>>> 
>>> Ranged delete is much simpler, because the "value" is the same tombstone 
>>> marker, and it also is guaranteed to expire and disappear eventually, so 
>>> the performance impact of dealing with them at read and compaction time 
>>> doesn't suffer in the long term.
>>> 
>>> 
>>> On 14/05/2024 16:59, Benjamin Lerer wrote:
 It should be like range tombstones ... in much worse ;-). A tombstone is a 
 simple marker (deleted). An update can be far more complex.  
 
 Le mar. 14 mai 2024 à 15:52, Jon Haddad  a écrit :
> Is there a technical limitation that would prevent a range write that 
> functions the same way as a range tombstone, other than probably needing 
> a version bump of the storage format?
> 
> 
> On Tue, May 14, 2024 at 12:03 AM Benjamin Lerer  wrote:
>> Range restrictions (>, >=, =<, < and BETWEEN) do not work on UPDATEs. 
>> They do work on DELETE because under the hood C* they get translated 
>> into range tombstones.
>> 
>> Le mar. 14 mai 2024 à 02:44, David Capwell  a écrit :
>>> I would also include in UPDATE… but yeah, <3 BETWEEN and welcome this 
>>> work.
>>> 
 On May 13, 2024, at 7:40 AM, Patrick McFadin  
 wrote:
 
 This is a great feature addition to CQL! I get asked about it from 
 time to time but then people figure out a workaround. It will be great 
 to just have it available. 
 
 And right on Simon! I think the only project I had as a high school 
 senior was figuring out how many parties I could go to and still 
 maintain a passing grade. Thanks for your work here. 
 
 Patrick 
 
 On Mon, May 13, 2024 at 1:35 AM Benjamin Lerer  
 wrote:
> Hi everybody,
> 
> Just raising awareness that Simon is working on adding support for 
> the BETWEEN operator in WHERE clauses (SELECT and DELETE) in 
> CASSANDRA-19604. We plan to add support for it in conditions in a 
> separate patch.
> 
> The pa

Re: [DISCUSS] ccm as a subproject

[Josh McKenzie]

> We do still have the issues of DSE-supporting code in it, as we do with the 
> drivers.  I doubt any of us strongly object to it: there's no trickery 
> happening here on the user; but we should be aware of it and have a rough 
> direction sketched out for when someone else comes along wanting to add 
> support for their proprietary product.
IMO as long as it's documented well at the outset and we have plans to slowly 
refactor to move it to clean boundaries (epic in JIRA anyone <3) so it can be 
extracted into a separately maintained module by folks that need it, I think 
we'd be in great shape. That'd also pave a path for others wanting to add 
support for their proprietary products as well. Win-win.

There's always this chicken or egg problem w/things like ccm. Do people not 
contribute to it because it's out of the umbrella, or is it out of the umbrella 
because people don't need to contribute to it?

I hadn't thought about other subprojects relying on it. That's a very good 
point.

On Thu, May 16, 2024, at 4:48 AM, Jacek Lewandowski wrote:
> +1 (my personal opinion)
> 
> How to deal with the DSE-supporting code is a separate discussion IMO
> 
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> czw., 16 maj 2024 o 10:21 Berenguer Blasi  
> napisał(a):
>> __
>> +1 ccm is super useful
>> 
>> On 16/5/24 10:09, Mick Semb Wever wrote:
>>> 
>>> 
>>> On Wed, 15 May 2024 at 16:24, Josh McKenzie  wrote:
>>>> Right now ccm isn't formally a subproject of Cassandra or under governance 
>>>> of the ASF. Given it's an integral components of our CI as well as for 
>>>> local testing for many devs, and we now have more experience w/our muscle 
>>>> on IP clearance and ingesting / absorbing subprojects where we can't track 
>>>> down every single contributor to get an ICLA, seems like it might be worth 
>>>> revisiting the topic of donation of ccm to Apache.
>>>> 
>>>> For what it's worth, Sylvain originally and then DataStax after transfer 
>>>> have both been incredible and receptive stewards of the projects and 
>>>> repos, so this isn't about any response to any behavior on their part. 
>>>> Structurally, however, it'd be better for the health of the project(s) 
>>>> long-term to have ccm promoted in. As far as I know there was strong 
>>>> receptivity to that donation in the past but the IP clearance was the 
>>>> primary hurdle.
>>>> 
>>>> Anyone have any thoughts for or against?
>>>> 
>>>> https://github.com/riptano/ccm
>>> 
>>> 
>>> 
>>> We've been working on this along with the python-driver (just haven't 
>>> raised it yet).  It is recognised, like the python-driver, as a key 
>>> dependency that would best be in the project.
>>> 
>>> Obtaining the CLAs should be much easier, the contributors to ccm are less 
>>> diverse, being more the people we know already.
>>> 
>>> We do still have the issues of DSE-supporting code in it, as we do with the 
>>> drivers.  I doubt any of us strongly object to it: there's no trickery 
>>> happening here on the user; but we should be aware of it and have a rough 
>>> direction sketched out for when someone else comes along wanting to add 
>>> support for their proprietary product.  We also don't want to be pushing 
>>> downstream users to be having to create their own forks either.
>>> 
>>> Great to see general consensus (so far) in receiving it :) 
>>>  

[DISCUSS] ccm as a subproject

[Josh McKenzie]

Right now ccm isn't formally a subproject of Cassandra or under governance of 
the ASF. Given it's an integral components of our CI as well as for local 
testing for many devs, and we now have more experience w/our muscle on IP 
clearance and ingesting / absorbing subprojects where we can't track down every 
single contributor to get an ICLA, seems like it might be worth revisiting the 
topic of donation of ccm to Apache.

For what it's worth, Sylvain originally and then DataStax after transfer have 
both been incredible and receptive stewards of the projects and repos, so this 
isn't about any response to any behavior on their part. Structurally, however, 
it'd be better for the health of the project(s) long-term to have ccm promoted 
in. As far as I know there was strong receptivity to that donation in the past 
but the IP clearance was the primary hurdle.

Anyone have any thoughts for or against?

https://github.com/riptano/ccm

Re: [DISCUSS] Adding support for BETWEEN operator

[Josh McKenzie]

> Is there a technical limitation that would prevent a range write that 
> functions the same way as a range tombstone, other than probably needing a 
> version bump of the storage format?
The technical limitation would be cost/benefit due to how this intersects w/our 
architecture I think.

Range tombstones have taught us that something that should be relatively simple 
(merge in deletion mask at read time) introduces a significant amount of 
complexity on all the paths Benjamin enumerated with a pretty long tail of bugs 
and data incorrectness issues and edge cases. The work to get there, at a high 
level glance, would be:
 1. Updates to CQL grammar, spec
 2. Updates to write path
 3. Updates to accord. And thinking about how this intersects w/accord's WAL / 
logic (I think? Consider me not well educated on details here)
 4. Updates to compaction w/consideration for edge cases on all the different 
compaction strategies
 5. Updates to iteration and merge logic
 6. Updates to paging logic
 7. Indexing
 8. repair, both full and incremental implications, support, etc
 9. the list probably goes on? There's always >= 1 thing we're not thinking of 
with a change like this. Usually more.
For all of the above we also would need unit, integration, and fuzz testing 
extensively to ensure the introduction of this new spanning concept on a write 
doesn't introduce edge cases where incorrect data is returned on merge.

All of which is to say: it's an interesting problem, but IMO given our 
architecture and what we know about the past of trying to introduce an 
architectural concept like this, the costs to getting something like this to 
production ready are pretty high.

To me the cost/benefit don't really balance out. Just my .02 though.

On Tue, May 14, 2024, at 2:50 PM, Benjamin Lerer wrote:
>> It would be a lot more constructive to apply our brains towards solving an 
>> interesting problem than pointing out all its potential flaws based on gut 
>> feelings.
> 
> It is not simply a gut feeling, Jon. This change impacts read, write, 
> indexing, storage, compaction, repair... The risk and cost associated with it 
> are pretty significant and I am not convinced at this point of its benefit.
> 
> Le mar. 14 mai 2024 à 19:05, Jon Haddad  a écrit :
>> Personally, I don't think that something being scary at first glance is a 
>> good reason not to explore an idea.  The scenario you've described here is 
>> tricky but I'm not expecting it to be any worse than say, SAI, which (the 
>> last I checked) has O(N) complexity on returning result sets with regard to 
>> rows returned.  We've also merged in Vector search which has O(N) overhead 
>> with the number of SSTables.  We're still fundamentally looking at, in most 
>> cases, a limited number of SSTables and some merging of values.
>> 
>> Write updates are essentially a timestamped mask, potentially overlapping, 
>> and I suspect potentially resolvable during compaction by propagating the 
>> values.  They could be eliminated or narrowed based on how they've 
>> propagated by using the timestamp metadata on the SSTable.
>> 
>> It would be a lot more constructive to apply our brains towards solving an 
>> interesting problem than pointing out all its potential flaws based on gut 
>> feelings.  We haven't even moved this past an idea.  
>> 
>> I think it would solve a massive problem for a lot of people and is 100% 
>> worth considering.  Thanks Patrick and David for raising this.
>> 
>> Jon
>> 
>> 
>> 
>> On Tue, May 14, 2024 at 9:48 AM Bowen Song via dev 
>>  wrote:
>>> __
>>> Ranged update sounds like a disaster for compaction and read performance.
>>> 
>>> Imagine compacting or reading some SSTables in which a large number of 
>>> overlapping but non-identical ranges were updated with different values. It 
>>> gives me a headache by just thinking about it.
>>> 
>>> Ranged delete is much simpler, because the "value" is the same tombstone 
>>> marker, and it also is guaranteed to expire and disappear eventually, so 
>>> the performance impact of dealing with them at read and compaction time 
>>> doesn't suffer in the long term.
>>> 
>>> 
>>> On 14/05/2024 16:59, Benjamin Lerer wrote:
 It should be like range tombstones ... in much worse ;-). A tombstone is a 
 simple marker (deleted). An update can be far more complex.  
 
 Le mar. 14 mai 2024 à 15:52, Jon Haddad  a écrit :
> Is there a technical limitation that would prevent a range write that 
> functions the same way as a range tombstone, other than probably needing 
> a version bump of the storage format?
> 
> 
> On Tue, May 14, 2024 at 12:03 AM Benjamin Lerer  wrote:
>> Range restrictions (>, >=, =<, < and BETWEEN) do not work on UPDATEs. 
>> They do work on DELETE because under the hood C* they get translated 
>> into range tombstones.
>> 
>> Le mar. 14 mai 2024 à 02:44, David Capwell  a écrit :
>>> I would also include in UPDATE… but yeah, <3 BETWEEN 

Re: Is there appetite to maintain the gocql driver (in the drivers subproject) ?

[Josh McKenzie]

Does anyone call out the need for a new CEP for bringing the gocql into the 
Driver's subproject ? 
> My suggestion is that this falls under CEP-8, even if it is not DataStax 
> donating this particular codebase, the process is largely the same and it is 
> the Drivers subproject receiving it.
+1 to future driver donations falling under CEP-8

On Tue, May 14, 2024, at 1:03 PM, Mick Semb Wever wrote:
> 
> Ok, so we're got confidence now on how to approach this, confirmation from 
> the project's maintainers supporting it, and interest from a handful of 
> people interested in maintaining and contributing to the project.
> 
> The proposed plan forward is…
> 
> We will go through a round of collecting CLAs along with agreements to donate 
> work to ASF from all gocql authors, over email and LinkedIn searches and 
> messages.  We will also open a github issue on the gocql project describing 
> the steps involved and mentioning all the authors.  A response on the GH 
> issue from everyone agreeing to the donation is the best single place to 
> collect the responses from everyone, but we'll accept and work with 
> whatever/however we get them.   These authors will also need to sign this 
> ICLA and submit it to the ASF.
> 
> After a four week grace period we will move ahead with the IP donation to 
> ASF, and make a list of all work (files) that we don't have CLAs for.  Such 
> work may remain with headers honouring their past MIT license and authors.  
> When the work is accepted and brought into the Cassandra Driver subproject we 
> will be looking to add committers to the subproject.  These may or may not be 
> people who have expressed interest in further contributing to the codebase, 
> but rather people we trust regardless when/if they come back to contribute.
> 
> Does anyone call out the need for a new CEP for bringing the gocql into the 
> Driver's subproject ? 
> My suggestion is that this falls under CEP-8, even if it is not DataStax 
> donating this particular codebase, the process is largely the same and it is 
> the Drivers subproject receiving it.
> 
>  
> 
> On Mon, 15 Apr 2024 at 12:31, Mick Semb Wever  wrote:
>> 
>>
>> 
 We can open an issue with LEGAL to see what they say at least?
>>> 
>>> 
>>> I will raise a LEGAL ticket.
>>> 
>>> My question here is whether we have to go through the process of 
>>> best-efforts to get approval to donate (transfer copyright), or whether we 
>>> can honour the copyright on the prior work and move forward ( by 
>>> referencing it in our NOTICE.txt, as per 
>>> https://infra.apache.org/licensing-howto.html )
>> 
>> 
>> https://issues.apache.org/jira/browse/LEGAL-674  

Re: [DISCUSS] guardail for global schema modifcations - CASSANDRA-19556 in the context of CASSANDRA-17495

[Josh McKenzie]

> Given that CASSANDRA-17495 was not released in any GA (just in 5.0-alpha1), I 
> think that the option 1) is still viable - we would drop CASSANDRA-17495 and 
> we would have CASSANDRA-19556 instead of that which would act as a global 
> on/off on all schema modifications however given that we go into beta2 I am 
> not sure if it is not just too late.
I think this is the best solution for our end-users long term excepting how 
close we are to a 5.0 GA. That said, guardrails aren't exactly super invasive 
destabilizing new features, so if you could get this patch in before we GA'd, 
I'd personally support making an exception for it.

We discussed a bit on the ticket; I fall in the ambivalent space of "I like 
modularity, except for when it's unnecessary complexity for a use-case or 
flexibility that users don't need", and not being sure whether operators would 
need different guardrails for functionality. Especially given security and 
roles.

On Mon, May 6, 2024, at 7:51 AM, Štefan Miklošovič wrote:
> Hi list,
> 
> there is a question in CASSANDRA-19556 we would like to have more feedback on 
> in order to move forward.
> 
> CASSANDRA-19556 wants to introduce two guardrails. One for forbidding / 
> allowing DCL statements - (Authentication|Authorization)Statement - and 
> another one 
> for DDL statements (all schema modifications).
> 
> However, there is already a guardrail implemented by CASSANDRA-17495 which 
> prevents only modifications of schema on a table level so CASSANDRA-19556 
> might be viewed as a superset of CASSANDRA-17495.
> 
> I am not sure why we stopped with tables only in CASSANDRA-17495, this might 
> be extended to keyspaces too, any schema modification really. I think we have 
> three options here:
> 
> 1) drop CASSANDRA-17495 and implement CASSANDRA-19556 which would cover _all_ 
> schema modifications, not just table-related ones
> 2) keep CASSANDRA-17495 and implement CASSANDRA-19556 as is currently 
> proposed - that means that we would be able to forbid all schema 
> modifications by CASSANDRA-19556 but once schema modifications are allowed, 
> we might further forbid table modifications as implemented by CASSANDRA-17495.
> 3) keep CASSANDRA-17495 but change the implementation of CASSANDRA-19556 in 
> such a way that it would be more granular. What I mean by the granularity is 
> that we would have separate guardrail for keyspace, for example. 
> 
> 2) is the least impactful approach but what I do not like is that, basically, 
> one guardrail (CASSANDRA-19556) would shadow CASSANDRA-17495. For example, if 
> the first one is disabled but the second one is enabled, we can modify 
> keyspaces but we can not modify tables. When the first one is enabled, we can 
> not modify tables even 17495 is not disabled which I find counterintuitive.
> 
> The pros of 3) would be that it would be more granular, indeed, but, is that 
> even necessary? There are a lot of ddl statements, creation of triggers, 
> views, indices, functions, aggregates ... How are we going to categorize it? 
> Do we want to have a guardrail per logical schema component? Is that not an 
> overkill? Can you come up with a scenario when an operator wants to disable 
> keyspace modifications but they would enable table modifications? Or 
> disabling just index, materialized view or function creations / modifications 
> but e.g keyspace modifications would be possible? Is it not easier to have 
> one guardrail for all schema modifications?
> 
> Given that CASSANDRA-17495 was not released in any GA (just in 5.0-alpha1), I 
> think that the option 1) is still viable - we would drop CASSANDRA-17495 and 
> we would have CASSANDRA-19556 instead of that which would act as a global 
> on/off on all schema modifications however given that we go into beta2 I am 
> not sure if it is not just too late.
> 
> Thank you for your opinions in advance.
> 
> Regards

Re: CI's working, and repeatable !!

[Josh McKenzie]

A huge amount of work and time went into this and it's going to have a big 
impact on the project. I want to offer a heartfelt thanks to all involved for 
the focus and energy that went into this!

As the author of the system David lovingly dubbed "JoshCI" (/sigh), I 
definitely want to see us all move to converge as much as possible on the CI 
code we're running. While I remain convinced something like CASSANDRA-18731 is 
vital for hygiene in the long run (unit testing our CI, declaratively defining 
atoms of build logic independently from flow), I also think there'd be 
significant value in more of us moving towards using the JenkinsFile where at 
all possible.

Seriously - thanks again for all this work everyone. CI on Cassandra is a Big 
Data Problem, and not an easy one.

On Sun, Apr 28, 2024, at 10:22 AM, Mick Semb Wever wrote:
> 
> Good news.
> 
> CI on 5.0 and trunk is working again, after an unexpected 6 weeks hiatus (and 
> a string of general problems since last year). 
> This includes pre-commit for 5.0 and trunk working again.
> 
> 
> More info…
> 
> From 5.0 we now have in-tree a Jenkinsfile that only relies on the in-tree 
> scripts – it does not depend upon cassandra-builds and all the individual dsl 
> created stage jobs. This aligns how pre-commit and post-commit works.  More 
> importantly, it makes our CI repeatable regardless of the fork/branch of the 
> code, or the jenkins installation.
> 
> For 5.0+ pre-commit use the Cassandra-devbranch-5 and make sure your patch is 
> after sha 3c85def
> The jenkinsfile now comes with pre-defined profiles, it's recommended to use 
> "skinny" until you need the final "pre-commit".  You can also use the custom 
> profile with a regexp when you need just specific test types.
> See https://ci-cassandra.apache.org/job/Cassandra-devbranch-5/build
> 
> For pre-commit on older branches, you now use Cassandra-devbranch-before-5
> 
> For both pre- and post-commit builds, each build now creates two new sharable 
> artefacts: ci_summary.html and results_details.tar.xz
> These are based on what apple contributors were sharing from builds from 
> their internal CI system.  The format and contents of these files is expected 
> to evolve.
> 
> Each build now archives its results and logs all under one location in 
> nightlies.
> 
> e.g. https://nightlies.apache.org/cassandra/Cassandra-5.0/227/ 
> 
> 
> 
> The post-commit pipeline profile remains *very* heavy, at 130k+ tests.  These 
> were previously ramped up to include everything in their pipelines, given 
> everything that's happening in both branches.   So they take time and 
> saturate everything they touch.  We need to re-evaluate what we need to be 
> testing to alleviate this.  There'll also be a new pattern of timeouts and 
> infra/script -related flakies, as happens whenever there's such a significant 
> change, all the patience and help possible is appreciated!
> 
> 
> 
> Now that the jenkinsfile can now be used on any jenkins server for any 
> fork/branch, the next work-in-progress is CASSANDRA-18145, to be able to run 
> the full pipeline with a single command line (given a k8s context 
> (~/.kube/config)).
>   
> We already have most of this working – it's possible to make a clone 
> ci-cassandra.apache.org on k8s using this wip helm chart: 
> https://github.com/thelastpickle/Cassius 
> And we are already using this on an auto-scaling gke k8s cluster – you might 
> have seen me posting the ci_summary.html and results_details.tar.xz files to 
> tickets for pre-commit CI instead of using the ci-cassandra.a.o or circleci 
> pre-commit liks.  Already, we have a full pipeline time down to two hours and 
> less than a third of the cost of CircleCI, and there's lhf to further improve 
> this.  For serious pre-commit testing we are still missing and need 
> repeatable test runs, ref CASSANDRA-18942.  On all this I'd like to give a 
> special shout out to Aleksandr Volochnev who was instrumental in the final 
> (and helm based) work of 18145 which was needed to be able to test its 
> prerequisite ticket CASSANDRA-18594 – ci-cassandra.a.o would not be running 
> again today without his recent time spent on it.
> 
> On a separate note, this new jenkinsfile is designed in preparation for 
> CASSANDRA-18731 ('Add declarative root CI structure'), to make it easier to 
> define profiles, tests, and their infrastructural requirements.
> 
> 
> To the community…
>   We are now in a place where we are looking and requesting further donations 
> of servers to the ci-cassandra.apache.org jenkins cluster.  We can now also 
> use cloud/instance credits to host auto-scaling k8s-based ci-cassandra.a.o 
> clones that would be available for community pre-commit testing.   
>   There's plenty of low-hanging fruit improvements available if folk want to 
> get involved.  Performance and throughput of splits is an important area as 
> it has a big impact on reducing costs of a whole pipeline run  (there's 
> nothing like knowing 

Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for Live Migrating Instances

[Josh McKenzie]

> might get roasted for scope creep
This community *would never*.

What you've outlined seems like a very reasonable stretch goal or v2 to keep in 
mind so we architect something in v1 that's also supportive of a v2 keyspace 
only migration.

On Thu, Apr 25, 2024, at 1:57 PM, Venkata Hari Krishna Nukala wrote:
> I have updated the CEP to use binary level file digest verification.
> 
> In the next iteration, I am going to address the below point. 
> > I would like to see more abstraction of how the files get moved / put in 
> > place with the proposed solution being the default implementation. That 
> > would allow others to plug in alternatives means of data movement like 
> > pulling down backups from S3 or rsync, etc. 
> 
> Thanks!
> Hari
> 
> On Wed, Apr 24, 2024 at 1:24 AM Patrick McFadin  wrote:
>> I finally got a chance to digest this CEP and am happy to see it raised. 
>> This feature has been left to the end user for far too long.
>> 
>> It might get roasted for scope creep, but here goes. Related and something 
>> that I've heard for years is the ability to migrate a single keyspace away 
>> from a set of hardware... online. Similar problem but a lot more 
>> coordination.
>>  - Create a Keyspace in Cluster B mimicking keyspace in Cluster A
>>  - Establish replication between keyspaces and sync schema
>>  - Move data from Cluster A to B
>>  - Decommission keyspace in Cluster A
>> 
>> In many cases, multiple tenants present cause the cluster to overpressure. 
>> The best solution in that case is to migrate the largest keyspace to a 
>> dedicated cluster.
>> 
>> Live migration but a bit more complicated. No chance of doing this manually 
>> without some serious brain surgery on c* and downtime.
>> 
>> Patrick
>> 
>> 
>> On Tue, Apr 23, 2024 at 11:37 AM Venkata Hari Krishna Nukala 
>>  wrote:
>>> Thank you all for the inputs and apologies for the late reply. I see good 
>>> points raised in this discussion. _Please allow me to reply to each point 
>>> individually._
>>> 
>>> To start with, let me focus on the point raised by Scott & Jon about file 
>>> content verification at the destination with the source in this reply. 
>>> Agree that just verifying the file name + size is not fool proof. The 
>>> reason why I called out binary level verification out of initial scope is 
>>> because of these two reasons: 1) Calculating digest for each file may 
>>> increase CPU utilisation and 2) Disk would also be under pressure as 
>>> complete disk content will also be read to calculate digest. As called out 
>>> in the discussion, I think we can't compromise on binary level check for 
>>> these two reasons. Let me update the CEP to include binary level 
>>> verification. During implementation, it can probably be made optional so 
>>> that it can be skipped if someone doesn't want it.
>>> 
>>> Thanks!
>>> Hari
>>> 
>>> On Mon, Apr 22, 2024 at 4:40 AM Slater, Ben via dev 
>>>  wrote:
 We use backup/restore for our implementation of this concept. It has the 
 added benefit that the backup / restore path gets exercised much more 
 regularly than it would in normal operations, finding edge case bugs at a 
 time when you still have other ways of recovering rather than in a full 
 disaster scenario.
 __ __
 Cheers
 Ben
 __ __
 __ __
 __ __
 __ __
 *From: *Jordan West 
 *Date: *Sunday, 21 April 2024 at 05:38
 *To: *dev@cassandra.apache.org 
 *Subject: *Re: [DISCUSS] CEP-40: Data Transfer Using Cassandra Sidecar for 
 Live Migrating Instances
 *EXTERNAL EMAIL - USE CAUTION when clicking links or attachments*
 
 
 I do really like the framing of replacing a node is restoring a node and 
 then kicking off a replace. That is effectively what we do internally. 
 __ __
 I also agree we should be able to do data movement well both internal to 
 Cassandra and externally for a variety of reasons. 
 __ __
 We’ve seen great performance with “ZCS+TLS” even though it’s not full zero 
 copy — nodes that previously took *days* to replace now take a few hours. 
 But we have seen it put pressure on nodes and drive up latencies which is 
 the main reason we still rely on an external data movement system by 
 default — falling back to ZCS+TLS as needed. 
 __ __
 Jordan 
 __ __
 On Fri, Apr 19, 2024 at 19:15 Jon Haddad  wrote:
> Jeff, this is probably the best explanation and justification of the idea 
> that I've heard so far.
> __ __
> I like it because
> __ __
> 1) we really should have something official for backups
> 2) backups / object store would be great for analytics
> 3) it solves a much bigger problem than the single goal of moving 
> instances.
> __ __
> I'm a huge +1 in favor of this perspective, with live migration being one 
> use case for backup / restore.
> __ __
>>

Re: Welcome Alexandre Dutra, Andrew Tolbert, Bret McGuire, Olivier Michallat as Cassandra Committers

[Josh McKenzie]

Congrats everyone and thanks for all the hard work to get things to this point!

On Wed, Apr 17, 2024, at 1:18 PM, Ekaterina Dimitrova wrote:
> Congrats and thank you for all your work on the drivers!
> 
> On Wed, 17 Apr 2024 at 13:17, Francisco Guerrero  wrote:
>> Congratulations everyone!
>> 
>> On 2024/04/17 17:14:34 Abe Ratnofsky wrote:
>> > Congrats everyone!
>> > 
>> > > On Apr 17, 2024, at 1:10 PM, Benjamin Lerer  wrote:
>> > > 
>> > > The Apache Cassandra PMC is pleased to announce that Alexandre Dutra, 
>> > > Andrew Tolbert, Bret McGuire and Olivier Michallat have accepted the 
>> > > invitation to become committers on the java driver sub-project. 
>> > > 
>> > > Thanks for your contributions to the Java driver during all those years!
>> > > Congratulations and welcome!
>> > > 
>> > > The Apache Cassandra PMC members
>> > 
>> >

Re: [DISCUSS] Modeling JIRA fix version for subprojects

[Josh McKenzie]

+1 to separate JIRA projects per subproject. Having workflows distinct to each 
project is reason enough for me, nevermind the global namespace pollution that 
occurs if you pack a bunch of disparate projects together into one instance.

On Mon, Apr 8, 2024, at 9:11 PM, Dinesh Joshi wrote:
> hi folks - sorry to have dropped the ball on responding to this thread.
> 
> My 2 cents are as follows - 
> 
> 1. Having a separate JIRA project for each sub-project will add management 
> overhead. This option, however, allows us to model unique workflows for the 
> sub-project.
> 
> 2. Managing the sub-project as part of the Cassandra JIRA project would imply 
> less management overhead but the sub-project would need to conform to the 
> same workflows.
> 
> I would pick option 1 unless there is a strong reason and desire to manage a 
> separate Jira project. We can always split out the Java Driver project if 
> things don't work out. OTOH merging a Jira project is harder.
> 
> Thanks,
> 
> Dinesh
> 
> On Thu, Apr 4, 2024 at 12:45 PM Abe Ratnofsky  wrote:
>> CEP-8 proposes using separate Jira projects per Cassandra sub-project:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>> 
>> > We suggest distinct Jira projects, one per driver, all to be created.
>> 
>> I don't see any discussion changing that from the [DISCUSS] or vote threads:
>> https://lists.apache.org/thread/01pljcncyjyo467l5orh8nf9okrh7oxm
>> https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
>> https://lists.apache.org/thread/crolkrhd4y6tt3k4hsy204xomshlcp4p
>> 
>> But looks like upon acceptance that was changed:
>> https://lists.apache.org/thread/dhov01s8dvvh3882oxhkmmfv4tqdd68o
>> 
>> > New issues will be tracked under the CASSANDRA project on Apache’s JIRA 
>> >  under the component 
>> > ‘Client/java-driver’.
>> 
>> I'm in favor of using the same Jira as Cassandra proper. Committership is 
>> project-wide, so having a standardized process (same ticket flow, review 
>> rules, labels, etc. is beneficial). But multiple votes happened based on the 
>> content of the CEP, so we should stick to what was voted on and move to a 
>> separate Jira.
>> 
>> --
>> Abe

Re: [DISCUSS] Fixing coverage reports for jvm-dtest-upgrade

[Josh McKenzie]

+1 from me

> If classloaders are appropriately named in the current versions of Cassandra, 
> we should be able to test upgrade paths to that version without updating the 
> older branches or building new jvm-dtest JARs for them.
Pretty sure Caleb was wrestling w/something recently that might have benefited 
from being able to differentiate which ClassLoader was sad; in general this 
seems like it'd be a big help to debugging startup / env issues, assuming it 
doesn't break anything. :)

On Fri, Mar 15, 2024, at 4:41 PM, Abe Ratnofsky wrote:
> Hey folks,
> 
> I'm working on gathering coverage data across all of our test suites. The 
> jvm-dtest-upgrade suite is currently unfriendly to Jacoco: it uses classes 
> from multiple versions of Cassandra but with the same class names, so 
> Jacoco's analysis fails due to "Can't add different class with same name"[1]. 
> We need a way to exclude alternate-version classes from Jacoco's analysis, so 
> we can get coverage for the current version of Cassandra.
> 
> Jacoco supports exclusion of classes based on class name or classloader name, 
> but the class names are frequently usually identical across Cassandra 
> versions. The jvm-dtest framework doesn't name classloaders, so we can't use 
> that either.
> 
> I'd like to propose that we name the jvm-dtest InstanceClassLoader instances 
> so that some can be excluded from Jacoco's analysis. Instances that create 
> new InstanceClassLoaders should be able to provide an immutable name in the 
> constructor. InstanceClassLoader names should indicate whether they're for 
> the current version of Cassandra (where coverage should be collected) or an 
> alternate version. If classloaders are appropriately named in the current 
> versions of Cassandra, we should be able to test upgrade paths to that 
> version without updating the older branches or building new jvm-dtest JARs 
> for them.
> 
> Any objections or alternate approaches?
> 
> --
> Abe
> 
> [1]: More specifically: Jacoco uses class IDs to map the analysis data that's 
> produced during text execution to .class files. I'm configuring the Jacoco 
> agent's classdumpdir to ensure that the classes loaded during execution are 
> the same classes that are analyzed during report generation, as is 
> recommended. When we build the alternate version JARs for jvm-dtest-upgrade, 
> we end up with multiple classes with the same name but different IDs.

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[Josh McKenzie]

Looks like we bumped from 3.6 requirement to 3.7 in CASSANDRA-18960 
 as well - similar 
thing. Vector support in python, though that patch took it from "return a 
simple blob" to "return something the python driver knows about, but apparently 
not variable types so we'll need to upgrade again."

> The version of the Python driver that is used by cqlsh (3.25.0) doesn't 
> entirely support the new vector data type introduced by CASSANDRA-18504 
> . While we can 
> perfectly write data, read vectors are presented as blobs:
> 

As far as I can tell, support for vector types in cqlsh is the sole reason 
we've bumped to 3.7 and 3.8 to support that python driver. That correct Andres 
/ Brandon?

On Mon, Mar 11, 2024, at 1:22 PM, Caleb Rackliffe wrote:
> The vector issues itself was a simple error message change: 
> https://github.com/datastax/python-driver/commit/e90c0f5d71f4cac94ed80ed72c8789c0818e11d0
> 
> Was there something else in 3.29.0 that actually necessitated the move to a 
> floor of Python 3.8? Do we generally change runtime requirements in minor 
> releases for the driver?
> 
> On Mon, Mar 11, 2024 at 12:12 PM Brandon Williams  wrote:
>> Given that 3.6 has been EOL for 2+ years[1], I don't think it makes
>> sense to add support for it back.
>> 
>> Kind Regards,
>> Brandon
>> 
>> [1] https://devguide.python.org/versions/
>> 
>> On Mon, Mar 11, 2024 at 12:08 PM David Capwell  wrote:
>> >
>> > Originally we had planned to support RHEL 7 but in testing 5.0 we found 
>> > out that cqlsh no longer works on RHEL 7[1].  This was changed in 
>> > CASSANDRA-19245 which upgraded python-driver from 3.28.0 to 3.29.0. For 
>> > some reason this minor version upgrade also dropped support for python 3.6 
>> > which is the supported python version on RHEL 7.
>> >
>> > We wanted to bring this to the attention of the community to figure out 
>> > next steps; do we wish to say that RHEL 7 is no longer supported (making 
>> > upgrades tied to OS upgrades, which can be very hard for users), or do we 
>> > want to add python 3.6 support back to python-driver?
>> >
>> >
>> > 1: the error seen by users is
>> > $ cqlsh
>> > Warning: unsupported version of Python, required 3.8-3.11 but found 3.6 
>> > Warning: unsupported version of Python, required 3.8-3.11 but found 2.7
>> > No appropriate Python interpreter found.
>> > $
>> >
>> >

Re: [Discuss] Repair inside C*

[Josh McKenzie]

> we're all willing to bikeshed for our personal preference on where it lives 
> and how it's implemented, and at the end of the day, code talks. I don't 
> think anyone's said they'll die on the hill of implementation details

:D

I don't think we're going to be able to reach a consensus on an email thread 
with higher level abstractions and indicative statements. For instance: "a lot 
of complexity around repair in the main process" vs. "a lot of complexity in 
signaling between a sidecar and a main process and supporting multiple versions 
of C*". Both resonate with me at face value and neither contain enough detail 
to weigh against one another.

A more granular, lower level CEP that includes a tradeoff of the two designs 
with a recommendation on a path forward might help unstick us from the ML 
back-and-forth.

We could also take an indicative vote on "in-process vs. in-sidecar" to see if 
we can get a read on temperature.

On Thu, Feb 22, 2024, at 2:06 PM, Paulo Motta wrote:
> Apologies, I just read the previous message and missed the previous 
> discussion on sidecar vs main process on this thread. :-)
> 
> It does not look like a final agreement was reached about this and there are 
> lots of good arguments for both sides, but perhaps it would be nice to agree 
> on this before a CEP is proposed since this will significantly influence the 
> initial design?
> 
> I tend to agree with Dinesh and Scott's pragmatic stance of providing initial 
> support to repair scheduling on the sidecar, since this has fewer 
> dependencies, and progressively move what makes sense to the main process as 
> TCM/Accord primitives become available and mature.
> 
> On Thu, Feb 22, 2024 at 1:44 PM Paulo Motta  wrote:
>> +1 to Josh's points,  The project has considered native repair scheduling 
>> for a long time but it was never made a reality due to the complex 
>> considerations involved and availability of custom implementations/tools 
>> like cassandra-reaper, which is a popular way of scheduling repairs in 
>> Cassandra.
>> 
>> Unfortunately I did not have cycles to review this proposal, but it looks 
>> promising from a quick glance.
>> 
>> One important consideration that I think we need to discuss is: where should 
>> repair scheduling live: in the main process or the sidecar?
>> 
>> I think there is a lot of complexity around repair in the main process and 
>> we need to be extra careful about adding additional complexity on top of 
>> that.
>> 
>> Perhaps this could be a good opportunity to consider the sidecar to host 
>> repair scheduling, since this looks to be a control plane responsibility? 
>> One downside is that this would not make repair scheduling available to 
>> users who do not use the sidecar.
>> 
>> What do you think? It would be great to have input from sidecar maintainers 
>> if this is something that would make sense for that subproject.
>> 
>> On Thu, Feb 22, 2024 at 12:33 PM Josh McKenzie  wrote:
>>> __
>>> Very late response from me here (basically necro'ing this thread).
>>> 
>>> I think it'd be useful to get this condensed into a CEP that we can then 
>>> discuss in that format. It's clearly something we all agree we need and 
>>> having an implementation that works, even if it's not in your preferred 
>>> execution domain, is vastly better than nothing IMO.
>>> 
>>> I don't have cycles (nor background ;) ) to do that, but it sounds like you 
>>> do Jaydeep given the implementation you have on a private fork + design.
>>> 
>>> A non-exhaustive list of things that might be useful incorporating into or 
>>> referencing from a CEP:
>>> Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
>>> Even older automatic repair scheduling: 
>>> https://issues.apache.org/jira/browse/CASSANDRA-10070
>>> Your design gdoc: 
>>> https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
>>> PR with automated repair: 
>>> https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c
>>> 
>>> My intuition is that we're all basically in agreement that this is 
>>> something the DB needs, we're all willing to bikeshed for our personal 
>>> preference on where it lives and how it's implemented, and at the end of 
>>> the day, code talks. I don't think anyone's sa

Re: [Discuss] Repair inside C*

[Josh McKenzie]

Very late response from me here (basically necro'ing this thread).

I think it'd be useful to get this condensed into a CEP that we can then 
discuss in that format. It's clearly something we all agree we need and having 
an implementation that works, even if it's not in your preferred execution 
domain, is vastly better than nothing IMO.

I don't have cycles (nor background ;) ) to do that, but it sounds like you do 
Jaydeep given the implementation you have on a private fork + design.

A non-exhaustive list of things that might be useful incorporating into or 
referencing from a CEP:
Slack thread: https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
Joey's old C* ticket: https://issues.apache.org/jira/browse/CASSANDRA-14346
Even older automatic repair scheduling: 
https://issues.apache.org/jira/browse/CASSANDRA-10070
Your design gdoc: 
https://docs.google.com/document/d/1CJWxjEi-mBABPMZ3VWJ9w5KavWfJETAGxfUpsViPcPo/edit#heading=h.r112r46toau0
PR with automated repair: 
https://github.com/jaydeepkumar1984/cassandra/commit/ef6456d652c0d07cf29d88dfea03b73704814c2c

My intuition is that we're all basically in agreement that this is something 
the DB needs, we're all willing to bikeshed for our personal preference on 
where it lives and how it's implemented, and at the end of the day, code talks. 
I don't think anyone's said they'll die on the hill of implementation details, 
so that feels like CEP time to me.

If you were willing and able to get a CEP together for automated repair based 
on the above material, given you've done the work and have the proof points 
it's working at scale, I think this would be a *huge contribution* to the 
community.

On Thu, Aug 24, 2023, at 7:26 PM, Jaydeep Chovatia wrote:
> Is anyone going to file an official CEP for this?
> As mentioned in this email thread, here is one of the solution's design doc 
> 
>  and source code on a private Apache Cassandra patch. Could you go through it 
> and let me know what you think?
> 
> Jaydeep
> 
> On Wed, Aug 2, 2023 at 3:54 PM Jon Haddad  wrote:
>> > That said I would happily support an effort to bring repair scheduling to 
>> > the sidecar immediately. This has nothing blocking it, and would 
>> > potentially enable the sidecar to provide an official repair scheduling 
>> > solution that is compatible with current or even previous versions of the 
>> > database.
>> 
>> This is something I hadn't thought much about, and is a pretty good argument 
>> for using the sidecar initially.  There's a lot of deployments out there and 
>> having an official repair option would be a big win. 
>> 
>> 
>> On 2023/07/26 23:20:07 "C. Scott Andreas" wrote:
>> > I agree that it would be ideal for Cassandra to have a repair scheduler 
>> > in-DB.
>> >
>> > That said I would happily support an effort to bring repair scheduling to 
>> > the sidecar immediately. This has nothing blocking it, and would 
>> > potentially enable the sidecar to provide an official repair scheduling 
>> > solution that is compatible with current or even previous versions of the 
>> > database.
>> >
>> > Once TCM has landed, we’ll have much stronger primitives for repair 
>> > orchestration in the database itself. But I don’t think that should block 
>> > progress on a repair scheduling solution in the sidecar, and there is 
>> > nothing that would prevent someone from continuing to use a sidecar-based 
>> > solution in perpetuity if they preferred.
>> >
>> > - Scott
>> >
>> > > On Jul 26, 2023, at 3:25 PM, Jon Haddad  
>> > > wrote:
>> > >
>> > > I'm 100% in favor of repair being part of the core DB, not the sidecar. 
>> > >  The current (and past) state of things where running the DB correctly 
>> > > *requires* running a separate process (either community maintained or 
>> > > official C* sidecar) is incredibly painful for folks.  The idea that 
>> > > your data integrity needs to be opt-in has never made sense to me from 
>> > > the perspective of either the product or the end user.
>> > >
>> > > I've worked with way too many teams that have either configured this 
>> > > incorrectly or not at all. 
>> > >
>> > > Ideally Cassandra would ship with repair built in and on by default.  
>> > > Power users can disable if they want to continue to maintain their own 
>> > > repair tooling for some reason.
>> > >
>> > > Jon
>> > >
>> > >> On 2023/07/24 20:44:14 German Eichberger via dev wrote:
>> > >> All,
>> > >> We had a brief discussion in [2] about the Uber article [1] where they 
>> > >> talk about having integrated repair into Cassandra and how great that 
>> > >> is. I expressed my disappointment that they didn't work with the 
>> > >> community on that (Uber, if you are listening time to make amends 🙂) 
>> > >> and it turns out Joey already had the idea and wrote the code [3] - so 
>> > >> I wanted to start a discussion to gauge interest and maybe how to 
>> >

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie]

> Do folks think we should file an official CEP and take it there?
+1 here.

Synthesizing your gdoc, Caleb's work, and the feedback from this thread into a 
draft seems like a solid next step.

On Wed, Feb 7, 2024, at 12:31 PM, Jaydeep Chovatia wrote:
> I see a lot of great ideas being discussed or proposed in the past to cover 
> the most common rate limiter candidate use cases. Do folks think we should 
> file an official CEP and take it there?
> 
> Jaydeep
> 
> On Fri, Feb 2, 2024 at 8:30 AM Caleb Rackliffe  
> wrote:
>> I just remembered the other day that I had done a quick writeup on the state 
>> of compaction stress-related throttling in the project:
>> 
>> https://docs.google.com/document/d/1dfTEcKVidRKC1EWu3SO1kE1iVLMdaJ9uY1WMpS3P_hs/edit?usp=sharing
>> 
>> I'm sure most of it is old news to the people on this thread, but I figured 
>> I'd post it just in case :)
>> 
>> On Tue, Jan 30, 2024 at 11:58 AM Josh McKenzie  wrote:
>>> __
>>>> 2.) We should make sure the links between the "known" root causes of 
>>>> cascading failures and the mechanisms we introduce to avoid them remain 
>>>> very strong.
>>> Seems to me that our historical strategy was to address individual known 
>>> cases one-by-one rather than looking for a more holistic load-balancing and 
>>> load-shedding solution. While the engineer in me likes the elegance of a 
>>> broad, more-inclusive *actual SEDA-like* approach, the pragmatist in me 
>>> wonders how far we think we are today from a stable set-point.
>>> 
>>> i.e. are we facing a handful of cases where nodes can still get pushed over 
>>> and then cascade that we can surgically address, or are we facing a broader 
>>> lack of back-pressure that rears its head in different domains (client -> 
>>> coordinator, coordinator -> replica, internode with other operations, etc) 
>>> at surprising times and should be considered more holistically?
>>> 
>>> On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
>>>> I almost forgot CASSANDRA-15817, which introduced 
>>>> reject_repair_compaction_threshold, which provides a mechanism to stop 
>>>> repairs while compaction is underwater.
>>>> 
>>>>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe  
>>>>> wrote:
>>>>> 
>>>>> Hey all,
>>>>> 
>>>>> I'm a bit late to the discussion. I see that we've already discussed 
>>>>> CASSANDRA-15013 <https://issues.apache.org/jira/browse/CASSANDRA-15013> 
>>>>> and CASSANDRA-16663 
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-16663> at least in 
>>>>> passing. Having written the latter, I'd be the first to admit it's a 
>>>>> crude tool, although it's been useful here and there, and provides a 
>>>>> couple primitives that may be useful for future work. As Scott mentions, 
>>>>> while it is configurable at runtime, it is not adaptive, although we did 
>>>>> make configuration easier in CASSANDRA-17423 
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17423>. It also is 
>>>>> global to the node, although we've lightly discussed some ideas around 
>>>>> making it more granular. (For example, keyspace-based limiting, or 
>>>>> limiting "domains" tagged by the client in requests, could be 
>>>>> interesting.) It also does not deal with inter-node traffic, of course.
>>>>> 
>>>>> Something we've not yet mentioned (that does address internode traffic) 
>>>>> is CASSANDRA-17324 
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-17324>, which I proposed 
>>>>> shortly after working on the native request limiter (and have just not 
>>>>> had much time to return to). The basic idea is this:
>>>>> 
>>>>>> When a node is struggling under the weight of a compaction backlog and 
>>>>>> becomes a cause of increased read latency for clients, we have two 
>>>>>> safety valves:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 1.) Disabling the native protocol server, which stops the node from 
>>>>>> coordinating reads and writes.
>>>>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch 
>>>>>> to avoid the node for reads from other coordinators.
&g

Welcome Brad Schoening as Cassandra Committer

[Josh McKenzie]

The Apache Cassandra PMC is pleased to announce that Brad Schoening has accepted
the invitation to become a committer.

Your work on the integrated python driver, launch script environment, and tests
has been a big help to many. Congratulations and welcome!

The Apache Cassandra PMC members

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[Josh McKenzie]

> Would it make sense to only block commits on the test strategy you've listed, 
> and shift the entire massive test suite to post-commit? 

> Lots and lots of other emails

;)

There's an interesting broad question of: What config do we consider 
"recommended" going forward, the "conservative" (i.e. old) or the "performant" 
(i.e. new)? And what JDK do we consider "recommended" going forward, the oldest 
we support or the newest?

Since those recommendations apply for new clusters, people need to qualify 
their setups, and we have a high bar of quality on testing pre-merge, my gut 
tells me "performant + newest JDK". This would impact what we'd test pre-commit 
IMO.

Having been doing a lot of CI stuff lately, some observations:
 • Our True North needs to be releasing a database that's free of defects that 
violate our core properties we commit to our users. No data loss, no data 
resurrection, transient or otherwise, due to defects in our code (meteors, 
tsunamis, etc notwithstanding).
 • The relationship of time spent on CI and stability of final full 
*post-commit* runs is asymptotic. It's not even 90/10; we're probably somewhere 
like 98% value gained from 10% of work, and the other 2% "stability" (i.e. 
green test suites, not "our database works") is a long-tail slog. Especially in 
the current ASF CI heterogenous env w/its current orchestration.
 • Thus: Pre-commit and post-commit should be different. The following points 
all apply to pre-commit:
 • The goal of pre-commit tests should be some number of 9's of no test 
failures post-commit (i.e. for every 20 green pre-commit we introduce 1 flake 
post-commit). Not full perfection; it's not worth the compute and complexity.
 • We should **build **all branches on all supported JDK's (8 + 11 for older, 
11 + 17 for newer, etc).
 • We should **run **all test suites with the *recommended **configuration* 
against the *highest versioned JDK a branch supports. *And we should formally 
recommend our users run on that JDK.
 • We should *at least* run all jvm-based configurations on the highest 
supported JDK version with the "not recommended but still supported" 
configuration.
 • I'm open to being persuaded that we should at least run jvm-unit tests on 
the older JDK w/the conservative config pre-commit, but not much beyond that.
That would leave us with the following distilled:

*Pre-commit:*
 • Build on all supported jdks
 • All test suites on highest supported jdk using recommended config
 • Repeat testing on new or changed tests on highest supported JDK 
w/recommended config
 • JDK-based test suites on highest supported jdk using other config
*Post-commit:*
 • Run everything. All suites, all supported JDK's, both config files.
With Butler + the *jenkins-jira* integration script  
(need
 to dust that off but it should remain good to go), we should have a pretty 
clear view as to when any consistent regressions are introduced and why. We'd 
remain exposed to JDK-specific flake introductions and flakes in unchanged 
tests, but there's no getting around the 2nd one and I expect the former to be 
rare enough to not warrant the compute to prevent it.

On Thu, Feb 15, 2024, at 10:02 AM, Jon Haddad wrote:
> Would it make sense to only block commits on the test strategy you've listed, 
> and shift the entire massive test suite to post-commit?  If there really is 
> only a small % of times the entire suite is useful this seems like it could 
> unblock the dev cycle but still have the benefit of the full test suite.  
> 
> 
> 
> On Thu, Feb 15, 2024 at 3:18 AM Berenguer Blasi  
> wrote:
>> __
>> On reducing circle ci usage during dev while iterating, not with the 
>> intention to replace the pre-commit CI (yet), we could do away with testing 
>> only dtests, jvm-dtests, units and cqlsh for a _single_ configuration imo. 
>> That would greatly reduce usage. I hacked it quickly here for illustration 
>> purposes: 
>> https://app.circleci.com/pipelines/github/bereng/cassandra/1164/workflows/3a47c9ef-6456-4190-b5a5-aea2aff641f1
>>  The good thing is that we have the tooling to dial in whatever we decide 
>> atm.
>> 
>> Changing pre-commit is a different discussion, to which I agree btw. But the 
>> above could save time and $ big time during dev and be done and merged in a 
>> matter of days imo.
>> 
>> I can open a DISCUSS thread if we feel it's worth it.
>> 
>> On 15/2/24 10:24, Mick Semb Wever wrote:
>>>  
 Mick and Ekaterina (and everyone really) - any thoughts on what test 
 coverage, if any, we should commit to for this new configuration? 
 Acknowledging that we already have *a lot* of CI that we run.
>>> 
>>> 
>>> 
>>> Branimir in this patch has already done some basic cleanup of test 
>>> variations, so this is not a duplication of the pipeline.  It's a 
>>> significant improvement.
>>> 
>>> I'm ok with cassandra_latest being committed and

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[Josh McKenzie]

> When we have failing tests people do not spend the time to figure out if 
> their logic caused a regression and merge, making things more unstable… so 
> when we merge failing tests that leads to people merging even more failing 
> tests...
What's the counter position to this Jacek / Berenguer?

Mick and Ekaterina (and everyone really) - any thoughts on what test coverage, 
if any, we should commit to for this new configuration? Acknowledging that we 
already have *a lot* of CI that we run.


On Wed, Feb 14, 2024, at 5:11 AM, Berenguer Blasi wrote:
> +1 to not doing, imo, the ostrich lol
> 
> On 14/2/24 10:58, Jacek Lewandowski wrote:
>> We should not block merging configuration changes given it is a valid 
>> configuration - which I understand as it is correct, passes all config 
>> validations, it matches documented rules, etc. And this provided latest 
>> config matches those requirements I assume.
>> 
>> The failures should block release or we should not advertise we have those 
>> features at all, and the configuration should be named "experimental" rather 
>> than "latest".
>> 
>> The config changes are not responsible for broken features and we should not 
>> bury our heads in the sand pretending that everything is ok.
>> 
>> Thanks,
>> 
>> śr., 14 lut 2024, 10:47 użytkownik Štefan Miklošovič 
>>  napisał:
>>> Wording looks good to me. I would also put that into NEWS.txt but I am not 
>>> sure what section. New features, Upgrading nor Deprecation does not seem to 
>>> be a good category. 
>>> 
>>> On Tue, Feb 13, 2024 at 5:42 PM Branimir Lambov  wrote:
 Hi All,
 
 CASSANDRA-18753 introduces a second set of defaults (in a separate 
 "cassandra_latest.yaml") that enable new features of Cassandra. The 
 objective is two-fold: to be able to test the database in this 
 configuration, and to point potential users that are evaluating the 
 technology to an optimized set of defaults that give a clearer picture of 
 the expected performance of the database for a new user. The objective is 
 to get this configuration into 5.0 to have the extra bit of confidence 
 that we are not releasing (and recommending) options that have not gone 
 through thorough CI.
 
 The implementation has already gone through review, but I'd like to get 
 people's opinion on two things:
 - There are currently a number of test failures when the new options are 
 selected, some of which appear to be genuine problems. Is the community 
 okay with committing the patch before all of these are addressed? This 
 should prevent the introduction of new failures and make sure we don't 
 release before clearing the existing ones.
 - I'd like to get an opinion on what's suitable wording and documentation 
 for the new defaults set. Currently, the patch proposes adding the 
 following text to the yaml (see 
 https://github.com/apache/cassandra/pull/2896/files):
 # NOTE:
 #   This file is provided in two versions:
 # - cassandra.yaml: Contains configuration defaults for a "compatible"
 #   configuration that operates using settings that are 
 backwards-compatible
 #   and interoperable with machines running older versions of 
 Cassandra.
 #   This version is provided to facilitate pain-free upgrades for 
 existing
 #   users of Cassandra running in production who want to gradually and
 #   carefully introduce new features.
 # - cassandra_latest.yaml: Contains configuration defaults that enable
 #   the latest features of Cassandra, including improved functionality 
 as
 #   well as higher performance. This version is provided for new users 
 of
 #   Cassandra who want to get the most out of their cluster, and for 
 users
 #   evaluating the technology.
 #   To use this version, simply copy this file over cassandra.yaml, or 
 specify
 #   it using the -Dcassandra.config system property, e.g. by running
 # cassandra 
 -Dcassandra.config=file:/$CASSANDRA_HOME/conf/cassandra_latest.yaml
 # /NOTE
 Does this sound sensible? Should we add a pointer to this defaults set 
 elsewhere in the documentation?
 
 Regards,
 Branimir

Re: [EXTERNAL] Re: [Discuss] Generic Purpose Rate Limiter in Cassandra

[Josh McKenzie]

> 2.) We should make sure the links between the "known" root causes of 
> cascading failures and the mechanisms we introduce to avoid them remain very 
> strong.
Seems to me that our historical strategy was to address individual known cases 
one-by-one rather than looking for a more holistic load-balancing and 
load-shedding solution. While the engineer in me likes the elegance of a broad, 
more-inclusive *actual SEDA-like* approach, the pragmatist in me wonders how 
far we think we are today from a stable set-point. 

i.e. are we facing a handful of cases where nodes can still get pushed over and 
then cascade that we can surgically address, or are we facing a broader lack of 
back-pressure that rears its head in different domains (client -> coordinator, 
coordinator -> replica, internode with other operations, etc) at surprising 
times and should be considered more holistically?

On Tue, Jan 30, 2024, at 12:31 AM, Caleb Rackliffe wrote:
> I almost forgot CASSANDRA-15817, which introduced 
> reject_repair_compaction_threshold, which provides a mechanism to stop 
> repairs while compaction is underwater.
> 
>> On Jan 26, 2024, at 6:22 PM, Caleb Rackliffe  
>> wrote:
>> 
>> Hey all,
>> 
>> I'm a bit late to the discussion. I see that we've already discussed 
>> CASSANDRA-15013  and 
>> CASSANDRA-16663  at 
>> least in passing. Having written the latter, I'd be the first to admit it's 
>> a crude tool, although it's been useful here and there, and provides a 
>> couple primitives that may be useful for future work. As Scott mentions, 
>> while it is configurable at runtime, it is not adaptive, although we did 
>> make configuration easier in CASSANDRA-17423 
>> . It also is global 
>> to the node, although we've lightly discussed some ideas around making it 
>> more granular. (For example, keyspace-based limiting, or limiting "domains" 
>> tagged by the client in requests, could be interesting.) It also does not 
>> deal with inter-node traffic, of course.
>> 
>> Something we've not yet mentioned (that does address internode traffic) is 
>> CASSANDRA-17324 , 
>> which I proposed shortly after working on the native request limiter (and 
>> have just not had much time to return to). The basic idea is this:
>> 
>>> When a node is struggling under the weight of a compaction backlog and 
>>> becomes a cause of increased read latency for clients, we have two safety 
>>> valves:
>>> 
>>> 
>>> 1.) Disabling the native protocol server, which stops the node from 
>>> coordinating reads and writes.
>>> 2.) Jacking up the severity on the node, which tells the dynamic snitch to 
>>> avoid the node for reads from other coordinators.
>>> 
>>> These are useful, but we don’t appear to have any mechanism that would 
>>> allow us to temporarily reject internode hint, batch, and mutation messages 
>>> that could further delay resolution of the compaction backlog.
>>> 
>> 
>> Whether it's done as part of a larger framework or on its own, it still 
>> feels like a good idea.
>> 
>> Thinking in terms of opportunity costs here (i.e. where we spend our finite 
>> engineering time to holistically improve the experience of operating this 
>> database) is healthy, but we probably haven't reached the point of 
>> diminishing returns on nodes being able to protect themselves from clients 
>> and from other nodes. I would just keep in mind two things:
>> 
>> 1.) The effectiveness of rate-limiting in the system (which includes the 
>> database and all clients) as a whole necessarily decreases as we move from 
>> the application to the lowest-level database internals. Limiting correctly 
>> at the client will save more resources than limiting at the native protocol 
>> server, and limiting correctly at the native protocol server will save more 
>> resources than limiting after we've dispatched requests to some thread pool 
>> for processing.
>> 2.) We should make sure the links between the "known" root causes of 
>> cascading failures and the mechanisms we introduce to avoid them remain very 
>> strong.
>> 
>> In any case, I'd be happy to help out in any way I can as this moves forward 
>> (especially as it relates to our past/current attempts to address this 
>> problem space).

Welcome Maxim Muzafarov as Cassandra Committer

[Josh McKenzie]

The Apache Cassandra PMC is pleased to announce that Maxim Muzafarov has 
accepted
the invitation to become a committer.

Thanks for all the hard work and collaboration on the project thus far, and 
we're all looking forward to working more with you in the future. 
Congratulations and welcome!

The Apache Cassandra PMC members

Re: [DISCUSSION] CEP-38: CQL Management API

[Josh McKenzie]

> Fundamentally, I think it's better for the project if administration is fully 
> done over CQL and we have a consistent, single way of doing things. 
Strongly agree here. With 2 caveats:
 1. Supporting backwards compat, especially for automated ops (i.e. nodetool, 
JMX, etc), is crucial. Painful, but crucial.
 2. We need something that's available for use before the node comes fully 
online; the point Jeff always brings up when we discuss moving away from JMX. 
So long as we have some kind of "out-of-band" access to nodes or accommodation 
for that, we should be good.
For context on point 2, see slack: 
https://the-asf.slack.com/archives/CK23JSY2K/p1688745128122749?thread_ts=1688662169.018449&cid=CK23JSY2K

> I point out that JMX works before and after the native protocol is running 
> (startup, shutdown, joining, leaving), and also it's semi-common for us to 
> disable the native protocol in certain circumstances, so at the very least, 
> we'd then need to implement a totally different cql protocol interface just 
> for administration, which nobody has committed to building yet.

I think this is a solvable problem, and I think the benefits of having a 
single, elegant way of interacting with a cluster and configuring it justifies 
the investment for us as a project. Assuming someone has the cycles to, you 
know, actually do the work. :D

On Sun, Jan 7, 2024, at 10:41 PM, Jon Haddad wrote:
> I like the idea of the ability to execute certain commands via CQL, but I 
> think it only makes sense for the nodetool commands that cause an action to 
> take place, such as compact or repair.  We already have virtual tables, I 
> don't think we need another layer to run informational queries.  I see little 
> value in having the following (I'm using exec here for simplicity):
> 
> cqlsh> exec tpstats
> 
> which returns a string in addition to:
> 
> cqlsh> select * from system_views.thread_pools
> 
> which returns structured data.  
> 
> I'd also rather see updatable configuration virtual tables instead of
> 
> cqlsh> exec setcompactionthroughput 128
> 
> Fundamentally, I think it's better for the project if administration is fully 
> done over CQL and we have a consistent, single way of doing things.  I'm not 
> dead set on it, I just think less is more in a lot of situations, this being 
> one of them.  
> 
> Jon
> 
> 
> On Wed, Jan 3, 2024 at 2:56 PM Maxim Muzafarov  wrote:
>> Happy New Year to everyone! I'd like to thank everyone for their
>> questions, because answering them forces us to move towards the right
>> solution, and I also like the ML discussions for the time they give to
>> investigate the code :-)
>> 
>> I'm deliberately trying to limit the scope of the initial solution
>> (e.g. exclude the agent part) to keep the discussion short and clear,
>> but it's also important to have a glimpse of what we can do next once
>> we've finished with the topic.
>> 
>> My view of the Command<> is that it is an abstraction in the broader
>> sense of an operation that can be performed on the local node,
>> involving one of a few internal components. This means that updating a
>> property in the settings virtual table via an update statement, or
>> executing e.g. the setconcurrentcompactors command are just aliases of
>> the same internal command via different APIs. Another example is the
>> netstats command, which simply aggregates the MessageService metrics
>> and returns them in a human-readable format (just another way of
>> looking at key-value metric pairs). More broadly, the command input is
>> Map and String as the result (or List).
>> 
>> As Abe mentioned, Command and CommandRegistry should be largely based
>> on the nodetool command set at the beginning. We have a few options
>> for how we can initially construct command metadata during the
>> registry implementation (when moving command metadata from the
>> nodetool to the core part), so I'm planning to consult with the
>> command representations of the k8cassandra project in the way of any
>> further registry adoptions have zero problems (by writing a test
>> openapi registry exporter and comparing the representation results).
>> 
>> So, the MVP is the following:
>> - Command
>> - CommandRegistry
>> - CQLCommandExporter
>> - JMXCommandExporter
>> - the nodetool uses the JMXCommandExporter
>> 
>> 
>> = Answers =
>> 
>> > What do you have in mind specifically there? Do you plan on rewriting a 
>> > brand new implementation which would be partially inspired by our agent? 
>> > Or would the project integrate our agent code in-tree or as a dependency?
>> 
>> Personally, I like the state of the k8ssandra project as it is now. My
>> understanding is that the server part of a database always lags behind
>> the client and sidecar parts in terms of the jdk version and the
>> features it provides. In contrast, sidecars should always be on top of
>> the market, so if we want to make an agent part in-tree, this should
>> be carefully considered for the flexibility wh

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Josh McKenzie]

> I don't even think we have to think about *new* SAI features to see where it 
> will benefit from further *local* optimization...
You make good points IMO. After Caleb's reasoning it makes sense to me to start 
working on query optimization w/even just our initial SAI feature-set given 
querying across multiple indices.

On Fri, Dec 22, 2023, at 8:42 AM, J. D. Jordan wrote:
> 
> The CEP-29 “rejected alternatives” section mentions one such use case.  Being 
> able to put NOT arbitrarily in a query.  Adding an OR operator is another 
> thing we are likely to want to do in the near future that would benefit from 
> this work, those benefit from the syntax tree and reordering parts of the 
> proposal.
> 
> But I think we already have enough complexity available to us to justify a 
> query optimizer in the fact of multi index queries today. Especially when you 
> have the new ANN OF operator in use combined with index queries.  Depending 
> on what order you query the indexes in, it can dramatically change the 
> performance of the query.  We are seeing and working through such issues in 
> Astra today.
> 
> -Jeremiah
> 
> 
>> On Dec 21, 2023, at 12:00 PM, Josh McKenzie  wrote:
>> 
>>> we are already late. We have several features running in production that we 
>>> chose to not open source yet because implementing phase 1 of the CEP would 
>>> have heavily simplify their designs. The cost of developing them was much 
>>> higher than what it would have been if the CEP had already been 
>>> implemented. We are also currently working on some SAI features that need 
>>> cost based optimization.
>> Are there DISCUSS threads or CEP's for any of that work? For us to have a 
>> useful discussion about whether we're at a point in the project where a 
>> query optimizer is appropriate for the project this information would be 
>> vital.
>> 
>> On Thu, Dec 21, 2023, at 12:33 PM, Benjamin Lerer wrote:
>>> Hey German,
>>> 
>>> To clarify things, we intend to push cardinalities across nodes, not costs. 
>>> It will be up to the Cost Model to estimate cost based on those 
>>> cardinalities. We will implement some functionalities to collect costs on 
>>> query execution to be able to provide them as the output of EXPLAIN ANALYZE.
>>> 
>>> We will provide more details on how we will collect and distribute 
>>> cardinalities. We will probably not go into details on how we will estimate 
>>> costs before the patch for it is ready. The main reason being that there 
>>> are a lot of different parts that you need to account for and that it will 
>>> require significant testing and experimentation.
>>> 
>>> Regarding multi-tenancy, even if you use query cost, do not forget that you 
>>> will have to account also for background tasks such as compaction, repair, 
>>> backup, ... which is not included in this CEP.  
>>> 
>>> Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev 
>>>  a écrit :
>>>> All,
>>>> 
>>>> very much agree with Scott's reasoning. 
>>>> 
>>>> It seems expedient given the advent of ACCORD transactions to be more like 
>>>> the other distributed SQL databases and just support SQL. But just because 
>>>> it's expedient it isn't right and we should work out the relational 
>>>> features in more detail before we embark on tying us to some query 
>>>> planning design.
>>>> 
>>>> The main problem in this space is pushing cost / across nodes based on 
>>>> data density. I understand that TCM will level out data density but the 
>>>> cost based optimizer proposal does a lot of hand waving when it comes to 
>>>> collecting/estimating costs for each node. I like to see more details on 
>>>> this since otherwise it will be fairly limiting.
>>>> 
>>>> I am less tied to ALLOW FILTERING - many of my customers find allowing 
>>>> filtering beneficial for their workloads so I think removing it makes 
>>>> sense to me (and yes we try to discourage them 🙂)
>>>> 
>>>> I am also intrigued by this proposal when I think about multi tenancy and 
>>>> resource governance: We have heard from several operator who run multiple 
>>>> internal teams on the same Cassandra cluster jut to optimize costs. Having 
>>>> a way to attribute those costs more fairly by adding up the costs the 
>>>> optimizer calculates might be hugely beneficial.  There could also be a 
>>>>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Josh McKenzie]

statement for clients to supply is 
>>>> perhaps simpler to implement and maintain, and has corollary benefits - 
>>>> such as providing a mechanism for users to specify their own execution 
>>>> plan.
>>>>  
>>>> Note, my proposal cuts across all of these elements of the CEP. There is 
>>>> no obvious need for a cross-cluster re-optimisation event or cross cluster 
>>>> statistic management.
>>>  
>>> I think that I am missing one part of your proposal. How do you plan to 
>>> build the initial execution plan for a prepared statement?
>>> 
>>> Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :
>>>> 
>>>> If we are to address that within the CEP itself then we should discuss it 
>>>> here, as I would like to fully understand the approach as well as how it 
>>>> relates to consistency of execution and the idea of triggering 
>>>> re-optimisation. These ideas are all interrelated.
>>>> 
>>>> I’m not sold on the proposed set of characteristics, and think my coupling 
>>>> an execution plan to a given prepared statement for clients to supply is 
>>>> perhaps simpler to implement and maintain, and has corollary benefits - 
>>>> such as providing a mechanism for users to specify their own execution 
>>>> plan.
>>>> 
>>>> Note, my proposal cuts across all of these elements of the CEP. There is 
>>>> no obvious need for a cross-cluster re-optimisation event or cross cluster 
>>>> statistic management.
>>>> 
>>>> We still also need to discuss more concretely how the base statistics 
>>>> themselves will be derived, as there is little detail here today in the 
>>>> proposal.
>>>> 
>>>> 
>>>>> On 20 Dec 2023, at 12:58, Benjamin Lerer  wrote:
>>>>> 
>>>>> After the second phase of the CEP, we will have two optimizer 
>>>>> implementations. One will be similar to what we have today and the other 
>>>>> one will be the CBO. As those implementations will be behind the new 
>>>>> Optimizer API interfaces they will both have support for EXPLAIN and they 
>>>>> will both benefit from the simplification/normalization rules. Such as 
>>>>> the ones that David mentioned.
>>>>> 
>>>>> Regarding functions, we are already able to determine which ones are 
>>>>> deterministic 
>>>>> (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/Function.java#L55).
>>>>>  We simply do not take advantage of it.
>>>>> 
>>>>> I removed the ALLOW FILTERING part and will open a discussion about it at 
>>>>> the beginning of next year.
>>>>> 
>>>>> Regarding the statistics management part, I would like to try to address 
>>>>> it within the CEP itself, if feasible. If it turns out to be too 
>>>>> complicated, I will separate it into its own CEP.
>>>>> 
>>>>> Le mar. 19 déc. 2023 à 22:23, David Capwell  a écrit :
>>>>>>> even if the only outcome of all this work were to tighten up 
>>>>>>> inconsistencies in our grammar and provide more robust EXPLAIN and 
>>>>>>> EXPLAIN ANALYZE functionality to our end users, I think that would be 
>>>>>>> highly valuable
>>>>>> 
>>>>>> In my mental model a no-op optimizer just becomes what we have today 
>>>>>> (since all new features really should be disabled by default, I would 
>>>>>> hope we support this), so we benefit from having a logical AST + ability 
>>>>>> to mutate it before we execute it and we can use this to make things 
>>>>>> nicer for users (as you are calling out)
>>>>>> 
>>>>>> Here is one example that stands out to me in accord
>>>>>> 
>>>>>> LET a = (select * from tbl where pk=0);
>>>>>> Insert into tbl2 (pk, …) values (a.pk, …); — this is not allowed as we 
>>>>>> don’t know the primary key… but this could trivially be written to 
>>>>>> replace a.pk with 0…
>>>>>> 
>>>>>> With this work we could also rethink what functions are deterministic 
>>>>>> and which ones are not (not trying to bike shed)… simple example is 
>>>>>> “now” (select now() from tbl; — each ro

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Josh McKenzie]

> One thing where this “could” come into play is that we currently run with 
> different configs at the CI level and we might be able to make this happen at 
> the class or method level instead..
It'd be great to be able to declaratively indicate which configurations a test 
needed to exercise and we just have 1 CI run that includes them as appropriate. 

On Mon, Dec 18, 2023, at 7:22 PM, David Capwell wrote:
>> A brief perusal shows jqwik as integrated with JUnit 5 taking a fairly 
>> interesting annotation-based approach to property testing. Curious if you've 
>> looked into or used that at all David (Capwell)? (link for the lazy: 
>> https://jqwik.net/docs/current/user-guide.html#detailed-table-of-contents).
> 
> I have not no.  Looking at your link it moves from lambdas to annotations, 
> and tries to define a API for stateful… I am neutral to that as its mostly 
> style…. One thing to call out is that the project documents it tries to 
> “shrink”… we ended up disabling this in QuickTheories as shrinking doesn’t 
> work well for many of our tests (too high resource demand and unable to 
> actually shrink once you move past trivial generators).  Looking at their 
> docs and their code, its hard for me to see how we actually create C* 
> generators… its so much class gen magic that I really don’t see how to create 
> AbstractType or TableMetadata… the only example they gave was not random data 
> but hand crafted data… 
> 
>> moving to JUnit 5
> 
> I am a fan of this.  If we add dependencies and don’t keep update with them 
> it becomes painful over time (missing features, lack of support, etc).  
> 
>> First of all - when you want to have a parameterized test case you do not 
>> have to make the whole test class parameterized - it is per test case. Also, 
>> each method can have different parameters.
> 
> I strongly prefer this, but never had it as a blocker from me doing param 
> tests…. One thing where this “could” come into play is that we currently run 
> with different configs at the CI level and we might be able to make this 
> happen at the class or method level instead..
> 
> @ServerConfigs(all) // can exclude unsupported configs
> public class InsertTest
> 
> It bothers me deeply that we run tests that don’t touch the configs we use in 
> CI, causing us to waste resources… Can we solve this in junit4 param logic… 
> no clue… 
> 
>> On Dec 15, 2023, at 6:52 PM, Josh McKenzie  wrote:
>> 
>>> First of all - when you want to have a parameterized test case you do not 
>>> have to make the whole test class parameterized - it is per test case. 
>>> Also, each method can have different parameters.
>> This is a pretty compelling improvement to me having just had to use the 
>> somewhat painful and blunt instrument of our current framework's 
>> parameterization; pretty clunky and broad.
>> 
>> It also looks like they moved to a "test engine abstracted away from test 
>> identification" approach to their architecture in 5 w/the "vintage" model 
>> providing native unchanged backwards-compatibility w/junit 4. Assuming they 
>> didn't bork up their architecture that *should* lower risk of the framework 
>> change leading to disruption or failure (famous last words...).
>> 
>> A brief perusal shows jqwik as integrated with JUnit 5 taking a fairly 
>> interesting annotation-based approach to property testing. Curious if you've 
>> looked into or used that at all David (Capwell)? (link for the lazy: 
>> https://jqwik.net/docs/current/user-guide.html#detailed-table-of-contents).
>> 
>> On Tue, Dec 12, 2023, at 11:39 AM, Jacek Lewandowski wrote:
>>> First of all - when you want to have a parameterized test case you do not 
>>> have to make the whole test class parameterized - it is per test case. 
>>> Also, each method can have different parameters.
>>> 
>>> For the extensions - we can have extensions which provide Cassandra 
>>> configuration, extensions which provide a running cluster and others. We 
>>> could for example apply some extensions to all test classes externally 
>>> without touching those classes, something like logging the begin and end of 
>>> each test case. 
>>> 
>>> 
>>> 
>>> wt., 12 gru 2023 o 12:07 Benedict  napisał(a):
>>>> 
>>>> Could you give (or link to) some examples of how this would actually 
>>>> benefit our test suites?
>>>> 
>>>> 
>>>>> On 12 Dec 2023, at 10:51, Jacek Lewandowski  
>>>>> wrote:
>>>>> 
>>>>> I have two major p

Re: Future direction for the row cache and OHC implementation

[Josh McKenzie]

Gotcha; wasn't sure given the earlier phrasing. Makes sense.

Dinesh's compromise position makes sense to me.

On Fri, Dec 15, 2023, at 11:21 PM, Ariel Weisberg wrote:
> Hi,
> 
> I did get one response from Robert indicating that he didn’t want to do the 
> work to contribute it.
> 
> I offered to do the work and asked for permission to contribute it and no 
> response. Followed up later with a ping and also no response.
> 
> Ariel
> 
> On Fri, Dec 15, 2023, at 9:58 PM, Josh McKenzie wrote:
>>> I have reached out to the original maintainer about it and it seems like if 
>>> we want to keep using it we will need to start releasing it under a new 
>>> package from a different repo.
>> 
>>> the current maintainer is not interested in donating it to the ASF
>> Is that the case Ariel or could you just not reach Robert?
>> 
>> On Fri, Dec 15, 2023, at 11:55 AM, Jeremiah Jordan wrote:
>>>> from a maintenance and
>>>> integration testing perspective I think it would be better to keep the
>>>> ohc in-tree, so we will be aware of any issues immediately after the
>>>> full CI run.
>>> 
>>> From the original email bringing OHC in tree is not an option because the 
>>> current maintainer is not interested in donating it to the ASF.  Thus the 
>>> option 1 of some set of people forking it to their own github org and 
>>> maintaining a version outside of the ASF C* project.
>>> 
>>> -Jeremiah
>>> 
>>> On Dec 15, 2023 at 5:57:31 AM, Maxim Muzafarov  wrote:
>>>> Ariel,
>>>> thank you for bringing this topic to the ML.
>>>> 
>>>> I may be missing something, so correct me if I'm wrong somewhere in
>>>> the management of the Cassandra ecosystem.  As I see it, the problem
>>>> right now is that if we fork the ohc and put it under its own root,
>>>> the use of that row cache is still not well tested (the same as it is
>>>> now). I am particularly emphasising the dependency management side, as
>>>> any version change/upgrade in Cassandra and, as a result of that
>>>> change a new set of libraries in the classpath should be tested
>>>> against this integration.
>>>> 
>>>> So, unless it is being widely used by someone else outside of the
>>>> community (which it doesn't seem to be), from a maintenance and
>>>> integration testing perspective I think it would be better to keep the
>>>> ohc in-tree, so we will be aware of any issues immediately after the
>>>> full CI run.
>>>> 
>>>> I'm also +1 for not deprecating it, even if it is used in narrow
>>>> cases, while the cost of maintaining its source code remains quite low
>>>> and it brings some benefits.
>>>> 
>>>> On Fri, 15 Dec 2023 at 05:39, Ariel Weisberg  wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> To add some additional context.
>>>>> 
>>>>> The row cache is disabled by default and it is already pluggable, but 
>>>>> there isn’t a Caffeine implementation present. I think one used to exist 
>>>>> and could be resurrected.
>>>>> 
>>>>> I personally also think that people should be able to scratch their own 
>>>>> itch row cache wise so removing it entirely just because it isn’t 
>>>>> commonly used isn’t the right move unless the feature is very far out of 
>>>>> scope for Cassandra.
>>>>> 
>>>>> Auto enabling/disabling the cache is a can of worms that could result in 
>>>>> performance and reliability inconsistency as the DB enables/disables the 
>>>>> cache based on heuristics when you don’t want it to. It being off by 
>>>>> default seems good enough to me.
>>>>> 
>>>>> RE forking, we could create a GitHub org for OHC and then add people to 
>>>>> it. There are some examples of dependencies that haven’t been contributed 
>>>>> to the project that live outside like CCM and JAMM.
>>>>> 
>>>>> Ariel
>>>>> 
>>>>> On Thu, Dec 14, 2023, at 5:07 PM, Dinesh Joshi wrote:
>>>>> 
>>>>> I would avoid taking away a feature even if it works in narrow set of 
>>>>> use-cases. I would instead suggest -
>>>>> 
>>>>> 1. Leave it disabled by default.
>>>>> 2. Detect when Row Cache has a low hit rate and warn the operator to turn 
>>>>> it off. Cassandra should ideally detect this and do it automatically.
>>>>> 3. Move to Caffeine instead of OHC.
>>>>> 
>>>>> I would suggest having this as the middle ground.
>>>>> 
>>>>> On Dec 14, 2023, at 4:41 PM, Mick Semb Wever  wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 3. Deprecate the row cache entirely in either 5.0 or 5.1 and remove it in 
>>>>> a later release
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> I'm for deprecating and removing it.
>>>>> It constantly trips users up and just causes pain.
>>>>> 
>>>>> Yes it works in some very narrow situations, but those situations often 
>>>>> change over time and again just bites the user.  Without the row-cache I 
>>>>> believe users would quickly find other, more suitable and lasting, 
>>>>> solutions.
>>>>> 
>>>>> 
>> 
> 

Re: Moving Semver4j from test to main dependencies

[Josh McKenzie]

+1

On Fri, Dec 15, 2023, at 1:29 PM, Derek Chen-Becker wrote:
> +1
> 
> Semver4j seems reasonable to me. I looked through the code and it's 
> relatively easy to understand. I'm not sure how easy it would be to replace 
> CassandraVersion, but that's not an immediate concern I guess.
> 
> Cheers,
> 
> Derek
> 
> On Fri, Dec 15, 2023 at 2:56 AM Jacek Lewandowski 
>  wrote:
>> Hi,
>> 
>> I'd like to add Semver4j to the production dependencies. It is currently on 
>> the test classpath. The library is pretty lightweight, licensed with MIT and 
>> has no transitive dependencies.
>> 
>> We need to represent the kernel version somehow in CASSANDRA-19196 and 
>> Semver4j looks as the right tool for it. Maybe at some point we can replace 
>> our custom implementation of CassandraVersion as well. 
>> 
>> Thanks,
>> - - -- --- -  -
>> Jacek Lewandowski
> 
> 
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
> 

Re: Future direction for the row cache and OHC implementation

[Josh McKenzie]

> I have reached out to the original maintainer about it and it seems like if 
> we want to keep using it we will need to start releasing it under a new 
> package from a different repo.

> the current maintainer is not interested in donating it to the ASF
Is that the case Ariel or could you just not reach Robert?

On Fri, Dec 15, 2023, at 11:55 AM, Jeremiah Jordan wrote:
>> from a maintenance and
>> integration testing perspective I think it would be better to keep the
>> ohc in-tree, so we will be aware of any issues immediately after the
>> full CI run.
> 
> From the original email bringing OHC in tree is not an option because the 
> current maintainer is not interested in donating it to the ASF.  Thus the 
> option 1 of some set of people forking it to their own github org and 
> maintaining a version outside of the ASF C* project.
> 
> -Jeremiah
> 
> On Dec 15, 2023 at 5:57:31 AM, Maxim Muzafarov  wrote:
>> Ariel,
>> thank you for bringing this topic to the ML.
>> 
>> I may be missing something, so correct me if I'm wrong somewhere in
>> the management of the Cassandra ecosystem.  As I see it, the problem
>> right now is that if we fork the ohc and put it under its own root,
>> the use of that row cache is still not well tested (the same as it is
>> now). I am particularly emphasising the dependency management side, as
>> any version change/upgrade in Cassandra and, as a result of that
>> change a new set of libraries in the classpath should be tested
>> against this integration.
>> 
>> So, unless it is being widely used by someone else outside of the
>> community (which it doesn't seem to be), from a maintenance and
>> integration testing perspective I think it would be better to keep the
>> ohc in-tree, so we will be aware of any issues immediately after the
>> full CI run.
>> 
>> I'm also +1 for not deprecating it, even if it is used in narrow
>> cases, while the cost of maintaining its source code remains quite low
>> and it brings some benefits.
>> 
>> On Fri, 15 Dec 2023 at 05:39, Ariel Weisberg  wrote:
>>> 
>>> Hi,
>>> 
>>> To add some additional context.
>>> 
>>> The row cache is disabled by default and it is already pluggable, but there 
>>> isn’t a Caffeine implementation present. I think one used to exist and 
>>> could be resurrected.
>>> 
>>> I personally also think that people should be able to scratch their own 
>>> itch row cache wise so removing it entirely just because it isn’t commonly 
>>> used isn’t the right move unless the feature is very far out of scope for 
>>> Cassandra.
>>> 
>>> Auto enabling/disabling the cache is a can of worms that could result in 
>>> performance and reliability inconsistency as the DB enables/disables the 
>>> cache based on heuristics when you don’t want it to. It being off by 
>>> default seems good enough to me.
>>> 
>>> RE forking, we could create a GitHub org for OHC and then add people to it. 
>>> There are some examples of dependencies that haven’t been contributed to 
>>> the project that live outside like CCM and JAMM.
>>> 
>>> Ariel
>>> 
>>> On Thu, Dec 14, 2023, at 5:07 PM, Dinesh Joshi wrote:
>>> 
>>> I would avoid taking away a feature even if it works in narrow set of 
>>> use-cases. I would instead suggest -
>>> 
>>> 1. Leave it disabled by default.
>>> 2. Detect when Row Cache has a low hit rate and warn the operator to turn 
>>> it off. Cassandra should ideally detect this and do it automatically.
>>> 3. Move to Caffeine instead of OHC.
>>> 
>>> I would suggest having this as the middle ground.
>>> 
>>> On Dec 14, 2023, at 4:41 PM, Mick Semb Wever  wrote:
>>> 
>>> 
>>> 
>>> 
>>> 3. Deprecate the row cache entirely in either 5.0 or 5.1 and remove it in a 
>>> later release
>>> 
>>> 
>>> 
>>> 
>>> I'm for deprecating and removing it.
>>> It constantly trips users up and just causes pain.
>>> 
>>> Yes it works in some very narrow situations, but those situations often 
>>> change over time and again just bites the user.  Without the row-cache I 
>>> believe users would quickly find other, more suitable and lasting, 
>>> solutions.
>>> 
>>> 

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Josh McKenzie]

> First of all - when you want to have a parameterized test case you do not 
> have to make the whole test class parameterized - it is per test case. Also, 
> each method can have different parameters.
This is a pretty compelling improvement to me having just had to use the 
somewhat painful and blunt instrument of our current framework's 
parameterization; pretty clunky and broad.

It also looks like they moved to a "test engine abstracted away from test 
identification" approach to their architecture in 5 w/the "vintage" model 
providing native unchanged backwards-compatibility w/junit 4. Assuming they 
didn't bork up their architecture that *should* lower risk of the framework 
change leading to disruption or failure (famous last words...).

A brief perusal shows jqwik as integrated with JUnit 5 taking a fairly 
interesting annotation-based approach to property testing. Curious if you've 
looked into or used that at all David (Capwell)? (link for the lazy: 
https://jqwik.net/docs/current/user-guide.html#detailed-table-of-contents).

On Tue, Dec 12, 2023, at 11:39 AM, Jacek Lewandowski wrote:
> First of all - when you want to have a parameterized test case you do not 
> have to make the whole test class parameterized - it is per test case. Also, 
> each method can have different parameters.
> 
> For the extensions - we can have extensions which provide Cassandra 
> configuration, extensions which provide a running cluster and others. We 
> could for example apply some extensions to all test classes externally 
> without touching those classes, something like logging the begin and end of 
> each test case. 
> 
> 
> 
> wt., 12 gru 2023 o 12:07 Benedict  napisał(a):
>> 
>> Could you give (or link to) some examples of how this would actually benefit 
>> our test suites?
>> 
>> 
>>> On 12 Dec 2023, at 10:51, Jacek Lewandowski  
>>> wrote:
>>> 
>>> I have two major pros for JUnit 5:
>>> - much better support for parameterized tests
>>> - global test hooks (automatically detectable extensions) + 
>>> multi-inheritance
>>> 
>>> 
>>> 
>>> 
>>> pon., 11 gru 2023 o 13:38 Benedict  napisał(a):
 
 Why do we want to move to JUnit 5? 
 
 I’m generally opposed to churn unless well justified, which it may be - 
 just not immediately obvious to me.
 
 
> On 11 Dec 2023, at 08:33, Jacek Lewandowski  
> wrote:
> 
> Nobody referred so far to the idea of moving to JUnit 5, what are the 
> opinions?
> 
> 
> 
> niedz., 10 gru 2023 o 11:03 Benedict  napisał(a):
>> 
>> Alex’s suggestion was that we meta randomise, ie we randomise the config 
>> parameters to gain better rather than lesser coverage overall. This 
>> means we cover these specific configs and more - just not necessarily on 
>> any single commit.
>> 
>> I strongly endorse this approach over the status quo.
>> 
>> 
>>> On 8 Dec 2023, at 13:26, Mick Semb Wever  wrote:
>>> 
>>>  
>>>  
>>>  
 
> I think everyone agrees here, but…. these variations are still 
> catching failures, and until we have an improvement or replacement we 
> do rely on them.   I'm not in favour of removing them until we have 
> proof /confidence that any replacement is catching the same failures. 
>  Especially oa, tries, vnodes. (Not tries and offheap is being 
> replaced with "latest", which will be valuable simplification.)  
 
 What kind of proof do you expect? I cannot imagine how we could prove 
 that because the ability of detecting failures results from the 
 randomness of those tests. That's why when such a test fail you 
 usually cannot reproduce that easily.
>>> 
>>> 
>>> Unit tests that fail consistently but only on one configuration, should 
>>> not be removed/replaced until the replacement also catches the failure.
>>>  
 We could extrapolate that to - why we only have those configurations? 
 why don't test trie / oa + compression, or CDC, or system memtable? 
>>> 
>>> 
>>> Because, along the way, people have decided a certain configuration 
>>> deserves additional testing and it has been done this way in lieu of 
>>> any other more efficient approach.
>>> 
>>> 
>>> 

Re: Custom FSError and CommitLog Error Handling

[Josh McKenzie]

Adding a poison-pill error option on finding of corrupt data makes sense to me. 
Not sure if there's enough demand / other customization being done in this 
space to justify the user customizable aspect; any immediate other approaches 
come to mind? If not, this isn't an area of the code that's changed all that 
much, so just adding a new option seems surgical and minimal to me.

On Tue, Dec 12, 2023, at 4:21 AM, Claude Warren, Jr via dev wrote:
> I can see this as a strong improvement in Cassandra management and support 
> it. 
> 
> +1 non binding
> 
> On Mon, Dec 11, 2023 at 8:28 PM Raymond Huffman  
> wrote:
>> Hello All,
>> 
>> On our fork of Cassandra, we've implemented some custom behavior for 
>> handling CommitLog and SSTable Corruption errors. Specifically, if a node 
>> detects one of those errors, we want the node to stop itself, and if the 
>> node is restarted, we want initialization to fail. This is useful in 
>> Kubernetes when you expect nodes to be restarted frequently and makes our 
>> corruption remediation workflows less error-prone. I think we could make 
>> this behavior more pluggable by allowing users to provide custom 
>> implementations of the FSErrorHandler, and the error handler that's 
>> currently implemented at 
>> org.apache.cassandra.db.commitlog.CommitLog#handleCommitError via config in 
>> the same way one can provide custom Partitioners and 
>> Authenticators/Authorizers.
>> 
>> Would you take as a contribution one of the following?
>> 1. user provided implementations of FSErrorHandler and 
>> CommitLogErrorHandler, set via config; and/or
>> 2. new commit failure and disk failure policies that write a poison pill 
>> file to disk and fail on startup if that file exists
>> 
>> The poison pill implementation is what we currently use - we call this a 
>> "Non Transient Error" and we want these states to always require manual 
>> intervention to resolve, including manual action to clear the error. I'd be 
>> happy to contribute this if other users would find it beneficial. I had 
>> initially shared this question in Slack, but I'm now sharing it here for 
>> broader visibility.
>> 
>> -Raymond Huffman

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Josh McKenzie]

> Goals
>  • Introduce a Cascades(2) query optimizer with rules easily extendable 
>  • Improve query performance for most common queries
>  • Add support for EXPLAIN and EXPLAIN ANALYZE to help with query 
> optimization and troubleshooting
>  • Lay the groundwork for the addition of features like joins, subqueries, 
> OR/NOT and index ordering
>  • Put in place some performance benchmarks to validate query optimizations
I think these are sensible goals. We're possibly going to face a chicken-or-egg 
problem with a feature like this that so heavily intersects with other as-yet 
written features where much of the value is in the intersection of them; if we 
continue down the current "one heuristic to rule them all" query planning 
approach we have now, we'll struggle to meaningfully explore or conceptualize 
the value of potential alternatives different optimizers could present us. Flip 
side, to Benedict's point, until SAI hits and/or some other potential future 
things we've all talked about, this cbo would likely fall directly into the 
same path that we effectively have hard-coded today (primary index path only).

One thing I feel pretty strongly about: even if the only outcome of all this 
work were to tighten up inconsistencies in our grammar and provide more robust 
EXPLAIN and EXPLAIN ANALYZE functionality to our end users, I think that would 
be highly valuable. This path of "only" would be predicated on us not having 
successful introduction of a robust secondary index implementation and a 
variety of other things we have a lot of interest in, so I find it unlikely, 
but worth calling out.

re: the removal of ALLOW FILTERING - is there room for compromise here and 
instead converting it to a guardrail that defaults to being enabled? That could 
theoretically give us a more gradual path to migration to a cost-based 
guardrail for instance, and would preserve the current robustness of the system 
while making it at least a touch more configurable.

On Fri, Dec 15, 2023, at 11:03 AM, Chris Lohfink wrote:
> Thanks for time in addressing concerns. At least with initial versions, as 
> long as there is a way to replace it with noop or disable it I would be 
> happy. This is pretty standard practice with features nowadays but I wanted 
> to highlight it as this might require some pretty tight coupling.
> 
> Chris
> 
> On Fri, Dec 15, 2023 at 7:57 AM Benjamin Lerer  wrote:
>> Hey Chris,
>> You raise some valid points.
>> 
>> I believe that there are 3 points that you mentioned:
>> 1) CQL restrictions are some form of safety net and should be kept
>> 2) A lot of Cassandra features do not scale and/or are too easy to use in a 
>> wrong way that can make the whole system collapse. We should not add more to 
>> that list. Especially not joins.
>> 
>> 3) Should we not start to fix features like secondary index rather than 
>> adding new ones? Which is heavily linked to 2).
>> 
>> Feel free to correct me if I got them wrong or missed one.
>> 
>> Regarding 1), I believe that you refer to the "Removing unnecessary CQL 
>> query limitations and inconsistencies" section. We are not planning to 
>> remove any safety net here.
>> What we want to remove is a certain amount of limitations which make things 
>> confusing for a user trying to write a query for no good reason. Like "why 
>> can I define a column alias but not use it anywhere in my query?" or "Why 
>> can I not create a list with 2 bind parameters?". While refactoring some CQL 
>> code, I kept on finding those types of exceptions that we can easily remove 
>> while simplifying the code at the same time.
>> 
>> For 2), I agree that at a certain scale or for some scenarios, some features 
>> simply do not scale or catch users by surprise. The goal of the CEP is to 
>> improve things in 2 ways. One is by making Cassandra smarter in the way it 
>> chooses how to process queries, hopefully improving its overall scalability. 
>> The other by being transparent about how Cassandra will execute the queries 
>> through the use of EXPLAIN. One problem of GROUP BY for example is that most 
>> users do not realize what is actually happening under the hood and therefore 
>> its limitations. I do not believe that EXPLAIN will change everything but it 
>> will help people to get a better understanding of the limitations of some 
>> features.
>> 
>> I do not know which features will be added in the future to C*. That will be 
>> discussed through some future CEPs. Nevertheless, I do not believe that it 
>> makes sense to write a CEP for a query optimizer without taking into account 
>> that we might at some point add some level of support for joins or 
>> subqueries. We have been too often delivering features without looking at 
>> what could be the possible evolutions which resulted in code where adding 
>> new features was more complex than it should have been. I do not want to 
>> make the same mistake. I want to create an optimizer that can be improved 
>> easily and 

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Josh McKenzie]

> Unit tests that fail consistently but only on one configuration, should not 
> be removed/replaced until the replacement also catches the failure.

> along the way, people have decided a certain configuration deserves 
> additional testing and it has been done this way in lieu of any other more 
> efficient approach.

Totally agree with these sentiments as well as the framing of our current unit 
tests as "bad fuzz-tests thanks to non-determinism".

To me, this reinforces my stance on a "pre-commit vs. post-commit" approach to 
testing *with our current constraints:*
 • Test the default configuration on all supported JDK's pre-commit
 • Post-commit, treat *consistent *failures on non-default configurations as 
immediate interrupts to the author that introduced them
 • Pre-release, push for no consistent failures on any suite in any 
configuration, and no regressions in flaky tests from prior release (in ASF CI 
env).
I think there's value in having the non-default configurations, but I'm not 
convinced the benefits outweigh the costs *specifically in terms of pre-commit 
work* due to flakiness in the execution of the software env itself, not to 
mention hardware env variance on the ASF side today.

All that said - if we got to a world where we could run our jvm-based tests 
deterministically within the simulator, my intuition is that we'd see a lot of 
the test-specific, non-defect flakiness reduced drastically. In such a world 
I'd be in favor of running :allthethings: pre-commit as we'd have *much* higher 
confidence that those failures were actually attributable to the author of 
whatever diff the test is run against. 

On Fri, Dec 8, 2023, at 8:25 AM, Mick Semb Wever wrote:
>  
>  
>  
>> 
>>> I think everyone agrees here, but…. these variations are still catching 
>>> failures, and until we have an improvement or replacement we do rely on 
>>> them.   I'm not in favour of removing them until we have proof /confidence 
>>> that any replacement is catching the same failures.  Especially oa, tries, 
>>> vnodes. (Not tries and offheap is being replaced with "latest", which will 
>>> be valuable simplification.)  
>> 
>> What kind of proof do you expect? I cannot imagine how we could prove that 
>> because the ability of detecting failures results from the randomness of 
>> those tests. That's why when such a test fail you usually cannot reproduce 
>> that easily.
> 
> 
> Unit tests that fail consistently but only on one configuration, should not 
> be removed/replaced until the replacement also catches the failure.
>  
>> We could extrapolate that to - why we only have those configurations? why 
>> don't test trie / oa + compression, or CDC, or system memtable? 
> 
> 
> Because, along the way, people have decided a certain configuration deserves 
> additional testing and it has been done this way in lieu of any other more 
> efficient approach.
> 
> 
> 

Re: Welcome Mike Adamson as Cassandra committer

[Josh McKenzie]

Congrats Mike! Good to see this recognition of your contributions to the 
project!

On Fri, Dec 8, 2023, at 10:02 AM, Patrick McFadin wrote:
> Yay! Congratulations Mike. Well deserved!
> 
> On Fri, Dec 8, 2023 at 7:00 AM Andrés de la Peña  wrote:
>> Congrats Mike!
>> 
>> On Fri, 8 Dec 2023 at 14:53, Jeremiah Jordan  
>> wrote:
>>> Congrats Mike!  Thanks for all your work on SAI and Vector index.  Well 
>>> deserved!
>>> 
>>> On Dec 8, 2023 at 8:52:07 AM, Brandon Williams  wrote:
 Congratulations Mike!
 
 Kind Regards,
 Brandon
 
 On Fri, Dec 8, 2023 at 8:41 AM Benjamin Lerer  wrote:
> 
> The PMC members are pleased to announce that Mike Adamson has accepted
> the invitation to become committer.
> 
> Thanks a lot, Mike, for everything you have done for the project.
> 
> Congratulations and welcome
> 
> The Apache Cassandra PMC members

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Josh McKenzie]

> that may be long-running and that could be run indefinitely
Perfect. That was the distinction I wasn't aware of. Also means having the burn 
target as part of regular CI runs is probably a mistake, yes? i.e. if someone 
adds a burn tests that runs indefinitely, are there any guardrails or built-in 
checks or timeouts to keep it from running right up to job timeout and then 
failing?

On Thu, Nov 30, 2023, at 1:11 PM, Benedict wrote:
> 
> A burn test is a randomised test targeting broad coverage of a single system, 
> subsystem or utility, that may be long-running and that could be run 
> indefinitely, each run providing incrementally more assurance of quality of 
> the system.
> 
> A long test is a unit test that sometimes takes a long time to run, no more 
> no less. I’m not sure any of these offer all that much value anymore, and 
> perhaps we could look to deprecate them.
> 
>> On 30 Nov 2023, at 17:20, Josh McKenzie  wrote:
>> 
>> Strongly agree. I started working on a declarative refactor out of our CI 
>> configuration so circle, ASFCI, and other systems could inherit from it (for 
>> instance, see pre-commit pipeline declaration here 
>> <https://github.com/apache/cassandra/pull/2554/files#diff-a4c4d1d91048841f76d124386858bda9944644cfef8ccb4ab84319cedaf5b3feR71-R89>);
>>  I had to set that down while I finished up implementing an internal CI 
>> system since the code in neither the ASF CI structure nor circle structure 
>> (.sh embedded in .yml /cry) was re-usable in their current form.
>> 
>> Having a jvm.options and cassandra.yaml file per suite and referencing them 
>> from a declarative job definition 
>> <https://github.com/apache/cassandra/pull/2554/files#diff-a4c4d1d91048841f76d124386858bda9944644cfef8ccb4ab84319cedaf5b3feR237-R267>
>>  would make things a lot easier to wrap our heads around and maintain I 
>> think.
>> 
>> As for what qualifies as burn vs. long... /shrug couldn't tell you. Would 
>> have to go down the git blame + dev ML + JIRA rabbit hole. :) Maybe someone 
>> else on-list knows.
>> 
>> On Thu, Nov 30, 2023, at 4:25 AM, Jacek Lewandowski wrote:
>>> Hi,
>>> 
>>> I'm getting a bit lost - what are the exact differences between those test 
>>> scenarios? What are the criteria for qualifying a test to be part of a 
>>> certain scenario?
>>> 
>>> I'm working a little bit with tests and build scripts and the number of 
>>> different configurations for which we have a separate target in the build 
>>> starts to be problematic, I cannot imagine how problematic it is for a new 
>>> contributor.
>>> 
>>> It is not urgent, but we should at least have a plan on how to simplify and 
>>> unify things.
>>> 
>>> I'm in favour of reducing the number of test targets to the minimum - for 
>>> different configurations I think we should provide a parameter pointing to 
>>> jvm options file and maybe to cassandra.yaml. I know that we currently do 
>>> some super hacky things with cassandra yaml for different configs - like 
>>> concatenting parts of it. I presume it is not necessary - we can have a 
>>> default test config yaml and a directory with overriding yamls; while 
>>> building we could have a tool which is able to load the default 
>>> configuration, apply the override and save the resulting yaml somewhere in 
>>> the build/test/configs for example. That would allows us to easily use 
>>> those yamls in IDE as well - currently it is impossible.
>>> 
>>> What do you think?
>>> 
>>> Thank you and my apologize for bothering about lower priority stuff while 
>>> we have a 5.0 release headache...
>>> 
>>> Jacek
>>> 
>> 

Re: Removal of deprecations added in Cassandra 3.x

[Josh McKenzie]

> Personally, I think the removal of the deprecated code which was marked like 
> that in 3.x is quite safe to do in 5.x but I have to ask broader audience to 
> have a consensus.
Safe for us, sure. Safe for our users, not so much. No amount of including it 
in release notes guarantees they'll see it, and to Mick's point:

> Anything that is public (user-facing) and is isolated code having little cost 
> to it should just be left.
Strong +1 to this sentiment.

On Thu, Nov 30, 2023, at 10:33 AM, Mick Semb Wever wrote:
>> Personally, I think the removal of the deprecated code which was marked like 
>> that in 3.x is quite safe to do in 5.x but I have to ask broader audience to 
>> have a consensus.
> 
> 
> Strawman:
> Evaluate the cost and risk to us by having to keep the code.
> Weigh that against the effort it takes for users to adjust their prod 
> systems, and assume they are many orders of magnitude more than us.
> 
> Anything that is public (user-facing) and is isolated code having little cost 
> to it should just be left.
>   
>>  I think that what is "private" might go away in 5.x easily.
> 
> 
> Yes.
> 
> 
> 

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[Josh McKenzie]

Strongly agree. I started working on a declarative refactor out of our CI 
configuration so circle, ASFCI, and other systems could inherit from it (for 
instance, see pre-commit pipeline declaration here 
);
 I had to set that down while I finished up implementing an internal CI system 
since the code in neither the ASF CI structure nor circle structure (.sh 
embedded in .yml /cry) was re-usable in their current form.

Having a jvm.options and cassandra.yaml file per suite and referencing them 
from a declarative job definition 

 would make things a lot easier to wrap our heads around and maintain I think.

As for what qualifies as burn vs. long... /shrug couldn't tell you. Would have 
to go down the git blame + dev ML + JIRA rabbit hole. :) Maybe someone else 
on-list knows.

On Thu, Nov 30, 2023, at 4:25 AM, Jacek Lewandowski wrote:
> Hi,
> 
> I'm getting a bit lost - what are the exact differences between those test 
> scenarios? What are the criteria for qualifying a test to be part of a 
> certain scenario?
> 
> I'm working a little bit with tests and build scripts and the number of 
> different configurations for which we have a separate target in the build 
> starts to be problematic, I cannot imagine how problematic it is for a new 
> contributor.
> 
> It is not urgent, but we should at least have a plan on how to simplify and 
> unify things.
> 
> I'm in favour of reducing the number of test targets to the minimum - for 
> different configurations I think we should provide a parameter pointing to 
> jvm options file and maybe to cassandra.yaml. I know that we currently do 
> some super hacky things with cassandra yaml for different configs - like 
> concatenting parts of it. I presume it is not necessary - we can have a 
> default test config yaml and a directory with overriding yamls; while 
> building we could have a tool which is able to load the default 
> configuration, apply the override and save the resulting yaml somewhere in 
> the build/test/configs for example. That would allows us to easily use those 
> yamls in IDE as well - currently it is impossible.
> 
> What do you think?
> 
> Thank you and my apologize for bothering about lower priority stuff while we 
> have a 5.0 release headache...
> 
> Jacek
> 

Re: [VOTE] Release Harry 0.0.2

[Josh McKenzie]

+1

On Wed, Nov 29, 2023, at 7:03 AM, Brandon Williams wrote:
> +1
> 
> Kind Regards,
> Brandon
> 
> On Wed, Nov 29, 2023 at 5:15 AM Alex Petrov  wrote:
> >
> > Even though we would like to bring harry in-tree, this is not an immediate 
> > priority. Meanwhile, we need to unblock RPM builds for trunk, which means 
> > no custom jars. We will have at least one more Harry release with 
> > outstanding features to avoid any blocking.
> >
> > Proposing the test build of cassandra-harry 0.0.2 for release, for TCM 
> > purposes.
> >
> > Repository:
> > https://gitbox.apache.org/repos/asf?p=cassandra-harry.git;a=shortlog;h=refs/tags/0.0.2
> >
> > Candidate SHA:
> > https://github.com/apache/cassandra-harry/commit/37761ce599242a34b027baff520e1185b7a7c3af
> > tagged with 0.0.2
> >
> > Artifacts:
> > https://repository.apache.org/content/repositories/orgapachecassandra-1320
> >
> > Key signature: A4C465FEA0C552561A392A61E91335D77E3E87CB
> >
> > Prominent changes:
> >
> > [CASSANDRA-18768] Improvements / changes required for Transactional 
> > Metadata testing:
> >   * Add an ability to run sequential r/w for more deterministic 
> > results
> >   * Implement Network Topology Strategy
> >   * Add all pds iterator to ops selector
> >   * Make sure to log when detecting that a run starts against a 
> > dirty table
> >   * Fix a concurrency issue with reorder buffer
> >   * Add some safety wheels / debugging instruments
> >   * Add a pd selector symmetry test
> >   * Make it simpler to write and log
> >   * Rename sequential rw to write before read
> >   * Avoid starving writers by readers and vice versa
> >   * Add a minimal guide for debugging falsifications
> >   * Fix select peers query for local state checker
> >   * Add examples for programmatic configuration
> >
> > [CASSANDRA-18318] Implement parsing schema provider
> > [CASSANDRA-18315] Trigger exception if we run out of partitions
> > [CASSANDRA-17603] Allow selecting subsets of columns and wilcard queries.
> > [CASSANDRA-17603] Open API for hand-crafting both mutation and read queries
> > [CASSANDRA-17603] Make it possible to run multiple Harry runners 
> > concurrently against the same keyspace
> > [CASSANDRA-17603] Implement concurrent quiescent checker
> > [CASSANDRA-17603] Pull in token util from Cassandra to avoid circular 
> > dependency
> > [CASSANDRA-17603] Pull in Cassandra concurrent utils until there is a 
> > common shared library
> >
> > The vote will be open for 24 hours. Everyone who has tested the build
> > is invited to vote. Votes by PMC members are considered binding. A
> > vote passes if there are at least three binding +1s.
> 

Re: Welcome Francisco Guerrero Hernandez as Cassandra Committer

[Josh McKenzie]

Congrats Francisco!

On Tue, Nov 28, 2023, at 2:37 PM, Melissa Logan wrote:
> Congrats Francisco!
> 
> On Tue, Nov 28, 2023 at 11:34 AM Vinay Chella  wrote:
>> Congratulations Francisco !!
>> 
>> Thanks,
>> Vinay Chella
>> 
>> 
>> On Tue, Nov 28, 2023 at 11:24 AM Mick Semb Wever  wrote:
>>> 
>>> 
>>> On Tue, 28 Nov 2023 at 19:54, Dinesh Joshi  wrote:
 The PMC members are pleased to announce that Francisco Guerrero Hernandez 
 has accepted
 the invitation to become committer today.
 
 Congratulations and welcome!
>>> 
>>> 
>>> Congrats !!

Re: [DISCUSS] CASSANDRA-19113: Publishing dtest-shaded JARs on release

[Josh McKenzie]

Building these jars every time we run every CI job is just silly.

+1.

On Tue, Nov 28, 2023, at 2:08 PM, Francisco Guerrero wrote:
> Hi Abe,
> 
> I'm +1 on this. Several Cassandra-ecosystem projects build the dtest jar in 
> CI. We'd very
> much prefer to just consumed shaded dtest jars from Cassandra releases for 
> testing
> purposes.
> 
> Best,
> - Francisco
> 
> On 2023/11/28 19:02:17 Abe Ratnofsky wrote:
> > Hey folks - wanted to raise a separate thread to discuss publishing of 
> > dtest-shaded JARs on release.
> > 
> > Currently, adjacent projects that want to use the jvm-dtest framework need 
> > to build the shaded JARs themselves. This is a decent amount of work, and 
> > is duplicated across each project. This is mainly relevant for projects 
> > like Sidecar and Driver. Currently, those projects need to clone and build 
> > apache/cassandra themselves, run ant dtest-jar, and move the JAR into the 
> > appropriate place. Different build systems treat local JARs differently, 
> > and the whole process can be a bit complicated. Would be great to be able 
> > to treat these as normal dependencies.
> > 
> > https://issues.apache.org/jira/browse/CASSANDRA-19113
> > 
> > Any objections?
> > 
> > --
> > Abe
> 

Re: CEP-21 - Transactional cluster metadata merged to trunk

[Josh McKenzie]

> on our internal CI system
Some more context:

This environment adheres to the requirements we laid out in pre-commit CI on 
Cassandra 

 with a couple required differences. We don't yet include the resource 
restriction detail in the test report; it's on my backlog of things to do but I 
can confirm that less CPU and <= equivalent ASFCI memory is being allocated for 
each test suite. I also had to go the route of extracting a blend of what's in 
circle and what's in ASF CI (in terms of test suites, filtering, etc) since 
neither represented a complete view of our CI ecosystem; there are currently 
things executed in either environment not executed in the other.

I've been tracking the upstreaming of that declarative combination in 
CASSANDRA-18731 but have had some other priorities take front-seat (i.e. 
getting a new CI system based on that working since neither upstream ASF CI nor 
circle are re-usable in their current form) and will be upstreaming that ASAP. 
https://issues.apache.org/jira/browse/CASSANDRA-18731

I've left a pretty long comment on CASSANDRA-18731 about the structure of 
things and where my opinion falls; *I think we need a separate DISCUSS thread 
on the ML about CI and what we require for pre-commit smoke* suites: 
https://issues.apache.org/jira/browse/CASSANDRA-18731?focusedCommentId=17790270&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17790270

The TL;DR:
> With an *incredibly large* patch in the form of TCM (88k+ LoC, 900+ files 
> touched), we have less than a .002% test failure injection rate using the 
> above restricted smoke heuristic, and many of them look to be circle ci env 
> specific and not asf ci.

>From a cursory inspection it looks like most of the breakages being tracked on 
>the ticket Sam linked for TCM are likely to be circle env specific (new *nix 
>optimized deletion having a race, OOM's, etc). The TCM merge is actually a 
>great forcing function for us to surface anything env specific in terms of 
>timing and resourcing up-front; I'm glad we have this opportunity but it's 
>unfortunate that it's been interpreted as merging w/out passing CI as opposed 
>to having some env-difference specific kinks to work out.

*This was an incredibly huge merge.* For comparison, I just did a --stat on the 
merge for CASSANDRA-8099:
> 645 files changed, 49381 insertions(+), 42227 deletions(-)

TCM from the C* repo:
>  934 files changed, 66185 insertions(+), 21669 deletions(-)
My gut tells me it's basically impossible to have a merge of this size that 
doesn't disrupt what it's merging into, or the authors just end up slowly dying 
in rebase hell. Or both. This was a massive undertaking and compared to our 
past on this project, has had an incredibly low impact on the target it was 
merged into and the authors are rapidly burning down failures.

To the authors - great work, and thanks for being so diligent on following up 
on any disruptions this body of work has caused to other contributors' 
environments.

To the folks who were disrupted - I get it. This is deeply frustrating, green 
CI has long been many of our white whale's, and having something merge over a 
US holiday week with an incredibly active project where we don't all have time 
to keep up with everything can make things like this feel like a huge surprise. 
It's incredibly unfortunate that the timing on us transitioning to this new CI 
system and working out the kinks is when this behemoth of a merge needed to 
come through, but silver-lining.

We're making great strides. Let's not lose sight of our growth because of the 
pain in the moment of it.

~Josh

p.s. - for the record, I don't think we should hold off on merging things just 
because some folks are on holiday. :)

On Mon, Nov 27, 2023, at 3:38 PM, Sam Tunnicliffe wrote:
> I ought to clarify, we did actually have green CI modulo 3 flaky tests on our 
> internal CI system. I've attached the test artefacts to CASSANDRA-18330 
> now[1][2]: 2 of the 3 failures are upgrade dtests, with 1 other python dtest 
> failure noted. None of these were reproducible in a dev setup, so we 
> suspected them to be environmental and intended to merge before returning to 
> confirm that. The "known" failures that we mentioned in the email that 
> started this thread were ones observed by Mick running the cep-21-tcm branch 
> through Circle before merging.  
> 
> As the CEP-21 changeset was approaching 88k LoC touching over 900 files, 
> permanently rebasing as we tried to eradicate every flaky test was simply 
> unrealistic, especially as other significant patches continued to land in 
> trunk. With that in mind, we took the decision to merge so that we could 
> focus on actually removing any remaining instability.
> 
> [1] https://issues.apache.org/jira/secure/attachment/13064727/ci_summary.html
> [2] 
> https://issues.apache.org/jira/secure/attachment/13064

Re: [DISCUSS] Harry in-tree

[Josh McKenzie]

Strong +1 to including harry in-tree and further, integrating a harry stress 
soak into our pre-commit and post-commit CI.

On Fri, Nov 24, 2023, at 5:10 PM, Alex Petrov wrote:
> Unfortunately my Harry talk got declined. Of course I’ll be happy to talk 
> about Harry and how it can be useful for contributors and about people’s 
> expectations. My talk is going to be about TCM again this time.
> 
> I will make sure examples are in place and are expressive by the summit.
> 
> On Fri, Nov 24, 2023, at 6:18 PM, Jeremy Hanna wrote:
>> I'm excited for Harry to come in-tree to improve the project stability and 
>> quality.  I know you're doing a talk at the Cassandra Summit about Harry to 
>> go over it.  If there's anything that can be done as part of this process to 
>> improve onboarding for Harry too, that would be great.  I'm just thinking 
>> about examples and things like that so people new to Harry can more easily 
>> write and run tests, test new features, and have a standard process for 
>> reporting findings.
>> 
>> Thanks Alex and all involved!
>> 
>> Jeremy
>> 
>>> On Nov 24, 2023, at 9:43 AM, Alex Petrov  wrote:
>>> 
>>> Hi everyone,
>>> 
>>> With TCM landed, there will be way more Harry tests in-tree: we are using 
>>> it for many coordination tests, and there's now a simulator test that uses 
>>> Harry. During development, Harry has allowed us to uncover and resolve 
>>> numerous elusive edge cases.
>>> 
>>> I had conversations with several folks, and wanted to propose to move 
>>> harry-core to Cassandra test tree. This will substantially 
>>> simplify/streamline co-development of Cassandra and Harry. With a new 
>>> HistoryBuilder API that has helped to find and trigger [1] [2] and [3], it 
>>> will also be much more approachable.
>>> 
>>> Besides making it easier for everyone to develop new fuzz tests, it will 
>>> also substantially lower the barrier to entry. Currently, debugging an 
>>> issue found by Harry involves a cumbersome process of rebuilding and 
>>> transferring jars between Cassandra and Harry, depending on which side you 
>>> modify. This not only hampers efficiency but also deters broader adoption. 
>>> By merging harry-core into the Cassandra test tree, we eliminate this 
>>> barrier.
>>> 
>>> Thank you,
>>> --Alex
>>> 
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19011
>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18993
>>> [3] https://issues.apache.org/jira/browse/CASSANDRA-18932
> 

Re: Road to 5.0-GA (was: [VOTE] Release Apache Cassandra 5.0-alpha2)

[Josh McKenzie]

> I think before we cut a beta we need to have diagnosed and fixed 18993 
> (assuming it is a bug).
Before a beta? I could see that for rc or GA definitely, but having a known 
(especially non-regressive) data loss bug in a beta seems like it's compatible 
with the guarantees we're providing for it: 
https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle

> This release is recommended for test/QA clusters where short(order of 
> minutes) downtime during upgrades is not an issue


On Sat, Nov 4, 2023, at 12:56 PM, Ekaterina Dimitrova wrote:
> Totally agree with the others. Such an issue on its own should be a priority 
> in any release. Looking forward to the reproduction test mentioned on the 
> ticket.
> 
> Thanks to Alex for his work on harry!
> 
> On Sat, 4 Nov 2023 at 12:47, Benedict  wrote:
>> Alex can confirm but I think it actually turns out to be a new bug in 5.0, 
>> but either way we should not cut a release with such a serious potential 
>> known issue.
>> 
>> > On 4 Nov 2023, at 16:18, J. D. Jordan  wrote:
>> > 
>> > Sounds like 18993 is not a regression in 5.0? But present in 4.1 as well? 
>> >  So I would say we should fix it with the highest priority and get a new 
>> > 4.1.x released. Blocking 5.0 beta voting is a secondary issue to me if we 
>> > have a “data not being returned” issue in an existing release?
>> > 
>> >> On Nov 4, 2023, at 11:09 AM, Benedict  wrote:
>> >> 
>> >> I think before we cut a beta we need to have diagnosed and fixed 18993 
>> >> (assuming it is a bug).
>> >> 
>>  On 4 Nov 2023, at 16:04, Mick Semb Wever  wrote:
>> >>> 
>> >>> 
>>  
>>  With the publication of this release I would like to switch the
>>  default 'latest' docs on the website from 4.1 to 5.0.  Are there any
>>  objections to this ?
>> >>> 
>> >>> 
>> >>> I would also like to propose the next 5.0 release to be 5.0-beta1
>> >>> 
>> >>> With the aim of reaching GA for the Summit, I would like to suggest we
>> >>> work towards the best-case scenario of 5.0-beta1 in two weeks and
>> >>> 5.0-rc1 first week Dec.
>> >>> 
>> >>> I know this is a huge ask with lots of unknowns we can't actually
>> >>> commit to.  But I believe it is a worthy goal, and possible if nothing
>> >>> sideswipes us – but we'll need all the help we can get this month to
>> >>> make it happen.
>> >>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

o knowingly go against it, or want to modify or 
> clarify what’s meant. This just falls naturally out of how we do things here 
> I think, and is how we go about a lot of business already. It retains the 
> agility you were talking about, setting norms cheaply.
> 
> It isn’t however a tightly held policy or legislative cudgel, it’s just what 
> those who were talking and paying attention at the time agreed. It can be 
> chucked out or rewoven at zero cost, but if the norms have taken hold and are 
> broadly understood in the same way, it won’t change much or at all, because 
> the actual glue is the norm, not the words, which only serve to broadcast 
> some formulation of the norm.
> 
> 
> 
>> On 1 Nov 2023, at 23:41, Josh McKenzie  wrote:
>> 
>>> but binding to the same extent 2 committers reviewing something we later 
>>> need to revert is binding.
>> To elaborate a bit - what I mean is "it's a bar we apply to help establish a 
>> baseline level of consensus but it's very much a 2-way door". Obviously 2 
>> committers +1'ing code is a formal agreed upon voting mechanism.
>> 
>> On Wed, Nov 1, 2023, at 7:26 PM, Josh McKenzie wrote:
>>>> Community voting is also entirely by consensus, there is no such thing as 
>>>> a simple majority community vote, technical or otherwise.
>>> Ah hah! You're absolutely correct in that this isn't one of our "blessed" 
>>> ways we vote. There's nothing written down about "committers are binding, 
>>> simple majority" for any specific category of discussion.
>>> 
>>> Are we ok with people creatively applying different ways to vote for things 
>>> where there's not otherwise guidance if they feel it helps capture 
>>> sentiment and engagement? Obviously the outcome of that isn't binding in 
>>> the same way other votes by the pmc are, but binding to the same extent 2 
>>> committers reviewing something we later need to revert is binding.
>>> 
>>> I'd rather we have a bunch of committers weigh in if we're talking about 
>>> changing import ordering, or Config.java structure, or refactoring out 
>>> singletons, or gatekeeping CI - things we've had come up over the years 
>>> where we've had a lot of people chime in and we benefit from more than just 
>>> "2 committers agree on it" but less than "We need a CEP or pmc vote for 
>>> this".
>>> 
>>> 
>>> On Wed, Nov 1, 2023, at 5:10 PM, Benedict wrote:
>>>> 
>>>> The project governance document does not list any kind of general purpose 
>>>> technical change vote. There are only three very specific kinds of 
>>>> community vote: code contributions, CEP and release votes.  Community 
>>>> voting is also entirely by consensus, there is no such thing as a simple 
>>>> majority community vote, technical or otherwise. I suggest carefully 
>>>> re-reading the document we both formulated!
>>>> 
>>>> If it is a technical contribution, as you contest, we only need a normal 
>>>> technical contribution vote to override it - i.e. two committer +1s. If 
>>>> that’s how we want to roll with it, I guess we’re not really in 
>>>> disagreement.
>>>> 
>>>> None of this really fundamentally changes anything. There’s a strong norm 
>>>> for a commit gate on CI, and nobody is going to go about breaking this 
>>>> norm willy-nilly. But equally there’s no need to panic and waste all this 
>>>> time debating hypothetical mechanisms to avoid this supposedly ironclad 
>>>> rule.
>>>> 
>>>> We clearly need to address confusion over governance though. The idea that 
>>>> agreeing things carefully costs us agility is one I cannot endorse. The 
>>>> project has leaned heavily into the consensus side of the Apache Way, as 
>>>> evidenced by our governance document. That doesn’t mean things can’t 
>>>> change quickly, it just means *before those changes become formal 
>>>> requirements *there needs to be *broad* consensus, as defined in the 
>>>> governing document. That’s it.
>>>> 
>>>> The norm existed before the vote, and it exists whether the vote was valid 
>>>> or not. That is how things evolve on the project, we just formalise them a 
>>>> little more slowly.
>>>> 
>>>> 
>>>>> On 1 Nov 2023, at 20:07, Josh McKenzie  wrote:
>>>>> 
>>>>>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> but binding to the same extent 2 committers reviewing something we later need 
> to revert is binding.
To elaborate a bit - what I mean is "it's a bar we apply to help establish a 
baseline level of consensus but it's very much a 2-way door". Obviously 2 
committers +1'ing code is a formal agreed upon voting mechanism.

On Wed, Nov 1, 2023, at 7:26 PM, Josh McKenzie wrote:
>> Community voting is also entirely by consensus, there is no such thing as a 
>> simple majority community vote, technical or otherwise.
> Ah hah! You're absolutely correct in that this isn't one of our "blessed" 
> ways we vote. There's nothing written down about "committers are binding, 
> simple majority" for any specific category of discussion.
> 
> Are we ok with people creatively applying different ways to vote for things 
> where there's not otherwise guidance if they feel it helps capture sentiment 
> and engagement? Obviously the outcome of that isn't binding in the same way 
> other votes by the pmc are, but binding to the same extent 2 committers 
> reviewing something we later need to revert is binding.
> 
> I'd rather we have a bunch of committers weigh in if we're talking about 
> changing import ordering, or Config.java structure, or refactoring out 
> singletons, or gatekeeping CI - things we've had come up over the years where 
> we've had a lot of people chime in and we benefit from more than just "2 
> committers agree on it" but less than "We need a CEP or pmc vote for this".
> 
> 
> On Wed, Nov 1, 2023, at 5:10 PM, Benedict wrote:
>> 
>> The project governance document does not list any kind of general purpose 
>> technical change vote. There are only three very specific kinds of community 
>> vote: code contributions, CEP and release votes.  Community voting is also 
>> entirely by consensus, there is no such thing as a simple majority community 
>> vote, technical or otherwise. I suggest carefully re-reading the document we 
>> both formulated!
>> 
>> If it is a technical contribution, as you contest, we only need a normal 
>> technical contribution vote to override it - i.e. two committer +1s. If 
>> that’s how we want to roll with it, I guess we’re not really in disagreement.
>> 
>> None of this really fundamentally changes anything. There’s a strong norm 
>> for a commit gate on CI, and nobody is going to go about breaking this norm 
>> willy-nilly. But equally there’s no need to panic and waste all this time 
>> debating hypothetical mechanisms to avoid this supposedly ironclad rule.
>> 
>> We clearly need to address confusion over governance though. The idea that 
>> agreeing things carefully costs us agility is one I cannot endorse. The 
>> project has leaned heavily into the consensus side of the Apache Way, as 
>> evidenced by our governance document. That doesn’t mean things can’t change 
>> quickly, it just means *before those changes become formal requirements 
>> *there needs to be *broad* consensus, as defined in the governing document. 
>> That’s it.
>> 
>> The norm existed before the vote, and it exists whether the vote was valid 
>> or not. That is how things evolve on the project, we just formalise them a 
>> little more slowly.
>> 
>> 
>>> On 1 Nov 2023, at 20:07, Josh McKenzie  wrote:
>>> 
>>> First off, I appreciate your time and attention on this stuff. Want to be 
>>> up front about that since these kinds of discussions can get prickly all 
>>> too easily. I'm *at least* as guilty as anyone else about getting my back 
>>> up on stuff like this. Figuring out the right things to "harden" as shared 
>>> contractual ways we behave and what to leave loose and case-by-case is 
>>> going to continue to be a challenge for us as we grow.
>>> 
>>> The last thing I personally want is for us to have too many extraneous 
>>> rules formalizing things that just serve to slow down peoples' ability to 
>>> contribute to the project. The flip side of that - for all of us to work in 
>>> a shared space and collectively remain maximally productive, some 
>>> individual freedoms (ability to merge a bunch of broken code and/or ninja 
>>> in things as we see fit, needing 2 committers' eyes on things, etc) will 
>>> have to be given up.
>>> 
>>> At it's core the discussion we had was prompted by divergence between 
>>> circle and ASF CI and our release process dragging on repeatedly during the 
>>> "stabilize ASF CI" phase.

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> Community voting is also entirely by consensus, there is no such thing as a 
> simple majority community vote, technical or otherwise.
Ah hah! You're absolutely correct in that this isn't one of our "blessed" ways 
we vote. There's nothing written down about "committers are binding, simple 
majority" for any specific category of discussion.

Are we ok with people creatively applying different ways to vote for things 
where there's not otherwise guidance if they feel it helps capture sentiment 
and engagement? Obviously the outcome of that isn't binding in the same way 
other votes by the pmc are, but binding to the same extent 2 committers 
reviewing something we later need to revert is binding.

I'd rather we have a bunch of committers weigh in if we're talking about 
changing import ordering, or Config.java structure, or refactoring out 
singletons, or gatekeeping CI - things we've had come up over the years where 
we've had a lot of people chime in and we benefit from more than just "2 
committers agree on it" but less than "We need a CEP or pmc vote for this".


On Wed, Nov 1, 2023, at 5:10 PM, Benedict wrote:
> 
> The project governance document does not list any kind of general purpose 
> technical change vote. There are only three very specific kinds of community 
> vote: code contributions, CEP and release votes.  Community voting is also 
> entirely by consensus, there is no such thing as a simple majority community 
> vote, technical or otherwise. I suggest carefully re-reading the document we 
> both formulated!
> 
> If it is a technical contribution, as you contest, we only need a normal 
> technical contribution vote to override it - i.e. two committer +1s. If 
> that’s how we want to roll with it, I guess we’re not really in disagreement.
> 
> None of this really fundamentally changes anything. There’s a strong norm for 
> a commit gate on CI, and nobody is going to go about breaking this norm 
> willy-nilly. But equally there’s no need to panic and waste all this time 
> debating hypothetical mechanisms to avoid this supposedly ironclad rule.
> 
> We clearly need to address confusion over governance though. The idea that 
> agreeing things carefully costs us agility is one I cannot endorse. The 
> project has leaned heavily into the consensus side of the Apache Way, as 
> evidenced by our governance document. That doesn’t mean things can’t change 
> quickly, it just means *before those changes become formal requirements 
> *there needs to be *broad* consensus, as defined in the governing document. 
> That’s it.
> 
> The norm existed before the vote, and it exists whether the vote was valid or 
> not. That is how things evolve on the project, we just formalise them a 
> little more slowly.
> 
> 
>> On 1 Nov 2023, at 20:07, Josh McKenzie  wrote:
>> 
>> First off, I appreciate your time and attention on this stuff. Want to be up 
>> front about that since these kinds of discussions can get prickly all too 
>> easily. I'm *at least* as guilty as anyone else about getting my back up on 
>> stuff like this. Figuring out the right things to "harden" as shared 
>> contractual ways we behave and what to leave loose and case-by-case is going 
>> to continue to be a challenge for us as we grow.
>> 
>> The last thing I personally want is for us to have too many extraneous rules 
>> formalizing things that just serve to slow down peoples' ability to 
>> contribute to the project. The flip side of that - for all of us to work in 
>> a shared space and collectively remain maximally productive, some individual 
>> freedoms (ability to merge a bunch of broken code and/or ninja in things as 
>> we see fit, needing 2 committers' eyes on things, etc) will have to be given 
>> up.
>> 
>> At it's core the discussion we had was prompted by divergence between circle 
>> and ASF CI and our release process dragging on repeatedly during the 
>> "stabilize ASF CI" phase. The "do we require green ci before merge of 
>> tickets" seems like it came along as an intuitive rider; best I can recall 
>> my thinking was "how else could we have a manageable load to stabilize in 
>> ASF CI if we didn't even require green circle before merging things in", but 
>> we didn't really dig into details; from a re-reading now, that portion of 
>> the discussion was just taken for granted as us being in alignment. Given it 
>> was a codifying a norm and everyone else in the discussion generally agreed, 
>> I don't think I or anyone thought to question it.
>> 
>> 
>>> “Votes on project structure *and 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

votes on project structure”
> 
> 
> 
> “Votes on project structure *and governance”*. Governance, per Wikipedia, is 
> "the way rules, norms and actions are structured and sustained.”
> 
> 
> 
> I do not see any ambiguity here. The community side provides no basis for a 
> vote of this kind, while the PMC side specifically reserves this kind of 
> decision. But evidently we need to make this clearer.
> 
> 
> 
> Regarding the legitimacy of questioning this now: I have not come up against 
> this legislation before. The norm of requiring green CI has been around for a 
> lot longer than this vote, so nothing much changed until we started 
> questioning the *specifics* of this legislation. At this point, the 
> legitimacy of the decision also matters. Clearly there is broad support for a 
> policy of this kind, but is this specific policy adequate?
> 
> 
> 
> While I endorse the general sentiment of the policy, I do not endorse a 
> policy that has no wiggle room. I have made every effort in all of my 
> policy-making to ensure there are loosely-defined escape hatches for the 
> community to use, in large part to minimise this kind of legalistic logjam, 
> which is just wasted cycles.
> 
> 
> 
> 
> 
>> On 1 Nov 2023, at 15:31, Josh McKenzie  wrote:
>> 
>>> That vote thread also did not reach the threshold; it was incorrectly 
>>> counted, as committer votes are not binding for procedural changes. I 
>>> counted at most 8 PMC +1 votes. 
>> This piqued my curiosity.
>> 
>> Link to how we vote: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Project+Governance
>> *STATUS: Ratified 2020/06/25*
>> 
>> Relevant bits here:
>>> On dev@:
>>> 
>>>  1. Discussion / binding votes on releases (Consensus: min 3 PMC +1, no -1)
>>>  2. Discussion / binding votes on project structure and governance changes 
>>> (adopting subprojects, how we vote and govern, etc). (super majority)
>> 
>> The thread where we voted on the CI bar Jeremiah referenced: 
>> https://lists.apache.org/thread/2shht9rb0l8fh2gfqx6sz9pxobo6sr60
>> Particularly relevant bit:
>>> Committer / pmc votes binding. Simple majority passes.
>> I think you're arguing that voting to change our bar for merging when it 
>> comes to CI falls under "votes on project structure"? I think when I called 
>> that vote I was conceptualizing it as a technical discussion about a shared 
>> norm on how we as committers deal with code contributions, where the 
>> "committer votes are binding, simple majority" applies.
>> 
>> I can see credible arguments in either direction, though I'd have expected 
>> those concerns or counter-arguments to have come up back in Jan of 2022 when 
>> we voted on the CI changes, not almost 2 years later after us operating 
>> under this new shared norm. The sentiments expressed on the discuss and vote 
>> thread were consistently positive and uncontentious; this feels to me like 
>> it falls squarely under the spirit of lazy consensus only at a much larger 
>> buy-in level than usual: 
>> https://community.apache.org/committers/decisionMaking.html#lazy-consensus
>> 
>> We've had plenty of time to call this vote and merge bar into question (i.e. 
>> every ticket we merge we're facing the "no regressions" bar), and the only 
>> reason I'd see us treating TCM or Accord differently would be because 
>> they're much larger bodies of work at merge so it's going to be a bigger 
>> lift to get to non-regression CI, and/or we would want a release cut from a 
>> formal branch rather than a feature branch for preview.
>> 
>> An alternative approach to keep this merge and CI burden lower would have 
>> been more incremental work merged into trunk periodically, an argument many 
>> folks in the community have made in the past. I personally have mixed 
>> feelings about it; there's pros and cons to both approaches.
>> 
>> All that said, I'm in favor of us continuing with this as a valid and 
>> ratified vote (technical norms == committer binding + simple majority). If 
>> we want to open a formal discussion about instead considering that a 
>> procedural change and rolling things back based on those grounds I'm fine 
>> with that, but we'll need to discuss that and think about the broader 
>> implications since things like changing import ordering, tooling, or other 
>> ecosystem-wide impacting changes (CI systems we all share, etc) would 
>> similarly potentially run afoul of needing superma

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

f CEPs.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 25 Oct 2023, at 21:55, Benedict  wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I am surprised this needs to be said, but - especially for 
>>>>>>>>>>>>>> long-running CEPs - you must involve yourself early, and 
>>>>>>>>>>>>>> certainly within some reasonable time of being notified the work 
>>>>>>>>>>>>>> is ready for broader input and review. In this case, more than 
>>>>>>>>>>>>>> six months ago.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This isn’t the first time this has happened, and it is 
>>>>>>>>>>>>>> disappointing to see it again. Clearly we need to make this 
>>>>>>>>>>>>>> explicit in the guidance docs.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Regarding the release of 5.1, I understood the proposal to be 
>>>>>>>>>>>>>> that we cut an actual alpha, thereby sealing the 5.1 release 
>>>>>>>>>>>>>> from new features. Only features merged before we cut the alpha 
>>>>>>>>>>>>>> would be permitted, and the alpha should be cut as soon as 
>>>>>>>>>>>>>> practicable. What exactly would we be waiting for? 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> If we don’t have a clear and near-term trigger for branching 5.1 
>>>>>>>>>>>>>> for its own release, shortly after Accord and TCM merge, then I 
>>>>>>>>>>>>>> am in favour of instead delaying 5.0.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> On 25 Oct 2023, at 19:40, Mick Semb Wever  
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> I'm open to the suggestions of not branching cassandra-5.1 
>>>>>>>>>>>>>>> and/or naming a preview release something other than 5.1-alpha1.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> But… the codebases and release process (and upgrade tests) do 
>>>>>>>>>>>>>>> not currently support releases with qualifiers that are not 
>>>>>>>>>>>>>>> alpha, beta, or rc.  We can add a preview qualifier to this 
>>>>>>>>>>>>>>> list, but I would not want to block getting a preview release 
>>>>>>>>>>>>>>> out only because this wasn't yet in place.  
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Hence the proposal used 5.1-alpha1 simply because that's what 
>>>>>>>>>>>>>>> we know we can do today.  An alpha release also means 
>>>>>>>>>>>>>>> (typically) the branch.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Is anyone up for looking into adding a "preview" qualifier to 
>>>>>>>>>>>>>>> our release process? 
>>>>>>>>>>>>>>> This may also solve our previous discussions and desire to have 
>>>>>>>>>>>>>>> quarterly releases that folk can use through the trunk dev 
>>>>>>>>>>>>>>> cycle.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Personally, with my understanding of timelines in front of us 
>>>>>>>>>>>>>>> to fully review and stabilise tcm, it makes sense to branch it 
>>>>>>>>>>>>>>> as soon as it's merged.  It's easiest to stabilise it on a 
>&

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> ACCORD in particular was hyped in numerous talks and presentations and noone 
> cautioned it might not hit 5.0, quite the opposite
We need to be very careful in the future about how we communicate the 
availability of future novel work, especially when the ones promoting the 
delivery of that work and timelines aren't the ones actively working on the 
code. And to be explicit - I don't think there's any bad actors here; I think 
this is a natural consequence of specialization of skills and focus in the 
community as well as disjoint between different groups of people.

Also, it's become clear to me that we still weren't all in alignment on our 
view of "do we ship 5.0 based on a date or do we ship 5.0 based on feature 
availability". Since we're still going through some evolution in our release 
philosophy (train vs. feature, etc), this is to be expected. We're getting 
there.

Having a marketing working group has helped bridge this gap, and getting more 
participation from other people in the community on that effort would help 
align more of us.


On Fri, Oct 27, 2023, at 5:00 PM, German Eichberger via dev wrote:
> Definitely want to second Josh. When I reached out on the ACCORD channel 
> about testing folks were super helpful and transparent about bugs, etc.
> 
> Frankly, I was pretty frustrated that ACCORD+TCM slipped. I was looking 
> forward to it and felt let down - but I also haven't done anything to help 
> other than trying it out. So, I only have myself to blame... 
> 
> That there was a surprise for many of us that it slipped is an indication 
> there wasn't enough communication - we should probably rethink how we 
> communicate progress, especially on long running and highly anticipated 
> initiatives. Maybe a paragraph in the "Project Status Update" (but then we 
> need more frequent updates 🙂) -- or send a separate update e-mail or as Maxim 
> is suggesting to some newly created release list. 
> 
> A highly anticipated feature has more visibility and we need to account for 
> that with more communication other than the usual channels. ACCORD in 
> particular was hyped in numerous talks and presentations and noone cautioned 
> it might not hit 5.0, quite the opposite --so we need to ask ourselves how 
> people who go on stage as Cassandra experts are not aware that it could slip. 
> That's where I think more communication could help -- 
> 
> 
> Thanks,
> German
> 
> 
> 
> 
> 
> *From:* Josh McKenzie 
> *Sent:* Friday, October 27, 2023 10:13 AM
> *To:* dev 
> *Subject:* [EXTERNAL] Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and 
> cut an immediate 5.1-alpha1)
>  
> Lots of threads of thought have popped up here. The big one I feel needs to 
> be clearly addressed and inspected is the implication of development not 
> happening transparently and not being inclusive or available for 
> participation by the community on major features.
> 
> The CEP process + dedicated development channels on ASF slack + public JIRA's 
> + feature branches in the ASF repo we've seen with specifically TCM and 
> Accord are the most transparent this kind of development has *ever been* on 
> this project, and I'd argue right at the sweet spot or past where the degree 
> of reaching out to external parties to get feedback starts to not just hit 
> diminishing returns but starts to actively hurt a small group of peoples' 
> ability to make rapid progress on something.
> 
> No-one can expect to review everything, and no-one can expect to follow every 
> JIRA, commit, or update. This is why we have the role of a committer; a 
> person in this community we've publicly communicated we trust based on earned 
> merit (and in our project's case, at least 2 people who's opinion we trust) 
> to do quality work, validate it, and reach our expected bar for correctness, 
> performance, and maintainability. If a CEP is voted in and 2 committers have 
> an implementation they feel meets the goals, CI is green, and nobody has a 
> serious technical concern that warrants a binding -1, we're good. It doesn't, 
> and shouldn't, matter who currently employs or sponsors their work. It 
> doesn't, and shouldn't, matter whether individuals on the project who were 
> interested in collaborating on that work missed one or multiple 
> announcements, or whether they saw those announcements and just didn't have 
> the cycles to engage when they wanted to.
> 
> Now - we can always improve. We can always try and be proactive, knowing each 
> other and our interests and reaching out to specific folks to make sure 
> they're aware that work has hit a collaboration point or inflection point. I 
> ca

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

t;>> >>> quarterly releases that folk can use through the trunk dev cycle.
>>>> >>>
>>>> >>> Personally, with my understanding of timelines in front of us to fully 
>>>> >>> review and stabilise tcm, it makes sense to branch it as soon as it's 
>>>> >>> merged.  It's easiest to stabilise it on a branch, and there's 
>>>> >>> definitely the desire and demand to do so, so it won't be getting 
>>>> >>> forgotten or down-prioritised.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Wed, 25 Oct 2023 at 18:07, Jeremiah Jordan 
>>>> >>>  wrote:
>>>> >>>>>
>>>> >>>>> If we do a 5.1 release why not take it as an opportunity to release 
>>>> >>>>> more things. I am not saying that we will. Just that we should let 
>>>> >>>>> that door open.
>>>> >>>>
>>>> >>>>
>>>> >>>> Agreed.  This is the reason I brought up the possibility of not 
>>>> >>>> branching off 5.1 immediately.
>>>> >>>>
>>>> >>>>
>>>> >>>> On Oct 25, 2023 at 3:17:13 AM, Benjamin Lerer  
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> The proposal includes 3 things:
>>>> >>>>> 1. Do not include TCM and Accord in 5.0 to avoid delaying 5.0
>>>> >>>>> 2. The next release will be 5.1 and will include only Accord and TCM
>>>> >>>>> 3. Merge TCM and Accord right now in 5.1 (making an initial release)
>>>> >>>>>
>>>> >>>>> I am fine with question 1 and do not have a strong opinion on which 
>>>> >>>>> way to go.
>>>> >>>>> 2. Means that every new feature will have to wait for post 5.1 even 
>>>> >>>>> if it is ready before 5.1 is stabilized and shipped. If we do a 5.1 
>>>> >>>>> release why not take it as an opportunity to release more things. I 
>>>> >>>>> am not saying that we will. Just that we should let that door open.
>>>> >>>>> 3. There is a need to merge TCM and Accord as maintaining those 
>>>> >>>>> separate branches is costly in terms of time and energy. I fully 
>>>> >>>>> understand that. On the other hand merging TCM and Accord will make 
>>>> >>>>> the TCM review harder and I do believe that this second round of 
>>>> >>>>> review is valuable as it already uncovered a valid issue. 
>>>> >>>>> Nevertheless, I am fine with merging TCM as soon as it passes CI and 
>>>> >>>>> continuing the review after the merge. If we cannot meet at least 
>>>> >>>>> that quality level (Green CI) we should not merge just for creating 
>>>> >>>>> an 5.1.alpha release for the summit.
>>>> >>>>>
>>>> >>>>> Now, I am totally fine with a preview release without numbering and 
>>>> >>>>> with big warnings that will only serve as a preview for the summit.
>>>> >>>>>
>>>> >>>>> Le mer. 25 oct. 2023 à 06:33, Berenguer Blasi 
>>>> >>>>>  a écrit :
>>>> >>>>>>
>>>> >>>>>> I also think there's many good new features in 5.0 already they'd 
>>>> >>>>>> make a
>>>> >>>>>> good release even on their own. My 2 cts.
>>>> >>>>>>
>>>> >>>>>> On 24/10/23 23:20, Brandon Williams wrote:
>>>> >>>>>> > The catch here is that we don't publish docker images currently.  
>>>> >>>>>> > The
>>>> >>>>>> > C* docker images available are not made by us.
>>>> >>>>>> >
>>>> >>>>>> > Kind Regards,
>>>> >>>>>> > Brandon
>>>> >>>>>> >
>>>> >>>>>> > On Tue, Oct 24, 2023 at 3:31 PM Patrick McFadin 
>>>> >>>>>> >  wrote:
>>>> >>>>>> >> Le

Project Status Update: 90-day catch-up edition [2023-10-27]

[Josh McKenzie]

In case you're keeping score on how frequently these are coming out: *please 
stop*. ;)

Silver lining - looks like we have a lot to discuss this round! Last update was 
late July and we've been churning through the 5.0 freeze and stabilization 
phase.


*[New Contributors Getting Started]
*
Check out https://the-asf.slack.com, channel #cassandra-dev. Reply directly to 
me on this email if you need an invite for your account, and reach out to the 
@cassandra_mentors alias in the channel if you need to get oriented.

We have a list of curated "getting started" tickets you can find here, filtered 
to "ToDo" (i.e. not yet worked): 
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=484&quickFilter=2160&quickFilter=2162&quickFilter=2652.

*Helpful links:**
*
- Getting Started with Development on C*: 
https://cassandra.apache.org/_/development/gettingstarted.html
- Building and IDE integration (worktrees are your friend; msg me on slack if 
you need pointers): https://cassandra.apache.org/_/development/ide.html
- Code Style: https://cassandra.apache.org/_/development/code_style.html


*[Dev mailing list]
*
https://lists.apache.org/list?dev@cassandra.apache.org:dfr=2023-7-20%7Cdto=2023-10-27:

My last email of shame was 35 threads. Drumroll for this one...
91. *Yeesh*. Let me stick to highlights.

Ekaterina pushed through dropping JDK8 support and adding JDK17 support... back 
in July. If you didn't know about it by know, consider yourself doubly 
notified. :) . https://lists.apache.org/thread/9pwz3vtpf88fly27psc7yxvcv0lwbz8k 
I think I can speak on behalf of all of us when I say: **Thank You Ekaterina.**

This came up recently on another thread about when to branch 5.1, but we 
discussed our freeze plans and exception rules for TCM and Accord here: 
https://lists.apache.org/thread/mzj3dq8b7mzf60k6mkby88b9n9ywmsgw. Mick was 
essentially looking for a similar waiver for Vector search since it was well 
abstracted, depended on SAI and external libs, and in general shouldn't be too 
big of a disruption to get into 5.0. General consensus at the time was "sure", 
and the work has since been completed. But here's the reminder and link for 
posterity (and in case you missed it).

Jaydeep reached out about a potential short-term solution to detecting 
token-ownership mismatch while we don't yet have TCM; this seems more pressing 
now as we're looking at a 5.0 without yet having TCM in it. The dev ML thread 
is here: https://lists.apache.org/thread/4p0orhom42g36osnknqj3fqmqhvqml1g, and 
he created https://issues.apache.org/jira/browse/CASSANDRA-18758 dealing with 
the topic. There's a relatively modest (7 files, just over 300 lines) PR 
available here: https://github.com/apache/cassandra/pull/2595/files; I haven't 
looked into it, but it might be worth considering getting this into 5.0 since 
it looks like we're moving to cutting w/out TCM. Any thoughts?

We had a pretty good discussion about automated repair scheduling, discussing 
whether it should live in the DB proper vs. in the sidecar, pros and cons, 
pressures, etc. Not sure if things moved beyond that; I know there's at least a 
few implementations out there that haven't yet made their way back to the ASF 
project proper. Thread: 
https://lists.apache.org/thread/glvmkwknf91rxc5l6w4d4m1kcvlr6mrv. My hope is we 
can avoid the gridlock we hit for a long time with the sidecar where there are 
multiple implementations with different tradeoffs and everyone's 
disincentivized from accepting a solution different from their own in-house one 
since it'd theoretically require re-tooling. Tough problem with no easy 
solutions, but would love to see this become a first class citizen in the 
ecosystem.

Paulo brought up a discussion about moving to disk_access_mode = 
mmap_index_only on 5.0. Seemed to be a consensus there but I'm not sure we 
actually changed that in the 5.0 branch? Thread: 
https://lists.apache.org/thread/nhp6vftc4kc3dxskngxy5rpo1lp19drw. Just pulled 
on cassandra-5.0 and it looks like auto + hasLargeAddressSpace() == .mmap 
rather than .mmap_index_only.

David Capwell worked on adding some retries to repair messages when they're 
failing to make the process more robust: 
https://lists.apache.org/thread/wxv6k6slljqcw73xcmpxj4kn5lz95jd1. Reception was 
positive enough that he went so far as to back-port it and also work on some 
for IR. Looks like he could use a reviewer here: 
https://issues.apache.org/jira/browse/CASSANDRA-18962 - and this is patch 
available.

Mike Adamson reached out about adding / taking a dependency on jvector: 
https://lists.apache.org/thread/zkqg7mk9hp35zn0cf1tvywc2m3l63jrn. The general 
gist of it was "looks good, written by committer(s) / pmc members, permissvely 
licensed. Go for it". Some discussion about copyright holders and whether that 
matters from an ASF perspective, and we've further had some good discussion 
about the application of generative AI tooling to not just code contributed to 
the ASF, but also in dep

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> If we cannot meet at least that quality level (Green CI) we should not merge
We should probably make it a formally agreed upon point to not merge things 
unless we're sure they won't destabilize, and thus block release of, a branch. 
So green CI for a feature (excepting feature-specific tests if it's still a 
work in progress), experimental flag if we don't consider it prod ready, should 
be absolute bare minimum for anything to merge really IMO.

On Wed, Oct 25, 2023, at 4:17 AM, Benjamin Lerer wrote:
> The proposal includes 3 things:
> 1. Do not include TCM and Accord in 5.0 to avoid delaying 5.0
> 2. The next release will be 5.1 and will include only Accord and TCM
> 3. Merge TCM and Accord right now in 5.1 (making an initial release)
> 
> I am fine with question 1 and do not have a strong opinion on which way to go.
> 2. Means that every new feature will have to wait for post 5.1 even if it is 
> ready before 5.1 is stabilized and shipped. If we do a 5.1 release why not 
> take it as an opportunity to release more things. I am not saying that we 
> will. Just that we should let that door open.
> 3. There is a need to merge TCM and Accord as maintaining those separate 
> branches is costly in terms of time and energy. I fully understand that. On 
> the other hand merging TCM and Accord will make the TCM review harder and I 
> do believe that this second round of review is valuable as it already 
> uncovered a valid issue. Nevertheless, I am fine with merging TCM as soon as 
> it passes CI and continuing the review after the merge. If we cannot meet at 
> least that quality level (Green CI) we should not merge just for creating an 
> 5.1.alpha release for the summit.
> 
> Now, I am totally fine with a preview release without numbering and with big 
> warnings that will only serve as a preview for the summit.
> 
> Le mer. 25 oct. 2023 à 06:33, Berenguer Blasi  a 
> écrit :
>> I also think there's many good new features in 5.0 already they'd make a 
>> good release even on their own. My 2 cts.
>> 
>> On 24/10/23 23:20, Brandon Williams wrote:
>> > The catch here is that we don't publish docker images currently.  The
>> > C* docker images available are not made by us.
>> >
>> > Kind Regards,
>> > Brandon
>> >
>> > On Tue, Oct 24, 2023 at 3:31 PM Patrick McFadin  wrote:
>> >> Let me make that really easy. Hell yes
>> >>
>> >> Not everybody runs CCM, I've tried but I've met resistance.
>> >>
>> >> Compiling your own version usually involves me saying the words "Yes, ant 
>> >> realclean exists. I'm not trolling you"
>> >>
>> >> docker pull  works on every OS and curates a single node 
>> >> experience.
>> >>
>> >>
>> >>
>> >> On Tue, Oct 24, 2023 at 12:37 PM Josh McKenzie  
>> >> wrote:
>> >>> In order for the project to advertise the release outside the dev@ list 
>> >>> it needs to be a formal release.
>> >>>
>> >>> That's my reading as well:
>> >>> https://www.apache.org/legal/release-policy.html#release-definition
>> >>>
>> >>> I wonder if there'd be value in us having a cronned job that'd do 
>> >>> nightly docker container builds on trunk + feature branches, archived 
>> >>> for N days, and we make that generally known to the dev@ list here so 
>> >>> folks that want to poke at the current state of trunk or other branches 
>> >>> could do so with very low friction. We'd probably see more engagement on 
>> >>> feature branches if it was turn-key easy for other C* devs to spin the 
>> >>> up and check them out.
>> >>>
>> >>> For what you're talking about here Patrick (a docker image for folks 
>> >>> outside the dev@ audience and more user-facing), we'd want to vote on it 
>> >>> and go through the formal process.
>> >>>
>> >>> On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:
>> >>>
>> >>> In order for the project to advertise the release outside the dev@ list 
>> >>> it needs to be a formal release.  That just means that there was a 
>> >>> release vote and at least 3 PMC members +1’ed it, and there are more +1 
>> >>> than there are -1, and we follow all the normal release rules.  The ASF 
>> >>> release process doesn’t care what branch you cut the artifacts from or 
>> >>> what version you c

Re: [DISCUSS] Allow UPDATE on settings virtual table to change running configuration

[Josh McKenzie]

Is the primary pain point you're trying to solve getting a 2nd committer 
reviewer Maxim? And / or making the review process simpler / cleaner for 
someone?

On Wed, Oct 18, 2023, at 5:06 PM, Maxim Muzafarov wrote:
> Hello everyone,
> 
> It has been a long time since the last update on this thread, so I
> wanted to share some status updates: The issue is still awaiting
> review, but all my hopes are pinned on Benjamin :-)
> 
> My question here is, is there anything I can do to facilitate the
> review for anyone who wants to delve into the patch?
> 
> I have a few thoughts to follow:
> - CEPify the changes - this will allow us to see the result of the
> discussion on a single page without having to re-read the whole
> thread;
> - Write a blog post with possible design solutions - this will both
> reveal the results of the discussion and potentially will draw some
> attention to the community;
> - Presenting and discussing slides at one of the Cassandra Town Halls;
> 
> I tend to the 1-st and/or 2-nd points. What are the best practices we
> have here for such cases though? Any thoughts?
> 
> On Tue, 11 Jul 2023 at 15:51, Maxim Muzafarov  wrote:
> >
> > Thank you for your comments and for sharing the ticket targeting
> > strategy, I'm really happy to see this page where I have found all the
> > answers to the questions I had. So, I tend towards your view and will
> > just land this ticket on the 5.0 release only for now as it makes
> > sense for me as well.
> >
> > I didn't add the feature flag for this feature because for 99% of the
> > source code changes it only works with Cassandra internals leaving the
> > public API unchanged. A few remarks on this are:
> > - the display format of the vtable property has changed to match the
> > yaml configuration style, this doesn't mean that we are displaying
> > property values in a completely different way in fact the formats
> > match with only 4 exceptions mentioned in the message above (this
> > should be fine for the major release I hope);
> > - a new column, which we've agreed to add (I'll fix the PR shortly);
> >
> >
> > I would also like to mention the follow-up todos required by this
> > issue to set the right expectations. Currently, we've brought a few
> > properties under the framework to make them updateable with the
> > SettingsTable, so that you can keep focusing on the framework itself
> > rather than on tagging the configuration properties themselves with
> > the @Mutable annotation. Although the solution is self-sufficient for
> > the already tagged properties, we still need to bring the rest of them
> > under the framework afterwards. I'll create an issue and do it right,
> > we'll be done with the inital patch.
> >
> >
> > On Fri, 7 Jul 2023 at 20:37, Josh McKenzie  wrote:
> > >
> > > This really is great work Maxim; definitely appreciate all the hard work 
> > > that's gone into it and I think the users will too.
> > >
> > > In terms of where it should land, we discussed this type of question at 
> > > length on the ML awhile ago and ended up codifying it in the wiki: 
> > > https://cwiki.apache.org/confluence/display/CASSANDRA/Patching%2C+versioning%2C+and+LTS+releases
> > >
> > > When working on a ticket, use the following guideline to determine which 
> > > branch to apply it to (Note: See How To Commit for details on the commit 
> > > and merge process)
> > >
> > > Bugfix: apply to oldest applicable LTS and merge up through latest GA to 
> > > trunk
> > >
> > > In the event you need to make changes on the merge commit, merge with -s 
> > > ours and revise the commit via --amend
> > >
> > > Improvement: apply to trunk only (next release)
> > >
> > > Note: refactoring and removing dead code qualifies as an Improvement; our 
> > > priority is stability on GA lines
> > >
> > > New Feature: apply to trunk only (next release)
> > >
> > > Our priority is to keep the 2 LTS releases and latest GA stable while 
> > > releasing new "latest GA" on a cadence that provides new improvements and 
> > > functionality to users soon enough to be valuable and relevant.
> > >
> > >
> > > So in this case, target whatever unreleased next feature release (i.e. 
> > > SEMVER MAJOR || MINOR) we have on deck.
> > >
> > > On Thu, Jul 6, 2023, at 1:21 PM, Ekaterina Dimitrova wrote:
> > >
> > > Hi,
> > >
> > > First of all, tha

Re: CASSANDRA-18775 (Cassandra supported OSs)

[Josh McKenzie]

+1 to drop if we're not using.

On Fri, Oct 20, 2023, at 6:58 PM, Ekaterina Dimitrova wrote:
> +1 on removal the whole lib if we are sure we don’t need it. Nothing better 
> than some healthy house cleaning 
> 
>  -1 on partial removals
> 
> On Fri, 20 Oct 2023 at 17:34, David Capwell  wrote:
>> +1 to drop the whole lib… 
>> 
>> 
>>> On Oct 20, 2023, at 7:55 AM, Jeremiah Jordan  
>>> wrote:
>>> 
>>> Agreed.  -1 on selectively removing any of the libs.  But +1 for removing 
>>> the whole thing if it is no longer used.
>>> 
>>> -Jeremiah
>>> 
>>> On Oct 20, 2023 at 9:28:55 AM, Mick Semb Wever  wrote:
> Does anyone see any reason _not_ to do this?
 
 
 Thanks for bring this to dev@
 
 I see reason not to do it, folk do submit patches for other archs despite 
 us not formally maintaining and testing the code for those archs.  Some 
 examples are PPC64 Big Endian (CASSANDRA-7476), s390x (CASSANDRA-17723), 
 PPC64 Little Endian (CASSANDRA-7381), sparcv9 (CASSANDRA-6628).  Wrote 
 this on the ticket too.
 
 +1 for removing sigar altogether (as Brandon points out). 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

 The pressure to ship is only going to increase and that's fertile ground
>>>>> for that sort of bug.
>>>>> 
>>>>> So, taking a step back and with a clearer picture, I support the 5.0 + 5.1
>>>>> plan mainly because I don't think 5.1 is (or should be) a fast follow.
>>>>> 
>>>>> For the user community, the communication should be straightforward. TCM +
>>>>> Accord are turning out to be much more complicated than was originally
>>>>> scoped, and for good reasons. Our first principle is to provide a stable
>>>>> and reliable system, so as a result, we'll be de-coupling TCM + Accord 
>>>>> from
>>>>> 5.0 into a 5.1 branch, which is available in parallel to 5.0 while
>>>>> additional hardening and testing is done. We can communicate this in a 
>>>>> blog
>>>>> post.,
>>>>> 
>>>>> To make this much more palatable to our use community, if we can get a
>>>>> build and docker image available ASAP with Accord, it will allow 
>>>>> developers
>>>>> to start playing with the syntax. Up to this point, that hasn't been 
>>>>> widely
>>>>> available unless you compile the code yourself. Developers need to
>>>>> understand how this will work in an application, and up to this point, the
>>>>> syntax is text they see in my slides. We need to get some hands-on and 
>>>>> that
>>>>> will get our user community engaged on Accord this calendar year. The
>>>>> feedback may even uncover some critical changes we'll need to make. Lack 
>>>>> of
>>>>> access to Accord by developers is a critical problem we can fix soon and
>>>>> there will be plenty of excitement there and start building use cases
>>>>> before the final code ships.
>>>>> 
>>>>> I'm bummed but realistic. It sucks that I won't have a pony for Christmas,
>>>>> but maybe one for my birthday?
>>>>> 
>>>>> Patrick
>>>>> 
>>>>> 
>>>>> 
>>>>> On Tue, Oct 24, 2023 at 7:23 AM Josh McKenzie  
>>>>> wrote:
>>>>> 
>>>>> > Maybe it won't be a glamorous release but shipping
>>>>> > 5.0 mitigates our worst case scenario.
>>>>> >
>>>>> > I disagree with this characterization of 5.0 personally. UCS, SAI, Trie
>>>>> > memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are
>>>>> > accurate, all combine to make 5.0 a pretty glamorous release IMO
>>>>> > independent of TCM and Accord. Accord is a true paradigm-shift 
>>>>> > game-changer
>>>>> > so it's easy to think of 5.0 as uneventful in comparison, and TCM helps
>>>>> > resolve one of the biggest pain-points in our system for over a decade, 
>>>>> > but
>>>>> > I think 5.0 is a very meaty release in its own right today.
>>>>> >
>>>>> > Anyway - I agree with you Brandon re: timelines. If things take longer
>>>>> > than we'd hope (which, if I think back, they do roughly 100% of the 
>>>>> > time on
>>>>> > this project), blocking on these features could both lead to a 
>>>>> > significant
>>>>> > delay in 5.0 going out as well as increasing pressure and risk of 
>>>>> > burnout
>>>>> > on the folks working on it. While I believe we all need some balanced
>>>>> > urgency to do our best work, being under the gun for something with a 
>>>>> > hard
>>>>> > deadline or having an entire project drag along blocked on you is not 
>>>>> > where
>>>>> > I want any of us to be.
>>>>> >
>>>>> > Part of why we talked about going to primarily annual calendar-based
>>>>> > releases was to avoid precisely this situation, where something that
>>>>> > *feels* right at the cusp of merging leads us to delay a release
>>>>> > repeatedly. We discussed this a couple times this year:
>>>>> > 1: https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3,
>>>>> > where Mick proposed a "soft-freeze" for everything w/out exception and 
>>>>> > 1st
>>>>> >

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
I disagree with this characterization of 5.0 personally. UCS, SAI, Trie 
memtables and sstables, maybe vector ANN if the sub-tasks on C-18715 are 
accurate, all combine to make 5.0 a pretty glamorous release IMO independent of 
TCM and Accord. Accord is a true paradigm-shift game-changer so it's easy to 
think of 5.0 as uneventful in comparison, and TCM helps resolve one of the 
biggest pain-points in our system for over a decade, but I think 5.0 is a very 
meaty release in its own right today.

Anyway - I agree with you Brandon re: timelines. If things take longer than 
we'd hope (which, if I think back, they do roughly 100% of the time on this 
project), blocking on these features could both lead to a significant delay in 
5.0 going out as well as increasing pressure and risk of burnout on the folks 
working on it. While I believe we all need some balanced urgency to do our best 
work, being under the gun for something with a hard deadline or having an 
entire project drag along blocked on you is not where I want any of us to be.

Part of why we talked about going to primarily annual calendar-based releases 
was to avoid precisely this situation, where something that *feels* right at 
the cusp of merging leads us to delay a release repeatedly. We discussed this a 
couple times this year:
1: https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3, where Mick 
proposed a "soft-freeze" for everything w/out exception and 1st week October 
"hard-freeze", and there was assumed to be lazy consensus
2: https://lists.apache.org/thread/mzj3dq8b7mzf60k6mkby88b9n9ywmsgw, where we 
kept along with what we discussed in 1 but added in CEP-30 to be waivered in as 
well.

So. We're at a crossroads here where we need to either follow through with what 
we all agreed to earlier this year, or acknowledge that our best intention of 
calendar-based releases can't stand up to our optimism and desire to get these 
features into the next major.

There's no immediate obvious better path to me in terms of what's best for our 
users. This is a situation of risk tolerance with a lot of unknowns that could 
go either way.

Any light that folks active on TCM and Accord could shed in terms of their best 
and worst-case scenarios on timelines for those features might help us narrow 
this down a bit. Otherwise, I'm inclined to defer to our past selves and fall 
back to "we agreed to yearly calendar releases for good reason. Let's stick to 
our guns."

On Tue, Oct 24, 2023, at 9:37 AM, Brandon Williams wrote:
> The concern I have with this is that is a big slippery 'if' that
> involves development time estimation, and if it tends to take longer
> than we estimate (as these things tend to do), then I can see a future
> where 5.0 is delayed until the middle of 2024, and I really don't want
> that to happen.  Maybe it won't be a glamorous release but shipping
> 5.0 mitigates our worst case scenario.
> 
> Kind Regards,
> Brandon
> 
> On Mon, Oct 23, 2023 at 4:02 PM Dinesh Joshi  wrote:
> >
> > I have a strong preference to move out the 5.0 date to have accord and TCM. 
> > I don’t see the point in shipping 5.0 without these features especially if 
> > 5.1 is going to follow close behind it.
> >
> > Dinesh
> >
> > On Oct 23, 2023, at 4:52 AM, Mick Semb Wever  wrote:
> >
> > 
> >
> > The TCM work (CEP-21) is in its review stage but being well past our 
> > cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would 
> > like to propose the following.
> >
> > We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and cut 
> > an immediate 5.1-alpha1 release.
> >
> > I see this as a win-win scenario for us, considering our current situation. 
> >  (Though it is unfortunate that Accord is included in this scenario because 
> > we agreed it to be based upon TCM.)
> >
> > This will mean…
> >  - We get to focus on getting 5.0 to beta and GA, which already has a ton 
> > of features users want.
> >  - We get an alpha release with TCM and Accord into users hands quickly for 
> > broader testing and feedback.
> >  - We isolate GA efforts on TCM and Accord – giving oss and downstream 
> > engineers time and patience reviewing and testing.  TCM will be the biggest 
> > patch ever to land in C*.
> >  - Give users a choice for a more incremental upgrade approach, given just 
> > how many new features we're putting on them in one year.
> >  - 5.1 w/ TCM and Accord will maintain its upgrade compatibility with all 
> > 4.x versions, just as if it had landed in 5.0.
> >
> >
> > The risks/costs this introduces are
> >  - If we cannot stabilise TCM and/or Accord on the cassandra-5.1 branch, 
> > and at some point decide to undo this work, while we can throw away the 
> > cassandra-5.1 branch we would need to do a bit of work reverting the 
> > changes in trunk.  This is a _very_ edge case, as confidence levels on the 
> > design and imple

Re: CASSANDRA-18941 produce size bounded SSTables from CQLSSTableWriter

[Josh McKenzie]

> to 4.0 and up to trunk
Think the proposal is all supported branches from 4.0 up.

+1 here.

On Mon, Oct 23, 2023, at 10:19 PM, guo Maxwell wrote:
> +1, but I want to know why only trunk and 4.0 ? not all the versions 
> involved, like 4.1 ,5.0 。
> 
> Francisco Guerrero  于2023年10月24日周二 07:47写道：
>> +1 (nb). I think this is a great addition to offline tools that use SSTable 
>> writer in general.
>> 
>> On 2023/10/23 23:21:13 Yifan Cai wrote:
>> > Hi,
>> > 
>> > I want to propose merging the patch in CASSANDRA-18941 to 4.0 and up to
>> > trunk and hope we are all OK with it.
>> > 
>> > In CASSANDRA-18941, I am adding the capability to produce size-bounded
>> > SSTables in CQLSSTableWriter for sorted data. It can greatly benefit
>> > Cassandra Analytics (https://github.com/apache/cassandra-analytics) for
>> > bulk writing SSTables, since it avoids buffering and sorting on flush,
>> > given the data source is sorted already in the bulk write process.
>> > Cassandra Analytics supports Cassandra 4.0 and depends on the cassandra-all
>> > 4.0.x library. Therefore, we are mostly interested in using the new
>> > capability in 4.0.
>> > 
>> > CQLSSTableWriter is only used in offline tools and never in the code path
>> > of Cassandra server.
>> > 
>> > Any objections to merging the patch to 4.0 and up to trunk?
>> > 
>> > - Yifan
>> >

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> If I had to pick a month of the year to release software used by large 
> enterprises, it probably would be something like March instead of December.
That's... a good point. If we end up on a cadence of major's in December (since 
we slipped to then for 4.1 and inherit that from that calendar year "pressure") 
we're setting ourselves up to release right in the largest consistent 
change-freeze window I know of for most users.

> It will be another 2.2 release.
Let me live on with the stories I tell myself about the hordes of Windows users 
that appreciated Windows support before the Storage Engine rewrite, thank you 
very much. :D

On Mon, Oct 23, 2023, at 1:57 PM, Caleb Rackliffe wrote:
> ...or like the end of January. Either way, feel free to ignore the "aside" :)
> 
> On Mon, Oct 23, 2023 at 12:53 PM Caleb Rackliffe  
> wrote:
>> Kind of in the same place as Benedict/Aleksey.
>> 
>> If we release a 5.1 in, let's say...March of next year, the number of 5.0 
>> users is going to be very minimal. Nobody is going to upgrade anything 
>> important from now through the first half of January anyway, right? They're 
>> going to be making sure their existing clusters aren't exploding.
>> 
>> (We still want TCM/Accord to be available to people to test by Summit, but 
>> that feels unrelated to whether we cut a 5.1 branch...)
>> 
>> Aside: If I had to pick a month of the year to release software used by 
>> large enterprises, it probably would be something like March instead of 
>> December. I have no good research to back that up, of course... 
>> 
>> On Mon, Oct 23, 2023 at 12:19 PM Benedict  wrote:
>>> 
>>> To be clear, I’m not making an argument either way about the path forwards 
>>> we should take, just concurring about a likely downside of this proposal. I 
>>> don’t have a strong opinion about how we should proceed.
>>> 
>>> 
>>>> On 23 Oct 2023, at 18:16, Benedict  wrote:
>>>> 
>>>> 
>>>> I agree. If we go this route we should essentially announce an immediate 
>>>> 5.1 alpha at the same time as 5.0 GA, and I can’t see almost anybody 
>>>> rolling out 5.0 with 5.1 so close on its heels.
>>>> 
>>>> 
>>>>> On 23 Oct 2023, at 18:11, Aleksey Yeshchenko  wrote:
>>>>> I’m not so sure that many folks will choose to go 4.0->5.0->5.1 path 
>>>>> instead of just waiting longer for TCM+Accord to be in, and go 4.0->5.1 
>>>>> in one hop.
>>>>> 
>>>>> Nobody likes going through these upgrades. So I personally expect 5.0 to 
>>>>> be a largely ghost release if we go this route, adopted by few, just a 
>>>>> permanent burden on the merge path to trunk.
>>>>> 
>>>>> Not to say that there isn’t valuable stuff in 5.0 without TCM and Accord 
>>>>> - there most certainly is - but with the expectation that 5.1 will follow 
>>>>> up reasonably shortly after with all that *and* two highly anticipated 
>>>>> features on top, I just don’t see the point. It will be another 2.2 
>>>>> release.
>>>>> 
>>>>> 
>>>>>> On 23 Oct 2023, at 17:43, Josh McKenzie  wrote:
>>>>>> 
>>>>>> We discussed that at length in various other mailing threads Jeff - kind 
>>>>>> of settled on "we're committing to cutting a major (semver MAJOR or 
>>>>>> MINOR) every 12 months but want to remain flexible for exceptions when 
>>>>>> appropriate".
>>>>>> 
>>>>>> And then we discussed our timeline for 5.0 this year and settled on the 
>>>>>> "let's try and get it out this calendar year so it's 12 months after 
>>>>>> 4.1, but we'll grandfather in TCM and Accord past freeze date if they 
>>>>>> can make it by October".
>>>>>> 
>>>>>> So that's the history for how we landed here.
>>>>>> 
>>>>>>> 2) Do we drop the support of 3.0 and 3.11 after 5.0.0 is out or after 
>>>>>>> 5.1.0 is?
>>>>>> This is my understanding, yes. Deprecation and support drop is 
>>>>>> predicated on the 5.0 release, not any specific features or anything.
>>>>>> 
>>>>>> On Mon, Oct 23, 2023, at 12:29 PM, Jeff Jirsa wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Mon, Oct 23, 2023 at 4:52 AM Mick Semb Wever  wrote:
>>>>>>>> 
>>>>>>>> The TCM work (CEP-21) is in its review stage but being well past our 
>>>>>>>> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I 
>>>>>>>> would like to propose the following.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> I think this presumes that 5.0 GA is date driven instead of feature 
>>>>>>> driven.
>>>>>>> 
>>>>>>> I'm sure there's a conversation elsewhere, but why isn't this date 
>>>>>>> movable?

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

We discussed that at length in various other mailing threads Jeff - kind of 
settled on "we're committing to cutting a major (semver MAJOR or MINOR) every 
12 months but want to remain flexible for exceptions when appropriate".

And then we discussed our timeline for 5.0 this year and settled on the "let's 
try and get it out this calendar year so it's 12 months after 4.1, but we'll 
grandfather in TCM and Accord past freeze date if they can make it by October".

So that's the history for how we landed here.

> 2) Do we drop the support of 3.0 and 3.11 after 5.0.0 is out or after 5.1.0 
> is?
This is my understanding, yes. Deprecation and support drop is predicated on 
the 5.0 release, not any specific features or anything.

On Mon, Oct 23, 2023, at 12:29 PM, Jeff Jirsa wrote:
> 
> 
> On Mon, Oct 23, 2023 at 4:52 AM Mick Semb Wever  wrote:
>> 
>> The TCM work (CEP-21) is in its review stage but being well past our cut-off 
>> date¹ for merging, and now jeopardising 5.0 GA efforts, I would like to 
>> propose the following.
>> 
> 
> 
> I think this presumes that 5.0 GA is date driven instead of feature driven.
> 
> I'm sure there's a conversation elsewhere, but why isn't this date movable?
> 
>  

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Josh McKenzie]

> This has to be in 5.0, even if it’s alpha and ships after December, or this 
> is going to be disaster that will take us much longer to unravel. 
I'm curious to hear more about this.

> What is it going to take to get it into 5.0? What is off track and how did we 
> get here?
I'm going to crystal-ball a combination of "we're in mythical man-month 
territory" and "we're doing something that's never been done before on a 
code-base that's a decade and a half old. In a distributed system. That takes 
(often unpredictable) time."

I'm +1 to what you've laid out here Mick. The idea of having another branch to 
merge through makes me sad, but it's worth it to both get 5.0 into users' hands 
around our committed cadence + also get an alpha of these new features into 
their hands as well IMO.

On Mon, Oct 23, 2023, at 11:14 AM, Paulo Motta wrote:
> From a user perspective I have to say I was excited to see Accord/TCM being 
> released on 5.0 but at the same time a little nervous about seeing so many 
> overhauling features being shipped on the same release.
> 
> I think rushing last minute features hurts the stability goals we set for the 
> project. As far as I understand, we have agreed to have a "release train" 
> model where everything ready by the release date is shipped and anything else 
> slips to the next version.
> 
> 5.0 will bring a number of exciting innovations and I don't think not 
> including TCM/Accord can be considered a disaster. I think letting the 
> community test the currently shipped features separately from TCM/Accord will 
> be a benefit from a stability perspective without hurting the project 
> momentum.
> 
> I think TCM/Accord are such major and long awaited improvements to the 
> project that deserve its own exclusive release, otherwise they can easily 
> shadow the other exciting features being shipped. I don't see any issue in 
> performing an earlier release next year if TCM/Accord is ready by then.
> 
> Regarding the versioning scheme, if we follow the versioning scheme we have 
> defined "by the book" then TCM/Accord would belong to a 6.0 version, which I 
> have to admit feels a bit weird but it would signal to the user community 
> that a major change is being introduced. I don't feel strongly about this so 
> would be fine with a 5.1 even though it would be a departure from the new 
> versioning scheme we have agreed upon.
> 
> On Mon, Oct 23, 2023 at 10:55 AM Patrick McFadin  wrote:
>> I’m going to be clearer in my statement. 
>> 
>> This has to be in 5.0, even if it’s alpha and ships after December, or this 
>> is going to be disaster that will take us much longer to unravel. 
>> 
>> On Mon, Oct 23, 2023 at 7:49 AM Jeremiah Jordan  
>> wrote:
>>> +1 from me assuming we have tickets and two committer +1’s on them for 
>>> everything being committed to trunk, and CI is working/passing before it 
>>> merges.  The usual things, but I want to make sure we do not compromise on 
>>> any of them as we try to “move fast” here.
>>> 
>>> -Jeremiah Jordan
>>> 
>>> On Oct 23, 2023 at 8:50:46 AM, Sam Tunnicliffe  wrote:
 
 +1 from me too. 
 
 Regarding Benedict's point, backwards incompatibility should be minimal; 
 we modified snitch behaviour slightly, so that local snitch config only 
 relates to the local node, all peer info is fetched from cluster metadata. 
 There is also a minor change to the way failed bootstraps are handled, as 
 with TCM they require an explicit cancellation step (running a nodetool 
 command). 
 
 Whether consensus decrees that this constitutes a major bump or not, I 
 think decoupling these major projects from 5.0 is the right move. 
  
 
> On 23 Oct 2023, at 12:57, Benedict  wrote:
> 
> 
> I’m cool with this.
> 
> We may have to think about numbering as I think TCM will break some 
> backwards compatibility and we might technically expect the follow-up 
> release to be 6.0
> 
> Maybe it’s not so bad to have such rapid releases either way.
> 
> 
>> On 23 Oct 2023, at 12:52, Mick Semb Wever  wrote:
>> 
>> 
>> The TCM work (CEP-21) is in its review stage but being well past our 
>> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would 
>> like to propose the following.
>> 
>> We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and 
>> cut an immediate 5.1-alpha1 release.
>> 
>> I see this as a win-win scenario for us, considering our current 
>> situation.  (Though it is unfortunate that Accord is included in this 
>> scenario because we agreed it to be based upon TCM.)
>> 
>> This will mean…
>>  - We get to focus on getting 5.0 to beta and GA, which already has a 
>> ton of features users want.
>>  - We get an alpha release with TCM and Accord into users hands quickly 
>> for broader testing and feedback.
>>  - We isolate GA efforts on TCM and A