The two examples that spawned this thread, JDK 21 support and auto-repair, seem like exactly the kind of things we would want to (and safely can) port to mainline cassandra-5.0 to provide a carrot for operators on 4.1 to at least get closer to (if not actually on) trunk. The benefits of doing that (with care) seem like they justify having a fallback plan of "downgrade to the previous 5.0.x patch release" if something goes wrong.
On Sun, Oct 12, 2025 at 2:13 PM Jaydeep Chovatia <[email protected]> wrote: > Here is my opinion. > > >– Unofficial branches will miss correctness and compatibility fixes > unless their maintenance is made a burden for all committers. If not, > they’ll be buggier and more prone to data loss than trunk. > > Typically, the backporting effort is handled by the author or co-author of > a given CEP. As long as they are motivated to pursue the backport, I don’t > anticipate this being a concern. In most cases, their motivation naturally > comes from the fact that they are themselves relying on or benefiting from > the backported version. > > >– The upgrade matrix becomes more complicated. As features are > backported, any change affecting internode messaging, config properties, > etc. becomes a potential compatibility breakage on upgrade, and these > upgrade paths will be untested and unexercised. > >– There’s an assumption in this thread that backports are easy to pick > up. Backporting is often not straightforward and requires a high degree of > understanding of the surrounding context, integration points, and what’s > changed across branches. > > As discussed earlier, we should conduct a formal vote on any proposed > backports and exercise caution with those that alter internal communication > mechanisms, Gossip protocols, or introduce backward incompatibilities. > Backports should meet a higher threshold—either by addressing fundamental > gaps in the database framework or by delivering substantial > reliability/efficiency improvements. For instance, CEP-37 and JDK 17/21 are > strong candidates for backporting: the former is essential to maintaining > data correctness in Cassandra, while the latter has become necessary as > much of the industry has already transitioned beyond JDK 11. > > >– The proposal runs counter to the goal of “people running the database > and finding + fixing issues.” I happily run trunk, but I don’t want to be > the only one running trunk if others are committing changes to it. > Committing changes to trunk then backporting them to releases considered > “stable” doesn’t produce a more stable database. > >– Backports branches don’t solve the “some people run forks” problem. > >– Increasing the user community’s confidence in running new releases of > the database. A lot of people are reluctant to upgrade, but it’s so much > safer and easier than since the 2.x/3.x days. We want people to be > confident running new releases of the database, not clinging to a branch. > >– Deploying trunk and reporting back. Contributors of new features they’d > like to backport should deploy and operate trunk. It’s the best way to > establish confidence and makes Cassandra better for everybody. > > I must acknowledge that the upgrade process has come a long way since the > 2.x and 3.x versions, but there’s still room for improvement. For instance, > when upgrading to 5.1, a lack of a straightforward rollback path can make > the process risky. This limitation often slows modernization efforts, as > teams are understandably hesitant to proceed without a reliable fallback. > Many businesses around the world run critical workloads on Cassandra, and > an outage caused by an upgrade would ultimately fall on the decision > makers—making them cautious about taking such risks. > This concern is precisely why many decision makers prefer to backport > features (such as CEP-37, JDK 17/21) and operate on private forks rather > than upgrade to 5.1. This proposal aims to make their lives easier by > providing an official and coordinated path for backporting, rather than > leaving each operator to maintain their own fork. For example, support for > JDK 17 or 21 on version 4.1 is already a widespread need among operators. > We should certainly begin a new discussion on how to make our upgrade/new > versions process safer, so that, in the long run, the need for backporting > and similar discussions is eliminated. > > >– Increasing release velocity. We do need to improve here and I’d be open > to 5.1. > > I am not sure that’s the case. For most decision makers, the primary > concern isn’t velocity but safety. The key question they ask themselves is, > ‘What is my fallback plan?’ If that plan appears uncertain or risky, they > are understandably hesitant to proceed with an upgrade. > > > Jaydeep > > On Sun, Oct 12, 2025 at 9:04 AM <[email protected]> wrote: > >> I don’t think we have consensus on this thread, but it feels like some >> are pushing forward as if we do (“If everybody is generally onboard with >> the proposal, we can start getting into the details of the logistics…,” >> followed by discussion of logistics). >> >> The thread also contains multiple different proposals: new feature >> backports branches, liberalizing feature backports to stable releases, >> cutting 5.1 now, or stay the course. >> >> I don’t support creation of new backports branches, but will keep my >> thoughts brief since there’s a lot of discussion: >> >> – The CI burden of existing branches is really high. Either new branches >> are treated as first-class and impose stability burdens on committers, or >> they fall into disrepair and are unsuitable for releases. Release >> engineering for a branch is nearly a full-time job. >> – Unofficial branches will miss correctness and compatibility fixes >> unless their maintenance is made a burden for all committers. If not, >> they’ll be buggier and more prone to data loss than trunk. >> – The upgrade matrix becomes more complicated. As features are >> backported, any change affecting internode messaging, config properties, >> etc. becomes a potential compatibility breakage on upgrade, and these >> upgrade paths will be untested and unexercised. >> – There’s an assumption in this thread that backports are easy to pick >> up. Backporting is often not straightforward and requires a high degree of >> understanding of the surrounding context, integration points, and what’s >> changed across branches. >> – The proposal runs counter to the goal of “people running the database >> and finding + fixing issues.” I happily run trunk, but I don’t want to be >> the only one running trunk if others are committing changes to it. >> Committing changes to trunk then backporting them to releases considered >> “stable” doesn’t produce a more stable database. >> – Pitching this as a limited-time pilot doesn't these problems, and it >> introduces new ones. The user community would fragment across these >> branches and have to be reconverged despite untested upgrade paths if the >> pilot were wound down. >> – Backports branches don’t solve the “some people run forks” problem. >> – Backports branches don’t solve the user community adoption problem >> either, unless we’re also publishing per-OS packages, Maven artifacts, etc. >> >> For me, the proposal wouldn't achieve its stated goal and introduces many >> new issues. But I do strongly support that goal. >> >> Toward bringing stable features into the user community’s hands more >> quickly, the fix for this seems like: >> >> – Increasing the user community’s confidence in running new releases of >> the database. A lot of people are reluctant to upgrade, but it’s so much >> safer and easier than since the 2.x/3.x days. We want people to be >> confident running new releases of the database, not clinging to a branch. >> – Deploying trunk and reporting back. Contributors of new features they’d >> like to backport should deploy and operate trunk. It’s the best way to >> establish confidence and makes Cassandra better for everybody. >> – Increasing release velocity. We do need to improve here and I’d be open >> to 5.1. >> – Exercising our existing consensus-based approach of backporting stable >> and well-contained enhancements to earlier branches following discussion on >> the mailing list. We could do this a little more often. >> >> – Scott >> >> On Oct 12, 2025, at 8:28 AM, Chris Lohfink <[email protected]> wrote: >> >> But it should include all features from trunk that we consider to be >>> production ready (that includes TCM in my book) >> >> >> Please no TCM/accord. That is why everyone will be on 5.0/5.1 for years. >> I'll be the person to say it outloud. I'm happy to be proven wrong but >> let's be realistic. >> >> Chris >> >> >>
