This is not a small change; true, hence the CEP which I think we should focus the discussion on. I think complexity is pretty minimal compared to say accord or tcm.
In fact in 4.1 where we have this, its very little changes because it always used the index on reads then. The newer features are what makes it complicated (bti/sai etc which could just disable this feature). The cassandra-analytics reader should require only minor updates, isolated in a new version-specific bridge module rather than a redesign. The on-disk format remains largely unchanged; the new SSTable version primarily acts as a compatibility marker. A bridge compiled against that release’s cassandra-all JAR will automatically inherit the necessary Descriptor, Version, and StatsMetadata support. Worth asking whether it's even worth supporting BTI and indexes in this first pass, versus going straight for BIG-format anticompaction and enabling zero-copy streaming across the board. Would drastically simplify it. Chris On Tue, Sep 15, 2026 at 7:12 AM Benedict Elliott Smith <[email protected]> wrote: > Let's just say your patch is "not small" (and the CEP will include several > more "not small" items of work), and that the primary factor is whether any > production code changes have been predominantly produced by a human > contributor or not. > > The project has debated AI usage from a legal point of view, but we need > to discuss it from a community perspective, now we all have sufficient > experience with the technology to have an informed discussion. > > I don't think any of us know for sure what the outcome of that discussion > will be. > > > On 2026/09/15 11:53:36 Chris Lohfink wrote: > > https://github.com/apache/cassandra/pull/5063 Isn't that large? > Especially > > compared to nearly 40k changes like > > https://github.com/apache/cassandra/pull/4967 that dont even have a JIRA > > attached > > > > Long term goal is sub second splitting something that takes up to an hour > > of heavy CPU currently with zero disk IO seems valuable regardless. > Making > > things like STCS, TWCS, and IR actually work (TWCS/STCS will not work > with > > IR) > > > > Chris > > > > > > On Tue, Sep 15, 2026 at 5:23 AM Benedict Elliott Smith < > [email protected]> > > wrote: > > > > > I think we need to settle a meta discussion first. We have had a > > > significant uptick in patches that are predominantly AI generated > recently, > > > and especially for large patches like this, we need to agree a > > > project-level policy before any more are merged. I will send a separate > > > email in a few days with my thoughts to kick things off. I'm sure the > > > discussion will be… productive, but it is overdue. We all have enough > > > experience with the technology now, and no doubt have sufficiently > > > diverging viewpoints that for the project to function we must agree on > a > > > shared understanding of what is expected and appropriate. > > > > > > Independent of this concern, regarding this patch I am in favour of > "zero > > > copy" range streaming, but I am unclear whether the complexity of > linking > > > file sub ranges is justified. > > > > > > > > > On 2026/09/14 19:22:43 Chris Lohfink wrote: > > > > Hi everyone, > > > > > > > > I'd like to open CEP-66, Zero-copy SSTable splitting, for discussion: > > > > > > > > * > > > > https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/451972773/draft+CEP-66+Zero-copy+SSTable+splitting > > > > < > > > > https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/451972773/draft+CEP-66+Zero-copy+SSTable+splitting > > > > > > > > * > > > > > > > > Anticompaction and partial-range streaming currently rewrite rows > whose > > > > encoded representation already exists on disk. This consumes CPU, > creates > > > > substantial heap churn and write amplification, and increases > temporary > > > > disk pressure. > > > > > > > > CEP-66 proposes splitting eligible compressed SSTables by retaining > > > > contiguous runs of their existing compression chunks. Cassandra would > > > > rebuild the child SSTables' indexes and other derived components > without > > > > deserializing, serializing, or recompressing their rows. > > > > > > > > "Zero-copy" here primarily means reusing the encoded bytes instead of > > > > rewriting rows. On filesystems that support range reflinks, > Cassandra can > > > > also share the underlying extents meaning no new data written. Other > > > > filesystems, including ext4, would copy the already-compressed bytes > and > > > > still avoid the row rewrite. > > > > > > > > The proposal is staged. It starts with an opt-in `sstablesplit > > > --zero-copy` > > > > mode for BIG-format SSTables in Cassandra 7.0/trunk. Later phases > add BTI > > > > support, secondary indexes, anticompaction, and partial-range > streaming. > > > > Existing implementations remain the default and provide the fallback > for > > > > unsupported inputs. > > > > > > > > I'd particularly appreciate feedback on: > > > > > > > > - The retained-prefix representation and proposed Cassandra 7.0 > SSTable > > > > format change > > > > - Rebuilding or conservatively deriving child metadata without > decoding > > > rows > > > > - The integrity and performance tradeoff around `Digest.crc32` > generation > > > > - The staged rollout, compatibility rules, and fallback behavior > > > > - Any correctness, operational, or filesystem concerns the proposal > has > > > > missed > > > > > > > > Thanks, and I look forward to the discussion. > > > > > > > > Regards, > > > > Chris Lohfink > > > > > > > > > >
