[
https://issues.apache.org/jira/browse/CASSANDRA-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lohfink updated CASSANDRA-21550:
--------------------------------------
Description:
## Context
This is phase 1 of
[CASSANDRA-21550](https://issues.apache.org/jira/browse/CASSANDRA-21550).
Anticompaction and range streaming currently redo work whose result already
exists on disk. They deserialize, serialize, and recompress rows even when the
required data is a contiguous run of the parent SSTable’s compression chunks.
CASSANDRA-21550 introduces machinery to retain those compressed chunks
verbatim—using extent sharing where available—and rebuild the remaining
components from index-only passes.
## Phase 1 scope
This PR deliberately limits the first phase to compressed BIG-format SSTables
and exposes the machinery through the opt-in:
```text
sstablesplit --zero-copy
```
Starting with `sstablesplit` provides a bounded, offline caller for validating
the new on-disk representation and splitting machinery before integrating it
into anticompaction and streaming.
This phase:
- Supports compressed BIG `pa`-or-later SSTables and writes `pb` children.
- Copies existing compression chunks without deserializing or recompressing
rows.
- Uses byte-range reflinks on supported Linux filesystems to share extents.
- Falls back to copying compressed bytes when reflinks are unavailable.
- Rebuilds indexes, summaries, filters, statistics, digests, and other child
components.
- Persists the first indexed partition position so readers can safely skip
retained compression-chunk prefixes.
- Updates loading, scanning, streaming, scrub, verification, and upgrade paths
to understand split-prefix SSTables.
- Requires `storage_compatibility_mode: NONE`.
- Rejects uncompressed SSTables, unsupported components, and SSTable-attached
indexes.
The requested split size remains a maximum unless one partition’s
compression-chunk span exceeds it. Compression chunks crossing a split boundary
may be retained by both adjacent children.
## Follow-up phases
The intended sequence for CASSANDRA-21550 is:
1. **BIG zero-copy splitting** — this PR.
2. Add the equivalent splitting implementation for **BTI**.
3. Add support for SSTables with **SAI and legacy secondary indexes (2i)**.
4. Use the machinery for **zero-copy anticompaction**, initially disabled by
default.
5. Use it to make **range streaming zero-copy**, also initially disabled by
default.
Anticompaction and general range streaming integration are intentionally
outside this PR.
## Initial Commits
1.
[ce328da5550a129d1d5d8377afd16da458129514](https://github.com/clohfink/cassandra/commit/ce328da5550a129d1d5d8377afd16da458129514)
— **Add zero-copy splitting for compressed BIG SSTables**
Adds the production implementation and documentation, including chunk
selection, index and metadata rebuilding, reflink support, split-prefix
metadata, and updates to affected SSTable consumers.
2.
[c07e4931783181e87054484b7b51626b21f6300b](https://github.com/clohfink/cassandra/commit/c07e4931783181e87054484b7b51626b21f6300b)
— **Add tests for zero-copy SSTable splitting**
Adds test coverage for splitting and sizing, reflinks and copy fallback,
metadata and version compatibility, corruption handling, streaming, scrub,
verification, upgrades, memory-mapped regions, and the command-line tool.
was:Anticompaction and range streaming both spend most of their cost re-doing
work whose result they already have on disk. Anticompaction rewrites every row
of an sstable through three writers only to place each partition into a
repaired, transient or unrepaired child. Streaming a subset of an sstable's
ranges takes the row-by-row path, where the sender is cheap (whole compression
chunks are sent verbatim) but the receiver decompresses, deserialises,
re-serialises and recompresses every row and then rebuilds the index, filter
and summary it could have been handed. Both are avoidable when the bytes wanted
are a contiguous run of the parent's compression chunks: the chunks can be
copied (or extent-shared) verbatim and every other component rebuilt from an
index-only pass, with no row deserialised on either side. This adds that
machinery and the two callers for it. Both are off by default.
> Zero-copy sstable splitting for anticompaction, and partial-sstable zero-copy
> streaming
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21550
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21550
> Project: Apache Cassandra
> Issue Type: New Feature
> Reporter: Chris Lohfink
> Priority: Normal
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> ## Context
> This is phase 1 of
> [CASSANDRA-21550](https://issues.apache.org/jira/browse/CASSANDRA-21550).
> Anticompaction and range streaming currently redo work whose result already
> exists on disk. They deserialize, serialize, and recompress rows even when
> the required data is a contiguous run of the parent SSTable’s compression
> chunks.
> CASSANDRA-21550 introduces machinery to retain those compressed chunks
> verbatim—using extent sharing where available—and rebuild the remaining
> components from index-only passes.
> ## Phase 1 scope
> This PR deliberately limits the first phase to compressed BIG-format SSTables
> and exposes the machinery through the opt-in:
> ```text
> sstablesplit --zero-copy
> ```
> Starting with `sstablesplit` provides a bounded, offline caller for
> validating the new on-disk representation and splitting machinery before
> integrating it into anticompaction and streaming.
> This phase:
> - Supports compressed BIG `pa`-or-later SSTables and writes `pb` children.
> - Copies existing compression chunks without deserializing or recompressing
> rows.
> - Uses byte-range reflinks on supported Linux filesystems to share extents.
> - Falls back to copying compressed bytes when reflinks are unavailable.
> - Rebuilds indexes, summaries, filters, statistics, digests, and other child
> components.
> - Persists the first indexed partition position so readers can safely skip
> retained compression-chunk prefixes.
> - Updates loading, scanning, streaming, scrub, verification, and upgrade
> paths to understand split-prefix SSTables.
> - Requires `storage_compatibility_mode: NONE`.
> - Rejects uncompressed SSTables, unsupported components, and SSTable-attached
> indexes.
> The requested split size remains a maximum unless one partition’s
> compression-chunk span exceeds it. Compression chunks crossing a split
> boundary may be retained by both adjacent children.
> ## Follow-up phases
> The intended sequence for CASSANDRA-21550 is:
> 1. **BIG zero-copy splitting** — this PR.
> 2. Add the equivalent splitting implementation for **BTI**.
> 3. Add support for SSTables with **SAI and legacy secondary indexes (2i)**.
> 4. Use the machinery for **zero-copy anticompaction**, initially disabled by
> default.
> 5. Use it to make **range streaming zero-copy**, also initially disabled by
> default.
> Anticompaction and general range streaming integration are intentionally
> outside this PR.
> ## Initial Commits
> 1.
> [ce328da5550a129d1d5d8377afd16da458129514](https://github.com/clohfink/cassandra/commit/ce328da5550a129d1d5d8377afd16da458129514)
> — **Add zero-copy splitting for compressed BIG SSTables**
> Adds the production implementation and documentation, including chunk
> selection, index and metadata rebuilding, reflink support, split-prefix
> metadata, and updates to affected SSTable consumers.
> 2.
> [c07e4931783181e87054484b7b51626b21f6300b](https://github.com/clohfink/cassandra/commit/c07e4931783181e87054484b7b51626b21f6300b)
> — **Add tests for zero-copy SSTable splitting**
> Adds test coverage for splitting and sizing, reflinks and copy fallback,
> metadata and version compatibility, corruption handling, streaming, scrub,
> verification, upgrades, memory-mapped regions, and the command-line tool.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
