[
https://issues.apache.org/jira/browse/CASSANDRA-21550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Lohfink updated CASSANDRA-21550:
--------------------------------------
Description:
h2. Motivation
Anticompaction and range streaming both spend most of their cost repeating work
whose result already exists on disk.
Anticompaction rewrites every row of an SSTable through multiple writers only
to place each partition into a repaired, transient, or unrepaired child.
When streaming a subset of an SSTable's ranges, the sender can transmit whole
compression chunks verbatim, but the receiver still decompresses, deserializes,
reserializes, and recompresses every row. It then rebuilds the index, filter,
summary, and metadata even though these components can be derived without
rewriting the row data.
This work is avoidable when the required bytes form contiguous runs of the
parent's compression chunks.
h2. Proposed approach
Introduce reusable zero-copy SSTable-splitting machinery that:
* Selects contiguous runs of compression chunks for each child SSTable.
* Retains the compressed bytes verbatim without deserializing or recompressing
rows.
* Uses byte-range reflinks to share filesystem extents when supported.
* Falls back to copying the compressed bytes when reflinks are unavailable.
* Rebuilds the index, filter, summary, statistics, and other components from
index-only passes.
* Represents compression-chunk prefixes that do not contain indexed partitions,
allowing boundary chunks to be retained safely.
* Preserves the validation, corruption detection, lifecycle, disk-space, and
compatibility guarantees expected by existing SSTable operations.
The same machinery can then be used by anticompaction and range streaming
instead of their existing row-by-row rewrite paths.
h2. Delivery plan
The work will be delivered incrementally so the on-disk representation and
splitting primitives can be validated before they are introduced into critical
online paths.
h3. Phase 1: BIG-format splitting
Add zero-copy splitting for compressed BIG-format SSTables and expose it
through an opt-in {{sstablesplit --zero-copy}} mode.
Using the offline splitter as the first caller provides a bounded way to
validate chunk selection, component rebuilding, extent sharing, compatibility,
verification, scrub, streaming, loading, and upgrade behavior.
This phase is limited to BIG SSTables without SSTable-attached indexes.
h3. Phase 2: BTI-format splitting
Add the equivalent zero-copy splitting implementation for BTI SSTables,
including the format-specific index rebuilding required by BTI.
h3. Phase 3: Secondary-index support
Add support for splitting SSTables that have Storage-Attached Indexes (SAI) or
legacy secondary indexes (2i). This phase must ensure that index components are
rebuilt, shared, or invalidated correctly for every generated child.
h3. Phase 4: Zero-copy anticompaction
Integrate the splitting machinery with anticompaction so repaired, transient,
and unrepaired children can retain existing compressed chunks instead of
rewriting every row.
The optimized anticompaction path will initially be disabled by default and
retain the existing rewrite path as a fallback.
h3. Phase 5: Zero-copy range streaming
Integrate the machinery with range streaming so streams can transfer compressed
chunk runs together with enough derived information for the receiver to
construct the resulting SSTables without deserializing and rewriting rows.
The optimized streaming path will initially be disabled by default and retain
the existing row-based path for compatibility and fallback.
h2. Compatibility and fallback
The new representation and protocol behavior must be explicitly version-gated.
Nodes must not create or transmit split-prefix SSTables to peers that cannot
interpret them.
Zero-copy operation must remain an optimization rather than a correctness
requirement. Unsupported formats, components, indexes, filesystems, peer
versions, or split layouts must either be rejected before modification or use
an existing safe fallback path.
Reflink support is optional. Filesystems without extent sharing still benefit
from avoiding row deserialization and recompression by copying the existing
compressed bytes.
h2. Expected outcome
For supported SSTables, anticompaction and range streaming will avoid row-level
deserialization, serialization, and recompression. The remaining work will
primarily be compression-chunk transfer or extent sharing plus rebuilding
derived components from indexes.
This should substantially reduce CPU use, write amplification, temporary disk
requirements, and elapsed time for both operations while preserving the
existing behavior as a fallback.
was:
# Zero-copy SSTable splitting for anticompaction and range streaming
## Motivation
Anticompaction and range streaming both spend most of their cost repeating work
whose result already exists on disk.
Anticompaction rewrites every row of an SSTable through multiple writers only
to place each partition into a repaired, transient, or unrepaired child.
When streaming a subset of an SSTable's ranges, the sender can transmit whole
compression chunks verbatim, but the receiver still decompresses, deserializes,
reserializes, and recompresses every row. It then rebuilds the index, filter,
summary, and metadata even though these components can be derived without
rewriting the row data.
This work is avoidable when the required bytes form contiguous runs of the
parent's compression chunks.
## Proposed approach
Introduce reusable zero-copy SSTable-splitting machinery that:
- Selects contiguous runs of compression chunks for each child SSTable.
- Retains the compressed bytes verbatim without deserializing or recompressing
rows.
- Uses byte-range reflinks to share filesystem extents when supported.
- Falls back to copying the compressed bytes when reflinks are unavailable.
- Rebuilds the index, filter, summary, statistics, and other components from
index-only passes.
- Represents compression-chunk prefixes that do not contain indexed partitions,
allowing boundary chunks to be retained safely.
- Preserves the validation, corruption detection, lifecycle, disk-space, and
compatibility guarantees expected by existing SSTable operations.
The same machinery can then be used by anticompaction and range streaming
instead of their existing row-by-row rewrite paths.
## Delivery plan
The work will be delivered incrementally so the on-disk representation and
splitting primitives can be validated before they are introduced into critical
online paths.
### Phase 1: BIG-format splitting
Add zero-copy splitting for compressed BIG-format SSTables and expose it
through an opt-in `sstablesplit --zero-copy` mode.
Using the offline splitter as the first caller provides a bounded way to
validate chunk selection, component rebuilding, extent sharing, compatibility,
verification, scrub, streaming, loading, and upgrade behavior.
This phase is limited to BIG SSTables without SSTable-attached indexes.
### Phase 2: BTI-format splitting
Add the equivalent zero-copy splitting implementation for BTI SSTables,
including the format-specific index rebuilding required by BTI.
### Phase 3: Secondary-index support
Add support for splitting SSTables that have Storage-Attached Indexes (SAI) or
legacy secondary indexes (2i). This phase must ensure that index components are
rebuilt, shared, or invalidated correctly for every generated child.
### Phase 4: Zero-copy anticompaction
Integrate the splitting machinery with anticompaction so repaired, transient,
and unrepaired children can retain existing compressed chunks instead of
rewriting every row.
The optimized anticompaction path will initially be disabled by default and
retain the existing rewrite path as a fallback.
### Phase 5: Zero-copy range streaming
Integrate the machinery with range streaming so eligible streams transfer
compressed chunk runs together with enough derived information for the receiver
to construct the resulting SSTables without deserializing and rewriting rows.
The optimized streaming path will initially be disabled by default and retain
the existing row-based path for compatibility and fallback.
## Compatibility and fallback
The new representation and protocol behavior must be explicitly version-gated.
Nodes must not create or transmit split-prefix SSTables to peers that cannot
interpret them.
Zero-copy operation must remain an optimization rather than a correctness
requirement. Unsupported formats, components, indexes, filesystems, peer
versions, or split layouts must either be rejected before modification or use
an existing safe fallback path.
Reflink support is optional. Filesystems without extent sharing still benefit
from avoiding row deserialization and recompression by copying the existing
compressed bytes.
## Expected outcome
For eligible SSTables, anticompaction and range streaming will avoid row-level
deserialization, serialization, and recompression. The remaining work will
primarily be compression-chunk transfer or extent sharing plus rebuilding
derived components from indexes.
This should substantially reduce CPU use, write amplification, temporary disk
requirements, and elapsed time for both operations while preserving the
existing behavior as a fallback.
> Zero-copy sstable splitting for anticompaction, and partial-sstable zero-copy
> streaming
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21550
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21550
> Project: Apache Cassandra
> Issue Type: New Feature
> Reporter: Chris Lohfink
> Priority: Normal
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> h2. Motivation
> Anticompaction and range streaming both spend most of their cost repeating
> work whose result already exists on disk.
> Anticompaction rewrites every row of an SSTable through multiple writers only
> to place each partition into a repaired, transient, or unrepaired child.
> When streaming a subset of an SSTable's ranges, the sender can transmit whole
> compression chunks verbatim, but the receiver still decompresses,
> deserializes, reserializes, and recompresses every row. It then rebuilds the
> index, filter, summary, and metadata even though these components can be
> derived without rewriting the row data.
> This work is avoidable when the required bytes form contiguous runs of the
> parent's compression chunks.
> h2. Proposed approach
> Introduce reusable zero-copy SSTable-splitting machinery that:
> * Selects contiguous runs of compression chunks for each child SSTable.
> * Retains the compressed bytes verbatim without deserializing or
> recompressing rows.
> * Uses byte-range reflinks to share filesystem extents when supported.
> * Falls back to copying the compressed bytes when reflinks are unavailable.
> * Rebuilds the index, filter, summary, statistics, and other components from
> index-only passes.
> * Represents compression-chunk prefixes that do not contain indexed
> partitions, allowing boundary chunks to be retained safely.
> * Preserves the validation, corruption detection, lifecycle, disk-space, and
> compatibility guarantees expected by existing SSTable operations.
> The same machinery can then be used by anticompaction and range streaming
> instead of their existing row-by-row rewrite paths.
> h2. Delivery plan
> The work will be delivered incrementally so the on-disk representation and
> splitting primitives can be validated before they are introduced into
> critical online paths.
> h3. Phase 1: BIG-format splitting
> Add zero-copy splitting for compressed BIG-format SSTables and expose it
> through an opt-in {{sstablesplit --zero-copy}} mode.
> Using the offline splitter as the first caller provides a bounded way to
> validate chunk selection, component rebuilding, extent sharing,
> compatibility, verification, scrub, streaming, loading, and upgrade behavior.
> This phase is limited to BIG SSTables without SSTable-attached indexes.
> h3. Phase 2: BTI-format splitting
> Add the equivalent zero-copy splitting implementation for BTI SSTables,
> including the format-specific index rebuilding required by BTI.
> h3. Phase 3: Secondary-index support
> Add support for splitting SSTables that have Storage-Attached Indexes (SAI)
> or legacy secondary indexes (2i). This phase must ensure that index
> components are rebuilt, shared, or invalidated correctly for every generated
> child.
> h3. Phase 4: Zero-copy anticompaction
> Integrate the splitting machinery with anticompaction so repaired, transient,
> and unrepaired children can retain existing compressed chunks instead of
> rewriting every row.
> The optimized anticompaction path will initially be disabled by default and
> retain the existing rewrite path as a fallback.
> h3. Phase 5: Zero-copy range streaming
> Integrate the machinery with range streaming so streams can transfer
> compressed chunk runs together with enough derived information for the
> receiver to construct the resulting SSTables without deserializing and
> rewriting rows.
> The optimized streaming path will initially be disabled by default and retain
> the existing row-based path for compatibility and fallback.
> h2. Compatibility and fallback
> The new representation and protocol behavior must be explicitly
> version-gated. Nodes must not create or transmit split-prefix SSTables to
> peers that cannot interpret them.
> Zero-copy operation must remain an optimization rather than a correctness
> requirement. Unsupported formats, components, indexes, filesystems, peer
> versions, or split layouts must either be rejected before modification or use
> an existing safe fallback path.
> Reflink support is optional. Filesystems without extent sharing still benefit
> from avoiding row deserialization and recompression by copying the existing
> compressed bytes.
> h2. Expected outcome
> For supported SSTables, anticompaction and range streaming will avoid
> row-level deserialization, serialization, and recompression. The remaining
> work will primarily be compression-chunk transfer or extent sharing plus
> rebuilding derived components from indexes.
> This should substantially reduce CPU use, write amplification, temporary disk
> requirements, and elapsed time for both operations while preserving the
> existing behavior as a fallback.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
