[
https://issues.apache.org/jira/browse/CASSANDRA-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-21480:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Triage Needed)
fixed in CASSANDRA-21245
> Compaction disk-space check over-reserves by the compression ratio
> (uncompressed vs on-disk units)
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21480
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21480
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Gil Ganz
> Priority: Normal
>
> `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace()` decides
> whether a compaction fits by comparing, *{*}per FileStore{*}*:
> ```
> needed = expectedNewWriteSize(this task) // ON-DISK
> (compressed) bytes
> + Σ estimatedRemainingWriteBytes(running comps) // UNCOMPRESSED bytes
> <-- mismatch
> against
> available = getUsableSpace() // ON-DISK
> (compressed) bytes
> ```
> The second term — the reservation for already-running compactions — is
> computed in *{*}uncompressed{*}* bytes, but it is summed with a compressed
> estimate and compared against *{*}compressed{*}* free space. For a table with
> compression ratio `r` (on-disk/uncompressed), the in-flight reservation is
> inflated by `1/r`.
> #
> ## Observed
> Cassandra 5.0.8, UnifiedCompactionStrategy, single data dir, table `SSTable
> Compression Ratio: 0.41839` (≈2.39× inflation). One in-flight tier compaction
> shows in `compactionstats`:
> ```
> 951b9c30-... Compaction ...MY_TABLE_NAME 684.33 GiB / 4.76 TiB bytes 14.05%
> ```
> That 4.76 TiB is the *{*}uncompressed{*}* input (~1.99 TiB on disk). Disk:
> ```
> /dev/mapper/data_vg-cass_lv 14T 9.5T 4.6T 68%
> ```
> Every new compaction is rejected:
> ```
> WARN Directories.java:553 - FileStore ... has only 4.32 TiB available, but
> 4.79 TiB is needed
> WARN CompactionTask.java:104 - insufficient space to compact all requested
> files. 2784.6711MiB required ... removing largest SSTable ...
> ... (scope reduced repeatedly down to a single 122 MB sstable, "needed"
> stays pinned ~4.78 TiB)
> WARN CompactionTask.java:434 - Not enough space for compaction ... estimated
> sstables = 1, expected write size = 122176278
> ERROR JVMStabilityInspector.java:70 - Exception in thread
> CompactionExecutor...
> java.lang.RuntimeException: Not enough space for compaction ... estimated
> sstables = 1, expected write size = 122176278
> at
> CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> ```
> The "needed" is dominated by the running compaction's remaining uncompressed
> bytes (~4.08 TiB). Its true on-disk write is ~1.71 TiB. Reducing scope of the
> *candidate* never helps because the binding term is the *other* compaction's
> (mis-unit'd) reservation, so the task strips to one sstable and aborts.
> #
> ##
> ### Numbers (r = 0.41839)
> |Quantity|Uncompressed (used by guard)|On disk (reality)|
> |—|—|—|
> |Running comp. remaining write|~4.08 TiB|~1.71 TiB|
> |Guard "needed" total|4.79 TiB|~2.0 TiB|
> |Available|—|4.32 TiB|
> Real on-disk requirement (~2.0 TiB) fits in 4.32 TiB free with ~2.3 TiB to
> spare; the guard rejects it purely due to the unit mismatch.
> #
> ## Root cause (code references, 5.0.8)
> 1. `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace` builds the
> two maps and calls the check:
> - `src/java/org/apache/cassandra/db/compaction/CompactionTask.java:402` —
> new-task write size:
> `writeSize = cfs.getExpectedCompactedFileSize(nonExpiredSSTables,
> compactionType)`
> - `:409` — running-compactions reservation:
> `Map<File,Long> expectedWriteSize =
> CompactionManager.instance.active.estimatedRemainingWriteBytes()`
> - `:412` —
> `cfs.getDirectories().hasDiskSpaceForCompactionsAndStreams(expectedNewWriteSize,
> expectedWriteSize)`
> 2. New-task term is *{*}on-disk (compressed){*}* — correct basis:
> - `ColumnFamilyStore.getExpectedCompactedFileSize`
> (`ColumnFamilyStore.java:1695`) → `SSTableReader.getTotalBytes`
> (`SSTableReader.java:541`) → `sstable.onDiskLength()`.
> 3. Running-compactions term is *{*}uncompressed{*}* — the bug:
> - `ActiveCompactions.estimatedRemainingWriteBytes`
> (`ActiveCompactions.java:58`) sums per-dir
> `CompactionInfo.estimatedRemainingWriteBytes()`.
> - `CompactionInfo.estimatedRemainingWriteBytes`
> (`CompactionInfo.java:188`) returns `getTotal() - getCompleted()`.
> - For a running compaction the `CompactionInfo` comes from
> `CompactionIterator.getCompactionInfo()` (`CompactionIterator.java:167`) with:
> - `total = totalBytes = Σ scanner.getLengthInBytes()`
> (`CompactionIterator.java:137-140`)
> - `SSTableScanner.getLengthInBytes()` returns
> *{*}`sstable.uncompressedLength()`{*}* (`SSTableScanner.java:153-156`).
> Note the sibling `getCompressedLengthInBytes()` → `onDiskLength()`
> (`:159-162`) exists but is unused here.
> - `completed = bytesRead = Σ scanner.getBytesScanned()`
> (`CompactionIterator.java:289-294`), and
> `bytesScanned` accumulates the *{*}uncompressed{*}* data-file pointer
> (`SSTableScanner.java:217`).
> - So both `total` and `completed` are uncompressed; their difference (the
> "remaining write") is uncompressed.
> 4. Available side is *{*}on-disk (compressed){*}*:
> - `Directories.getAvailableSpaceForCompactions` (`Directories.java:563`) =
> `(FileStore.getUsableSpace() - min_free_space_per_drive) *
> max_space_usable_for_compactions_in_percentage` (default 0.95).
> - `hasDiskSpaceForCompactionsAndStreams` (`Directories.java:517-561`) sums
> the two maps and compares to that.
> Result: a compressed-bytes free-space budget is charged an uncompressed-bytes
> reservation for every in-flight compaction → over-reservation by
> `1/compressionRatio` (here 2.39×).
> The same uncompressed reservation also feeds the streaming disk check via
> `StreamSession.java:927`
> (`Directories.perFileStore(CompactionManager.instance.active.estimatedRemainingWriteBytes(),
> ...)`), so streams are over-throttled on compressed tables too.
>
> #
> ## Why this is a unit bug and not a deliberate worst-case policy
> A reasonable objection would be : compaction *output* is not guaranteed to
> compress like existing data (incompressible output, or inputs written with a
> stronger compressor), so reserving the uncompressed size could be intended as
> a safe worst-case bound. Two facts refute that for compaction:
> 1. *{*}Compaction output is a recompression of data already on disk at ratio
> r, not new ingest.{*}* The merged rows are the same column values already
> stored compressed; running the same compressor over them yields ~the same
> ratio. Unlike a flush (genuinely new data, no prior on-disk evidence),
> compaction has direct evidence — the inputs' own realized on-disk sizes.
> 2. *{*}The new-task side of the *same check{*} already assumes output-on-disk
> ≈ input-on-disk (compressed).** `expectedNewWriteSize` comes from
> `getExpectedCompactedFileSize` → `getTotalBytes` → `onDiskLength()`
> (compressed). So the code already predicts compaction output in compressed
> terms; only the *in-flight* reservation for running compactions uses a
> different (uncompressed) measure for the identical quantity. That is an
> internal inconsistency — had uncompressed been a considered worst-case
> policy, the new-task side would use it too, and it does not.
> #
> ## Proposed fix
> Make the running-compactions reservation use the inputs' realized *{*}on-disk
> (compressed){*}* size so both terms and the free-space comparison share
> units. Preferred:
> - Base `estimatedRemainingWriteBytes()` on `onDiskLength()` rather than
> `uncompressedLength()` — e.g. have `CompactionIterator` track remaining via
> the already-exposed `ISSTableScanner.getCompressedLengthInBytes()` →
> `onDiskLength()` (`SSTableScanner.java:159-162`) and report `CompactionInfo`
> total/completed in compressed terms.
> Lower-effort alternative (slightly less precise, extrapolates from the table
> average): multiply each holder's `estimatedRemainingWriteBytes()` by its CFS
> `metric.compressionRatio` when > 0 — mirroring what
> `ColumnFamilyStore.getExpectedCompactedFileSize` already does for the CLEANUP
> path (`ColumnFamilyStore.java:1712-1714`).
> Either way, residual recompression risk (output marginally larger than input
> on disk) is already covered by the existing margins
> `max_space_usable_for_compactions_in_percentage` (0.95) and
> `min_free_space_per_drive`; a small explicit fudge factor could be added if
> desired. Neither approach requires assuming *new* data compresses like
> existing data — both measure the data actually being recompacted.
> #
> ## Repro
> 1. Table with compression (ratio ~0.4) and several GB+ of data under UCS (or
> STCS with large sstables).
> 2. Fill the FileStore so that `free_on_disk < Σ
> uncompressed_remaining_of_running_compactions` but `free_on_disk > Σ
> on-disk_remaining`.
> 3. Observe new compactions abort with `Not enough space for compaction` even
> though the real on-disk writes fit. `compactionstats` "total" (uncompressed)
> vs `SSTable Compression Ratio` from `tablestats` shows the gap.
> #
> ## Workaround
> Per-table `ColumnFamilyStoreMBean.compactionDiskSpaceCheck(false)`
> (`ColumnFamilyStoreMBean.java:317`) disables the guard; keep
> `concurrent_compactors` low while disabled. Safe in practice because real
> on-disk write << the uncompressed reservation and `SSTableRewriter` releases
> inputs progressively.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
