[
https://issues.apache.org/jira/browse/CASSANDRA-21480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gil Ganz updated CASSANDRA-21480:
---------------------------------
Description:
`CompactionTask.buildCompactionCandidatesForAvailableDiskSpace()` decides
whether a compaction fits by comparing, *{*}per FileStore{*}*:
```
needed = expectedNewWriteSize(this task) // ON-DISK (compressed)
bytes
+ Σ estimatedRemainingWriteBytes(running comps) // UNCOMPRESSED bytes
<-- mismatch
against
available = getUsableSpace() // ON-DISK (compressed)
bytes
```
The second term — the reservation for already-running compactions — is computed
in *{*}uncompressed{*}* bytes, but it is summed with a compressed estimate and
compared against *{*}compressed{*}* free space. For a table with compression
ratio `r` (on-disk/uncompressed), the in-flight reservation is inflated by
`1/r`.
#
## Observed
Cassandra 5.0.8, UnifiedCompactionStrategy, single data dir, table `SSTable
Compression Ratio: 0.41839` (≈2.39× inflation). One in-flight tier compaction
shows in `compactionstats`:
```
951b9c30-... Compaction ...MY_TABLE_NAME 684.33 GiB / 4.76 TiB bytes 14.05%
```
That 4.76 TiB is the *{*}uncompressed{*}* input (~1.99 TiB on disk). Disk:
```
/dev/mapper/data_vg-cass_lv 14T 9.5T 4.6T 68%
```
Every new compaction is rejected:
```
WARN Directories.java:553 - FileStore ... has only 4.32 TiB available, but
4.79 TiB is needed
WARN CompactionTask.java:104 - insufficient space to compact all requested
files. 2784.6711MiB required ... removing largest SSTable ...
... (scope reduced repeatedly down to a single 122 MB sstable, "needed" stays
pinned ~4.78 TiB)
WARN CompactionTask.java:434 - Not enough space for compaction ... estimated
sstables = 1, expected write size = 122176278
ERROR JVMStabilityInspector.java:70 - Exception in thread CompactionExecutor...
java.lang.RuntimeException: Not enough space for compaction ... estimated
sstables = 1, expected write size = 122176278
at
CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
```
The "needed" is dominated by the running compaction's remaining uncompressed
bytes (~4.08 TiB). Its true on-disk write is ~1.71 TiB. Reducing scope of the
*candidate* never helps because the binding term is the *other* compaction's
(mis-unit'd) reservation, so the task strips to one sstable and aborts.
#
##
### Numbers (r = 0.41839)
|Quantity|Uncompressed (used by guard)|On disk (reality)|
|—|—|—|
|Running comp. remaining write|~4.08 TiB|~1.71 TiB|
|Guard "needed" total|4.79 TiB|~2.0 TiB|
|Available|—|4.32 TiB|
Real on-disk requirement (~2.0 TiB) fits in 4.32 TiB free with ~2.3 TiB to
spare; the guard rejects it purely due to the unit mismatch.
#
## Root cause (code references, 5.0.8)
1. `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace` builds the
two maps and calls the check:
- `src/java/org/apache/cassandra/db/compaction/CompactionTask.java:402` —
new-task write size:
`writeSize = cfs.getExpectedCompactedFileSize(nonExpiredSSTables,
compactionType)`
- `:409` — running-compactions reservation:
`Map<File,Long> expectedWriteSize =
CompactionManager.instance.active.estimatedRemainingWriteBytes()`
- `:412` —
`cfs.getDirectories().hasDiskSpaceForCompactionsAndStreams(expectedNewWriteSize,
expectedWriteSize)`
2. New-task term is *{*}on-disk (compressed){*}* — correct basis:
- `ColumnFamilyStore.getExpectedCompactedFileSize`
(`ColumnFamilyStore.java:1695`) → `SSTableReader.getTotalBytes`
(`SSTableReader.java:541`) → `sstable.onDiskLength()`.
3. Running-compactions term is *{*}uncompressed{*}* — the bug:
- `ActiveCompactions.estimatedRemainingWriteBytes`
(`ActiveCompactions.java:58`) sums per-dir
`CompactionInfo.estimatedRemainingWriteBytes()`.
- `CompactionInfo.estimatedRemainingWriteBytes` (`CompactionInfo.java:188`)
returns `getTotal() - getCompleted()`.
- For a running compaction the `CompactionInfo` comes from
`CompactionIterator.getCompactionInfo()` (`CompactionIterator.java:167`) with:
- `total = totalBytes = Σ scanner.getLengthInBytes()`
(`CompactionIterator.java:137-140`)
- `SSTableScanner.getLengthInBytes()` returns
*{*}`sstable.uncompressedLength()`{*}* (`SSTableScanner.java:153-156`).
Note the sibling `getCompressedLengthInBytes()` → `onDiskLength()`
(`:159-162`) exists but is unused here.
- `completed = bytesRead = Σ scanner.getBytesScanned()`
(`CompactionIterator.java:289-294`), and
`bytesScanned` accumulates the *{*}uncompressed{*}* data-file pointer
(`SSTableScanner.java:217`).
- So both `total` and `completed` are uncompressed; their difference (the
"remaining write") is uncompressed.
4. Available side is *{*}on-disk (compressed){*}*:
- `Directories.getAvailableSpaceForCompactions` (`Directories.java:563`) =
`(FileStore.getUsableSpace() - min_free_space_per_drive) *
max_space_usable_for_compactions_in_percentage` (default 0.95).
- `hasDiskSpaceForCompactionsAndStreams` (`Directories.java:517-561`) sums
the two maps and compares to that.
Result: a compressed-bytes free-space budget is charged an uncompressed-bytes
reservation for every in-flight compaction → over-reservation by
`1/compressionRatio` (here 2.39×).
The same uncompressed reservation also feeds the streaming disk check via
`StreamSession.java:927`
(`Directories.perFileStore(CompactionManager.instance.active.estimatedRemainingWriteBytes(),
...)`), so streams are over-throttled on compressed tables too.
#
## Why this is a unit bug and not a deliberate worst-case policy
A reasonable objection would be : compaction *output* is not guaranteed to
compress like existing data (incompressible output, or inputs written with a
stronger compressor), so reserving the uncompressed size could be intended as a
safe worst-case bound. Two facts refute that for compaction:
1. *{*}Compaction output is a recompression of data already on disk at ratio r,
not new ingest.{*}* The merged rows are the same column values already stored
compressed; running the same compressor over them yields ~the same ratio.
Unlike a flush (genuinely new data, no prior on-disk evidence), compaction has
direct evidence — the inputs' own realized on-disk sizes.
2. *{*}The new-task side of the *same check{*} already assumes output-on-disk ≈
input-on-disk (compressed).** `expectedNewWriteSize` comes from
`getExpectedCompactedFileSize` → `getTotalBytes` → `onDiskLength()`
(compressed). So the code already predicts compaction output in compressed
terms; only the *in-flight* reservation for running compactions uses a
different (uncompressed) measure for the identical quantity. That is an
internal inconsistency — had uncompressed been a considered worst-case policy,
the new-task side would use it too, and it does not.
#
## Proposed fix
Make the running-compactions reservation use the inputs' realized *{*}on-disk
(compressed){*}* size so both terms and the free-space comparison share units.
Preferred:
- Base `estimatedRemainingWriteBytes()` on `onDiskLength()` rather than
`uncompressedLength()` — e.g. have `CompactionIterator` track remaining via the
already-exposed `ISSTableScanner.getCompressedLengthInBytes()` →
`onDiskLength()` (`SSTableScanner.java:159-162`) and report `CompactionInfo`
total/completed in compressed terms.
Lower-effort alternative (slightly less precise, extrapolates from the table
average): multiply each holder's `estimatedRemainingWriteBytes()` by its CFS
`metric.compressionRatio` when > 0 — mirroring what
`ColumnFamilyStore.getExpectedCompactedFileSize` already does for the CLEANUP
path (`ColumnFamilyStore.java:1712-1714`).
Either way, residual recompression risk (output marginally larger than input on
disk) is already covered by the existing margins
`max_space_usable_for_compactions_in_percentage` (0.95) and
`min_free_space_per_drive`; a small explicit fudge factor could be added if
desired. Neither approach requires assuming *new* data compresses like existing
data — both measure the data actually being recompacted.
#
## Repro
1. Table with compression (ratio ~0.4) and several GB+ of data under UCS (or
STCS with large sstables).
2. Fill the FileStore so that `free_on_disk < Σ
uncompressed_remaining_of_running_compactions` but `free_on_disk > Σ
on-disk_remaining`.
3. Observe new compactions abort with `Not enough space for compaction` even
though the real on-disk writes fit. `compactionstats` "total" (uncompressed) vs
`SSTable Compression Ratio` from `tablestats` shows the gap.
#
## Workaround
Per-table `ColumnFamilyStoreMBean.compactionDiskSpaceCheck(false)`
(`ColumnFamilyStoreMBean.java:317`) disables the guard; keep
`concurrent_compactors` low while disabled. Safe in practice because real
on-disk write << the uncompressed reservation and `SSTableRewriter` releases
inputs progressively.
was:
`CompactionTask.buildCompactionCandidatesForAvailableDiskSpace()` decides
whether a compaction fits by comparing, **per FileStore**:
```
needed = expectedNewWriteSize(this task) // ON-DISK (compressed)
bytes
+ Σ estimatedRemainingWriteBytes(running comps) // UNCOMPRESSED bytes
<-- mismatch
against
available = getUsableSpace() // ON-DISK (compressed)
bytes
```
The second term — the reservation for already-running compactions — is computed
in **uncompressed** bytes, but it is summed with a compressed estimate and
compared against **compressed** free space. For a table with compression ratio
`r` (on-disk/uncompressed), the in-flight reservation is inflated by `1/r`.
## Observed
Cassandra 5.0.8, UnifiedCompactionStrategy, single data dir, table `SSTable
Compression Ratio: 0.41839` (≈2.39× inflation). One in-flight tier compaction
shows in `compactionstats`:
```
951b9c30-... Compaction ...MY_TABLE_NAME 684.33 GiB / 4.76 TiB bytes 14.05%
```
That 4.76 TiB is the **uncompressed** input (~1.99 TiB on disk). Disk:
```
/dev/mapper/data_vg-cass_lv 14T 9.5T 4.6T 68%
```
Every new compaction is rejected:
```
WARN Directories.java:553 - FileStore ... has only 4.32 TiB available, but
4.79 TiB is needed
WARN CompactionTask.java:104 - insufficient space to compact all requested
files. 2784.6711MiB required ... removing largest SSTable ...
... (scope reduced repeatedly down to a single 122 MB sstable, "needed" stays
pinned ~4.78 TiB)
WARN CompactionTask.java:434 - Not enough space for compaction ... estimated
sstables = 1, expected write size = 122176278
ERROR JVMStabilityInspector.java:70 - Exception in thread CompactionExecutor...
java.lang.RuntimeException: Not enough space for compaction ... estimated
sstables = 1, expected write size = 122176278
at
CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
```
The "needed" is dominated by the running compaction's remaining uncompressed
bytes (~4.08 TiB). Its true on-disk write is ~1.71 TiB. Reducing scope of the
*candidate* never helps because the binding term is the *other* compaction's
(mis-unit'd) reservation, so the task strips to one sstable and aborts.
### Numbers (r = 0.41839)
| Quantity | Uncompressed (used by guard) | On disk (reality) |
|---|---|---|
| Running comp. remaining write | ~4.08 TiB | ~1.71 TiB |
| Guard "needed" total | 4.79 TiB | ~2.0 TiB |
| Available | — | 4.32 TiB |
Real on-disk requirement (~2.0 TiB) fits in 4.32 TiB free with ~2.3 TiB to
spare; the guard rejects it purely due to the unit mismatch.
## Root cause (code references, 5.0.8)
1. `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace` builds the
two maps and calls the check:
- `src/java/org/apache/cassandra/db/compaction/CompactionTask.java:402` —
new-task write size:
`writeSize = cfs.getExpectedCompactedFileSize(nonExpiredSSTables,
compactionType)`
- `:409` — running-compactions reservation:
`Map<File,Long> expectedWriteSize =
CompactionManager.instance.active.estimatedRemainingWriteBytes()`
- `:412` —
`cfs.getDirectories().hasDiskSpaceForCompactionsAndStreams(expectedNewWriteSize,
expectedWriteSize)`
2. New-task term is **on-disk (compressed)** — correct basis:
- `ColumnFamilyStore.getExpectedCompactedFileSize`
(`ColumnFamilyStore.java:1695`) → `SSTableReader.getTotalBytes`
(`SSTableReader.java:541`) → `sstable.onDiskLength()`.
3. Running-compactions term is **uncompressed** — the bug:
- `ActiveCompactions.estimatedRemainingWriteBytes`
(`ActiveCompactions.java:58`) sums per-dir
`CompactionInfo.estimatedRemainingWriteBytes()`.
- `CompactionInfo.estimatedRemainingWriteBytes` (`CompactionInfo.java:188`)
returns `getTotal() - getCompleted()`.
- For a running compaction the `CompactionInfo` comes from
`CompactionIterator.getCompactionInfo()` (`CompactionIterator.java:167`) with:
- `total = totalBytes = Σ scanner.getLengthInBytes()`
(`CompactionIterator.java:137-140`)
- `SSTableScanner.getLengthInBytes()` returns
**`sstable.uncompressedLength()`** (`SSTableScanner.java:153-156`).
Note the sibling `getCompressedLengthInBytes()` → `onDiskLength()`
(`:159-162`) exists but is unused here.
- `completed = bytesRead = Σ scanner.getBytesScanned()`
(`CompactionIterator.java:289-294`), and
`bytesScanned` accumulates the **uncompressed** data-file pointer
(`SSTableScanner.java:217`).
- So both `total` and `completed` are uncompressed; their difference (the
"remaining write") is uncompressed.
4. Available side is **on-disk (compressed)**:
- `Directories.getAvailableSpaceForCompactions` (`Directories.java:563`) =
`(FileStore.getUsableSpace() - min_free_space_per_drive) *
max_space_usable_for_compactions_in_percentage` (default 0.95).
- `hasDiskSpaceForCompactionsAndStreams` (`Directories.java:517-561`) sums
the two maps and compares to that.
Result: a compressed-bytes free-space budget is charged an uncompressed-bytes
reservation for every in-flight compaction → over-reservation by
`1/compressionRatio` (here 2.39×).
The same uncompressed reservation also feeds the streaming disk check via
`StreamSession.java:927`
(`Directories.perFileStore(CompactionManager.instance.active.estimatedRemainingWriteBytes(),
...)`), so streams are over-throttled on compressed tables too.
## Why this is a unit bug and not a deliberate worst-case policy
A reasonable objection would be : compaction *output* is not guaranteed to
compress like existing data (incompressible output, or inputs written with a
stronger compressor), so reserving the uncompressed size could be intended as a
safe worst-case bound. Two facts refute that for compaction:
1. **Compaction output is a recompression of data already on disk at ratio r,
not new ingest.** The merged rows are the same column values already stored
compressed; running the same compressor over them yields ~the same ratio.
Unlike a flush (genuinely new data, no prior on-disk evidence), compaction has
direct evidence — the inputs' own realized on-disk sizes.
2. **The new-task side of the *same check* already assumes output-on-disk ≈
input-on-disk (compressed).** `expectedNewWriteSize` comes from
`getExpectedCompactedFileSize` → `getTotalBytes` → `onDiskLength()`
(compressed). So the code already predicts compaction output in compressed
terms; only the *in-flight* reservation for running compactions uses a
different (uncompressed) measure for the identical quantity. That is an
internal inconsistency — had uncompressed been a considered worst-case policy,
the new-task side would use it too, and it does not.
## Proposed fix
Make the running-compactions reservation use the inputs' realized **on-disk
(compressed)** size so both terms and the free-space comparison share units.
Preferred:
- Base `estimatedRemainingWriteBytes()` on `onDiskLength()` rather than
`uncompressedLength()` — e.g. have `CompactionIterator` track remaining via the
already-exposed `ISSTableScanner.getCompressedLengthInBytes()` →
`onDiskLength()` (`SSTableScanner.java:159-162`) and report `CompactionInfo`
total/completed in compressed terms.
Lower-effort alternative (slightly less precise, extrapolates from the table
average): multiply each holder's `estimatedRemainingWriteBytes()` by its CFS
`metric.compressionRatio` when > 0 — mirroring what
`ColumnFamilyStore.getExpectedCompactedFileSize` already does for the CLEANUP
path (`ColumnFamilyStore.java:1712-1714`).
Either way, residual recompression risk (output marginally larger than input on
disk) is already covered by the existing margins
`max_space_usable_for_compactions_in_percentage` (0.95) and
`min_free_space_per_drive`; a small explicit fudge factor could be added if
desired. Neither approach requires assuming *new* data compresses like existing
data — both measure the data actually being recompacted.
## Repro
1. Table with compression (ratio ~0.4) and several GB+ of data under UCS (or
STCS with large sstables).
2. Fill the FileStore so that `free_on_disk < Σ
uncompressed_remaining_of_running_compactions` but `free_on_disk > Σ
on-disk_remaining`.
3. Observe new compactions abort with `Not enough space for compaction` even
though the real on-disk writes fit. `compactionstats` "total" (uncompressed) vs
`SSTable Compression Ratio` from `tablestats` shows the gap.
## Workaround
Per-table `ColumnFamilyStoreMBean.compactionDiskSpaceCheck(false)`
(`ColumnFamilyStoreMBean.java:317`) disables the guard; keep
`concurrent_compactors` low while disabled. Safe in practice because real
on-disk write << the uncompressed reservation and `SSTableRewriter` releases
inputs progressively.
> Compaction disk-space check over-reserves by the compression ratio
> (uncompressed vs on-disk units)
> --------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21480
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21480
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Gil Ganz
> Priority: Normal
>
> `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace()` decides
> whether a compaction fits by comparing, *{*}per FileStore{*}*:
> ```
> needed = expectedNewWriteSize(this task) // ON-DISK
> (compressed) bytes
> + Σ estimatedRemainingWriteBytes(running comps) // UNCOMPRESSED bytes
> <-- mismatch
> against
> available = getUsableSpace() // ON-DISK
> (compressed) bytes
> ```
> The second term — the reservation for already-running compactions — is
> computed in *{*}uncompressed{*}* bytes, but it is summed with a compressed
> estimate and compared against *{*}compressed{*}* free space. For a table with
> compression ratio `r` (on-disk/uncompressed), the in-flight reservation is
> inflated by `1/r`.
> #
> ## Observed
> Cassandra 5.0.8, UnifiedCompactionStrategy, single data dir, table `SSTable
> Compression Ratio: 0.41839` (≈2.39× inflation). One in-flight tier compaction
> shows in `compactionstats`:
> ```
> 951b9c30-... Compaction ...MY_TABLE_NAME 684.33 GiB / 4.76 TiB bytes 14.05%
> ```
> That 4.76 TiB is the *{*}uncompressed{*}* input (~1.99 TiB on disk). Disk:
> ```
> /dev/mapper/data_vg-cass_lv 14T 9.5T 4.6T 68%
> ```
> Every new compaction is rejected:
> ```
> WARN Directories.java:553 - FileStore ... has only 4.32 TiB available, but
> 4.79 TiB is needed
> WARN CompactionTask.java:104 - insufficient space to compact all requested
> files. 2784.6711MiB required ... removing largest SSTable ...
> ... (scope reduced repeatedly down to a single 122 MB sstable, "needed"
> stays pinned ~4.78 TiB)
> WARN CompactionTask.java:434 - Not enough space for compaction ... estimated
> sstables = 1, expected write size = 122176278
> ERROR JVMStabilityInspector.java:70 - Exception in thread
> CompactionExecutor...
> java.lang.RuntimeException: Not enough space for compaction ... estimated
> sstables = 1, expected write size = 122176278
> at
> CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:436)
> ```
> The "needed" is dominated by the running compaction's remaining uncompressed
> bytes (~4.08 TiB). Its true on-disk write is ~1.71 TiB. Reducing scope of the
> *candidate* never helps because the binding term is the *other* compaction's
> (mis-unit'd) reservation, so the task strips to one sstable and aborts.
> #
> ##
> ### Numbers (r = 0.41839)
> |Quantity|Uncompressed (used by guard)|On disk (reality)|
> |—|—|—|
> |Running comp. remaining write|~4.08 TiB|~1.71 TiB|
> |Guard "needed" total|4.79 TiB|~2.0 TiB|
> |Available|—|4.32 TiB|
> Real on-disk requirement (~2.0 TiB) fits in 4.32 TiB free with ~2.3 TiB to
> spare; the guard rejects it purely due to the unit mismatch.
> #
> ## Root cause (code references, 5.0.8)
> 1. `CompactionTask.buildCompactionCandidatesForAvailableDiskSpace` builds the
> two maps and calls the check:
> - `src/java/org/apache/cassandra/db/compaction/CompactionTask.java:402` —
> new-task write size:
> `writeSize = cfs.getExpectedCompactedFileSize(nonExpiredSSTables,
> compactionType)`
> - `:409` — running-compactions reservation:
> `Map<File,Long> expectedWriteSize =
> CompactionManager.instance.active.estimatedRemainingWriteBytes()`
> - `:412` —
> `cfs.getDirectories().hasDiskSpaceForCompactionsAndStreams(expectedNewWriteSize,
> expectedWriteSize)`
> 2. New-task term is *{*}on-disk (compressed){*}* — correct basis:
> - `ColumnFamilyStore.getExpectedCompactedFileSize`
> (`ColumnFamilyStore.java:1695`) → `SSTableReader.getTotalBytes`
> (`SSTableReader.java:541`) → `sstable.onDiskLength()`.
> 3. Running-compactions term is *{*}uncompressed{*}* — the bug:
> - `ActiveCompactions.estimatedRemainingWriteBytes`
> (`ActiveCompactions.java:58`) sums per-dir
> `CompactionInfo.estimatedRemainingWriteBytes()`.
> - `CompactionInfo.estimatedRemainingWriteBytes`
> (`CompactionInfo.java:188`) returns `getTotal() - getCompleted()`.
> - For a running compaction the `CompactionInfo` comes from
> `CompactionIterator.getCompactionInfo()` (`CompactionIterator.java:167`) with:
> - `total = totalBytes = Σ scanner.getLengthInBytes()`
> (`CompactionIterator.java:137-140`)
> - `SSTableScanner.getLengthInBytes()` returns
> *{*}`sstable.uncompressedLength()`{*}* (`SSTableScanner.java:153-156`).
> Note the sibling `getCompressedLengthInBytes()` → `onDiskLength()`
> (`:159-162`) exists but is unused here.
> - `completed = bytesRead = Σ scanner.getBytesScanned()`
> (`CompactionIterator.java:289-294`), and
> `bytesScanned` accumulates the *{*}uncompressed{*}* data-file pointer
> (`SSTableScanner.java:217`).
> - So both `total` and `completed` are uncompressed; their difference (the
> "remaining write") is uncompressed.
> 4. Available side is *{*}on-disk (compressed){*}*:
> - `Directories.getAvailableSpaceForCompactions` (`Directories.java:563`) =
> `(FileStore.getUsableSpace() - min_free_space_per_drive) *
> max_space_usable_for_compactions_in_percentage` (default 0.95).
> - `hasDiskSpaceForCompactionsAndStreams` (`Directories.java:517-561`) sums
> the two maps and compares to that.
> Result: a compressed-bytes free-space budget is charged an uncompressed-bytes
> reservation for every in-flight compaction → over-reservation by
> `1/compressionRatio` (here 2.39×).
> The same uncompressed reservation also feeds the streaming disk check via
> `StreamSession.java:927`
> (`Directories.perFileStore(CompactionManager.instance.active.estimatedRemainingWriteBytes(),
> ...)`), so streams are over-throttled on compressed tables too.
>
> #
> ## Why this is a unit bug and not a deliberate worst-case policy
> A reasonable objection would be : compaction *output* is not guaranteed to
> compress like existing data (incompressible output, or inputs written with a
> stronger compressor), so reserving the uncompressed size could be intended as
> a safe worst-case bound. Two facts refute that for compaction:
> 1. *{*}Compaction output is a recompression of data already on disk at ratio
> r, not new ingest.{*}* The merged rows are the same column values already
> stored compressed; running the same compressor over them yields ~the same
> ratio. Unlike a flush (genuinely new data, no prior on-disk evidence),
> compaction has direct evidence — the inputs' own realized on-disk sizes.
> 2. *{*}The new-task side of the *same check{*} already assumes output-on-disk
> ≈ input-on-disk (compressed).** `expectedNewWriteSize` comes from
> `getExpectedCompactedFileSize` → `getTotalBytes` → `onDiskLength()`
> (compressed). So the code already predicts compaction output in compressed
> terms; only the *in-flight* reservation for running compactions uses a
> different (uncompressed) measure for the identical quantity. That is an
> internal inconsistency — had uncompressed been a considered worst-case
> policy, the new-task side would use it too, and it does not.
> #
> ## Proposed fix
> Make the running-compactions reservation use the inputs' realized *{*}on-disk
> (compressed){*}* size so both terms and the free-space comparison share
> units. Preferred:
> - Base `estimatedRemainingWriteBytes()` on `onDiskLength()` rather than
> `uncompressedLength()` — e.g. have `CompactionIterator` track remaining via
> the already-exposed `ISSTableScanner.getCompressedLengthInBytes()` →
> `onDiskLength()` (`SSTableScanner.java:159-162`) and report `CompactionInfo`
> total/completed in compressed terms.
> Lower-effort alternative (slightly less precise, extrapolates from the table
> average): multiply each holder's `estimatedRemainingWriteBytes()` by its CFS
> `metric.compressionRatio` when > 0 — mirroring what
> `ColumnFamilyStore.getExpectedCompactedFileSize` already does for the CLEANUP
> path (`ColumnFamilyStore.java:1712-1714`).
> Either way, residual recompression risk (output marginally larger than input
> on disk) is already covered by the existing margins
> `max_space_usable_for_compactions_in_percentage` (0.95) and
> `min_free_space_per_drive`; a small explicit fudge factor could be added if
> desired. Neither approach requires assuming *new* data compresses like
> existing data — both measure the data actually being recompacted.
> #
> ## Repro
> 1. Table with compression (ratio ~0.4) and several GB+ of data under UCS (or
> STCS with large sstables).
> 2. Fill the FileStore so that `free_on_disk < Σ
> uncompressed_remaining_of_running_compactions` but `free_on_disk > Σ
> on-disk_remaining`.
> 3. Observe new compactions abort with `Not enough space for compaction` even
> though the real on-disk writes fit. `compactionstats` "total" (uncompressed)
> vs `SSTable Compression Ratio` from `tablestats` shows the gap.
> #
> ## Workaround
> Per-table `ColumnFamilyStoreMBean.compactionDiskSpaceCheck(false)`
> (`ColumnFamilyStoreMBean.java:317`) disables the guard; keep
> `concurrent_compactors` low while disabled. Safe in practice because real
> on-disk write << the uncompressed reservation and `SSTableRewriter` releases
> inputs progressively.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
