[
https://issues.apache.org/jira/browse/CASSANDRA-21666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-21666:
------------------------------------------
Description:
h2. Summary
{{nodetool compressiondictionary train}} fails with {{Insufficient samples for
training: N (minimum required: 11)}} on tables that have a large
{{chunk_length_in_kb}} and many SSTables, even when there are gigabytes of data
available to sample.
h2. Root cause
{{SSTableChunkSampler}}
({_}src/java/org/apache/cassandra/db/compression/SSTableChunkSampler.java{_})
allocates the number of chunks to sample from each SSTable using integer
division that truncates to zero for most SSTables when the sample target is
small.
{{{}calculateTargetChunkCount(){}}}:
{code:java}
int averageChunkSize = totalDataSize / totalChunks; // ~ chunk_length_in_kb
return config.maxTotalSampleSize / averageChunkSize; // target shrinks as
chunk size grows
{code}
With the default {{maxTotalSampleSize = 8 MiB}} and 64 KiB chunks:
{{{}targetChunkCount = 8 MiB / 64 KiB = 128{}}}.
{{{}sampleChunksFromSSTables(){}}}:
{code:java}
long chunksFromThisSSTable = Math.min((targetChunkCount * info.chunkCount) /
totalChunks, remainingTarget);
if (chunksFromThisSSTable <= 0)
continue; // <-- SSTable skipped entirely
{code}
{{(targetChunkCount * info.chunkCount) / totalChunks}} is integer division. An
SSTable contributes at least one chunk only when {{{}info.chunkCount >=
totalChunks / targetChunkCount{}}}. With {{targetChunkCount = 128}} and ~387
SSTables (~100k total chunks), that threshold is ~780 chunks (~50 MB) — so
every SSTable below it rounds to 0 and is skipped, leaving ~8 samples, under
the 11-sample minimum, and training fails.
Because {{targetChunkCount}} is 16x larger at 4 KiB chunks (2048), small-chunk
tables clear the threshold and are unaffected — which is why this only
manifests at larger {{{}chunk_length{}}}.
||chunk_length_in_kb||targetChunkCount||samples collected||result||
|4|2048|~2418|COMPLETED|
|64|128|8|FAILED (minimum required: 11)|
h2. Steps to reproduce
# Create a table with {{ZstdDictionaryCompressor}} and {{{}chunk_length_in_kb:
64{}}}.
# Write several GB so the table has many SSTables (LCS L0, TWCS, or any table
not yet fully compacted; ~387 SSTables / ~6.5 GB in this repro).
# {{nodetool compressiondictionary train <ks> <table>}}
*Expected:* the sampler collects a representative sample and a dictionary is
trained.
*Actual:*
{noformat}
Status: FAILED | Samples: 8 | Size: 0.22 MiB | Elapsed: 1s
Training failed for <ks>.<table>
Reason: Insufficient samples for training: 8 (minimum required: 11)
{noformat}
The identical dataset with {{chunk_length_in_kb: 4}} trains successfully (~2418
samples).
h2. Suggested fix
* In {{{}sampleChunksFromSSTables(){}}}, guarantee at least one chunk from
each suitable SSTable, bounded by the remaining budget — e.g. {{Math.max(1,
(targetChunkCount * info.chunkCount) / totalChunks)}} while still respecting
{{{}remainingTarget{}}}/{{{}maxTotalSampleSize{}}}.
* And/or floor {{targetChunkCount}} in {{calculateTargetChunkCount()}} at
{{max(MIN_SAMPLES, sstableCount)}} so it can never fall below what is needed to
reach the minimum.
* Separately, either make {{--force}} actually bypass the sample-count
minimum, or correct the message that recommends it.
h2. Environment
Cassandra 6.0-alpha-SNAPSHOT (trunk), single node.
was:
h2. Summary
{{nodetool compressiondictionary train}} fails with {{Insufficient samples for
training: N (minimum required: 11)}} on tables that have a large
{{chunk_length_in_kb}} and many SSTables, even when there are gigabytes of data
available to sample.
h2. Root cause
{{SSTableChunkSampler}}
({_}src/java/org/apache/cassandra/db/compression/SSTableChunkSampler.java{_})
allocates the number of chunks to sample from each SSTable using integer
division that truncates to zero for most SSTables when the sample target is
small.
{{{}calculateTargetChunkCount(){}}}:
{code:java}
int averageChunkSize = totalDataSize / totalChunks; // ~ chunk_length_in_kb
return config.maxTotalSampleSize / averageChunkSize; // target shrinks as
chunk size grows
{code}
With the default {{maxTotalSampleSize = 8 MiB}} and 64 KiB chunks:
{{{}targetChunkCount = 8 MiB / 64 KiB = 128{}}}.
{{{}sampleChunksFromSSTables(){}}}:
{code:java}
long chunksFromThisSSTable = Math.min((targetChunkCount * info.chunkCount) /
totalChunks, remainingTarget);
if (chunksFromThisSSTable <= 0)
continue; // <-- SSTable skipped entirely
{code}
{{(targetChunkCount * info.chunkCount) / totalChunks}} is integer division. An
SSTable contributes at least one chunk only when {{{}info.chunkCount >=
totalChunks / targetChunkCount{}}}. With {{targetChunkCount = 128}} and ~387
SSTables (~100k total chunks), that threshold is ~780 chunks (~50 MB) — so
every SSTable below it rounds to 0 and is skipped, leaving ~8 samples, under
the 11-sample minimum, and training fails.
Because {{targetChunkCount}} is 16x larger at 4 KiB chunks (2048), small-chunk
tables clear the threshold and are unaffected — which is why this only
manifests at larger {{{}chunk_length{}}}.
||chunk_length_in_kb||targetChunkCount||samples collected||result||
|4|2048|~2418|COMPLETED|
|64|128|8|FAILED (minimum required: 11)|
h2. Steps to reproduce
# Create a table with {{ZstdDictionaryCompressor}} and {{{}chunk_length_in_kb:
64{}}}.
# Write several GB so the table has many SSTables (LCS L0, TWCS, or any table
not yet fully compacted; ~387 SSTables / ~6.5 GB in this repro).
# {{nodetool compressiondictionary train <ks> <table>}}
*Expected:* the sampler collects a representative sample and a dictionary is
trained.
*Actual:*
{noformat}
Status: FAILED | Samples: 8 | Size: 0.22 MiB | Elapsed: 1s
Training failed for <ks>.<table>
Reason: Insufficient samples for training: 8 (minimum required: 11)
{noformat}
The identical dataset with {{chunk_length_in_kb: 4}} trains successfully (~2418
samples).
h2. Secondary issues
# {{{-}f/{-}-force}} does *not* bypass the 11-sample minimum (verified: it
still fails with {{{}minimum required: 11{}}}). The failure message {{Use
--force to train anyway}} is therefore misleading.
# The working operator workaround — {{--max-total-sample-size 128MiB}} (lifts
{{targetChunkCount}} to ~2048 -> 2040 samples -> {{{}COMPLETED{}}}) — is
non-obvious for what is a sampler defect.
# Any automatic/scheduled training built on this sampler inherits the bug and
will silently fail to (re)train large-chunk tables, with no error surfaced to
the operator.
h2. Suggested fix
* In {{{}sampleChunksFromSSTables(){}}}, guarantee at least one chunk from
each suitable SSTable, bounded by the remaining budget — e.g. {{Math.max(1,
(targetChunkCount * info.chunkCount) / totalChunks)}} while still respecting
{{{}remainingTarget{}}}/{{{}maxTotalSampleSize{}}}.
* And/or floor {{targetChunkCount}} in {{calculateTargetChunkCount()}} at
{{max(MIN_SAMPLES, sstableCount)}} so it can never fall below what is needed to
reach the minimum.
* Separately, either make {{--force}} actually bypass the sample-count
minimum, or correct the message that recommends it.
h2. Environment
Cassandra 6.0-alpha-SNAPSHOT (trunk), single node.
> Compression dictionary training fails with "insufficient samples" on
> large-chunk tables (SSTableChunkSampler per-SSTable allocation rounds to zero)
> ---------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21666
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21666
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/Compression
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 6.0-alpha3, 7.x
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h2. Summary
> {{nodetool compressiondictionary train}} fails with {{Insufficient samples
> for training: N (minimum required: 11)}} on tables that have a large
> {{chunk_length_in_kb}} and many SSTables, even when there are gigabytes of
> data available to sample.
> h2. Root cause
> {{SSTableChunkSampler}}
> ({_}src/java/org/apache/cassandra/db/compression/SSTableChunkSampler.java{_})
> allocates the number of chunks to sample from each SSTable using integer
> division that truncates to zero for most SSTables when the sample target is
> small.
> {{{}calculateTargetChunkCount(){}}}:
> {code:java}
> int averageChunkSize = totalDataSize / totalChunks; // ~
> chunk_length_in_kb
> return config.maxTotalSampleSize / averageChunkSize; // target shrinks as
> chunk size grows
> {code}
> With the default {{maxTotalSampleSize = 8 MiB}} and 64 KiB chunks:
> {{{}targetChunkCount = 8 MiB / 64 KiB = 128{}}}.
> {{{}sampleChunksFromSSTables(){}}}:
> {code:java}
> long chunksFromThisSSTable = Math.min((targetChunkCount * info.chunkCount) /
> totalChunks, remainingTarget);
> if (chunksFromThisSSTable <= 0)
> continue; // <-- SSTable skipped entirely
> {code}
> {{(targetChunkCount * info.chunkCount) / totalChunks}} is integer division.
> An SSTable contributes at least one chunk only when {{{}info.chunkCount >=
> totalChunks / targetChunkCount{}}}. With {{targetChunkCount = 128}} and ~387
> SSTables (~100k total chunks), that threshold is ~780 chunks (~50 MB) — so
> every SSTable below it rounds to 0 and is skipped, leaving ~8 samples, under
> the 11-sample minimum, and training fails.
> Because {{targetChunkCount}} is 16x larger at 4 KiB chunks (2048),
> small-chunk tables clear the threshold and are unaffected — which is why this
> only manifests at larger {{{}chunk_length{}}}.
> ||chunk_length_in_kb||targetChunkCount||samples collected||result||
> |4|2048|~2418|COMPLETED|
> |64|128|8|FAILED (minimum required: 11)|
> h2. Steps to reproduce
> # Create a table with {{ZstdDictionaryCompressor}} and
> {{{}chunk_length_in_kb: 64{}}}.
> # Write several GB so the table has many SSTables (LCS L0, TWCS, or any
> table not yet fully compacted; ~387 SSTables / ~6.5 GB in this repro).
> # {{nodetool compressiondictionary train <ks> <table>}}
> *Expected:* the sampler collects a representative sample and a dictionary is
> trained.
> *Actual:*
> {noformat}
> Status: FAILED | Samples: 8 | Size: 0.22 MiB | Elapsed: 1s
> Training failed for <ks>.<table>
> Reason: Insufficient samples for training: 8 (minimum required: 11)
> {noformat}
> The identical dataset with {{chunk_length_in_kb: 4}} trains successfully
> (~2418 samples).
> h2. Suggested fix
> * In {{{}sampleChunksFromSSTables(){}}}, guarantee at least one chunk from
> each suitable SSTable, bounded by the remaining budget — e.g. {{Math.max(1,
> (targetChunkCount * info.chunkCount) / totalChunks)}} while still respecting
> {{{}remainingTarget{}}}/{{{}maxTotalSampleSize{}}}.
> * And/or floor {{targetChunkCount}} in {{calculateTargetChunkCount()}} at
> {{max(MIN_SAMPLES, sstableCount)}} so it can never fall below what is needed
> to reach the minimum.
> * Separately, either make {{--force}} actually bypass the sample-count
> minimum, or correct the message that recommends it.
> h2. Environment
> Cassandra 6.0-alpha-SNAPSHOT (trunk), single node.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
