[
https://issues.apache.org/jira/browse/CASSANDRA-21615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118882#comment-18118882
]
Johnny Miller edited comment on CASSANDRA-21615 at 9/24/26 4:13 PM:
--------------------------------------------------------------------
[~mck] [~brandon.williams] Ive rebased that commit onto cassandra-5.0 branch (
[https://github.com/apache/cassandra/compare/cassandra-5.0...axonops:cassandra:millerjp/CASSANDRA-21615/cassandra-5.0]
) - no PR yet.
I think I have done teh original author etc.. stuff ok and I only changed what
would not compile on 5.0 (there is no CompactionRealm so the partition count
comes from TableMetrics via ShardManager.create).
All the existing compaction unit tests and the UCS dtests pass with it.
I wrote some additional dtests in my previous attempt doing repairs (to
replicate the problem before fixing) and when I ran them against this I found
another bug - this commit only works on the bti format (which has been the
default at datastax for a while). It breaks on big (default on 5.0)
How do you want to handle this?
* Should I sumbit this PR now and open a separate ticket for the big problem
(once I find a fix) or try and fix big in the same PR? Should we just say UCS
is only supported on bti?
* Do you want my additional repair dtests that found the bug included in the
pr?
* Also, do you want me to try and get this into trunk and/or 6.0?
was (Author: johnny15676):
[~mck] Ive rebased the commit onto cassandra-5.0 branch (
[https://github.com/apache/cassandra/compare/cassandra-5.0...axonops:cassandra:millerjp/CASSANDRA-21615/cassandra-5.0]
) - no PR yet.
I think I have done teh original author etc.. stuff ok and I only changed what
would not compile on 5.0 (there is no CompactionRealm so the partition count
comes from TableMetrics via ShardManager.create).
All the existing compaction unit tests and the UCS dtests pass with it.
I wrote some additional dtests in my previous attempt doing repairs (to
replicate the problem before fixing) and when I ran them against this I found
another bug - this commit only works on the bti format (which has been the
default at datastax for a while). It breaks on big (default on 5.0)
How do you want to handle this?
* Should I sumbit this PR now and open a separate ticket for the big problem
(once I find a fix) or try and fix big in the same PR? Should we just say UCS
is only supported on bti?
* Do you want my additional repair dtests that found the bug included in the
pr?
* Also, do you want me to try and get this into trunk and/or 6.0?
> UCS never compacts tiny high-density SSTables streamed by repair
> ----------------------------------------------------------------
>
> Key: CASSANDRA-21615
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21615
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction/UCS
> Reporter: Hayato Shimizu
> Assignee: Johnny Miller
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> Cassandra Version: 5.0.8
> This was observed on a production cluster, and it took a while to rule out
> misconfiguration before it became clear that UCS has no configuration that
> will compact these repair-streamed SSTables.
> On tables that receive frequent repair streams, UCS accumulates a very large
> number of tiny SSTables that never get compacted. The live SSTable count
> climbs into the thousands per node while total disk usage barely increases,
> because each streamed file is only hundreds of bytes to a few KB.
> We could see in the metrics the SSTable count increased only during repair
> activity and did not decrease afterwards, and pending compactions stayed near
> zero throughout. So this was definitely not a compaction backlog. UCS did not
> appear to be selecting these SSTables for compaction in the first place.
> The following was seen on a production node, via {{{}sstablemetadata{}}}:
> * a file of a few hundred bytes, local token space coverage on the order of
> 1e-11, so density (size ÷ coverage) on the order of 1e13
> * a file of a few hundred bytes, coverage on the order of 1e-10, density on
> the order of 1e12
> * a file of a few thousand bytes, coverage on the order of 1e-11, density on
> the order of 1e14
> At these densities each file was assigned to a very high level, far above the
> table's normal SSTables. Each covered such a narrow token range that it
> intersected no other SSTable in its level, so no overlap set of size >= 2
> ever formed and the threshold was never met.
> Lowering the threshold was the obvious first thing to try, but unfortunately
> it did not help. Setting scaling_parameters to 'T4, T4, N' (threshold 2 in
> the upper levels), and separately updated min_sstable_size to 0MiB, then only
> a handful of files whose ranges happened to intersect compacted, so the table
> stayed at a very large SSTable count. This makes sense since a threshold of 2
> does nothing when nothing intersects.
> overlap_inclusion_method was on its default of TRANSITIVE throughout, so
> transitive inclusion was already in effect and did not pull these files in.
> As far as we can tell that's because they are denser than the table's larger
> SSTables, so a lower-level compaction never seems to include them, though we
> have not confirmed this in the code.
> Checked JIRA before filing and didn't find this — CASSANDRA-16552 and 16638
> are the closest but they're about the anticompaction race under incremental
> repair, which is different (this is fully unrepaired, no anticompaction).
> We'd like to know whether the above explanation is correct, whether this is a
> bug or a known limitation of UCS, and whether there's a recommended way to
> handle repair-streamed SSTables like these on UCS tables in the meantime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
