Hayato Shimizu created CASSANDRA-21615:
------------------------------------------
Summary: UCS never compacts tiny high-density SSTables streamed by
repair
Key: CASSANDRA-21615
URL: https://issues.apache.org/jira/browse/CASSANDRA-21615
Project: Apache Cassandra
Issue Type: Bug
Components: Local/Compaction/UCS
Reporter: Hayato Shimizu
Cassandra Version: 5.0.8
This was observed on a production cluster, and it took a while to rule out
misconfiguration before it became clear that UCS has no configuration that will
compact these repair-streamed SSTables.
On tables that receive frequent repair streams, UCS accumulates a very large
number of tiny SSTables that never get compacted. The live SSTable count climbs
into the thousands per node while total disk usage barely increases, because
each streamed file is only hundreds of bytes to a few KB.
We could see in the metrics the SSTable count increased only during repair
activity and did not decrease afterwards, and pending compactions stayed near
zero throughout. So this was definitely not a compaction backlog. UCS did not
appear to be selecting these SSTables for compaction in the first place.
The following was seen on a production node, via {{{}sstablemetadata{}}}:
* a file of a few hundred bytes, local token space coverage on the order of
1e-11, so density (size ÷ coverage) on the order of 1e13
* a file of a few hundred bytes, coverage on the order of 1e-10, density on
the order of 1e12
* a file of a few thousand bytes, coverage on the order of 1e-11, density on
the order of 1e14
At these densities each file was assigned to a very high level, far above the
table's normal SSTables. Each covered such a narrow token range that it
intersected no other SSTable in its level, so no overlap set of size >= 2 ever
formed and the threshold was never met.
Lowering the threshold was the obvious first thing to try, but unfortunately it
did not help. Setting scaling_parameters to 'T4, T4, N' (threshold 2 in the
upper levels), and separately updated min_sstable_size to 0MiB, then only a
handful of files whose ranges happened to intersect compacted, so the table
stayed at a very large SSTable count. This makes sense since a threshold of 2
does nothing when nothing intersects.
overlap_inclusion_method was on its default of TRANSITIVE throughout, so
transitive inclusion was already in effect and did not pull these files in. As
far as we can tell that's because they are denser than the table's larger
SSTables, so a lower-level compaction never seems to include them, though we
have not confirmed this in the code.
Checked JIRA before filing and didn't find this — CASSANDRA-16552 and 16638 are
the closest but they're about the anticompaction race under incremental repair,
which is different (this is fully unrepaired, no anticompaction).
We'd like to know whether the above explanation is correct, whether this is a
bug or a known limitation of UCS, and whether there's a recommended way to
handle repair-streamed SSTables like these on UCS tables in the meantime.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
