Jon Haddad created CASSANDRA-21667:
--------------------------------------
Summary: Reduce allocation from metrics collection
Key: CASSANDRA-21667
URL: https://issues.apache.org/jira/browse/CASSANDRA-21667
Project: Apache Cassandra
Issue Type: Improvement
Reporter: Jon Haddad
Assignee: Jon Haddad
The {{EstimatedPartitionCount}} table gauge recomputes its value from scratch
every time it is read. Each read opens and deserialises the {{Statistics.db}}
component of every live SSTable in the table, builds a HyperLogLog++ estimator
from each one, and merges them. Nothing is cached.
A metrics agent scrapes this gauge on every collection interval. {{nodetool
tablestats}} reads it as well. The work is therefore repeated at the scrape
rate, per table, for the life of the node.
The inputs are immutable. An SSTable's cardinality estimator never changes
after the file is written. The merged value can only change when the SSTable
set changes.
{{TableMetrics}} registers the gauge:
{code:java}
estimatedPartitionCount = createTableGauge("EstimatedPartitionCount",
"EstimatedRowCount", new Gauge<Long>()
{
public Long getValue()
{
long memtablePartitions = 0;
for (Memtable memtable : cfs.getTracker().getView().getAllMemtables())
memtablePartitions += memtable.partitionCount();
try(ColumnFamilyStore.RefViewFragment refViewFragment =
cfs.selectAndReference(View.selectFunction(SSTableSet.CANONICAL)))
{
return
SSTableReader.getApproximateKeyCount(refViewFragment.sstables) +
memtablePartitions;
}
}
}, null);
{code}
{{SSTableReader.getApproximateKeyCount}} then does this per SSTable:
{code:java}
CompactionMetadata metadata =
StatsComponent.load(sstable.descriptor).compactionMetadata();
...
if (cardinality == null)
cardinality = metadata.cardinalityEstimator;
else
cardinality = cardinality.merge(metadata.cardinalityEstimator);
{code}
Three costs come out of those two lines.
* {{StatsComponent.load(descriptor)}} passes {{MetadataType.values()}}. It
deserialises {{VALIDATION}}, {{COMPACTION}}, {{STATS}} and {{HEADER}}. Only
{{COMPACTION}} is used here. The other three are decoded and discarded.
* Every call reopens the file. {{RandomAccessReader.open}} is called per
SSTable, per scrape.
* {{ICardinality.merge}} allocates a whole new estimator per SSTable and copies
the running one into it.
The per-table overhead was measured with JMH, {{AverageTime}} mode plus the
{{gc}} profiler, on a table with 1000 partitions per SSTable and
auto-compaction disabled. The benchmark reads the gauge and reports the cost
of one call.
||SSTables in the table||Time per read||Heap allocated per read||
|1|~49 us|~79 KiB|
|10|~0.49 ms|~0.79 MiB|
|100|4.9 ms|7.9 MiB|
The cost is linear in the SSTable count. It does not depend on the partition
count, because the estimator is a fixed-width structure once it leaves sparse
mode.
Per SSTable, per scrape: *~49 us of CPU and ~79 KiB of garbage*.
That ~79 KiB divides into two parts.
*Cardinality estimator: ~29 KiB per SSTable.* Measured separately with
{{ThreadMXBean.getThreadAllocatedBytes}} over a deserialise-and-merge loop, at
the precision Cassandra uses ({{new HyperLogLogPlus(13, 25)}}). A dense
estimator at that precision serialises to 5473 bytes. Three copies of it exist
per SSTable during the merge:
* the {{byte[]}} read out of {{Statistics.db}}, ~5.5 KiB;
* the register set of the deserialised estimator, {{int[1366]}}, ~5.5 KiB;
* the fresh estimator that {{merge()}} allocates to hold the result, ~5.5 KiB.
The rest is temporaries inside the deserialise and the merge.
*The remaining metadata: ~50 KiB per SSTable.* This is the {{VALIDATION}},
{{STATS}} and {{HEADER}} components that the caller never reads.
{{StatsMetadata}} alone carries two {{EstimatedHistogram}} instances, a
tombstone histogram, an {{IntervalSet}} of commit log positions, the covered
clustering slice, the first and last keys and the encoding stats.
{{SerializationHeader.Component}} carries the column maps. All of it is built
and dropped.
At the node level the arithmetic is {{tables x SSTables per table x 79 KiB /
scrape interval}}.
||Node||Garbage per scrape||At a 10 s scrape interval||
|100 tables, 20 SSTables each|~154 MiB|~15 MiB/s|
|200 tables, 50 SSTables each|~790 MiB|~79 MiB/s|
|500 tables, 50 SSTables each|~1.9 GiB|~193 MiB/s|
This is short-lived garbage, so it lands in the young generation. It still
sets a floor on the young collection rate that no workload asked for. On a
node with many tables the scrape alone can dominate the allocation rate at idle.
The CPU cost lands on the JMX or agent thread, not on a read path. The
allocation cost lands on every thread, because it drives the collector.
Two more per-scrape costs sit next to this one. Both read data that changes
far less often than the scrape rate.
* {{SecondaryIndexManager.getAllIndexColumnFamilyStores()}} builds a new
{{HashSet}} and walks every index on every call. The backing tables only
change when an index is registered or dropped. {{TableMetrics}} reads it per
scrape, and {{ColumnFamilyStore.concatWithIndexes()}} reads it on the flush and
compaction paths.
* {{CompactionStrategyManager.getEstimatedRemainingTasks()}} calls
{{getAllStrategies()}}, which builds a concat and a transform chain per call,
to sum a handful of integers. It backs the {{PendingCompactions}} gauge.
A per-table cache of the merged partition count should remove most of this.
The value is a pure function of the SSTable set, so the cache holds until that
set changes, and it must be invalidated whenever the SSTables change for any
reason: flush, compaction, streaming, bulk load, index build, cleanup,
secondary index rebuild, and any other lifecycle transaction. A cache that
only watches compaction is wrong.
With that in place a scrape reads a cached long instead of touching the file
system, and the ~79 KiB per SSTable per scrape becomes ~79 KiB per SSTable set
change.
A JMH benchmark that measures both the cached and the uncached path is on the
{{metrics-gauge-caching}} branch as {{EstimatedPartitionCountBench}}. It
parameterises on the SSTable count.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
