Jon Haddad created CASSANDRA-21655:
--------------------------------------
Summary: async compaction pipeline
Key: CASSANDRA-21655
URL: https://issues.apache.org/jira/browse/CASSANDRA-21655
Project: Apache Cassandra
Issue Type: Improvement
Reporter: Jon Haddad
Compaction compresses each chunk, writes it, and updates its CRC serially on
the compaction thread, inside \{{SequentialWriter.doFlush}}. A CPU profile of
a 4.5 GiB compaction puts that write side at 37.2% of the total, so the thread
that merges rows spends over a third of its time not merging.
\{{trickle_fsync}} compounds it: the 10 MiB byte interval fires about 450 times
over that compaction, and each \{{fdatasync}} forces the whole growing file
rather than a range.
{noformat}
compaction, 100%
|<---------------- merge side, 62.8% ---------------->|<---- doFlush, 37.2%
---->|
|<-LZ4 21.1%->|<CRC
7.6%>|io|
8.5%
{noformat}
Compression plus CRC is 77% of the write side. Moving the whole 37.2% off the
compaction thread makes the merge side the slowest stage, a ceiling of 1.6x.
The patch gives each SSTable writer its own writer thread and hands chunks to
it through a pool of reusable chunk-sized off-heap slots, 4 MiB by default; the
producer takes a fresh slot instead of waiting for the one it just filled.
Compression, the write and the checksum all run on that thread. The
byte-interval fsync is replaced by a periodic force, about six per compaction
instead of hundreds, which removes work rather than merely overlapping it. The
pipeline is a collaborator both \{{CompressedSequentialWriter}} and
\{{DirectCompressedSequentialWriter}} own, so direct IO gets it too, and
\{{DataComponent.buildWriter}} routes flush, streaming and index builds through
the same path. It sits behind \{{async_compaction_writer_enabled}}.
Measured on a 4543 MiB dataset across 16 SSTables, 16 cores, JDK 21, throttling
off:
||path||before||after||change||
|cursor|15809 ms, 287 MiB/s|5952 ms, 763 MiB/s|+166%|
|iterator|18476 ms, 246 MiB/s|9129 ms, 498 Mi
The result beats the 1.6x ceiling because thatotal; the redundant fsyncs were
part of it,and removing them shrank the work itself.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
