[
https://issues.apache.org/jira/browse/CASSANDRA-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Lightfoot updated CASSANDRA-21487:
--------------------------------------
Description:
h2. What's wrong
{{trickle_fsync: true}} is meant to fsync a large sequential write at
intervals, so a flush trickles its dirty pages to disk instead of dumping them
all at once and spiking read latency. On compressed SSTables it does nothing.
Compression is the default, so the setting has been silently ignored on the
data file of every flush and every buffered compaction. No error, no log line -
it just doesn't work.
h2. Why
The compressed writer throws the setting away.
{{CompressedSequentialWriter.buildOption()}} rebuilds the writer's options and
copies only three of them:
{code:java}
return SequentialWriterOption.newBuilder()
.bufferSize(parameters.chunkLength())
.bufferType(parameters.getSstableCompressor().preferredBufferType())
.finishOnClose(option.finishOnClose()) //
trickleFsync + interval lost here
.build();
{code}
{{trickleFsync}} and {{trickleFsyncByteInterval}} never make it through, so the
rebuilt option defaults to {{{}trickleFsync = false{}}}. The trickle branch in
{{SequentialWriter.doFlush}}
({{{}if (option.trickleFsync()) …{}}}) never runs, and no interval fsync ever
fires.
{{IOOptions}} and {{BigTableWriter}} set the options correctly; only the
compressed writer drops them. Commit {{fb221095cb}} (June 2016) moved trickle
config into the passed-in option but forgot this path. It has been broken ever
since.
h2. What still works
The primary index (plain {{{}SequentialWriter{}}}) and uncompressed data files
({{{}ChecksummedSequentialWriter{}}}) pass the option through untouched, so
trickle works there. But the index is tiny; the data file is where the bytes
and the writeback storm are.
h2. Fix
Pass the two fields through:
{code:java}
.trickleFsync(option.trickleFsync())
.trickleFsyncByteInterval(option.trickleFsyncByteInterval())
{code}
The interval counter in {{doFlush}} already tracks uncompressed bytes
correctly; only the enable flag was missing.
h2. How it was found
Found while benchmarking flush write-pacing: {{trickle_fsync}} on and off gave
an identical dirty window, which led here. Fix this on its own. The proposed
\{{trickle_fsync_mode: writeback }}improvement is separate work that adds a
third field to propagate through the same method.
was:
h2. What's wrong
{{trickle_fsync: true}} is meant to fsync a large sequential write at
intervals, so a flush
trickles its dirty pages to disk instead of dumping them all at once and
spiking read latency.
On compressed SSTables it does nothing. Compression is the default, so the
setting has been silently ignored on the data file of every flush and every
buffered compaction. No error, no log line - it just doesn't work.
h2. Why
The compressed writer throws the setting away.
{{CompressedSequentialWriter.buildOption()}} rebuilds the writer's options and
copies only three of them:
{code:java}
return SequentialWriterOption.newBuilder()
.bufferSize(parameters.chunkLength())
.bufferType(parameters.getSstableCompressor().preferredBufferType())
.finishOnClose(option.finishOnClose()) //
trickleFsync + interval lost here
.build();
{code}
{{trickleFsync}} and {{trickleFsyncByteInterval}} never make it through, so the
rebuilt option defaults to {{{}trickleFsync = false{}}}. The trickle branch in
{{SequentialWriter.doFlush}}
({{{}if (option.trickleFsync()) …{}}}) never runs, and no interval fsync ever
fires.
{{IOOptions}} and {{BigTableWriter}} set the options correctly; only the
compressed writer drops them. Commit {{fb221095cb}} (June 2016) moved trickle
config into the passed-in option but forgot this path. It has been broken ever
since.
h2. What still works
The primary index (plain {{{}SequentialWriter{}}}) and uncompressed data files
({{{}ChecksummedSequentialWriter{}}}) pass the option through untouched, so
trickle works there. But the index is tiny; the data file is where the bytes
and the writeback storm are.
h2. Fix
Pass the two fields through:
{code:java}
.trickleFsync(option.trickleFsync())
.trickleFsyncByteInterval(option.trickleFsyncByteInterval())
{code}
The interval counter in {{doFlush}} already tracks uncompressed bytes
correctly; only the enable flag was missing.
h2. How it was found
Found while benchmarking flush write-pacing: {{trickle_fsync}} on and off gave
an identical dirty window, which led here. Fix this on its own. The proposed
{{trickle_fsync_mode: writeback }}improvement is separate work that adds a
third field to propagate through the same method.
> trickle_fsync is silently ignored for compressed SSTables
> ---------------------------------------------------------
>
> Key: CASSANDRA-21487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21487
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Sam Lightfoot
> Assignee: Sam Lightfoot
> Priority: Normal
>
> h2. What's wrong
> {{trickle_fsync: true}} is meant to fsync a large sequential write at
> intervals, so a flush trickles its dirty pages to disk instead of dumping
> them all at once and spiking read latency. On compressed SSTables it does
> nothing. Compression is the default, so the setting has been silently ignored
> on the data file of every flush and every buffered compaction. No error, no
> log line - it just doesn't work.
> h2. Why
> The compressed writer throws the setting away.
> {{CompressedSequentialWriter.buildOption()}} rebuilds the writer's options
> and copies only three of them:
> {code:java}
> return SequentialWriterOption.newBuilder()
> .bufferSize(parameters.chunkLength())
>
> .bufferType(parameters.getSstableCompressor().preferredBufferType())
> .finishOnClose(option.finishOnClose()) //
> trickleFsync + interval lost here
> .build();
> {code}
> {{trickleFsync}} and {{trickleFsyncByteInterval}} never make it through, so
> the rebuilt option defaults to {{{}trickleFsync = false{}}}. The trickle
> branch in {{SequentialWriter.doFlush}}
> ({{{}if (option.trickleFsync()) …{}}}) never runs, and no interval fsync ever
> fires.
> {{IOOptions}} and {{BigTableWriter}} set the options correctly; only the
> compressed writer drops them. Commit {{fb221095cb}} (June 2016) moved trickle
> config into the passed-in option but forgot this path. It has been broken
> ever since.
> h2. What still works
> The primary index (plain {{{}SequentialWriter{}}}) and uncompressed data
> files ({{{}ChecksummedSequentialWriter{}}}) pass the option through
> untouched, so trickle works there. But the index is tiny; the data file is
> where the bytes and the writeback storm are.
> h2. Fix
> Pass the two fields through:
> {code:java}
> .trickleFsync(option.trickleFsync())
> .trickleFsyncByteInterval(option.trickleFsyncByteInterval())
> {code}
> The interval counter in {{doFlush}} already tracks uncompressed bytes
> correctly; only the enable flag was missing.
> h2. How it was found
> Found while benchmarking flush write-pacing: {{trickle_fsync}} on and off
> gave an identical dirty window, which led here. Fix this on its own. The
> proposed \{{trickle_fsync_mode: writeback }}improvement is separate work that
> adds a third field to propagate through the same method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
