[
https://issues.apache.org/jira/browse/CASSANDRA-15452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821919#comment-17821919
]
Jon Haddad edited comment on CASSANDRA-15452 at 2/29/24 2:34 AM:
-----------------------------------------------------------------
I recently ran some benchmarks on EBS with fio, testing sequential access, to
get some hard numbers around this problem with the disk configured at 3K IOPS
and 125MB throughput. Here's a summary of what I'm seeing:
||Request Size||IOPS||Throughput||
|4K|3049|11.9 MB/s|
|8K|3012|23 MB/s|
|16K|3013|47 MB/s|
|32K|3013|94 MB/s|
|64K|1938|121 MB/s|
|128K|957|120 MB/s|
|256K|478|120 MB/s|
Consistent with the documentation, we see that small reads don't perform well
at all, throughput wise, as they hit their IOPS limit early on. Using larger
reads allows us to achieve higher throughput at the same number of I/O
operations.
EBS allows you to perform an I/O operation up to 256KB [(docs
here)|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html#ebs-io-iops]
so we should try to ensure we don't waste IOPS by performing small reads.
was (Author: rustyrazorblade):
I recently ran some benchmarks on EBS with fio, testing sequential access, to
get some hard numbers around this problem with the disk configured at 3K IOPS
and 125MB throughput. Here's a summary of what I'm seeing:
||Request Size||IOPS||Throughput||
|4K|3049|11.9 MB/s|
|8K|3012|23 MB/s|
|16K|3013|47 MB/s|
|32K|3013|94 MB/s|
|64K|1938|121 MB/s|
|128K|957|120 MB/s|
|256K|478|120 MB/s|
EBS allows you to perform an I/O operation up to 256KB [(docs
here),|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html#ebs-io-iops]),]
so we should try to ensure we don't waste IOPS by performing small reads.
> Improve disk access patterns during compaction compaction
> ---------------------------------------------------------
>
> Key: CASSANDRA-15452
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15452
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Local Write-Read Paths, Local/Compaction
> Reporter: Jon Haddad
> Priority: Normal
> Attachments: results.txt, sequential.fio
>
>
> On read heavy workloads Cassandra performs much better when using a low read
> ahead setting. In my tests I've seen an 5x improvement in throughput and
> more than a 50% reduction in latency. However, I've also observed that it
> can have a negative impact on compaction and streaming throughput. It
> especially negatively impacts cloud environments where small reads incur high
> costs in IOPS due to tiny requests.
> # We should investigate using POSIX_FADV_DONTNEED on files we're compacting
> to see if we can improve performance and reduce page faults.
> # This should be combined with an internal read ahead style buffer that
> Cassandra manages, similar to a BufferedInputStream but with our own
> machinery. This buffer should read fairly large blocks of data off disk at
> at time. EBS, for example, allows 1 IOP to be up to 256KB. A considerable
> amount of time is spent in blocking I/O during compaction and streaming.
> Reducing the frequency we read from disk should speed up all sequential I/O
> operations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
