[ https://issues.apache.org/jira/browse/CASSANDRA-15452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17844826#comment-17844826 ]
Jordan West commented on CASSANDRA-15452: ----------------------------------------- I wrote a [prototype|https://github.com/jrwest/cassandra/tree/jwest/15452-5.0] that uses a 256KB off-heap buffer for non-mmap'd, compressed data files only and tested it with [~rustyrazorblade]. The test uses "big" format. We have yet to validate with BTI format. *To summarize our findings when running a single compaction:* * *on NVMe: equal throughput with an approx 5x reduction in IOPS* * *on EBS: 3x throughput with an approx 5x reduction in IOPS* Attached are graphs of the iostat data and the raw iostat output from the EBS tests. *EBS (patched):* !iostat-ebs-15452.png! *EBS (5.0-HEAD):* !iostat-ebs-head.png! *NVMe (patched):* !iostat-instance-15452.png! *NVMe (5.0-HEAD):* !iostat-instance-head.png! [^iostat-5.0-head.output] [^iostat-5.0-patched.output] > Improve disk access patterns during compaction and streaming > ------------------------------------------------------------ > > Key: CASSANDRA-15452 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15452 > Project: Cassandra > Issue Type: Improvement > Components: Legacy/Local Write-Read Paths, Local/Compaction > Reporter: Jon Haddad > Assignee: Jordan West > Priority: Normal > Attachments: everyfs.txt, iostat-5.0-head.output, > iostat-5.0-patched.output, iostat-ebs-15452.png, iostat-ebs-head.png, > iostat-instance-15452.png, iostat-instance-head.png, results.txt, > sequential.fio > > > On read heavy workloads Cassandra performs much better when using a low read > ahead setting. In my tests I've seen an 5x improvement in throughput and > more than a 50% reduction in latency. However, I've also observed that it > can have a negative impact on compaction and streaming throughput. It > especially negatively impacts cloud environments where small reads incur high > costs in IOPS due to tiny requests. > # We should investigate using POSIX_FADV_DONTNEED on files we're compacting > to see if we can improve performance and reduce page faults. > # This should be combined with an internal read ahead style buffer that > Cassandra manages, similar to a BufferedInputStream but with our own > machinery. This buffer should read fairly large blocks of data off disk at > at time. EBS, for example, allows 1 IOP to be up to 256KB. A considerable > amount of time is spent in blocking I/O during compaction and streaming. > Reducing the frequency we read from disk should speed up all sequential I/O > operations. > # We can reduce system calls by buffering writes as well, but I think it > will have less of an impact than the reads -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org