[
https://issues.apache.org/jira/browse/CASSANDRA-19979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jon Haddad updated CASSANDRA-19979:
-----------------------------------
Summary: Use larger internal buffers on streaming slow path (was: Use
internal buffers on streaming slow path)
> Use larger internal buffers on streaming slow path
> --------------------------------------------------
>
> Key: CASSANDRA-19979
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19979
> Project: Cassandra
> Issue Type: Improvement
> Reporter: Jon Haddad
> Priority: Normal
> Attachments: image-2024-10-04-12-40-26-727.png
>
>
> CASSANDRA-15452 is introducing an internal buffer to compaction in order to
> increase throughput while reducing IOPS. We do the same thing with our
> streaming slow path, although it's not optimal. There's a common
> misconception that the overhead comes from serde overhead, but I've found on
> a lot of devices the overhead is due to our read patterns. This is most
> commonly found on non-NVMe drives, especially disaggregated storage such as
> EBS where the latency is higher and more variable.
> Attached is a perf profile showing the cost of streaming is dominated by
> pread. The team I was working with was seeing they could stream only 12MB
> per streaming session. Reducing the number of read operations by using
> larger buffered reads should improve this by at least 3-5x on some systems,
> as well as reduce CPU overhead from reduced system calls.
> I think we need to do a few things:
> * Use a larger internal buffer on disk reads.
> * Buffer writes to the network. Writing constant small values to the
> network has a very high latency cost, we'd be better off flushing larger
> values more often
> * Move the blocking network part to a separate thread. We don't need to
> wait on the network transfer in order to read more data off disk. Once we
> improve the internal buffer on reads I think we'll see this as the next
> problem so let's tackle it now.
>
>
>
>
>
>
>
>
> !image-2024-10-04-12-40-26-727.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
