It may help with some, but it's compaction and memtable flushes that generate the most IO. You also run the risk of not having data fully committed to disk if something bad were to happen. It might be ok for time series data that you can afford to lose, but not for mission critical things.
On Fri, Feb 21, 2025 at 11:25 PM Long Pan <panlong...@gmail.com> wrote: > > Perhaps a basic question: In the context of a remote disk, wouldn't it make > even more sense to use commitlog_sync = periodic rather than batch? Since > periodic decouples disk I/O from client write latency, it seems better suited > for mitigating the additional overhead of remote storage. > > On Fri, Feb 21, 2025 at 8:25 PM Long Pan <panlong...@gmail.com> wrote: >> >> Thank you all very much, Guo, Patrick and Jon! I will take a close look at >> the resources you are sharing. >> >> On Thu, Feb 20, 2025 at 10:06 AM Patrick McFadin <pmcfa...@gmail.com> wrote: >>> >>> I'll give you the general guidance around any type of storage you >>> pick. This even applies to local disks but it will directly apply to >>> your question. >>> >>> The key to success with storage and Cassandra is sequential, >>> concurrent IO. Most of the large IO operations are either writing and >>> reading a large file from disk. Sometimes, and in the harder case to >>> manage, at the same time. Storage systems that bias more to reads or >>> writes will create an imbalance that can lead to issues. And worth >>> emphasizing. These are sequential reads and writes. IOPs are mostly >>> irrelevant. The second aspect to manage is latency. Latency from disk >>> directly correlates to query performance. >>> >>> With respect to remote storage, they tend to have more issues with >>> these requirements. NFS, for example, has far too much latency and >>> concurrency. Just don't use it. The best thing you can do when looking >>> at choices is run some simple tests. Another Jon Haddad resource but >>> great: https://www.youtube.com/watch?v=dPpEORxoMRU You don't even need >>> to run Cassandra in the test. Just do some IO testing and verify that >>> it can read and write in a balanced manner, observe the latency and >>> watch for any IOWait that creeps up. >>> >>> If you have a specific technology combination, just ask here. >>> Collectively we have probably seen it all. >>> >>> Patrick >>> >>> On Wed, Feb 19, 2025 at 10:27 PM Long Pan <panlong...@gmail.com> wrote: >>> > >>> > Hi Cassandra Community, >>> > >>> > I’m exploring the feasibility of running Cassandra with remote storage, >>> > primarily block storage (e.g., AWS EBS, OCI Block Volume, Google >>> > Persistent Disk) and possibly even file storage (e.g., NFS, EFS, FSx). >>> > While local SSDs are the typical recommendation for optimal performance, >>> > I’d like to understand if anyone has experience or insights on using >>> > remote disks in production. >>> > >>> > Specifically, I’m looking for guidance on: >>> > >>> > Feasibility – Has anyone successfully run Cassandra with remote storage? >>> > If so, what use cases worked well? >>> > Major Downsides & Caveats – Are there any known performance bottlenecks, >>> > consistency issues? >>> > Configuration Tuning – Are there any special settings (e.g., compaction, >>> > memtable flush thresholds, disk I/O tuning) that can help mitigate >>> > potential drawbacks? >>> > Monitoring & Alerting – What are the key metrics and failure scenarios to >>> > watch out for when using remote storage? >>> > >>> > I’d appreciate any insights, war stories, or best practices from those >>> > who have experimented with or deployed Cassandra on remote storage. >>> > >>> > Thanks, >>> > Long Pan
