Thanks for the explanation. I tried again with commitlog_sync_group_window at 2ms, concurrent_writes at 512, and doing 1000 individual inserts at a time with the same loop + semaphore approach. This only nets 9k / second.
I got much higher throughput for the other modes with BatchStatement of 100 inserts rather than 100x more individual inserts. On Tue, Apr 23, 2024 at 10:45 AM Bowen Song via user < user@cassandra.apache.org> wrote: > I suspect you are abusing batch statements. Batch statements should only > be used where atomicity or isolation is needed. Using batch statements > won't make inserting multiple partitions faster. In fact, it often will > make that slower. > > Also, the liner relationship between commitlog_sync_group_window and > write throughput is expected. That's because the max number of uncompleted > writes is limited by the write concurrency, and a write is not considered > "complete" before it is synced to disk when commitlog sync is in group or > batch mode. That means within each interval, only limited number of writes > can be done. The ways to increase that including: add more nodes, sync > commitlog at shorter intervals and allow more concurrent writes. > > > On 23/04/2024 20:43, Nathan Marz wrote: > > Thanks. I raised concurrent_writes to 128 and > set commitlog_sync_group_window to 20ms. This causes a single execute of a > BatchStatement containing 100 inserts to succeed. However, the throughput > I'm seeing is atrocious. > > With these settings, I'm executing 10 BatchStatement concurrently at a > time using the semaphore + loop approach I showed in my first message. So > as requests complete, more are sent out such that there are 10 in-flight at > a time. Each BatchStatement has 100 individual inserts. I'm seeing only 730 > inserts / second. Again, with periodic mode I see 38k / second and with > batch I see 14k / second. My expectation was that group commit mode > throughput would be somewhere between those two. > > If I set commitlog_sync_group_window to 100ms, the throughput drops to 14 > / second. > > If I set commitlog_sync_group_window to 10ms, the throughput increases to > 1587 / second. > > If I set commitlog_sync_group_window to 5ms, the throughput increases to > 3200 / second. > > If I set commitlog_sync_group_window to 1ms, the throughput increases to > 13k / second, which is slightly less than batch commit mode. > > Is group commit mode supposed to have better performance than batch mode? > > > On Tue, Apr 23, 2024 at 8:46 AM Bowen Song via user < > user@cassandra.apache.org> wrote: > >> The default commitlog_sync_group_window is very long for SSDs. Try >> reduce it if you are using SSD-backed storage for the commit log. 10-15 ms >> is a good starting point. You may also want to increase the value of >> concurrent_writes, consider at least double or quadruple it from the >> default. You'll need even higher write concurrency for longer >> commitlog_sync_group_window. >> >> On 23/04/2024 19:26, Nathan Marz wrote: >> >> "batch" mode works fine. I'm having trouble with "group" mode. The only >> config for that is "commitlog_sync_group_window", and I have that set to >> the default 1000ms. >> >> On Tue, Apr 23, 2024 at 8:15 AM Bowen Song via user < >> user@cassandra.apache.org> wrote: >> >>> Why would you want to set commitlog_sync_batch_window to 1 second long >>> when commitlog_sync is set to batch mode? The documentation >>> <https://cassandra.apache.org/doc/stable/cassandra/architecture/storage_engine.html> >>> on this says: >>> >>> *This window should be kept short because the writer threads will be >>> unable to do extra work while waiting. You may need to increase >>> concurrent_writes for the same reason* >>> >>> If you want to use batch mode, at least ensure >>> commitlog_sync_batch_window is reasonably short. The default is 2 >>> millisecond. >>> >>> >>> On 23/04/2024 18:32, Nathan Marz wrote: >>> >>> I'm doing some benchmarking of Cassandra on a single m6gd.large >>> instance. It works fine with periodic or batch commitlog_sync options, but >>> I'm having tons of issues when I change it to "group". I have >>> "commitlog_sync_group_window" set to 1000ms. >>> >>> My client is doing writes like this (pseudocode): >>> >>> Semaphore sem = new Semaphore(numTickets); >>> while(true) { >>> >>> sem.acquire(); >>> session.executeAsync(insert.bind(genUUIDStr(), genUUIDStr(), >>> genUUIDStr()) >>> .whenComplete((t, u) -> sem.release()) >>> >>> } >>> >>> If I set numTickets higher than 20, I get tons of timeout errors. >>> >>> I've also tried doing single commands with BatchStatement with many >>> inserts at a time, and that fails with timeout when the batch size gets >>> more than 20. >>> >>> Increasing the write request timeout in cassandra.yaml makes it time out >>> at slightly higher numbers of concurrent requests. >>> >>> With periodic I'm able to get about 38k writes / second, and with batch >>> I'm able to get about 14k / second. >>> >>> Any tips on what I should be doing to get group commitlog_sync to work >>> properly? I didn't expect to have to do anything other than change the >>> config. >>> >>>
