Yadong Chen created CASSANDRA-15932:
---------------------------------------
Summary: CommitLogSegmentManager#sync is doing sync one by one
need to be improved
Key: CASSANDRA-15932
URL: https://issues.apache.org/jira/browse/CASSANDRA-15932
Project: Cassandra
Issue Type: Improvement
Components: Local/Commit Log
Reporter: Yadong Chen
Assignee: Yadong Chen
Attachments: image-2020-07-08-12-05-07-746.png,
image-2020-07-08-12-21-10-369.png
test machine settings:
# 24 cores
# 45G memory
# maximum disk thoughput can over 500MB/s
I was running PeriodicCommitLogStressTest#testFixedSize in that test machine
and notice both cpu and io can't be full utilized
PeriodicCommitLogStressTest#testFixedSize was running with CommitLogSyncPeriod
set to 10000 (I changed it to match production environment setting) and no
compression and no encryption
!image-2020-07-08-12-05-07-746.png!
After some digging I find out it's the sync process can't keep up and when
lastSyncedAt is far behind all thread doing CommitLog#add will wait until it
catch up.
Once catch up, all threads go back continue writing and so on.
The overall IO utilization I guess is about 60% more or less. After introduce a
thread pool to do sync parallel instead of doing it one by one. The IO
utilization is improved
!image-2020-07-08-12-21-10-369.png!
the IO throughput is more steady (almost always above 400MB) and need less time
to reach that throughput
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
