[
https://issues.apache.org/jira/browse/KUDU-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147725#comment-18147725
]
ASF subversion and git services commented on KUDU-2175:
-------------------------------------------------------
Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]
[consensus] async cmeta-flushing in RaftConsensus
With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].
The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.
Existing Raft consensus tests pass. Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:
* a tablet server crashes after the WAL is synced but before the
asynchronous cmeta flush completes in RaftConsensus::Update(),
so the term persisted in the cmeta file is behind the term
persisted in the latest WAL segments:
** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers
* when RaftConsensus::Shutdown() is called while a cmeta flush is still
pending, the voting information is correctly persisted to disk before
Shutdown() returns:
** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown
* when a duplicate vote request arrives from a peer that's been
granted 'yes' vote already, the response is sent only after the
information on the granted 'yes' vote is persisted in the
replica's consensus metadata file on disk:
** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending
* even with asynchronous cmeta flushing, the metadata is flushed
exactly once per Raft term advance, and repeated RaftConsensus::Update()
do not trigger additional flushes:
** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance
I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:
* perf record -e raw_syscalls:* -- \
./bin/same_tablet_concurrent_writes-itest \
--gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
* perf trace --input perf.data --summary > perf-trace.out 2>&1
The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls. The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.
NOTE: it's expected that Raft workers now spend a bit more time
on fsync()/fdatasync() calls, where the delta is transferred
from RPC workers to Raft workers
Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 95733.6 | 58460.8
raft worker, total max | 13859.7 | 5443.3
raft worker, max max | 500.8 | 500.5
| |
rpc worker, total sum | 2720960.0 | 1028690.0
rpc worker, total max | 28678.1 | 5879.7
rpc worker, max max | 24346.2 | 5141.0
Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 7.7 | 17.1
raft worker, total max | 3.4 | 3.3
raft worker, max max | 3.4 | 3.3
| |
rpc worker, total sum | 40.5 | 30.4
rpc worker, total max | 5.3 | 5.6
rpc worker, max max | 2.8 | 3.1
Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 2.6 | 12.2
raft worker, total max | 1.1 | 1.9
raft worker, max max | 1.1 | 1.9
| |
rpc worker, total sum | 25.3 | 22.0
rpc worker, total max | 3.6 | 3.7
rpc worker, max max | 1.8 | 1.9
[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708
Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
> Do not hold the RaftConsensus lock while performing I/O
> -------------------------------------------------------
>
> Key: KUDU-2175
> URL: https://issues.apache.org/jira/browse/KUDU-2175
> Project: Kudu
> Issue Type: Improvement
> Components: consensus
> Reporter: Mike Percy
> Assignee: Alexey Serbin
> Priority: Major
>
> Do not hold the RaftConsensus lock while performing I/O. This optimization
> would improve stability and reliability as it could make a tablet server less
> sensitive to disk latency spikes and election storms. On the other hand, we
> would need to carefully ensure that the ordering invariants currently
> enforced by this lock continue to hold.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)