[
https://issues.apache.org/jira/browse/KUDU-2244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147726#comment-18147726
]
ASF subversion and git services commented on KUDU-2244:
-------------------------------------------------------
Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]
[consensus] async cmeta-flushing in RaftConsensus
With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].
The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.
Existing Raft consensus tests pass. Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:
* a tablet server crashes after the WAL is synced but before the
asynchronous cmeta flush completes in RaftConsensus::Update(),
so the term persisted in the cmeta file is behind the term
persisted in the latest WAL segments:
** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers
* when RaftConsensus::Shutdown() is called while a cmeta flush is still
pending, the voting information is correctly persisted to disk before
Shutdown() returns:
** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown
* when a duplicate vote request arrives from a peer that's been
granted 'yes' vote already, the response is sent only after the
information on the granted 'yes' vote is persisted in the
replica's consensus metadata file on disk:
** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending
* even with asynchronous cmeta flushing, the metadata is flushed
exactly once per Raft term advance, and repeated RaftConsensus::Update()
do not trigger additional flushes:
** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance
I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:
* perf record -e raw_syscalls:* -- \
./bin/same_tablet_concurrent_writes-itest \
--gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
* perf trace --input perf.data --summary > perf-trace.out 2>&1
The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls. The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.
NOTE: it's expected that Raft workers now spend a bit more time
on fsync()/fdatasync() calls, where the delta is transferred
from RPC workers to Raft workers
Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 95733.6 | 58460.8
raft worker, total max | 13859.7 | 5443.3
raft worker, max max | 500.8 | 500.5
| |
rpc worker, total sum | 2720960.0 | 1028690.0
rpc worker, total max | 28678.1 | 5879.7
rpc worker, max max | 24346.2 | 5141.0
Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 7.7 | 17.1
raft worker, total max | 3.4 | 3.3
raft worker, max max | 3.4 | 3.3
| |
rpc worker, total sum | 40.5 | 30.4
rpc worker, total max | 5.3 | 5.6
rpc worker, max max | 2.8 | 3.1
Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type | Before | After
------------------------------------------------
raft worker, total sum | 2.6 | 12.2
raft worker, total max | 1.1 | 1.9
raft worker, max max | 1.1 | 1.9
| |
rpc worker, total sum | 25.3 | 22.0
rpc worker, total max | 3.6 | 3.7
rpc worker, max max | 1.8 | 1.9
[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708
Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
> spinlock contention in raft_consensus
> -------------------------------------
>
> Key: KUDU-2244
> URL: https://issues.apache.org/jira/browse/KUDU-2244
> Project: Kudu
> Issue Type: Improvement
> Components: consensus
> Reporter: Andrew Wong
> Priority: Major
> Labels: performance
>
> I was going through the logs of a cluster that was seeing a bunch of
> kernel_stack_watchdog traces, and the slowness seemed to be caused by a lot
> of activity in consensus requests. E.g.
> W1214 18:57:29.514219 36138 kernel_stack_watchdog.cc:145] Thread 36317 stuck
> at
> /data/jenkins/workspace/generic-package-centos64-7-0-impala/topdir/BUILD/kudu-1.3.0-cdh5.11.0/src/kudu/rpc/outbound_call.cc:192
> for 123ms:
> Kernel stack:
> [<ffffffff810b7ef5>] sys_sched_yield+0x65/0xd0
> [<ffffffff81645909>] system_call_fastpath+0x16/0x1b
> [<ffffffffffffffff>] 0xffffffffffffffff
> User stack:
> @ 0x7f72fab92057 __GI___sched_yield
> @ 0x19498bf kudu::Thread::StartThread()
> @ 0x1952e7d kudu::ThreadPool::CreateThreadUnlocked()
> @ 0x19534d3 kudu::ThreadPool::Submit()
> @ 0x1953a27 kudu::ThreadPool::SubmitFunc()
> @ 0x1953ecb kudu::ThreadPool::SubmitClosure()
> @ 0x9c94ec kudu::consensus::RaftConsensus::ElectionCallback()
> @ 0x9e6032 kudu::consensus::LeaderElection::CheckForDecision()
> @ 0x9e78c3
> kudu::consensus::LeaderElection::VoteResponseRpcCallback()
> @ 0xa8b137 kudu::rpc::OutboundCall::CallCallback()
> @ 0xa8c2bc kudu::rpc::OutboundCall::SetResponse()
> @ 0xa822c0 kudu::rpc::Connection::HandleCallResponse()
> @ 0xa83ffc ev::base<>::method_thunk<>()
> @ 0x198a07f ev_invoke_pending
> @ 0x198af71 ev_run
> @ 0xa5e049 kudu::rpc::ReactorThread::RunThread()
> So it seemed to be cause by some slowness in getting threads. Upon perusing
> the logs a bit more, there were a sizable number of spinlock profiling traces:
> W1214 18:54:27.897955 36379 rpcz_store.cc:238] Trace:
> 1214 18:54:26.766922 (+ 0us) service_pool.cc:143] Inserting onto call
> queue
> 1214 18:54:26.771135 (+ 4213us) service_pool.cc:202] Handling call
> 1214 18:54:26.771138 (+ 3us) raft_consensus.cc:1126] Updating replica for
> 0 ops
> 1214 18:54:27.897699 (+1126561us) raft_consensus.cc:1165] Early marking
> committed up to index 0
> 1214 18:54:27.897700 (+ 1us) raft_consensus.cc:1170] Triggering prepare
> for 0 ops
> 1214 18:54:27.897701 (+ 1us) raft_consensus.cc:1282] Marking committed up
> to 1766
> 1214 18:54:27.897702 (+ 1us) raft_consensus.cc:1332] Filling consensus
> response to leader.
> 1214 18:54:27.897736 (+ 34us) spinlock_profiling.cc:255] Waited 991 ms on
> lock 0x120b3540. stack: 00000000019406c5 00000000009c60d7 00000000009c75f7
> 00000000007dc628 0000000000a7adfc 0000000000a7b9cd 000000000194d059
> 00007f72fbcc2dc4 00007f72fabad1cc ffffffffffffffff
> 1214 18:54:27.897737 (+ 1us) raft_consensus.cc:1327] UpdateReplicas()
> finished
> 1214 18:54:27.897741 (+ 4us) inbound_call.cc:130] Queueing success
> response
> Metrics: {"spinlock_wait_cycles":2478395136}
> Each of the traces noted on the order of 500-1000ms of waiting on spinlocks.
> Upon looking at raft_consensus.cc, it seems we're holding a spinlock
> (update_lock_) while we call RaftConsensus::UpdateReplica(), which according
> to its header, "won't return until all operations have been stored in the log
> and all Prepares() have been completed". While locking may be necessary, it's
> worth considering using a different kind of lock here.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)