[ 
https://issues.apache.org/jira/browse/KUDU-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147728#comment-18147728
 ] 

ASF subversion and git services commented on KUDU-2708:
-------------------------------------------------------

Commit 32653a5767ec1769d6cf48021b649ec1061f93fc in kudu's branch 
refs/heads/master from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=32653a576 ]

[consensus] async cmeta-flushing in RaftConsensus

With this changelist, RequestVote, UpdateReplica, and other
RaftConsensus methods don't hold lock_ and update_lock_ while flushing
Raft consensus metadata to the backing storage. This should help during
Raft election storms and other situations described in [1][2][3][4][5].

The WAL segments are still being flushed/synced while holding
update_lock_ in RaftConsensus::Update() (but lock_ isn't held, though),
and this aspect should be taken care in a separate changelist.

Existing Raft consensus tests pass.  Also, several new tests scenarios
added to cover corner cases introduced by the new functionality:

  * a tablet server crashes after the WAL is synced but before the
    asynchronous cmeta flush completes in RaftConsensus::Update(),
    so the term persisted in the cmeta file is behind the term
    persisted in the latest WAL segments:
      ** RaftConsensusITest.FollowerCrashBeforeCmetaFlushRecovers

  * when RaftConsensus::Shutdown() is called while a cmeta flush is still
    pending, the voting information is correctly persisted to disk before
    Shutdown() returns:
      ** RaftConsensusQuorumTest.PendingVoteInfoPersistedOnShutdown
      ** RaftConsensusQuorumTest.TombstonedVotePersistsOnShutdown

  * when a duplicate vote request arrives from a peer that's been
    granted 'yes' vote already, the response is sent only after the
    information on the granted 'yes' vote is persisted in the
    replica's consensus metadata file on disk:
      ** RaftConsensusQuorumTest.DuplicateRequestVoteWhileCmetaFlushPending

  * even with asynchronous cmeta flushing, the metadata is flushed
    exactly once per Raft term advance, and repeated RaftConsensus::Update()
    do not trigger additional flushes:
      ** RaftConsensusQuorumTest.FollowerCmetaFlushedExactlyOncePerTermAdvance

I ran modified same_tablet_concurrent_writes-itest before and after the
update under perf:

  * perf record -e raw_syscalls:* -- \
      ./bin/same_tablet_concurrent_writes-itest \
      --gtest_filter=SameTabletConcurrentWritesTest.InsertsOnly
  * perf trace --input perf.data --summary > perf-trace.out 2>&1

The collected data came through post-processing using awk to find
the amount of time spent by Raft and RPC worker threads on futex(),
fsync(), and fdatasync() syscalls.  The results show significant
reduction in time RPC and Raft worker threads spent waiting on
futex (e.g. waiting on simple_spinlock), and that was even without
injection of extra delays into fsync()/fdatasync() paths.

NOTE: it's expected that Raft workers now spend a bit more time
      on fsync()/fdatasync() calls, where the delta is transferred
      from RPC workers to Raft workers

      Time spent on futex() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |   95733.6 |   58460.8
raft worker, total max |   13859.7 |    5443.3
raft worker, max max   |     500.8 |     500.5
                       |           |
rpc worker, total sum  | 2720960.0 | 1028690.0
rpc worker, total max  |   28678.1 |    5879.7
rpc worker, max max    |   24346.2 |    5141.0

      Time spent on fdatasync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       7.7 |      17.1
raft worker, total max |       3.4 |       3.3
raft worker, max max   |       3.4 |       3.3
                       |           |
rpc worker, total sum  |      40.5 |      30.4
rpc worker, total max  |       5.3 |       5.6
rpc worker, max max    |       2.8 |       3.1

      Time spent on fsync() syscall (ms)
------------------------------------------------
Thread Type            |   Before  |   After
------------------------------------------------
raft worker, total sum |       2.6 |      12.2
raft worker, total max |       1.1 |       1.9
raft worker, max max   |       1.1 |       1.9
                       |           |
rpc worker, total sum  |      25.3 |      22.0
rpc worker, total max  |       3.6 |       3.7
rpc worker, max max    |       1.8 |       1.9

[1] https://issues.apache.org/jira/browse/KUDU-1064
[2] https://issues.apache.org/jira/browse/KUDU-2175
[3] https://issues.apache.org/jira/browse/KUDU-2244
[4] https://issues.apache.org/jira/browse/KUDU-2370
[5] https://issues.apache.org/jira/browse/KUDU-2708

Change-Id: Ia11b326a96da40735f87decb04b2db3229e0a59b
Assisted-by: Cursor <[email protected]> (Claude Sonnet 4.6)
Reviewed-on: https://gerrit.cloudera.org/25160
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>


> Possible contention creating temporary files while flushing cmeta during an 
> election storm
> ------------------------------------------------------------------------------------------
>
>                 Key: KUDU-2708
>                 URL: https://issues.apache.org/jira/browse/KUDU-2708
>             Project: Kudu
>          Issue Type: Improvement
>          Components: perf
>            Reporter: William Berkeley
>            Priority: Major
>
> Doing investigation into consensus queue overflows that happen under heavy 
> write load, I noticed 6/10 service threads at the time of overflow have 
> stacks like
> {noformat}
> 0x3b6720f710 <unknown>
>            0x1fb900a base::internal::SpinLockDelay()
>            0x1fb8ea7 base::SpinLock::SlowLock()
>             0xb82e25 kudu::consensus::RaftConsensus::RequestVote()
>             0x931555 
> kudu::tserver::ConsensusServiceImpl::RequestConsensusVote()
>            0x1e28a2c kudu::rpc::GeneratedServiceIf::Handle()
>            0x1e2935a kudu::rpc::ServicePool::RunThread()
>            0x1f9bd91 kudu::Thread::SuperviseThread()
>         0x3b672079d1 start_thread
>         0x3b66ee88fd clone
> {noformat}
> They are waiting on some tablet's Raft consensus instance's {{lock_}} in 
> order to vote. Looking into what might be holding that lock, I see stacks like
> {noformat}
> 0x3b6720f710 <unknown>
>         0x3b66edb2ed __GI_open64
>         0x3b66e63caa __gen_tempname
>            0x1f1cf35 kudu::(anonymous namespace)::PosixEnv::MkTmpFile()
>            0x1f1f662 kudu::(anonymous namespace)::PosixEnv::NewTempRWFile()
>            0x1f8305e kudu::pb_util::WritePBContainerToPath()
>             0xb47932 kudu::consensus::ConsensusMetadata::Flush()
>             0xb74164 
> kudu::consensus::RaftConsensus::SetVotedForCurrentTermUnlocked()
>             0xb783aa 
> kudu::consensus::RaftConsensus::RequestVoteRespondVoteGranted()
>             0xb836a1 kudu::consensus::RaftConsensus::RequestVote()
>             0x931555 
> kudu::tserver::ConsensusServiceImpl::RequestConsensusVote()
>            0x1e28a2c kudu::rpc::GeneratedServiceIf::Handle()
>            0x1e2935a kudu::rpc::ServicePool::RunThread()
>            0x1f9bd91 kudu::Thread::SuperviseThread()
>         0x3b672079d1 start_thread
>         0x3b66ee88fd clone
> {noformat}
> Doing some junior spelunking into glibc code, one hypothesis is that we are 
> generating lots of collisions of proposed temporary file names in the cmeta 
> folder because many threads are attempting to flush cmeta at once. The glibc 
> code looks like
> Maybe we could put the thread id into the temporary file name when a thread 
> does a cmeta flush.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to