[
https://issues.apache.org/jira/browse/KUDU-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148085#comment-18148085
]
ASF subversion and git services commented on KUDU-3822:
-------------------------------------------------------
Commit 5fa7a62e63770c48f6fdd4c37edced40c990e13b in kudu's branch
refs/heads/branch-1.17.x from Alexey Serbin
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=5fa7a62e6 ]
KUDU-3822 fix flaky test scenario in raft_consensus-itest
This changelist addresses intermittent failures of the
RaftConsensusITest.TestReplicaBehaviorViaRPC scenario in
raft_consensus-itest.
The root cause of the issue was in the assumption that amending the
timeout of a scan RPC by 10ms [1] provided good enough margin to let
the RPC worker thread respond before the client-side RPC layer would
detect that the call had timed out.
In fact, in the presence of scheduler anomalies and process pauses,
there isn't any guarantee that pthread_cond_timedwait() or
pthread_cond_timedwait_relative_np() will awake exactly at the
timeout mark when no event was registered at the underlying
synchronization primitive -- it might take longer, of course.
Increasing the margin over 10ms [1] doesn't seem to be a bullet-proof
approach anyway. Also, there isn't a way to set timeout for a scan
operation separate from RPC timeout itself (e.g., the scenario could
use very long RPC timeout and relatively short scan timeout if that
were the case). Instead, let's update the test scenario to check for
the expected error message logged in the tserver's logs if it's not
delivered by application-level RPC error status.
To verify, I ran the updated test bits in the environment where
RaftConsensusITest.TestReplicaBehaviorViaRPC failed at least once
every 10 runs. There hasn't been a single failure in 100 runs.
[1] 'client_deadline' in TabletServiceImpl::HandleScanAtSnapshot()
defined by deducting 10ms from the actual RPC timeout, so there
is time to send the response a bit earlier in attempt to reach the
client side before the client's own timeout detector triggers
DONT_BUILD
Change-Id: Ic2e3b479e8f3576865c2a5e430222633de962137
Reviewed-on: http://gerrit.cloudera.org:8080/25248
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
(cherry picked from commit 6375051662fb3ccfb73ccb5cacbc24162bdb67eb)
Reviewed-on: https://gerrit.cloudera.org/25329
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Attila Bukor <[email protected]>
(cherry picked from commit 7ea20ab2467de663b8c78d925c385c2e17e15129)
Reviewed-on: https://gerrit.cloudera.org/25542
> RaftConsensusITest.TestReplicaBehaviorViaRPC fails from time to time
> --------------------------------------------------------------------
>
> Key: KUDU-3822
> URL: https://issues.apache.org/jira/browse/KUDU-3822
> Project: Kudu
> Issue Type: Bug
> Components: consensus, test
> Affects Versions: 1.19.0
> Reporter: Alexey Serbin
> Priority: Minor
> Fix For: 1.19.0, 1.18.2
>
>
> The RaftConsensusITest.TestReplicaBehaviorViaRPC scenario from
> raft_consensus-itest.cc fails from time to time, at least in DEBUG build
> configuration on macOS 26.7.2 aarch64.
> {noformat}
> W20261027 17:20:34.563573 14846398 tablet_service.cc:3107] error setting up
> scanner 95e6ab2793f14defad7c6b3d107cb88a wit
> h request: Timed out: Timed out waiting for ts: P: 1793146834463561 usec, L:
> 0 to be safe (mode: NON-LEADER). Current sa
> fe time: P: 1793146834459376 usec, L: 3138 Physical time difference: 0.004s:
> new_scan_request { tablet_id: "a14ab11c3864410bb739e814b065e907"
> projected_columns { name: "key" type: INT32 is_key: true is_nullable: false
> encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0
> immutable: false } projected_columns { name: "int_val" type: INT32 is_key:
> false is_nullable: false encoding: AUTO_ENCODING compression:
> DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } projected_columns
> { name: "string_val" type: STRING is_key: false is_nullable: true encoding:
> AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable:
> false } read_mode: READ_AT_SNAPSHOT } batch_size_bytes: 0
> W20261027 17:20:34.563669 14846398 rpcz_store.cc:267] Call
> kudu.tserver.TabletServerService.Scan from 127.0.0.1:53608 (request call id
> 13) took 100 ms (client timeout 100 ms). Trace:
> W20261027 17:20:34.563687 14846398 rpcz_store.cc:269] 1027 17:20:34.463506 (+
> 0us) service_pool.cc:167] Inserting onto call queue
> 1027 17:20:34.463513 (+ 7us) service_pool.cc:224] Handling call
> 1027 17:20:34.463528 (+ 15us) tablet_service.cc:2908] Created scanner
> 95e6ab2793f14defad7c6b3d107cb88a for tablet a14ab11c3864410bb739e814b065e907,
> query id is
> 1027 17:20:34.463550 (+ 22us) tablet_service.cc:3048] Creating iterator
> 1027 17:20:34.463553 (+ 3us) tablet_service.cc:3442] Waiting safe time to
> advance
> 1027 17:20:34.563562 (+100009us) tablet_service.cc:3082] Iterator created
> 1027 17:20:34.563571 (+ 9us) tablet_service.cc:3104] Iterator init: Timed
> out: Timed out waiting for ts: P: 1793146834463561 usec, L: 0 to be safe
> (mode: NON-LEADER). Current safe time: P: 1793146834459376 usec, L: 3138
> Physical time difference: 0.004s
> 1027 17:20:34.563669 (+ 98us) inbound_call.cc:177] Queueing success
> response
> Metrics: {}
> /Users/aserbin/Projects/kudu/src/kudu/integration-tests/raft_consensus-itest.cc:1384:
> Failure
> Failed
> Bad status: Timed out: Scan RPC to 127.0.0.1:53584 timed out after 0.100s
> (SENT)
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)