[
https://issues.apache.org/jira/browse/KUDU-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125323#comment-18125323
]
Abhishek Chennaka commented on KUDU-3747:
-----------------------------------------
{quote}the client still calls KuduClient::Data::GetTabletServer() as a part of
KuduScanner::Data::OpenTablet() when commencing read/scan operation.
{quote}
This call is normal. It runs once for every tablet on every scan, and it sends
no request to the master. GetTabletServer() does two things which are both
local:
- It picks a replica from the tablet entry already in the meta-cache
(SelectTServer).
- It calls InitProxy(), which returns immediately when a connection object
already exists (If there were previous writes onto this tablet).
I reproduced the repro.cc workload in a mini-cluster test (3 tservers,
--raft_heartbeat_interval_ms=5000, 1 and 10 range partitions, LEADER_ONLY and
CLOSEST_REPLICA). In every case the client sent zero
GetTableLocations/GetTabletLocations RPCs to the master during Open() and
NextBatch(), so the meta-cache served all lookups.
With a single partition there is no scan latency. With 10 partitions,
NextBatch() takes ~4–6s. The number of partitions is not the cause but whether
a tablet was written to. A newly elected leader doesn't contact its followers
until a write arrives or its heartbeat timer fires (up to
--raft_heartbeat_interval_ms), and until its first op commits its MVCC clean
time is uninitialized
In case of the original 10 tablet situation the time goes into the
sleep-and-retry loop at client side scanner
[here|https://github.com/apache/kudu/blob/bdfb2b10cf5c5ce9eb67d0f669a5309d16b57ff4/src/kudu/client/scanner-internal.cc#L604-L616]:
a. The client sends the scan to the leader (followers reject the same way
with CLOSEST_REPLICA).
b. The leader rejects it with clean time has not yet been initialized
(TABLET_NOT_RUNNING).
c. The client marks that leader as unusable (blacklists it).
d. GetTabletServer() then finds no usable leader and returns No LEADER for
tablet … (blacklist replicas …).
e. The client sleeps and retries.
{quote}it would be natural for the client at least to rely on the information
in the scan tokens when opening a table for scanning
{quote}
Even with tokens, the client would send the scan to the same leader and get the
same rejection as tokens carry the locations the cache already has. This can be
added as an enhancement changelist but doesn't change the behavior in repro.cc.
So I'd suggest doing the below in this Jira:
In scope of this Jira, we can add the following enhancement - Avoid roundtrips
to master if leader replica is not needed by relying on meta-cache and let the
scan-tokens prime the cache when cache is leaderless and the scan-token has a
leader.
Create a follow-up Jira to fix the actual issue:
The safetime has to be advanced independent of the --raft_heartbeat_interval
period.
Have the client report the underlying TABLET_NOT_RUNNING / clean-time error
instead of "No LEADER for tablet …".
> Clarify on the C++ client meta-cache logic for
> create-table-and-immediate-write-and-scan use case
> -------------------------------------------------------------------------------------------------
>
> Key: KUDU-3747
> URL: https://issues.apache.org/jira/browse/KUDU-3747
> Project: Kudu
> Issue Type: Task
> Components: client
> Affects Versions: 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.17.1,
> 1.18.1
> Reporter: Alexey Serbin
> Priority: Major
> Attachments: repro.cc
>
>
> With KUDU-1802 implemented, the Kudu C++ client is able to obtain information
> on tablet replica locations from scan tokens, avoiding extra calls to Kudu
> masters when such information in present. It works very well in the majority
> of use cases. However, a couple of surprises were discovered in a relatively
> rare use case when a Kudu client application creates a new table, immediately
> start writing data into the newly created table, and then reads data back
> from the table.
> The two unexpected items are:
> # After populating its metacache with the information on the location of
> tablet replicas and using it while writing data into the newly created table,
> the client still calls {{KuduClient::Data::GetTabletServer()}} as a part of
> {{KuduScanner::Data::OpenTablet()}} when commencing read/scan operation.
> # Even if ignoring the already existing tablet location information in the
> metacache (say, if it's short-lived and already expired), it would be natural
> for the client at least to rely on the information in the scan tokens when
> opening a table for scanning (given KUDU-1802 provisions are already there)
> instead of re-resolving tablet locations.
> This became apparent once increasing the setting for the
> {{\-\-raft_heartbeat_interval_ms}} setting that Kudu tablet servers run
> tenfold (i.e. from default 500 ms to 5000 ms): the client application that
> tries to exercise the described use case exhibited high latency when calling
> {{KuduScanner::Open()}} right after writing a few rows into the newly created
> table.
> It's necessary to clarify what's going on.
> [Attached|https://issues.apache.org/jira/secure/attachment/13081007/repro.cc]
> is a sample Kudu client application in C++. Compile it similarly to
> {{$KUDU_HOME/examples/cpp/example.cc}} and run against a Kudu cluster having
> at least three tablet servers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
