Hello Jean-Daniel Cryans, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5498
to look at the new patch set (#3).
Change subject: KUDU-1806. java: fetching scan tokens should fetch larger
batches
......................................................................
KUDU-1806. java: fetching scan tokens should fetch larger batches
This changes the number of tablets fetched in a single GetTableLocations
RPC from 10 to 1000 for the case of scans or scan token generation. On a
stress test on 200 nodes with 40 concurrent query streams, this
substantially reduced the time spent in scan token generation by the
Impala planner.
This keeps the existing batch size (10) for cases where the client is
looking up a tablet for the purposes of a normal operation, but extends
it to 1000 for cases where we are explicitly fetching a range of tablet
locations. The aim here is that this will not increase network traffic
or master load for the case of random write workloads, but will increase
performance for "query planner" type workloads.
Although this will slightly increase the amount of work done by a
GetTableLocations RPC, my guess is that the majority of the RPC cost is
dominated by fixed per-RPC costs and not the linear cost based on the
number of tablets. This is especially true when taking into account the
typical RTT within a large/busy cluster (~1ms). So, it is a lot cheaper,
both in wall clock and total resources consumed, to process one larger
RPC rather than tens or hundreds of small ones.
This test also modifies the scan token generation test case to set the
fetch size down to a low value. This ensures that the code path to go
back and fetch more locations is still exercised, rather than always
fetching all of the tablets in one RPC.
Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
M
java/kudu-client/src/main/java/org/apache/kudu/client/GetTableLocationsRequest.java
M java/kudu-client/src/main/java/org/apache/kudu/client/KuduTable.java
M java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
6 files changed, 107 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/5498/3
--
To view, visit http://gerrit.cloudera.org:8080/5498
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Adar Dembo <[email protected]>
Gerrit-Reviewer: Dan Burkert <[email protected]>
Gerrit-Reviewer: Jean-Daniel Cryans <[email protected]>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Matthew Jacobs <[email protected]>
Gerrit-Reviewer: Todd Lipcon <[email protected]>
