Jean-Daniel Cryans has submitted this change and it was merged. Change subject: KUDU-1806. java: fetching scan tokens should fetch larger batches ......................................................................
KUDU-1806. java: fetching scan tokens should fetch larger batches This changes the number of tablets fetched in a single GetTableLocations RPC from 10 to 1000 for the case of scans or scan token generation. On a stress test on 200 nodes with 40 concurrent query streams, this substantially reduced the time spent in scan token generation by the Impala planner. This keeps the existing batch size (10) for cases where the client is looking up a tablet for the purposes of a normal operation, but extends it to 1000 for cases where we are explicitly fetching a range of tablet locations. The aim here is that this will not increase network traffic or master load for the case of random write workloads, but will increase performance for "query planner" type workloads. Although this will slightly increase the amount of work done by a GetTableLocations RPC, my guess is that the majority of the RPC cost is dominated by fixed per-RPC costs and not the linear cost based on the number of tablets. This is especially true when taking into account the typical RTT within a large/busy cluster (~1ms). So, it is a lot cheaper, both in wall clock and total resources consumed, to process one larger RPC rather than tens or hundreds of small ones. This test also modifies the scan token generation test case to set the fetch size down to a low value. This ensures that the code path to go back and fetch more locations is still exercised, rather than always fetching all of the tablets in one RPC. Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9 Reviewed-on: http://gerrit.cloudera.org:8080/5498 Reviewed-by: Dan Burkert <[email protected]> Tested-by: Kudu Jenkins Reviewed-by: Jean-Daniel Cryans <[email protected]> (cherry picked from commit 01e6871bca38ea2bed5c17290d43739a0c88094e) Reviewed-on: http://gerrit.cloudera.org:8080/5516 --- M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java M java/kudu-client/src/main/java/org/apache/kudu/client/GetTableLocationsRequest.java M java/kudu-client/src/main/java/org/apache/kudu/client/KuduTable.java M java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java 6 files changed, 107 insertions(+), 53 deletions(-) Approvals: Jean-Daniel Cryans: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5516 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9 Gerrit-PatchSet: 2 Gerrit-Project: kudu Gerrit-Branch: branch-1.2.x Gerrit-Owner: Todd Lipcon <[email protected]> Gerrit-Reviewer: Jean-Daniel Cryans <[email protected]> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Todd Lipcon <[email protected]>
