Hello Jean-Daniel Cryans,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/5498
to review the following change.
Change subject: KUDU-1806. java: fetching scan cokens should fetch larger
batches
......................................................................
KUDU-1806. java: fetching scan cokens should fetch larger batches
This changes the number of tablets fetched in a single GetTableLocations
RPC from 10 to 1000. On a stress test on 200 nodes with 40 concurrent
query streams, this substantially reduced the time spent in scan token
generation by the Impala planner.
Initially, I was going to try to set the fetch size parameter in a
context-dependent way, fetching few locations for point lookups (i.e.
writes) and many locations for scans. However, it turns out that point
lookups already set an 'endPartitionKey' parameter in the RPC, and the
master will stop returning locations after reaching this point. So,
setting a batch size of 1000 for such queries will not change the amount
of work done by the master for point queries.
Although this will slightly increase the amount of work done by a
GetTableLocations RPC, my guess is that the majority of the RPC cost is
dominated by fixed per-RPC costs and not the linear cost based on the
number of tablets. This is especially true when taking into account the
typical RTT within a large/busy cluster (~1ms). So, it is a lot cheaper,
both in wall clock and total resources consumed, to process one larger
RPC rather than tens or hundreds of small ones.
In addition to changing the constant, this test also modifies the scan
token generation test case to set the fetch size down to a low value.
This ensures that the code path to go back and fetch more locations is
still exercised, rather than always fetching all of the tablets in one
RPC.
Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestKuduClient.java
2 files changed, 42 insertions(+), 27 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/98/5498/1
--
To view, visit http://gerrit.cloudera.org:8080/5498
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I46260a96dfd0847f70146496e48c2766b8e17ea9
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <[email protected]>
Gerrit-Reviewer: Jean-Daniel Cryans <[email protected]>
