Hello Grant Henke, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16385
to look at the new patch set (#3).
Change subject: IMPALA-9792: Add ability to split kudu scan ranges
......................................................................
IMPALA-9792: Add ability to split kudu scan ranges
This patch adds the ability to split kudu scan token via the provided
kudu java API. A query option "TARGETED_KUDU_SCAN_RANGE_LENGTH" has
been added to set the scan range length used in this implementation.
Potential benefit:
This helps increase parallelism during scanning which can
result in more efficient use of CPU with higher mt_dop.
Limitation:
- The scan range length sent to kudu is just a hint and does not
guarantee that the token will be split at that limit.
- Comes at an added cost of an RPC to tablet server per token in
order to split it. A slow tablet server which can already slow
down scanning during execution can now also potentially slow
down planning.
- Also adds the cost of an RPC per token to open a new scanner for
it on the kudu side. Therefore, scanning many smaller split
tokens can slow down scanning and we can also lose benefits
of scanning a single large token sequentially with a single scanner.
Testing:
- Added an e2e test
Change-Id: Ia02fd94cc1d13c61bc6cb0765dd2cbe90e9a5ce8
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M tests/query_test/test_kudu.py
7 files changed, 85 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/16385/3
--
To view, visit http://gerrit.cloudera.org:8080/16385
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia02fd94cc1d13c61bc6cb0765dd2cbe90e9a5ce8
Gerrit-Change-Number: 16385
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Bikramjeet Vig <[email protected]>
Gerrit-Reviewer: Grant Henke <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Tim Armstrong <[email protected]>
