Hello Michael Smith, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20850
to look at the new patch set (#16).
Change subject: IMPALA-12533: Support row materialization outside of fetch lock
......................................................................
IMPALA-12533: Support row materialization outside of fetch lock
This patch supports an alternative result materialization path where
results are copied to a temporary row batch then materialized into thrift
outside of the fetch lock. This can improve fetch throughput when the
client is fetching concurrently from multiple result streams.
New metrics FetchLockWaitTimer and ResultFlushTimer have been added to
account for synchrounous time waiting for the fetch lock and cumulative
time materializing results outside of the fetch lock.
New query option delay_materialize_results_threshold has been added to
automatically enable delayed result materialization when the fetch lock
wait time exceeds the specified fraction of the materialization time
(default 0.2).
Once enabled, delayed materialization will be used for the remainder of
the rows fetched for the current query. It is expected that the default
threshold will never be reached with a single fetch stream. This
optimization is disabled when query result caching is active since the
cache stores materialized rows and is populated inside of the fetch lock.
Testing:
-Tests added to tests/query_test/test_observability.py
-Manual testing with Multi-stream Hive JDBC driver. This showed a 20%
improvement in fetch time with a wide table and 5 streams. On the
mini-cluster, performance is now limited by root sink exchange
throughput. Production clusters should be able to achieve higher gains.
Change-Id: If9512aa6022dbcf81e7eb5f559853b89fe80bd23
---
M be/src/benchmarks/hash-benchmark.cc
M be/src/codegen/llvm-codegen-cache-test.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/exchange-node.cc
M be/src/exec/sort-node.cc
M be/src/exprs/expr-codegen-test.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/fragment-state.cc
M be/src/runtime/fragment-state.h
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-hs2-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/service/query-result-set.cc
M be/src/service/query-result-set.h
M be/src/util/runtime-profile-counters.h
M common/thrift/ImpalaService.thrift
M common/thrift/Query.thrift
M tests/query_test/test_observability.py
28 files changed, 450 insertions(+), 123 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/20850/16
--
To view, visit http://gerrit.cloudera.org:8080/20850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If9512aa6022dbcf81e7eb5f559853b89fe80bd23
Gerrit-Change-Number: 20850
Gerrit-PatchSet: 16
Gerrit-Owner: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
