Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20850
Change subject: IMPALA-12533: Support row materialization outside of fetch lock ...................................................................... IMPALA-12533: Support row materialization outside of fetch lock This patch supports an alternative result materialization path where results are copied to a temporary row batch then materialized into thrift outside of the fetch lock. This can improve fetch throughput when the client is fetching concurrently from multiple result streams. New metrics FetchLockWaitTimer and ResultFlushTimer and have been added to account for synchrounous time waiting for the fetch lock and cumulative time materializing results outside of the fetch lock. New query option delay_materialize_results_threshold has been added to automatically enable delayed result materialization when the fetch lock wait time exceeds the specified fraction of the materialization time (default 0.2). Once enabled, delayed materialization will be used for the remainder of the rows fetched for the current query. It is expected that the default threshold will never be reached with a single fetch stream. This optimization is disabled when query result caching is active since the cache stores materialized rows and is populated inside of the fetch lock. Testing: -Tests added to tests/query_test/test_observability.py -Manual testing with Multi-stream Hive JDBC driver. This showed a 20% improvement in fetch time with a wide table and 5 streams. On the mini-cluster, performance is now limited by root sink exchange throughput. Production clusters should be able to achieve higher gains. Change-Id: If9512aa6022dbcf81e7eb5f559853b89fe80bd23 --- M be/src/exec/buffered-plan-root-sink.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-hs2-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/service/query-result-set.cc M be/src/service/query-result-set.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M tests/query_test/test_observability.py 13 files changed, 294 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/20850/1 -- To view, visit http://gerrit.cloudera.org:8080/20850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If9512aa6022dbcf81e7eb5f559853b89fe80bd23 Gerrit-Change-Number: 20850 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler <[email protected]>
