Hello Riza Suminto, David Rorke, Surya Hebbar, Csaba Ringhofer, Michael Smith,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/20377
to look at the new patch set (#7).
Change subject: IMPALA-12385: Enable Periodic metrics by default
......................................................................
IMPALA-12385: Enable Periodic metrics by default
This patch enables periodic metrics in query profiles by default and
changes the metric collectors to be more suitable for mixed workloads.
-Changed default of resource_trace_ratio to 1.
-Changed profile metrics to use sampling counters which can automatically
resize for long queries.
-Reduced profile metric samping interval to 50ms to support short-running
queries.
-Changed fragment metrics to use the same sampling interval as periodic
metrics.
-Added singleton PeriodicCounterUpdater and thread for updating system
(KRPC) metrics at a different period than fragment metrics.
-Added new flag periodic_system_counter_update_period_ms for configuring
system metric update period. Default is 500ms.
Example profile output:
- HostCpuIoWaitPercentage (400.000ms): 1, 0, 2, 3, 4, 6, 5, 2, 1, ...
- HostCpuSysPercentage (400.000ms): 1, 12, 10, 11, 11, 5, 3, 3, 3, ...
- HostCpuUserPercentage (400.000ms): 8, 46, 39, 39, 39, 29, 22, 23, ...
Testing:
-Updated runtime-profile-test and test_observability.py for new defaults
-Manual inspection of query profile metrics for long-running queries
-Tested for performance regression using 50 concurrent TPCH Q1 and with
periodic_counter_update_period_ms=1 to verify that the periodic update
loop does not affect peformance or become a bottleneck
Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1
---
M be/src/runtime/exec-env.cc
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/query-state.cc
M be/src/util/periodic-counter-updater.cc
M be/src/util/periodic-counter-updater.h
M be/src/util/runtime-profile-counters.h
M be/src/util/runtime-profile-test.cc
M be/src/util/runtime-profile.cc
M be/src/util/runtime-profile.h
M be/src/util/streaming-sampler.h
M common/thrift/Query.thrift
M tests/query_test/test_observability.py
12 files changed, 147 insertions(+), 81 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/20377/7
--
To view, visit http://gerrit.cloudera.org:8080/20377
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1
Gerrit-Change-Number: 20377
Gerrit-PatchSet: 7
Gerrit-Owner: Kurt Deschler <kdesc...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <csringho...@cloudera.com>
Gerrit-Reviewer: David Rorke <dro...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Surya Hebbar <sheb...@cloudera.com>
