Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/20377 )
Change subject: IMPALA-12385: Enable Periodic metrics by default ...................................................................... Patch Set 1: (5 comments) http://gerrit.cloudera.org:8080/#/c/20377/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/20377/1//COMMIT_MSG@12 PS1, Line 12: resource_trace_ratio to 1 > AFAIK, there is a pretty significant overhead on always sampling this metri I didn't see any significant overhead, even with sampling at 10ms. Can you please provide an examples of a query that is slower? http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/runtime/query-state.cc File be/src/runtime/query-state.cc: http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/runtime/query-state.cc@221 PS1, Line 221: AddSamplingTimeSeriesCounter > Will this cause interpretation problem if different host happen to resize i The code appears to handle this already. Note that SamplingTimeSeriesCounter is already being used for Fragment metrics. http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/periodic-counter-updater.cc File be/src/util/periodic-counter-updater.cc: http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/periodic-counter-updater.cc@30 PS1, Line 30: periodic_counter_update_period_ms, 50 > I'm a bit concern about lowering this to 10x. Can the code in PeriodicCount 50ms doesn't appear to create performance issues with single-user queries. I will test with concurrent queries. Even at 100ms, values are too far apart for detailed analysis of short queries. http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/runtime-profile-counters.h@807 PS1, Line 807: typedef StreamingSampler<int64_t, 64> StreamingCounterSampler; > If initial_period = 50ms, and MAX_SAMPLES = 64, that means it will take 320 Queries on the order of 1sec were not affected. I will test more with shorter queries. http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/streaming-sampler.h File be/src/util/streaming-sampler.h: http://gerrit.cloudera.org:8080/#/c/20377/1/be/src/util/streaming-sampler.h@40 PS1, Line 40: int initial_period > I'd rather keep this default to 500, but then add new parameter in AddSampl Memory and thread usage need to use the lower interval to short-running queries. I can understand adding a different switch to preserve the 500ms default for KRPC. -- To view, visit http://gerrit.cloudera.org:8080/20377 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic8e5cbfd4b324081158574ceb8f4b3a062a69fd1 Gerrit-Change-Number: 20377 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Kurt Deschler <kdesc...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Reviewer: Surya Hebbar <sheb...@cloudera.com> Gerrit-Comment-Date: Fri, 18 Aug 2023 22:11:07 +0000 Gerrit-HasComments: Yes
