[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Reviewed-on: http://gerrit.cloudera.org:8080/11894 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 92 insertions(+), 54 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 13 Nov 2018 21:32:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 13 Nov 2018 17:40:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3454/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 13 Nov 2018 17:40:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 13 Nov 2018 17:36:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11894 to look at the new patch set (#3). Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 92 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11894/3 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/hdfs-scan-node-base.h@133 PS2, Line 133: /// ScannerIoWaitTime - total amount of time scanner threads spent waiting for > Can this be larger than the wall clock time if multiple threads are waiting Updated the comment to mention what it's comparable to. For multithreaded scans, it does include the time that multiple threads are waiting, but that's what I think is most useful since other metrics in the scan node are aggregates across all threads. http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.h File be/src/exec/kudu-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.h@67 PS2, Line 67: const TupleDescriptor* tuple_desc_; > While you're here, want to initialize this one and client_, too? Done http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.cc File be/src/exec/kudu-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.cc@57 PS2, Line 57: client_(nullptr), > While you're here, initialize in header, remove here? Done http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@62 PS2, Line 62: CANCEL_SAFE_SCOPED_TIMER3(c1, nullptr nullptr); > missing comma? And is_cancelled seems to be missing (but I'm no macro exper Done http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@495 PS2, Line 495: if (c1 == nullptr && c2 == nullptr && c3 == nullptr) return; > Is this something that we do (pass all nullptr)? Probably not? krpc-data-stream-recvr did pass in a nullptr to a SCOPED_TIMER() before this patch, but I removed that one spot. I don't think it's worth keeping around this as an "optimisation" though, since by far the most common case will be that one counter is non-null. http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@502 PS2, Line 502: ScopedTimer(RuntimeProfile::Counter* counter, const bool* is_cancelled = nullptr) > Can we replace this one by making c2 default to nullptr in the previous cto Done http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@525 PS2, Line 525: NULL > nit: nullptr Updated in whole file -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 08 Nov 2018 18:12:29 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 08 Nov 2018 18:46:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/hdfs-scan-node-base.h File be/src/exec/hdfs-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/hdfs-scan-node-base.h@133 PS2, Line 133: /// ScannerIoWaitTime - total amount of time scanner threads spent waiting for Can this be larger than the wall clock time if multiple threads are waiting? If so, can we mention this in the comment? Alternatively, would it make sense to only measure the blocking time once? That way a value of 30 seconds would mean that query execution would have been 30 seconds faster if we had not waited for I/O. I cannot think of an easy way to measure that though, unless we want to move the accounting into the scan range queues. http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.h File be/src/exec/kudu-scan-node-base.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.h@67 PS2, Line 67: const TupleDescriptor* tuple_desc_; While you're here, want to initialize this one and client_, too? http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.cc File be/src/exec/kudu-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/exec/kudu-scan-node-base.cc@57 PS2, Line 57: client_(nullptr), While you're here, initialize in header, remove here? http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@62 PS2, Line 62: CANCEL_SAFE_SCOPED_TIMER3(c1, nullptr nullptr); missing comma? And is_cancelled seems to be missing (but I'm no macro expert). http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@495 PS2, Line 495: if (c1 == nullptr && c2 == nullptr && c3 == nullptr) return; Is this something that we do (pass all nullptr)? http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@502 PS2, Line 502: ScopedTimer(RuntimeProfile::Counter* counter, const bool* is_cancelled = nullptr) Can we replace this one by making c2 default to nullptr in the previous ctor? http://gerrit.cloudera.org:8080/#/c/11894/2/be/src/util/runtime-profile-counters.h@525 PS2, Line 525: NULL nit: nullptr -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 08 Nov 2018 01:41:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1304/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:52:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/1303/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:41:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h@59 PS1, Line 59: ScopedTimer MACRO_CONCAT(SCOPED_TIMER_COUNTER, __COUNTER__)(c1, c2) > line too long (93 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:17:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11894 to look at the new patch set (#2). Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 85 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11894/2 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h File be/src/util/runtime-profile-counters.h: http://gerrit.cloudera.org:8080/#/c/11894/1/be/src/util/runtime-profile-counters.h@59 PS1, Line 59: ScopedTimer MACRO_CONCAT(SCOPED_TIMER_COUNTER, __COUNTER__)(c1, c2) line too long (93 > 90) -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:04:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11894 ) Change subject: IMPALA-7819: add per-node storage wait time to profile .. Patch Set 1: Lars, some of the observability work reminded me of this counter that I've wished we had a few times. -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Nov 2018 00:04:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7819: add per-node storage wait time to profile
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11894 Change subject: IMPALA-7819: add per-node storage wait time to profile .. IMPALA-7819: add per-node storage wait time to profile Adds ScannerIoWaitTime and KuduClientTime to measure time spent waiting for the I/O manager and time spent in the Kudu client, respectively. To minimise the additional overhead, add support to the timer infrastructure to update multiple counters with one stopwatch. Apply this in some other places too. Testing: Ran core tests. Ran some queries manually with and without mt_dop and sanity-checked the profiles. Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scanner.cc M be/src/exec/scanner-context.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-recvr.cc M be/src/util/runtime-profile-counters.h 10 files changed, 84 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/11894/1 -- To view, visit http://gerrit.cloudera.org:8080/11894 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I74a496efdbe9e804db779c9753b00a303a270580 Gerrit-Change-Number: 11894 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
