Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15931 )
Change subject: IMPALA-9711: incrementally update aggregate profile ...................................................................... IMPALA-9711: incrementally update aggregate profile In order to not cause additional work in the default mode, we still only compute the average once per instance, when it completes or when the query finishes. When --gen_experimental_profile=true, we update the aggregated profile for each status report, so that the live profile can be viewed as the query executes. The implications of this are as follows: * More work is done on the KRPC control service RPC thread (although this is largely moot after part 2 of IMPALA-9382 where we merge into the aggregated profile directly, so avoid the extra update). * For complex multi-stage queries, the profile merging work is done earlier as each stage completes, therefore the critical path of the query is shortened * Multiple RPC threads may be merging profiles concurrently * Multiple threads may be calling AggregatedRuntimeProfile::Update() on the same profile, whereas previously all merging was done by a single thread. I looked through the locking in that function to check correctness. Testing: Ran core tests. Ran a subset of the Python tests under TSAN, confirmed no races were introduced in this code. Change-Id: Ib03e79a40a33d8e74464640ae5f95a1467a6713a Reviewed-on: http://gerrit.cloudera.org:8080/15931 Reviewed-by: Tim Armstrong <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/util/runtime-profile.cc 4 files changed, 85 insertions(+), 27 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib03e79a40a33d8e74464640ae5f95a1467a6713a Gerrit-Change-Number: 15931 Gerrit-PatchSet: 15 Gerrit-Owner: Tim Armstrong <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]>
