Tim Armstrong has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16057 )
Change subject: WIP - IMPALA-9382: part 2/2: aggregate profiles sent to coordinator ...................................................................... WIP - IMPALA-9382: part 2/2: aggregate profiles sent to coordinator This is a WIP only and depends on a merge commit. I will clean up and make the history linear once some of the preceding patches get merged. This reworks the status reporting so that serialized AggregatedRuntimeProfile objects are sent from executors to coordinators. These profiles are substantially denser and faster to process for higher mt_dop values. The changes required were: * A new Update() method for AggregatedRuntimeProfile that updates the profile from a serialised AggregateRuntimeProfile for a subset of the instances. The code is generalized from the existing InitFromThrift() code path. * Per-fragment reports included in the status report protobuf when --gen_experimental_profile=true. * Logic on the coordinator that either consumes serialized AggregatedRuntimeProfile per fragment, when --gen_experimental_profile=true, or consumes a serialized RuntimeProfile per finstance otherwise. This also adds support for event sequences and time series in the aggregated profile, so the amount of information in the aggregated profile is now on par with the basic profile. We also finish off support for JSON profile. The JSON profile is more stripped down because we do not need to round-trip profiles via JSON and it is a much less dense profile representation. TODO: * Should we use the flag on the coordinator to decide how to aggregate them, or set a flag in the status report. Testing: TODO * Can we test the report seq number stuff? * Profile regex/aggregation tests are breaking with experimental profile Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/fragment-state.cc M be/src/runtime/fragment-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/protobuf/control_service.proto M common/thrift/RuntimeProfile.thrift 13 files changed, 811 insertions(+), 183 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/16057/6 -- To view, visit http://gerrit.cloudera.org:8080/16057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca Gerrit-Change-Number: 16057 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong <[email protected]>
