[
https://issues.apache.org/jira/browse/IMPALA-10235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-10235.
------------------------------------
Fix Version/s: Impala 4.0
Assignee: Tim Armstrong
Resolution: Fixed
This was introduced by part 1 of IMPALA-9382 and fixed by this part of the
below patch.
@@ -1842,7 +2137,8 @@ RuntimeProfileBase::AveragedCounter::GetStats() const {
if (!vals.empty()) {
sort(vals.begin(), vals.end());
result.num_vals = vals.size();
- result.mean = std::accumulate(vals.begin(), vals.end(), 0) /
result.num_vals;
+ T sum = std::accumulate(vals.begin(), vals.end(), 0L);
+ result.mean = sum / result.num_vals;
result.min = vals[0];
result.max = vals.back();
int end_idx = vals.size() - 1;
@@ -1902,6 +2198,50 @@ void
RuntimeProfileBase::AveragedCounter::PrettyPrintImpl(
commit 9429bd779de986d3e61858bef7e258bd73a2cacd
Author: Tim Armstrong <[email protected]>
Date: Sun May 17 16:37:46 2020 -0700
IMPALA-9382: part 2/3: aggregate profiles sent to coordinator
This reworks the status reporting so that serialized
AggregatedRuntimeProfile objects are sent from executors
to coordinators. These profiles are substantially denser
and faster to process for higher mt_dop values. The aggregation
is also done in a single step, merging the aggregated thrift
profile from the executor directly into the final aggregated
profile, instead of converting it to an unaggregated profile
first.
The changes required were:
* A new Update() method for AggregatedRuntimeProfile that
updates the profile from a serialised AggregateRuntimeProfile
for a subset of the instances. The code is generalized from the
existing InitFromThrift() code path.
* Per-fragment reports included in the status report protobuf
when --gen_experimental_profile=true.
* Logic on the coordinator that either consumes serialized
AggregatedRuntimeProfile per fragment, when
--gen_experimental_profile=true, or consumes a serialized
RuntimeProfile per finstance otherwise.
This also adds support for event sequences and time series
in the aggregated profile, so the amount of information
in the aggregated profile is now on par with the basic profile.
We also finish off support for JSON profile. The JSON profile is
more stripped down because we do not need to round-trip profiles
via JSON and it is a much less dense profile representation.
Part 3 will clean up and improve the display of the profile.
Testing:
* Add sanity tests for aggregated runtime profile.
* Add unit tests to exercise aggregation of the various counter types
* Ran core tests.
Change-Id: Ic680cbfe94c939c2a8fad9d0943034ed058c6bca
Reviewed-on: http://gerrit.cloudera.org:8080/16057
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Averaged timer profile counters can be negative for trivial queries
> -------------------------------------------------------------------
>
> Key: IMPALA-10235
> URL: https://issues.apache.org/jira/browse/IMPALA-10235
> Project: IMPALA
> Issue Type: Bug
> Reporter: Sahil Takiar
> Assignee: Tim Armstrong
> Priority: Major
> Labels: newbie, ramp-up
> Fix For: Impala 4.0
>
> Attachments: profile-output.txt
>
>
> Steps to reproduce on master:
> {code:java}
> stakiar @ stakiar-desktop -bash ~/Impala 2020-10-13 11:13:02 master
> [74] → ./bin/impala-shell.sh -q "select sleep(100) from functional.alltypes
> limit 25" -p > profile-output.txt
> ...
> Query: select sleep(100) from functional.alltypes limit 25
> Query submitted at: 2020-10-13 11:13:07 (Coordinator:
> http://stakiar-desktop:25000)
> Query progress can be monitored at:
> http://stakiar-desktop:25000/query_plan?query_id=694f94671571d4d1:cdec9db900000000
> Fetched 25 row(s) in 2.64s
> {code}
> Attached the contents of {{profile-output.txt}}
> Relevant portion of the profile:
> {code:java}
> Averaged Fragment F00:(Total: 2s603ms, non-child: 272.519us, % non-child:
> 0.01%)
> ...
> - CompletionTime: -1665218428.000ns
> ...
> - TotalThreadsTotalWallClockTime: -1686005515.000ns
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 2.151ms
> ...
> - TotalTime: -1691524485.000ns
> {code}
> For whatever reason, this only affects the averaged fragment profile. For
> this query, there was only one coordinator fragment and thus only one
> fragment instance. The coordinator fragment instance showed normal timer
> values:
> {code:java}
> Coordinator Fragment F00:
> ...
> - CompletionTime: 2s629ms
> ...
> - TotalThreadsTotalWallClockTime: 2s608ms
> - TotalThreadsSysTime: 0.000ns
> - TotalThreadsUserTime: 2.151ms
> ...
> - TotalTime: 2s603ms
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
