[
https://issues.apache.org/jira/browse/IMPALA-9846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surya Hebbar updated IMPALA-9846:
---------------------------------
Description:
The traditional query profile generated bigger forests of runtime profiles and
child counters, from fragments and operators upto instance levels.
This structure of the runtime profile can potentially stress the memory
allocator and use up a lot more memory and cache than is really necessary.
To mitigate these issues, the aggregated profiles were introduced, which are
substantially denser and faster to process for higher 'mt_dop' values.
In the aggregated profile, the depth of the forests have been reduced by
transforming instance-level counters into operator level arrays or maps.
The aggregation is also done in a single step, merging the aggregated thrift
profiles from the executor directly into the final aggregated profile, without
converting it to an unaggregated profile first.
This representation also helps with producing nice, high-level, readable text
profile by default with the option to produce more detailed profiles and
alternate views of the profile when required.
With this change, aggregated runtime profile will be enabled by default with
the 'aggregated_profile' flag set to 'true'. This will serves as replacement
for the previous 'gen_experimental_profile' flag.
For enabling the aggregated profile, more than 2700+ tests, both backend and
end-to-end tests need to be be implemented/modified/corrected to accommodate
the usage of both the aggregated runtime profile and the traditional runtime
profile.
We also need to ensure that the aggregated profile is an adequate replacement,
then shift over the default.
was:We need to ensure that the aggregated profile is an adequate replacement,
then switch over the default.
> Shift to Aggregated Runtime Profile representation
> --------------------------------------------------
>
> Key: IMPALA-9846
> URL: https://issues.apache.org/jira/browse/IMPALA-9846
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Surya Hebbar
> Priority: Critical
> Labels: multithreading
>
> The traditional query profile generated bigger forests of runtime profiles
> and child counters, from fragments and operators upto instance levels.
> This structure of the runtime profile can potentially stress the memory
> allocator and use up a lot more memory and cache than is really necessary.
> To mitigate these issues, the aggregated profiles were introduced, which are
> substantially denser and faster to process for higher 'mt_dop' values.
> In the aggregated profile, the depth of the forests have been reduced by
> transforming instance-level counters into operator level arrays or maps.
> The aggregation is also done in a single step, merging the aggregated thrift
> profiles from the executor directly into the final aggregated profile,
> without converting it to an unaggregated profile first.
> This representation also helps with producing nice, high-level, readable text
> profile by default with the option to produce more detailed profiles and
> alternate views of the profile when required.
> With this change, aggregated runtime profile will be enabled by default with
> the 'aggregated_profile' flag set to 'true'. This will serves as replacement
> for the previous 'gen_experimental_profile' flag.
> For enabling the aggregated profile, more than 2700+ tests, both backend and
> end-to-end tests need to be be implemented/modified/corrected to accommodate
> the usage of both the aggregated runtime profile and the traditional runtime
> profile.
> We also need to ensure that the aggregated profile is an adequate
> replacement, then shift over the default.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
