[
https://issues.apache.org/jira/browse/IMPALA-15287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18111301#comment-18111301
]
Joe McDonnell commented on IMPALA-15287:
----------------------------------------
Existing prototype that pulls in some Google Benchmark code and builds against
libpfm to print some counters for a unit test:
[http://gerrit.cloudera.org:8080/23917]
I think one way to implement this for profiles is to integrate it into
be/src/util/system-state-info.*, which periodically samples CPU/Network/Disk
statistics. These are incorporated into this profile via sampling counters
[https://github.com/apache/impala/blob/7a3cb82b49d845bc5ef597e598f589593dd13549/be/src/runtime/query-state.cc#L220-L252].
The SystemStateInfo could sample these performance counters with the same
frequency and provide it in the profile. Unlike the CPU/Network/Disk
statistics, we want to be able to pick the performance counters at startup via
a startup flag, so we won't have a fixed list of metrics. We would need the
profile labels to be dynamically generated based on the metrics we want to
collect.
> Add ability to collect performance counters directly in Impalad
> ---------------------------------------------------------------
>
> Key: IMPALA-15287
> URL: https://issues.apache.org/jira/browse/IMPALA-15287
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> Google benchmark has integrated ability to sample performance counters. This
> uses a dependency on libpfm to help accomplish this. See
> [https://abseil.io/fast/53]
> Adding an option like this to Impala itself could provide useful information.
> One option would be to have a background thread sampling the counters
> periodically (e.g. every 100ms or 250ms), then grab the overlapping daemon
> level counters and pull them into the runtime profile. For single-query at a
> time benchmarking, this provides useful information at the query level.
> The performance metrics that are probably most interesting to start with
> would be LLC misses or TLB misses.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
