[
https://issues.apache.org/jira/browse/IMPALA-13304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17951539#comment-17951539
]
ASF subversion and git services commented on IMPALA-13304:
----------------------------------------------------------
Commit 7ad7a86c0eedbb2d056b8f8dfb3d4d7cfc145993 in impala's branch
refs/heads/master from Surya Hebbar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=7ad7a86c0 ]
IMPALA-13624: Implement textual representation for aggregate event sequences
This adds support for a summarized textual representation of timestamps
for the event sequences present in the aggregated profile.
With the verbose format present in profile V1 and V2, it becomes
difficult to analyze an event's timestamps across instances.
The event sequences are now displayed in a histogram format, based on
the number of timestamps present, in order to support an easier view
for skew analysis and other possible use cases.
(i.e. based on json_profile_event_timestamp_limit)
The summary generated from aggregated instance-level timestamps
(i.e. IMPALA-13304) is used to achieve this within the profile V2,
which covers the possbility of missing events.
Example,
Verbosity::DEFAULT
json_profile_event_timestamp_limit = 5 (default)
Case #1, Number of instances exceeded limit
Node Lifecycle Event Timeline Summary :
- Open Started (4s880ms):
Min: 2s312ms, Avg: 3s427ms, Max: 4s880ms, Count: 12
HistogramCount: 4, 4, 0, 0, 4
Case #2, Number of instances within the limit
Node Lifecycle Event Timeline:
- Open Started: 5s885ms, 1s708ms, 3s434ms
- Open Finished: 5s885ms, 1s708ms, 3s435ms
- First Batch Requested: 5s885ms, 1s708ms, 3s435ms
- First Batch Returned: 6s319ms, 2s123ms, 3s570ms
- Last Batch Returned: 7s878ms, 2s123ms, 3s570ms
With Verbosity::EXTENDED or more, all events and timestamps are printed
with full verbosity as before.
Tests:
For test_profile_tool.py, updated the generated outputs for text
and JSON profiles.
Change-Id: I4bcc0e2e7fccfa8a184cfa8a3a96d68bfe6035c0
Reviewed-on: http://gerrit.cloudera.org:8080/22245
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Riza Suminto <[email protected]>
> Include aggregate instance-level metrics within experimental profile(V2)
> ------------------------------------------------------------------------
>
> Key: IMPALA-13304
> URL: https://issues.apache.org/jira/browse/IMPALA-13304
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Surya Hebbar
> Assignee: Surya Hebbar
> Priority: Major
>
> Currently, instance-level details of fragment events are completely omitted
> from the experimental profile(V2) in contrast with the traditional profile.
> This is to stop the profile size from blowing up with increasing number of
> nodes.
> Due to this, the query timeline's fragment timing diagram does not function.
> The generated profiles can be switched by setting
> {{gen_experimental_profile}} to {{true}} or {{{}false{}}}.
> It would be beneficial to include aggregate instance-level metrics without
> blowing up the profile size in case of large number of nodes. This would
> further help in plotting the query timeline's fragment timing diagram as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
