Hello Quanlong Huang, Surya Hebbar, Csaba Ringhofer, Michael Smith, Impala
Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24451
to look at the new patch set (#7).
Change subject: IMPALA-15097: Show RF effectiveness in plan graph
......................................................................
IMPALA-15097: Show RF effectiveness in plan graph
This updates plan-graph.py to visualize runtime filter
effectiveness from scan-node runtime filter counters.
Runtime filter consumer edges are marked as dashed when the target
scan processed input but the filter rejected nothing. At higher
verbosity, the graph also shows compact rejected/total counters for
rows, row groups, files, or splits when those counters are available.
Per-instance scan-node counters are summed when present because the
graph should show total skipped input. Averaged Fragment counters are
used only as fallback for profiles without per-instance filter
counters. If no usable counters are available, runtime filter edges are
left unchanged.
Recognizing the instance section takes more than "Fragment F<n>:". The
coordinator fragment is named "Coordinator Fragment F<n>" and a
fragment holding more than one profile carries the count, as in
"Fragment F<n> [12 instances]"; both forms appear in the profiles under
testdata/impala-profiles. An unrecognized header leaves the parser
inside the preceding Averaged Fragment, so the instance counters are
added to the averaged ones instead of replacing them: on the test
profile RF001 read "rows 150/300 rejected" where the instances alone
say 100/200.
The counters themselves are read as an unaggregated profile writes
them. An aggregated profile has one fragment section per fragment and
renders a counter as "mean=... min=... max=...", of which
str_to_counter_value() takes the mean, so the sum would come out per
instance rather than per query. That is noted where the parsing lives
rather than worked around here.
The filter table parser now handles the current profile format with
the Eff. Tgt. Node(s) column while keeping compatibility with the older
format.
Testing:
- python3 -m py_compile bin/diagnostics/experimental/plan-graph.py \
tests/unittests/test_plan_graph.py
- python3 -m pytest --confcutdir=tests/unittests \
tests/unittests/test_plan_graph.py -q
- Reverted the header regex and checked the two new tests are the ones
that fail
- bin/diagnostics/experimental/plan-graph.py -v1 rf_profile.txt | \
dot -Tsvg -o query-plan-v1.svg
- bin/diagnostics/experimental/plan-graph.py -v2 rf_profile.txt | \
dot -Tsvg -o query-plan-v2.svg
- Smoke-tested --no-color, --no-rf, and an older profile-table format
Change-Id: Ic61ba09b9977da4545f3bfd1489e0856a7c0af40
---
M bin/diagnostics/experimental/plan-graph.py
A tests/unittests/test_plan_graph.py
2 files changed, 476 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/24451/7
--
To view, visit http://gerrit.cloudera.org:8080/24451
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic61ba09b9977da4545f3bfd1489e0856a7c0af40
Gerrit-Change-Number: 24451
Gerrit-PatchSet: 7
Gerrit-Owner: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Aleksandr Efimov <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Surya Hebbar <[email protected]>
