[
https://issues.apache.org/jira/browse/IMPALA-14796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18088412#comment-18088412
]
ASF subversion and git services commented on IMPALA-14796:
----------------------------------------------------------
Commit 8f6fdc0f3910503556fc088cc4ef306ac5e96009 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=8f6fdc0f3 ]
IMPALA-14796: Show effective runtime filter targets in profile
This patch adds an "Eff. Tgt. Node(s)" (Effective Target Node(s)) column
to the "Final filter table" in the query profile. This shows which scan
nodes actually had rows rejected by each runtime filter, distinguishing
filters that were effective from those that were applied but rejected no
data. E.g.
ID Src. Node Tgt. Node(s) Eff. Tgt. Node(s) Target type ...
--------------------------------------------------------------------
10 6 2 2 LOCAL ...
8 7 1 1 REMOTE ...
5 8 2 2 LOCAL ...
4 8 0 N REMOTE ...
2 9 0, 3 0, 3 REMOTE, REMOTE ...
0 10 4 4 REMOTE ...
In the above example, filter 4 has "N" in the "Eff. Tgt. Node(s)"
column, which means it doesn't filter out any rows, i.e. effective
target node is "None". All the other filters are effective.
Implementation
- In ScanNode::Close(), collect the effective runtime filter ids by
checking the "rejected" counters of all the FilterStats. These
counters correspond to "Files rejected", "RowGroups rejected", "Rows
rejected", "Splits rejected" in the query profile. If any of them is
non-zero, the filter has rejected some data so it's effective.
- Executor reports this info to coordinator via ReportExecStatus RPCs.
A list of (filter_id, scan_node_id) pairs is added in
ReportExecStatusRequestPB to carry this info.
- Coordinator aggregates the effective filter targets when processing
the status reports.
- In FilterDebugString(), add a column to show the node ids where the
runtime filter is effective.
Other minor changes
- In coordinator.cc, move the code of setting the "Final filter table"
from ReleaseExecResources() to ComputeQuerySummary() to ensure the
final status reports from backends all arrive.
- Removed temp_object_pool and temp_mem_tracker from
FilterDebugString() as they have been unused since commit a985e11.
- Replaced boost::lexical_cast<string> with std::to_string in
converting int to string which is more optimized.
- Sort node ids in "Tgt. Node(s)" and "Eff. Tgt. Node(s)" columns to
make the output consistent across different runs.
Limitation
- Kudu scanner doesn't expose metrics reflecting effect of individual
filters so we can't detect effective runtime filters on KuduScanNode.
Currently the "Eff. Tgt. Node(s)" column of them always has value "N"
(IMPALA-15002).
Tests
- Added e2e test for TPCH-Q5 where some filters are ineffective in
both the original profile and aggregated profile modes.
- Added checks in runtime_filters.test for queries that have only one
runtime filter.
- Updated in_list_filters.test for the new column.
- Ran tests on both the original planner and the calcite planner.
Assisted-by: Claude Sonnet 4.5
Change-Id: Iccf4b87ac4579a70273f3306ec7b58850f06b17c
Reviewed-on: http://gerrit.cloudera.org:8080/24123
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Add "effective" column in "Final filter table" in query profile
> ---------------------------------------------------------------
>
> Key: IMPALA-14796
> URL: https://issues.apache.org/jira/browse/IMPALA-14796
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
> Labels: ramp-up
>
> In the query profile, there is a section about runtime filters, e.g., for
> TPCH-Q5:
> {noformat}
> Final filter table:
> ID Src. Node Tgt. Node(s) Target type Partition filter Pending
> (Expected) First arrived Completed Enabled Bloom Size Est fpp Min
> value Max value In-list size
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 10 6 2 LOCAL false
> 0 (3) N/A N/A true 1.00 MB 2.06e-06
>
> 8 7 1 REMOTE false
> 0 (3) 392.242ms 393.338ms true 1.00 MB 2.06e-06
>
> 5 8 2 LOCAL false
> 0 (3) N/A N/A true 1.00 MB 3.54e-11
>
> 4 8 0 REMOTE false
> 0 (3) 351.647ms 351.978ms true 1.00 MB 7.53e-16
>
> 2 9 0, 3 REMOTE, REMOTE false, false
> 0 (3) 347.219ms 347.494ms true 1.00 MB 7.53e-16
>
> 0 10 4 REMOTE false
> 0 (3) 342.907ms 343.293ms true 1.00 MB 2.79e-17{noformat}
> It'd be helpful to add a boolean column "effective" to show whether the
> filter actually rejects any data (rows/RowGroups/splits/files).
> Currently, we have to check the "rejected" counters of the ScanNodes, e.g.,
> {noformat}
> Filter 2 (1.00 MB):
> - Files processed: 0 (0)
> - Files rejected: 0 (0)
> - Files total: 0 (0)
> - RowGroups processed: 1 (1)
> - RowGroups rejected: 0 (0)
> - RowGroups total: 1 (1)
> - Rows processed: 150.00K (150000)
> - Rows rejected: 119.82K (119817)
> - Rows total: 150.00K (150000)
> - Splits processed: 0 (0)
> - Splits rejected: 0 (0)
> - Splits total: 0 (0)
> Filter 4 (1.00 MB):
> - Files processed: 0 (0)
> - Files rejected: 0 (0)
> - Files total: 0 (0)
> - RowGroups processed: 1 (1)
> - RowGroups rejected: 0 (0)
> - RowGroups total: 1 (1)
> - Rows processed: 16.38K (16384)
> - Rows rejected: 0 (0)
> - Rows total: 30.18K (30183)
> - Splits processed: 0 (0)
> - Splits rejected: 0 (0)
> - Splits total: 0 (0){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
