[ 
https://issues.apache.org/jira/browse/PHOENIX-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Kyle Purtell updated PHOENIX-8066:
-----------------------------------------
    Description: 
Queries executing against uncovered secondary indexes—both global and 
local—currently defer the entire {{WHERE}} clause to a post-join evaluation 
phase. Because no predicate is pushed down to the underlying data table lookup, 
the region server must execute a point lookup, evaluate tuple projector 
expressions, deep copy joined cells, and construct a post-join tuple for every 
candidate identified in the index. 

When the filter on uncovered columns is highly selective, this pipeline 
performs substantial, wasted work. Non-qualifying rows incur the full overhead 
of cell construction, region operations, and coprocessor hook dispatches. 
Compounding this cost, the scanner pipeline emits an empty result for each 
discarded row, forcing the client and server through an entire 
{{RegionScanner.next()}} round trip per rejected candidate.

To eliminate unnecessary joins and materialization overhead, we introduce an 
early data row filtering phase prior to joined tuple construction. During 
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate 
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the 
uncovered index scanner. The pushdown compiler inspects the candidate 
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict 
safety constraints. Filtering executes after the data row has been fetched and 
passed verification, ensuring background index repair mechanisms operate 
unhindered. A row is discarded if and only if the early filter 
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or 
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or 
unsupported condition, the row is retained. The full post-join filter in 
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early 
filtering undergo standard post-join evaluation, preserving canonical 
short-circuiting rules and error reporting orders.

The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from 
single row step iterations. Rather than surfacing an empty result to the caller 
whenever a row is rejected, fails verification, or lacks a corresponding data 
row, the scanner continuously advances to subsequent index candidates within 
the same invocation. The loop processes candidates until it identifies an 
accepted row, exhausts the current page, or reaches the regional execution time 
limit. When interrupted by a time limit, the scanner returns a dummy row at the 
last consumed index key, ensuring restarted scans do not reread processed 
ranges. 

This optimization applies to server-side uncovered global and local index 
scans, including shared aggregate scan paths. Scans with hash joins, CDC 
scanners, analyze scans, and scans lacking tuple projectors bypass the early 
filter.

No wire format changes. Existing clients continue passing standard filter 
attributes, and result sets remain unchanged.

New tests exercise the predicate classifier across data-only, mixed, 
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, BSON 
paths). Tests verify type/sort-order compatibility, error fallback safety, and 
per-row state resets across different storage encodings. Verifies single call 
batch consumption of rejected rows, boundary resumption markers on paging 
timeouts, single row behavior under {{DISTINCT}} queries, index repair 
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}}, 
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}}, 
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths 
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table 
moves, and cluster upgrade profiles.

 

  was:
Queries executing against uncovered secondary indexes—both global and 
local—currently defer the entire {{WHERE}} clause to a post-join evaluation 
phase. Because no predicate is pushed down to the underlying data table lookup, 
the region server must execute a point lookup, evaluate tuple projector 
expressions, deep copy joined cells, and construct a post-join tuple for every 
candidate identified in the index. 

When the filter on uncovered columns is highly selective, this pipeline 
performs substantial, wasted work. Non-qualifying rows incur the full overhead 
of cell construction, region operations, and coprocessor hook dispatches. 
Compounding this cost, the scanner pipeline emits an empty result for each 
discarded row, forcing the client and server through an entire 
{{RegionScanner.next()}} round trip per rejected candidate.

To eliminate unnecessary joins and materialization overhead, we introduce an 
early data row filtering phase prior to joined tuple construction. During 
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate 
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the 
uncovered index scanner. The pushdown compiler inspects the candidate 
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict 
safety constraints. Filtering executes after the data row has been fetched and 
passed verification, ensuring background index repair mechanisms operate 
unhindered. A row is discarded if and only if the early filter 
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or 
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or 
unsupported condition, the row is retained. The full post-join filter in 
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early 
filtering undergo standard post-join evaluation, preserving canonical 
short-circuiting rules and error reporting orders.

The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from 
single row step iterations. Rather than surfacing an empty result to the caller 
whenever a row is rejected, fails verification, or lacks a corresponding data 
row, the scanner continuously advances to subsequent index candidates within 
the same invocation. The loop processes candidates until it identifies an 
accepted row, exhausts the current page, or reaches the regional execution time 
limit. When interrupted by a time limit, the scanner returns a dummy row at the 
last consumed index key, ensuring restarted scans do not reread processed 
ranges. 

This optimization applies to server-side uncovered global and local index 
scans, including shared aggregate scan paths. Scans with hash joins, CDC 
scanners, analyze scans, and scans lacking tuple projectors bypass the early 
filter.

No wire format changes. Existing clients continue passing standard filter 
attributes, and result sets remain unchanged.

New tests exercise the predicate classifier across data-only, mixed, 
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, missing 
BSON paths). Tests verify type/sort-order compatibility, error fallback safety, 
and per-row state resets across different storage encodings. Verifies single 
call batch consumption of rejected rows, boundary resumption markers on paging 
timeouts, single row behavior under {{DISTINCT}} queries, index repair 
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}}, 
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}}, 
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths 
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table 
moves, and cluster upgrade profiles.


 


> Filter uncovered index data rows before the join
> ------------------------------------------------
>
>                 Key: PHOENIX-8066
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-8066
>             Project: Phoenix
>          Issue Type: Improvement
>    Affects Versions: 5.3.2
>            Reporter: Andrew Kyle Purtell
>            Assignee: Andrew Kyle Purtell
>            Priority: Major
>             Fix For: 5.4.0, 5.3.3
>
>
> Queries executing against uncovered secondary indexes—both global and 
> local—currently defer the entire {{WHERE}} clause to a post-join evaluation 
> phase. Because no predicate is pushed down to the underlying data table 
> lookup, the region server must execute a point lookup, evaluate tuple 
> projector expressions, deep copy joined cells, and construct a post-join 
> tuple for every candidate identified in the index. 
> When the filter on uncovered columns is highly selective, this pipeline 
> performs substantial, wasted work. Non-qualifying rows incur the full 
> overhead of cell construction, region operations, and coprocessor hook 
> dispatches. Compounding this cost, the scanner pipeline emits an empty result 
> for each discarded row, forcing the client and server through an entire 
> {{RegionScanner.next()}} round trip per rejected candidate.
> To eliminate unnecessary joins and materialization overhead, we introduce an 
> early data row filtering phase prior to joined tuple construction. During 
> scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate 
> conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the 
> uncovered index scanner. The pushdown compiler inspects the candidate 
> expressions, retaining only top-level {{AND}} conjuncts that satisfy strict 
> safety constraints. Filtering executes after the data row has been fetched 
> and passed verification, ensuring background index repair mechanisms operate 
> unhindered. A row is discarded if and only if the early filter 
> deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or 
> {{{}NULL{}}}). If evaluation encounters an error, incomplete state, or 
> unsupported condition, the row is retained. The full post-join filter in 
> {{RegionScannerFactory}} remains untouched. Rows that bypass or pass early 
> filtering undergo standard post-join evaluation, preserving canonical 
> short-circuiting rules and error reporting orders.
> The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from 
> single row step iterations. Rather than surfacing an empty result to the 
> caller whenever a row is rejected, fails verification, or lacks a 
> corresponding data row, the scanner continuously advances to subsequent index 
> candidates within the same invocation. The loop processes candidates until it 
> identifies an accepted row, exhausts the current page, or reaches the 
> regional execution time limit. When interrupted by a time limit, the scanner 
> returns a dummy row at the last consumed index key, ensuring restarted scans 
> do not reread processed ranges. 
> This optimization applies to server-side uncovered global and local index 
> scans, including shared aggregate scan paths. Scans with hash joins, CDC 
> scanners, analyze scans, and scans lacking tuple projectors bypass the early 
> filter.
> No wire format changes. Existing clients continue passing standard filter 
> attributes, and result sets remain unchanged.
> New tests exercise the predicate classifier across data-only, mixed, 
> index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, BSON 
> paths). Tests verify type/sort-order compatibility, error fallback safety, 
> and per-row state resets across different storage encodings. Verifies single 
> call batch consumption of rejected rows, boundary resumption markers on 
> paging timeouts, single row behavior under {{DISTINCT}} queries, index repair 
> precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}}, 
> {{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}}, 
> {{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths 
> ({{{}NO_INDEX{}}}) and validate stable execution across region splits, table 
> moves, and cluster upgrade profiles.
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to