Andrew Kyle Purtell created PHOENIX-8066:
--------------------------------------------

             Summary: Filter uncovered index data rows before the join
                 Key: PHOENIX-8066
                 URL: https://issues.apache.org/jira/browse/PHOENIX-8066
             Project: Phoenix
          Issue Type: Improvement
    Affects Versions: 5.3.2
            Reporter: Andrew Kyle Purtell
            Assignee: Andrew Kyle Purtell
             Fix For: 5.4.0, 5.3.3


Queries executing against uncovered secondary indexes—both global and 
local—currently defer the entire {{WHERE}} clause to a post-join evaluation 
phase. Because no predicate is pushed down to the underlying data table lookup, 
the region server must execute a point lookup, evaluate tuple projector 
expressions, deep copy joined cells, and construct a post-join tuple for every 
candidate identified in the index. 

When the filter on uncovered columns is highly selective, this pipeline 
performs substantial, wasted work. Non-qualifying rows incur the full overhead 
of cell construction, region operations, and coprocessor hook dispatches. 
Compounding this cost, the scanner pipeline emits an empty result for each 
discarded row, forcing the client and server through an entire 
{{RegionScanner.next()}} round trip per rejected candidate.

To eliminate unnecessary joins and materialization overhead, we introduce an 
early data row filtering phase prior to joined tuple construction. During 
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate 
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the 
uncovered index scanner. The pushdown compiler inspects the candidate 
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict 
safety constraints. Filtering executes after the data row has been fetched and 
passed verification, ensuring background index repair mechanisms operate 
unhindered. A row is discarded if and only if the early filter 
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or 
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or 
unsupported condition, the row is retained. The full post-join filter in 
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early 
filtering undergo standard post-join evaluation, preserving canonical 
short-circuiting rules and error reporting orders.

The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from 
single row step iterations. Rather than surfacing an empty result to the caller 
whenever a row is rejected, fails verification, or lacks a corresponding data 
row, the scanner continuously advances to subsequent index candidates within 
the same invocation. The loop processes candidates until it identifies an 
accepted row, exhausts the current page, or reaches the regional execution time 
limit. When interrupted by a time limit, the scanner returns a dummy row at the 
last consumed index key, ensuring restarted scans do not reread processed 
ranges. 

This optimization applies to server-side uncovered global and local index 
scans, including shared aggregate scan paths. Scans with hash joins, CDC 
scanners, analyze scans, and scans lacking tuple projectors bypass the early 
filter.

No wire format changes. Existing clients continue passing standard filter 
attributes, and result sets remain unchanged.

New tests exercise the predicate classifier across data-only, mixed, 
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, missing 
BSON paths). Tests verify type/sort-order compatibility, error fallback safety, 
and per-row state resets across different storage encodings. Verifies single 
call batch consumption of rejected rows, boundary resumption markers on paging 
timeouts, single row behavior under {{DISTINCT}} queries, index repair 
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}}, 
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}}, 
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths 
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table 
moves, and cluster upgrade profiles.


 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to