Andrew Kyle Purtell created PHOENIX-8066:
--------------------------------------------
Summary: Filter uncovered index data rows before the join
Key: PHOENIX-8066
URL: https://issues.apache.org/jira/browse/PHOENIX-8066
Project: Phoenix
Issue Type: Improvement
Affects Versions: 5.3.2
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
Fix For: 5.4.0, 5.3.3
Queries executing against uncovered secondary indexes—both global and
local—currently defer the entire {{WHERE}} clause to a post-join evaluation
phase. Because no predicate is pushed down to the underlying data table lookup,
the region server must execute a point lookup, evaluate tuple projector
expressions, deep copy joined cells, and construct a post-join tuple for every
candidate identified in the index.
When the filter on uncovered columns is highly selective, this pipeline
performs substantial, wasted work. Non-qualifying rows incur the full overhead
of cell construction, region operations, and coprocessor hook dispatches.
Compounding this cost, the scanner pipeline emits an empty result for each
discarded row, forcing the client and server through an entire
{{RegionScanner.next()}} round trip per rejected candidate.
To eliminate unnecessary joins and materialization overhead, we introduce an
early data row filtering phase prior to joined tuple construction. During
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the
uncovered index scanner. The pushdown compiler inspects the candidate
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict
safety constraints. Filtering executes after the data row has been fetched and
passed verification, ensuring background index repair mechanisms operate
unhindered. A row is discarded if and only if the early filter
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or
unsupported condition, the row is retained. The full post-join filter in
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early
filtering undergo standard post-join evaluation, preserving canonical
short-circuiting rules and error reporting orders.
The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from
single row step iterations. Rather than surfacing an empty result to the caller
whenever a row is rejected, fails verification, or lacks a corresponding data
row, the scanner continuously advances to subsequent index candidates within
the same invocation. The loop processes candidates until it identifies an
accepted row, exhausts the current page, or reaches the regional execution time
limit. When interrupted by a time limit, the scanner returns a dummy row at the
last consumed index key, ensuring restarted scans do not reread processed
ranges.
This optimization applies to server-side uncovered global and local index
scans, including shared aggregate scan paths. Scans with hash joins, CDC
scanners, analyze scans, and scans lacking tuple projectors bypass the early
filter.
No wire format changes. Existing clients continue passing standard filter
attributes, and result sets remain unchanged.
New tests exercise the predicate classifier across data-only, mixed,
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, missing
BSON paths). Tests verify type/sort-order compatibility, error fallback safety,
and per-row state resets across different storage encodings. Verifies single
call batch consumption of rejected rows, boundary resumption markers on paging
timeouts, single row behavior under {{DISTINCT}} queries, index repair
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}},
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}},
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table
moves, and cluster upgrade profiles.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)