[
https://issues.apache.org/jira/browse/PHOENIX-8066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-8066:
-----------------------------------------
Description:
Queries executing against uncovered secondary indexes currently defer the
entire {{WHERE}} clause to a post-join evaluation phase. Because no predicate
is pushed down to the underlying data table lookup, the region server must
execute a point lookup, evaluate tuple projector expressions, deep copy joined
cells, and construct a post-join tuple for every candidate identified in the
index.
When the filter on uncovered columns is highly selective, this pipeline
performs substantial, wasted work. Non-qualifying rows incur the full overhead
of cell construction, region operations, and coprocessor hook dispatches.
Compounding this cost, the scanner pipeline emits an empty result for each
discarded row, forcing the client and server through an entire
{{RegionScanner.next()}} round trip per rejected candidate.
To eliminate unnecessary joins and materialization overhead, we introduce an
early data row filtering phase prior to joined tuple construction. During
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the
uncovered index scanner. The pushdown compiler inspects the candidate
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict
safety constraints. Filtering executes after the data row has been fetched and
passed verification, ensuring background index repair mechanisms operate
unhindered. A row is discarded if and only if the early filter
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or
unsupported condition, the row is retained. The full post-join filter in
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early
filtering undergo standard post-join evaluation, preserving canonical
short-circuiting rules and error reporting orders.
The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from
single row step iterations. Rather than surfacing an empty result to the caller
whenever a row is rejected, fails verification, or lacks a corresponding data
row, the scanner continuously advances to subsequent index candidates within
the same invocation. The loop processes candidates until it identifies an
accepted row, exhausts the current page, or reaches the regional execution time
limit. When interrupted by a time limit, the scanner returns a dummy row at the
last consumed index key, ensuring restarted scans do not reread processed
ranges.
This optimization applies to server-side uncovered global and local index
scans, including shared aggregate scan paths. Scans with hash joins, CDC
scanners, analyze scans, and scans lacking tuple projectors bypass the early
filter.
No wire format changes. Existing clients continue passing standard filter
attributes, and result sets remain unchanged.
New tests exercise the predicate classifier across data-only, mixed,
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, BSON
paths). Tests verify type/sort-order compatibility, error fallback safety, and
per-row state resets across different storage encodings. Verifies single call
batch consumption of rejected rows, boundary resumption markers on paging
timeouts, single row behavior under {{DISTINCT}} queries, index repair
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}},
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}},
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table
moves, and cluster upgrade profiles.
was:
Queries executing against uncovered secondary indexes—both global and
local—currently defer the entire {{WHERE}} clause to a post-join evaluation
phase. Because no predicate is pushed down to the underlying data table lookup,
the region server must execute a point lookup, evaluate tuple projector
expressions, deep copy joined cells, and construct a post-join tuple for every
candidate identified in the index.
When the filter on uncovered columns is highly selective, this pipeline
performs substantial, wasted work. Non-qualifying rows incur the full overhead
of cell construction, region operations, and coprocessor hook dispatches.
Compounding this cost, the scanner pipeline emits an empty result for each
discarded row, forcing the client and server through an entire
{{RegionScanner.next()}} round trip per rejected candidate.
To eliminate unnecessary joins and materialization overhead, we introduce an
early data row filtering phase prior to joined tuple construction. During
scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate
conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the
uncovered index scanner. The pushdown compiler inspects the candidate
expressions, retaining only top-level {{AND}} conjuncts that satisfy strict
safety constraints. Filtering executes after the data row has been fetched and
passed verification, ensuring background index repair mechanisms operate
unhindered. A row is discarded if and only if the early filter
deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or
{{{}NULL{}}}). If evaluation encounters an error, incomplete state, or
unsupported condition, the row is retained. The full post-join filter in
{{RegionScannerFactory}} remains untouched. Rows that bypass or pass early
filtering undergo standard post-join evaluation, preserving canonical
short-circuiting rules and error reporting orders.
The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from
single row step iterations. Rather than surfacing an empty result to the caller
whenever a row is rejected, fails verification, or lacks a corresponding data
row, the scanner continuously advances to subsequent index candidates within
the same invocation. The loop processes candidates until it identifies an
accepted row, exhausts the current page, or reaches the regional execution time
limit. When interrupted by a time limit, the scanner returns a dummy row at the
last consumed index key, ensuring restarted scans do not reread processed
ranges.
This optimization applies to server-side uncovered global and local index
scans, including shared aggregate scan paths. Scans with hash joins, CDC
scanners, analyze scans, and scans lacking tuple projectors bypass the early
filter.
No wire format changes. Existing clients continue passing standard filter
attributes, and result sets remain unchanged.
New tests exercise the predicate classifier across data-only, mixed,
index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, BSON
paths). Tests verify type/sort-order compatibility, error fallback safety, and
per-row state resets across different storage encodings. Verifies single call
batch consumption of rejected rows, boundary resumption markers on paging
timeouts, single row behavior under {{DISTINCT}} queries, index repair
precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}},
{{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}},
{{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths
({{{}NO_INDEX{}}}) and validate stable execution across region splits, table
moves, and cluster upgrade profiles.
> Filter uncovered index data rows before the join
> ------------------------------------------------
>
> Key: PHOENIX-8066
> URL: https://issues.apache.org/jira/browse/PHOENIX-8066
> Project: Phoenix
> Issue Type: Improvement
> Affects Versions: 5.3.2
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 5.4.0, 5.3.3
>
>
> Queries executing against uncovered secondary indexes currently defer the
> entire {{WHERE}} clause to a post-join evaluation phase. Because no predicate
> is pushed down to the underlying data table lookup, the region server must
> execute a point lookup, evaluate tuple projector expressions, deep copy
> joined cells, and construct a post-join tuple for every candidate identified
> in the index.
> When the filter on uncovered columns is highly selective, this pipeline
> performs substantial, wasted work. Non-qualifying rows incur the full
> overhead of cell construction, region operations, and coprocessor hook
> dispatches. Compounding this cost, the scanner pipeline emits an empty result
> for each discarded row, forcing the client and server through an entire
> {{RegionScanner.next()}} round trip per rejected candidate.
> To eliminate unnecessary joins and materialization overhead, we introduce an
> early data row filtering phase prior to joined tuple construction. During
> scanner initialization in {{{}RegionScannerFactory{}}}, safe predicate
> conjuncts are extracted from {{INDEX_FILTER}} and pushed directly into the
> uncovered index scanner. The pushdown compiler inspects the candidate
> expressions, retaining only top-level {{AND}} conjuncts that satisfy strict
> safety constraints. Filtering executes after the data row has been fetched
> and passed verification, ensuring background index repair mechanisms operate
> unhindered. A row is discarded if and only if the early filter
> deterministically evaluates to non-{{{}TRUE{}}} (evaluating to {{FALSE}} or
> {{{}NULL{}}}). If evaluation encounters an error, incomplete state, or
> unsupported condition, the row is retained. The full post-join filter in
> {{RegionScannerFactory}} remains untouched. Rows that bypass or pass early
> filtering undergo standard post-join evaluation, preserving canonical
> short-circuiting rules and error reporting orders.
> The internal scanning loop ({{{}getNextCoveredIndexRow{}}}) is decoupled from
> single row step iterations. Rather than surfacing an empty result to the
> caller whenever a row is rejected, fails verification, or lacks a
> corresponding data row, the scanner continuously advances to subsequent index
> candidates within the same invocation. The loop processes candidates until it
> identifies an accepted row, exhausts the current page, or reaches the
> regional execution time limit. When interrupted by a time limit, the scanner
> returns a dummy row at the last consumed index key, ensuring restarted scans
> do not reread processed ranges.
> This optimization applies to server-side uncovered global and local index
> scans, including shared aggregate scan paths. Scans with hash joins, CDC
> scanners, analyze scans, and scans lacking tuple projectors bypass the early
> filter.
> No wire format changes. Existing clients continue passing standard filter
> attributes, and result sets remain unchanged.
> New tests exercise the predicate classifier across data-only, mixed,
> index-only, and complex expressions ({{{}CASE{}}}, {{{}COALESCE{}}}, BSON
> paths). Tests verify type/sort-order compatibility, error fallback safety,
> and per-row state resets across different storage encodings. Verifies single
> call batch consumption of rejected rows, boundary resumption markers on
> paging timeouts, single row behavior under {{DISTINCT}} queries, index repair
> precedence, and server side limits. End to end ITs ({{{}ServerPagingIT{}}},
> {{{}UncoveredGlobalIndexRegionScannerIT{}}}, {{{}GlobalIndexCheckerIT{}}},
> {{{}LocalIndexIT{}}}) confirm parity against non-indexed scan paths
> ({{{}NO_INDEX{}}}) and validate stable execution across region splits, table
> moves, and cluster upgrade profiles.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)