[
https://issues.apache.org/jira/browse/PHOENIX-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-8044:
-----------------------------------------
Summary: During read repair Phoenix skip scan queries can fail with
IllegalStateException (was: Phoenix skip scan queries during read repair can
fail with IllegalStateException)
> During read repair Phoenix skip scan queries can fail with
> IllegalStateException
> --------------------------------------------------------------------------------
>
> Key: PHOENIX-8044
> URL: https://issues.apache.org/jira/browse/PHOENIX-8044
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.3.2
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 5.4.0, 5.3.3
>
>
> During read repair of unverified rows on covered global indexes, Phoenix skip
> scan queries can fail with {{IllegalStateException}} indicating that a cell
> seek hint does not strictly advance ({{{}"The next hint must come after
> previous hint"{}}}). This failure is due to state pollution when the primary
> scan and a nested read repair scan concurrently mutate a shared chain of
> {{Filter}} structures.
> The failure happens under a set of specific conditions:
> - *Covered Global Index Traversal:* The query targets a covered global index
> where {{GlobalIndexChecker}} wraps the underlying region scanner.
> - *Skip-Scan Optimization:* The query execution plan employs a
> {{SkipScanFilter}} (such as evaluating an {{IN}} predicate over leading
> indexed columns).
> - *Unverified Point Mutations:* An intervening index row physically sorted
> between two active skip scan targets remains unverified because of an earlier
> failed secondary write phase. Point rebuilding fails to verify its latest
> cell version.
> - *Immediate Boundary Proximity:* The unverified row is the first entry
> evaluated by the main scanner immediately following a matched skip scan key.
> - *Column Encoded Row Layout:* The empty column cell sorts first (guaranteed
> on tables with column encoding). {{UnverifiedRowFilter}} evaluates this cell,
> identifies the row as unverified, and immediately returns
> {{{}INCLUDE_AND_SEEK_NEXT_ROW{}}}. Consequently, the downstream
> {{SkipScanFilter}} in the filter chain is bypassed on this initial pass.
> - *Trailing Intermediary Rows:* At least one subsequent index row exists
> that also sorts prior to the next skip scan seek target.
> {{GlobalIndexScanner}} constructs its internal scanners, including those for
> read repair, using HBase's copy constructor {{{}new Scan(scan){}}}. This
> constructor only performs a shallow copy of the configured {{Filter}}
> reference. As a result, the primary scanner and all internal downstream
> scanners share the same {{Filter}} instances. While sharing a filter chain is
> valid for scanners that sequentially execute scans, this is not safe for
> concurrent use cases.
> Here is an example of this going awry: When main scanner encounters the
> unverified row, because the empty column appears first,
> {{UnverifiedRowFilter}} intercepts the row and issues an immediate
> row-advance directive without invoking client side filters, so the shared
> {{SkipScanFilter}} remains unaware of this row. When point rebuilding leaves
> the row unverified, {{repairIndexRows()}} launches {{singleRowIndexScan}}
> over the shared filter chain to inspect previous cell versions. This nested
> scan delegates directly to {{{}SkipScanFilter{}}}. The filter evaluates the
> key, computes a seek hint targeting the next skip-scan interval, caches the
> generated hint, and terminates the single row scan. The primary scanner
> resumes and reads the next index row, which still precedes the target skip
> scan key. When the main scanner calls {{{}filterCell(){}}}, the shared
> {{SkipScanFilter}} emits the same seek hint cached during the repair scan.
> HBase's scanner infrastructure enforces monotonic progression via
> {{{}setNextCellHint(){}}}, requiring every hint to sort strictly after the
> prior hint ({{{}next > prev{}}}). Because {{{}next == prev{}}}, the scanner
> aborts with an unhandled exception.
> The fix is to avoid sharing {{Filter}} instances. This is a significant
> general safety improvement because these structures are not designed to be
> shared.
> During initialization of the repair path, {{GlobalIndexChecker}} serializes
> the client filter hierarchy once via {{{}ProtobufUtil{}}}. Each subsequent
> {{singleRowIndexScan}} deserializes a fresh filter instance from this byte
> payload and wraps it with dedicated {{UnverifiedRowFilter}} and
> {{PagingFilter}} instances. This provides complete state isolation for skip
> scan cursors across point lookups at the cost of a single protobuf parse per
> repair, which is negligible given the rarity of read repairs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
