[
https://issues.apache.org/jira/browse/PHOENIX-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-7893:
-----------------------------------------
Summary: False stale region boundary cache condition and unlimited
recursion when reverse scanning over local indexes (was: Populate
EXPECTED_UPPER_REGION_KEY on local index scans)
> False stale region boundary cache condition and unlimited recursion when
> reverse scanning over local indexes
> ------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-7893
> URL: https://issues.apache.org/jira/browse/PHOENIX-7893
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.4.0, 5.3.1
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
>
> {{LocalIndexIT}} {{testLocalIndexReverseScanShouldReturnAllRows}} and
> {{testLocalIndexUsedForUncoveredOrderBy}} can fail with a
> {{StackOverflowError}}.
> The issue dates back to PHOENIX-4967 and PHOENIX-4964 when a reverse scan
> runs over a multiregion, pre-split local index. The query fails during
> {{executeQuery()}} with:
> {noformat}
> java.lang.StackOverflowError
> at
> org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1732)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1635)
> at
> org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
> at
> org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
> ... (repeats thousands of times) ...
> {noformat}
> Test logs contain the same {{StaleRegionBoundaryCacheException}} more than
> 10,000 times against the same region.
> There are two related bugs. The first is a server-side boundary-check bug
> that causes a permanent false-positive {{StaleRegionBoundaryCacheException}}.
> The second is an unbounded client-side retry that produces a
> {{StackOverflowError}} instead of a clean failure.
> The trigger is a server-side check in {{BaseScannerRegionObserver.java}}:
> {noformat}
> if (isLocalIndex) {
> byte[] expectedUpperRegionKey =
> scan.getAttribute(EXPECTED_UPPER_REGION_KEY) == null
> ? scan.getStopRow() // <-- fallback used by
> ALL regular queries
> : scan.getAttribute(EXPECTED_UPPER_REGION_KEY);
> byte[] actualStartRow = scan.getAttribute(SCAN_ACTUAL_START_ROW);
> isStaleRegionBoundaries =
> (expectedUpperRegionKey != null
> && Bytes.compareTo(upperExclusiveRegionKey, expectedUpperRegionKey)
> != 0)
> || (actualStartRow != null
> && Bytes.compareTo(actualStartRow, lowerInclusiveRegionKey) < 0);
> }
> {noformat}
> When the client builds a local index scan in
> {{ScanUtil.setLocalIndexAttributes}}, {{SCAN_ACTUAL_START_ROW}} is set, but
> {{EXPECTED_UPPER_REGION_KEY}} is not. A repository wide search confirms
> {{EXPECTED_UPPER_REGION_KEY}} is only ever set in {{PhoenixInputFormat}}. The
> server always falls back to {{scan.getStopRow()}} for the regular query path.
> But for a reversed scan, {{startRow}} is the high bound and {{stopRow}} is
> the lower bound.
> There is evidence of this problem in the test logs, e.g.:
> {noformat}
> Throwing StaleRegionBoundaryCacheException due to mismatched scan boundaries.
> Region: ...,o\x00...\x00,...
> lowerInclusiveScanKey: (empty -> high end of reverse
> scan)
> upperExclusiveScanKey: o\x00\x00...\x00 (= scan.getStopRow(), the LOW
> bound)
> lowerInclusiveRegionKey: o\x00\x00...\x00
> upperExclusiveRegionKey: (empty -> last region)
> scan reversed: true
> {noformat}
> {{expectedUpperRegionKey = scan.getStopRow() = o\x00…}} is wrong, this is the
> lower bound.
> {{upperExclusiveRegionKey = ""}} is empty, the real upper boundary of the
> last region
> so {{isStaleRegionBoundaries = true}} but the region boundaries are not
> actually stale.
> The second issue, leading to a crash, is unbounded retry recursion in
> {{BaseResultIterators}}:
> {noformat}
> } catch (StaleRegionBoundaryCacheException | HashJoinCacheNotFoundException
> e2) {
> if (!clearedCache) {
> services.clearTableRegionCache(TableName.valueOf(physicalTableName));
> context.getOverallQueryMetrics().cacheRefreshedDueToSplits();
> }
> Scan oldScan = scanPair.getFirst();
> byte[] startKey = oldScan.getAttribute(SCAN_ACTUAL_START_ROW);
> if (e2 instanceof HashJoinCacheNotFoundException) {
> if (retryCount <= 0) { // <-- guard EXISTS for the hash-join
> path only
> throw e2;
> }
> // ... re-add hash cache ...
> }
> concatIterators = recreateIterators(services, isLocalIndex, allIterators,
> iterators,
> isReverse, maxQueryEndTime, previousScan, clearedCache, concatIterators,
> scanPairItr, scanPair, retryCount - 1); // <-- decrements, but
> nothing checks it
> }
> {noformat}
> {{recreateIterators}} derives new scans and calls {{getIterators}} again,
> which reissues the scan, hits the same persistent
> {{StaleRegionBoundaryCacheException}}, and reenters the handler. The
> {{retryCount}} decrement proves a bound was intended, but the
> {{StaleRegionBoundaryCacheException}} path has no check like {{retryCount <=
> 0}}. Only the {{HashJoinCacheNotFoundException}} branch has a bound on
> retries. With a persistent stale condition the mutual recursion between
> {{getIterators}} and {{recreateIterators}} never terminates until finally the
> stack is exhausted and the JVM throws a {{StackOverflowError}}.
> The trigger requires a reverse or descending or uncovered {{ORDER BY}} local
> index scan over a table whose region layout has an open-ended last region.
> The bug surfaces intermittently depending on split timing and region layout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
