[
https://issues.apache.org/jira/browse/PHOENIX-7893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-7893:
-----------------------------------------
Description:
{{LocalIndexIT}} {{testLocalIndexReverseScanShouldReturnAllRows}} and
{{testLocalIndexUsedForUncoveredOrderBy}} can fail with a
{{StackOverflowError}}.
The issue dates back to PHOENIX-4967 and PHOENIX-4964 when a reverse scan runs
over a multiregion, pre-split local index. The query fails during
{{executeQuery()}} with:
{noformat}
java.lang.StackOverflowError
at
org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1732)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1635)
at
org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
at
org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
... (repeats thousands of times) ...
{noformat}
Test logs contain the same {{StaleRegionBoundaryCacheException}} more than
10,000 times against the same region.
There are two related bugs. The first is a server-side boundary-check bug that
causes a permanent false-positive {{StaleRegionBoundaryCacheException}}. The
second is an unbounded client-side retry that produces a {{StackOverflowError}}
instead of a clean failure.
The trigger is a server-side check in {{BaseScannerRegionObserver.java}}:
{noformat}
if (isLocalIndex) {
byte[] expectedUpperRegionKey =
scan.getAttribute(EXPECTED_UPPER_REGION_KEY) == null
? scan.getStopRow() // <-- fallback used by ALL
regular queries
: scan.getAttribute(EXPECTED_UPPER_REGION_KEY);
byte[] actualStartRow = scan.getAttribute(SCAN_ACTUAL_START_ROW);
isStaleRegionBoundaries =
(expectedUpperRegionKey != null
&& Bytes.compareTo(upperExclusiveRegionKey, expectedUpperRegionKey)
!= 0)
|| (actualStartRow != null
&& Bytes.compareTo(actualStartRow, lowerInclusiveRegionKey) < 0);
}
{noformat}
When the client builds a local index scan in
{{ScanUtil.setLocalIndexAttributes}}, {{SCAN_ACTUAL_START_ROW}} is set, but
{{EXPECTED_UPPER_REGION_KEY}} is not. A repository wide search confirms
{{EXPECTED_UPPER_REGION_KEY}} is only ever set in {{PhoenixInputFormat}}. The
server always falls back to {{scan.getStopRow()}} for the regular query path.
But for a reversed scan, {{startRow}} is the high bound and {{stopRow}} is the
lower bound.
There is evidence of this problem in the test logs, e.g.:
{noformat}
Throwing StaleRegionBoundaryCacheException due to mismatched scan boundaries.
Region: ...,o\x00...\x00,...
lowerInclusiveScanKey: (empty -> high end of reverse
scan)
upperExclusiveScanKey: o\x00\x00...\x00 (= scan.getStopRow(), the LOW
bound)
lowerInclusiveRegionKey: o\x00\x00...\x00
upperExclusiveRegionKey: (empty -> last region)
scan reversed: true
{noformat}
{{expectedUpperRegionKey = scan.getStopRow() = o\x00…}} is wrong, this is the
lower bound.
{{upperExclusiveRegionKey = ""}} is empty, the real upper boundary of the last
region
so {{isStaleRegionBoundaries = true}} but the region boundaries are not
actually stale.
The second issue, leading to a crash, is unbounded retry recursion in
{{BaseResultIterators}}:
{noformat}
} catch (StaleRegionBoundaryCacheException | HashJoinCacheNotFoundException e2)
{
if (!clearedCache) {
services.clearTableRegionCache(TableName.valueOf(physicalTableName));
context.getOverallQueryMetrics().cacheRefreshedDueToSplits();
}
Scan oldScan = scanPair.getFirst();
byte[] startKey = oldScan.getAttribute(SCAN_ACTUAL_START_ROW);
if (e2 instanceof HashJoinCacheNotFoundException) {
if (retryCount <= 0) { // <-- guard EXISTS for the hash-join path
only
throw e2;
}
// ... re-add hash cache ...
}
concatIterators = recreateIterators(services, isLocalIndex, allIterators,
iterators,
isReverse, maxQueryEndTime, previousScan, clearedCache, concatIterators,
scanPairItr, scanPair, retryCount - 1); // <-- decrements, but nothing
checks it
}
{noformat}
{{recreateIterators}} derives new scans and calls {{getIterators}} again, which
reissues the scan, hits the same persistent
{{StaleRegionBoundaryCacheException}}, and reenters the handler. The
{{retryCount}} decrement proves a bound was intended, but the
{{StaleRegionBoundaryCacheException}} path has no check like {{retryCount <=
0}}. Only the {{HashJoinCacheNotFoundException}} branch has a bound on retries.
With a persistent stale condition the mutual recursion between {{getIterators}}
and {{recreateIterators}} never terminates until finally the stack is exhausted
and the JVM throws a {{StackOverflowError}}.
was:
{{LocalIndexIT}} {{testLocalIndexReverseScanShouldReturnAllRows}} and
{{testLocalIndexUsedForUncoveredOrderBy}} can fail with a
{{StackOverflowError}}.
The issue dates back to PHOENIX-4967 and PHOENIX-4964 when a reverse scan runs
over a multiregion, pre-split local index. The query fails during
{{executeQuery()}} with:
{noformat}
java.lang.StackOverflowError
at
org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1732)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1635)
at
org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
at
org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
at
org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
... (repeats thousands of times) ...
{noformat}
Test logs contain the same {{StaleRegionBoundaryCacheException}} more than
10,000 times against the same region.
There are two related bugs. The first is a server-side boundary-check bug that
causes a permanent false-positive {{StaleRegionBoundaryCacheException}}. The
second is an unbounded client-side retry that produces a {{StackOverflowError}}
instead of a clean failure.
The trigger is a server-side check in {{BaseScannerRegionObserver.java}}:
{noformat}
if (isLocalIndex) {
byte[] expectedUpperRegionKey =
scan.getAttribute(EXPECTED_UPPER_REGION_KEY) == null
? scan.getStopRow() // <-- fallback used by ALL
regular queries
: scan.getAttribute(EXPECTED_UPPER_REGION_KEY);
byte[] actualStartRow = scan.getAttribute(SCAN_ACTUAL_START_ROW);
isStaleRegionBoundaries =
(expectedUpperRegionKey != null
&& Bytes.compareTo(upperExclusiveRegionKey, expectedUpperRegionKey)
!= 0)
|| (actualStartRow != null
&& Bytes.compareTo(actualStartRow, lowerInclusiveRegionKey) < 0);
}
{noformat}
When the client builds a local index scan in
{{ScanUtil.setLocalIndexAttributes}}, {{SCAN_ACTUAL_START_ROW}} is set, but
{{EXPECTED_UPPER_REGION_KEY}} is not. A repository wide search confirms
{{EXPECTED_UPPER_REGION_KEY}} is only ever set in {{PhoenixInputFormat}}. The
server always falls back to {{scan.getStopRow()}} for the regular query path.
But for a reversed scan, {{startRow}} is the high bound and {{stopRow}} is the
lower bound.
There is evidence of this problem in the test logs, e.g.:
{noformat}
Throwing StaleRegionBoundaryCacheException due to mismatched scan boundaries.
Region: ...,o\x00...\x00,...
lowerInclusiveScanKey: (empty -> high end of reverse
scan)
upperExclusiveScanKey: o\x00\x00...\x00 (= scan.getStopRow(), the LOW
bound)
lowerInclusiveRegionKey: o\x00\x00...\x00
upperExclusiveRegionKey: (empty -> last region)
scan reversed: true
{noformat}
{{expectedUpperRegionKey = scan.getStopRow() = o\x00…}} is wrong, this is the
lower bound.
{{upperExclusiveRegionKey = ""}} is empty, the real upper boundary of the last
region
so {{isStaleRegionBoundaries = true}} but the region boundaries are not
actually stale.
> Populate EXPECTED_UPPER_REGION_KEY on local index scans
> -------------------------------------------------------
>
> Key: PHOENIX-7893
> URL: https://issues.apache.org/jira/browse/PHOENIX-7893
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.4.0, 5.3.1
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
>
> {{LocalIndexIT}} {{testLocalIndexReverseScanShouldReturnAllRows}} and
> {{testLocalIndexUsedForUncoveredOrderBy}} can fail with a
> {{StackOverflowError}}.
> The issue dates back to PHOENIX-4967 and PHOENIX-4964 when a reverse scan
> runs over a multiregion, pre-split local index. The query fails during
> {{executeQuery()}} with:
> {noformat}
> java.lang.StackOverflowError
> at
> org.apache.phoenix.iterate.BaseResultIterators.close(BaseResultIterators.java:1732)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1635)
> at
> org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
> at
> org.apache.phoenix.iterate.BaseResultIterators.recreateIterators(BaseResultIterators.java:1688)
> at
> org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1584)
> ... (repeats thousands of times) ...
> {noformat}
> Test logs contain the same {{StaleRegionBoundaryCacheException}} more than
> 10,000 times against the same region.
> There are two related bugs. The first is a server-side boundary-check bug
> that causes a permanent false-positive {{StaleRegionBoundaryCacheException}}.
> The second is an unbounded client-side retry that produces a
> {{StackOverflowError}} instead of a clean failure.
> The trigger is a server-side check in {{BaseScannerRegionObserver.java}}:
> {noformat}
> if (isLocalIndex) {
> byte[] expectedUpperRegionKey =
> scan.getAttribute(EXPECTED_UPPER_REGION_KEY) == null
> ? scan.getStopRow() // <-- fallback used by
> ALL regular queries
> : scan.getAttribute(EXPECTED_UPPER_REGION_KEY);
> byte[] actualStartRow = scan.getAttribute(SCAN_ACTUAL_START_ROW);
> isStaleRegionBoundaries =
> (expectedUpperRegionKey != null
> && Bytes.compareTo(upperExclusiveRegionKey, expectedUpperRegionKey)
> != 0)
> || (actualStartRow != null
> && Bytes.compareTo(actualStartRow, lowerInclusiveRegionKey) < 0);
> }
> {noformat}
> When the client builds a local index scan in
> {{ScanUtil.setLocalIndexAttributes}}, {{SCAN_ACTUAL_START_ROW}} is set, but
> {{EXPECTED_UPPER_REGION_KEY}} is not. A repository wide search confirms
> {{EXPECTED_UPPER_REGION_KEY}} is only ever set in {{PhoenixInputFormat}}. The
> server always falls back to {{scan.getStopRow()}} for the regular query path.
> But for a reversed scan, {{startRow}} is the high bound and {{stopRow}} is
> the lower bound.
> There is evidence of this problem in the test logs, e.g.:
> {noformat}
> Throwing StaleRegionBoundaryCacheException due to mismatched scan boundaries.
> Region: ...,o\x00...\x00,...
> lowerInclusiveScanKey: (empty -> high end of reverse
> scan)
> upperExclusiveScanKey: o\x00\x00...\x00 (= scan.getStopRow(), the LOW
> bound)
> lowerInclusiveRegionKey: o\x00\x00...\x00
> upperExclusiveRegionKey: (empty -> last region)
> scan reversed: true
> {noformat}
> {{expectedUpperRegionKey = scan.getStopRow() = o\x00…}} is wrong, this is the
> lower bound.
> {{upperExclusiveRegionKey = ""}} is empty, the real upper boundary of the
> last region
> so {{isStaleRegionBoundaries = true}} but the region boundaries are not
> actually stale.
> The second issue, leading to a crash, is unbounded retry recursion in
> {{BaseResultIterators}}:
> {noformat}
> } catch (StaleRegionBoundaryCacheException | HashJoinCacheNotFoundException
> e2) {
> if (!clearedCache) {
> services.clearTableRegionCache(TableName.valueOf(physicalTableName));
> context.getOverallQueryMetrics().cacheRefreshedDueToSplits();
> }
> Scan oldScan = scanPair.getFirst();
> byte[] startKey = oldScan.getAttribute(SCAN_ACTUAL_START_ROW);
> if (e2 instanceof HashJoinCacheNotFoundException) {
> if (retryCount <= 0) { // <-- guard EXISTS for the hash-join
> path only
> throw e2;
> }
> // ... re-add hash cache ...
> }
> concatIterators = recreateIterators(services, isLocalIndex, allIterators,
> iterators,
> isReverse, maxQueryEndTime, previousScan, clearedCache, concatIterators,
> scanPairItr, scanPair, retryCount - 1); // <-- decrements, but
> nothing checks it
> }
> {noformat}
> {{recreateIterators}} derives new scans and calls {{getIterators}} again,
> which reissues the scan, hits the same persistent
> {{StaleRegionBoundaryCacheException}}, and reenters the handler. The
> {{retryCount}} decrement proves a bound was intended, but the
> {{StaleRegionBoundaryCacheException}} path has no check like {{retryCount <=
> 0}}. Only the {{HashJoinCacheNotFoundException}} branch has a bound on
> retries. With a persistent stale condition the mutual recursion between
> {{getIterators}} and {{recreateIterators}} never terminates until finally the
> stack is exhausted and the JVM throws a {{StackOverflowError}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
