[
https://issues.apache.org/jira/browse/PHOENIX-7003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh Shah updated PHOENIX-7003:
----------------------------------
Description:
Encountered an issue in production cluster where the region inconsistency check
didn't catch region hole and full region overlaps and the range scan query
returned incorrect results (result were out of range). We were running hbck
repair operation _exactly_ at the same time the query was running.
This is the check
[here|https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L707-L720]
but this will check whether current regions end key is greater than previous
region's end key.
{code}
if (Bytes.compareTo(regionLocation.getRegionInfo().getEndKey(),
currentKey) <= 0
&& !Bytes.equals(currentKey, HConstants.EMPTY_START_ROW)
&& !Bytes.equals(regionLocation.getRegionInfo().getEndKey(),
HConstants.EMPTY_END_ROW)) {
GLOBAL_HBASE_COUNTER_METADATA_INCONSISTENCY.increment();
String regionNameString =
new String(regionLocation.getRegionInfo().getRegionName(),
StandardCharsets.UTF_8);
throw new IOException(String.format(
"HBase region information overlap/inconsistencies on region
%s", regionNameString));
}
{code}
The above check will NOT check if there any region holes and full region
overlaps since hbase will always return 2 region locations for the same start
key/end key combination.
was:
Encountered an issue in production cluster where the region inconsistency check
didn't catch region hole and full region overlaps and the range scan query
returned incorrect results (result were out of range). We ran hbck repair
operation _exactly_ at the same time the query was running.
This is the check
[here|https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L707-L720]
but this will check whether current regions end key is greater than previous
region's end key.
{code}
if (Bytes.compareTo(regionLocation.getRegionInfo().getEndKey(),
currentKey) <= 0
&& !Bytes.equals(currentKey, HConstants.EMPTY_START_ROW)
&& !Bytes.equals(regionLocation.getRegionInfo().getEndKey(),
HConstants.EMPTY_END_ROW)) {
GLOBAL_HBASE_COUNTER_METADATA_INCONSISTENCY.increment();
String regionNameString =
new String(regionLocation.getRegionInfo().getRegionName(),
StandardCharsets.UTF_8);
throw new IOException(String.format(
"HBase region information overlap/inconsistencies on region
%s", regionNameString));
}
{code}
The above check will NOT check if there any region holes and full region
overlaps since hbase will always return 2 region locations for the same start
key/end key combination.
> Harden hbase region inconsistencies check in CQSI#getAllTableRegions method.
> ----------------------------------------------------------------------------
>
> Key: PHOENIX-7003
> URL: https://issues.apache.org/jira/browse/PHOENIX-7003
> Project: Phoenix
> Issue Type: Bug
> Components: core
> Reporter: Rushabh Shah
> Priority: Major
>
> Encountered an issue in production cluster where the region inconsistency
> check didn't catch region hole and full region overlaps and the range scan
> query returned incorrect results (result were out of range). We were running
> hbck repair operation _exactly_ at the same time the query was running.
> This is the check
> [here|https://github.com/apache/phoenix/blob/master/phoenix-core/src/main/java/org/apache/phoenix/query/ConnectionQueryServicesImpl.java#L707-L720]
> but this will check whether current regions end key is greater than previous
> region's end key.
> {code}
> if (Bytes.compareTo(regionLocation.getRegionInfo().getEndKey(),
> currentKey) <= 0
> && !Bytes.equals(currentKey, HConstants.EMPTY_START_ROW)
> && !Bytes.equals(regionLocation.getRegionInfo().getEndKey(),
> HConstants.EMPTY_END_ROW)) {
> GLOBAL_HBASE_COUNTER_METADATA_INCONSISTENCY.increment();
> String regionNameString =
> new
> String(regionLocation.getRegionInfo().getRegionName(),
> StandardCharsets.UTF_8);
> throw new IOException(String.format(
> "HBase region information overlap/inconsistencies on
> region %s", regionNameString));
> }
> {code}
> The above check will NOT check if there any region holes and full region
> overlaps since hbase will always return 2 region locations for the same start
> key/end key combination.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
