[
https://issues.apache.org/jira/browse/PHOENIX-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807832#comment-17807832
]
ASF GitHub Bot commented on PHOENIX-7106:
-----------------------------------------
virajjasani commented on code in PR #1736:
URL: https://github.com/apache/phoenix/pull/1736#discussion_r1456192978
##########
phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/UncoveredIndexRegionScanner.java:
##########
@@ -404,4 +421,37 @@ public boolean next(List<Cell> result) throws IOException {
region.closeRegionOperation();
}
}
+
+ /**
+ * Add dummy cell to the result list based on either the previous rowkey
returned to the
+ * client or the start rowkey and start rowkey include params.
+ *
+ * @param result result to add the dummy cell to.
+ * @param initStartRowKey scan start rowkey.
+ * @param includeInitStartRowKey scan start rowkey included.
+ * @param scan scan object.
+ */
+ private void updateDummyWithPrevRowKey(List<Cell> result, byte[]
initStartRowKey,
+ boolean includeInitStartRowKey,
Scan scan) {
+ result.clear();
+ if (previousResultRowKey != null) {
+ getDummyResult(previousResultRowKey, result);
+ } else {
+ if (includeInitStartRowKey && initStartRowKey.length > 0) {
+ byte[] prevKey;
+ if (Bytes.compareTo(initStartRowKey, initStartRowKey.length -
1,
+ 1, Bytes.toBytesBinary("\\x00"), 0, 1) == 0) {
+ prevKey = new byte[initStartRowKey.length - 1];
+ System.arraycopy(initStartRowKey, 0, prevKey, 0,
prevKey.length);
+ } else {
+ prevKey =
ByteUtil.previousKeyWithLength(ByteUtil.concat(initStartRowKey,
+ new byte[10]), initStartRowKey.length + 10);
Review Comment:
Finally, the build results look good.
However, this is very interesting point and something hbase should document
it well. The idea is to be able to return large rowkeys upto length: (2^15 -
1), but that does not mean client should be able to store rowkey of this max
length. The max length of the rowkey should be determined based on:
1. table name bytes
2. rowkey bytes
3. length of string "99999999999999" (used by hbase to formulate region name
during meta lookup)
4. whether meta lookup uses new format of region encoding (if yes, it will
add MD5_HEX_LENGTH into calculation)
Concatenation of above params should be of max length (2^15 - 1), not just
rowkey bytes alone.
This is important to ensure when hbase client performs meta lookup for the
large rowkey, it is able to keep scan start rowkey length <= (2^15 - 1).
> Data Integrity issues due to invalid rowkeys returned by various coprocessors
> -----------------------------------------------------------------------------
>
> Key: PHOENIX-7106
> URL: https://issues.apache.org/jira/browse/PHOENIX-7106
> Project: Phoenix
> Issue Type: Improvement
> Affects Versions: 5.2.0, 5.1.4
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Blocker
> Fix For: 5.2.0, 5.1.4
>
>
> HBase scanner interface expects server to perform scan of the cells from
> HFile or Block cache and return consistent data i.e. rowkey of the cells
> returned should stay in the range of the scan boundaries. When a region moves
> and scanner needs reset, or if the current row is too large and the server
> returns partial row, the subsequent scanner#next is supposed to return
> remaining cells. When this happens, cell rowkeys returned by servers i.e. any
> coprocessors is expected to be in the scan boundary range so that server can
> reliably perform its validation and return remaining cells as expected.
> Phoenix client initiates serial or parallel scans from the aggregators based
> on the region boundaries and the scan boundaries are sometimes adjusted based
> on where optimizer provided key ranges, to include tenant boundaries, salt
> boundaries etc. After the client opens the scanner and performs scan
> operation, some of the coprocs return invalid rowkey for the following cases:
> # Grouped aggregate queries
> # Some Ungrouped aggregate queries
> # Offset queries
> # Dummy cells returned with empty rowkey
> # Update statistics queries
> # Uncovered Index queries
> # Ordered results at server side
> # ORDER BY DESC on rowkey
> # Global Index read-repair
> # Paging region scanner with HBase scanner reopen
> # ORDER BY on non-pk column(s) with/without paging
> # GROUP BY on non-pk column(s) with/without paging
> Since many of these cases return reserved rowkeys, they are likely not going
> to match scan or region boundaries. It has potential to cause data integrity
> issues in certain scenarios as explained above. Empty rowkey returned by
> server can be treated as end of the region scan by HBase client.
> With the paging feature enabled, if the page size is kept low, we have higher
> chances of scanners returning dummy cell, resulting in increased num of RPC
> calls for better latency and timeouts. We should return only valid rowkey in
> the scan range for all the cases where we perform above mentioned operations
> like complex aggregate or offset queries etc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
