[
https://issues.apache.org/jira/browse/HBASE-28692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862810#comment-17862810
]
Robin Infant A edited comment on HBASE-28692 at 7/4/24 1:52 PM:
----------------------------------------------------------------
Hi [~zhangduo] [~apurtell] ,
In StoreScanner.java, we are applying filters only if a valid cell is present
at the top (dummy key which is mentioned in above case). Hence, we can reach
the required cells using hints returned by the matcher (SEEK_NEXT_USING_HINT).
If the deleted cells are the top cells, each deleted cell needs to be skipped
without applying the filter. Thus, each iteration has to skip all 100,000 cells
(Match code: SKIP) as it is unable to apply filters. Consequently, each
iteration needs to process 100,000 deleted cells and the required valid cells
set by the limit, which is the actual bottleneck.
Total unwanted processing cells may be equal to the number of iterations
multiplied by the number of deleted cells. In the above-mentioned issue, this
equates to 200 iterations (500 per iteration) * 100,000 deleted cells, which is
around 20 million unwanted processing of cells.
Is there any way we can overcome this?
was (Author: JIRAUSER305458):
Hi [~zhangduo] [~apurtell] ,
In StoreScanner.java, we are applying filters only if a valid cell is present
at the top. Hence, we can reach the required cells using hints returned by the
matcher (SEEK_NEXT_USING_HINT).
If the deleted cells are the top cells, each deleted cell needs to be skipped
without applying the filter. Thus, each iteration has to skip all 100,000 cells
(Match code: SKIP) as it is unable to apply filters. Consequently, each
iteration needs to process 100,000 deleted cells and the required valid cells
set by the limit, which is the actual bottleneck.
Total unwanted processing cells may be equal to the number of iterations
multiplied by the number of deleted cells. In the above-mentioned issue, this
equates to 200 iterations (500 per iteration) * 100,000 deleted cells, which is
around 20 million unwanted processing of cells.
Is there any way we can overcome this?
> The GET call for column keys becomes significantly slower when it follows
> lexicographically prior deleted cells. [PERFORMANCE BLOAT]
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-28692
> URL: https://issues.apache.org/jira/browse/HBASE-28692
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 2.5.5
> Environment: Ubuntu 22.04.4 LTS - 64 Bit
> Reporter: Robin Infant A
> Priority: Critical
> Attachments: DataFlow.png, HbaseGet.java,
> RegionServerDummyKeyAdded.png, RegionServerFlame.png
>
>
> *Steps to Reproduce :*
> 1. Ensure the column family of the table is configured with MIN_VERSION = 1
> and KEEP_DELETED_CELLS = true to ensure that deleted cells remain even after
> deletion (this is to ensure that flush/compaction doesn't remove deleted
> cells).
> 2. Add multiple column keys against a single row key. In this case, use a
> timestamp (SET1).
> 3. Use a GET call (GET1) with an iterator-based design to get all the keys.
> Use the ColumnPaginationFilter and RangeKeyFilter to achieve this (refer to
> the attached HbaseGet.java file for implementation details).
> 4. Note the time taken for GET1 (e.g., ~1 second for 100,000 entries).
> 5. Delete all the added column keys (SET1).
> 6. Add another set of 100,000 entries (SET2).
> 7. Use the GET call again (GET2) to retrieve the data and note the time taken.
> *Expected Behavior:*
> The time taken for GET2 should be similar to GET1, approximately ~1 second.
> *Actual Behavior:*
> The time taken for GET2 increases significantly, approximately 60 seconds,
> which is ~50 times longer than GET1.
> *Additional Information:*
> i) If, after the first delete (SET1), data with lexicographically prior
> characters (e.g., "0") are added before the deleted cells (timestamp), the
> GET call time remains consistent with GET1, approximately 1200 ms.
> ii) If all 200,000 entries are retrieved without deleting the first set
> (SET1), the total time is around 2 to 3 seconds. However, deleting SET1
> causes GET2 to take 50 times longer than GET1.
> *Suggested Fix:*
> Temporarily, to fix this issue, add a dummy column key with a
> lexicographically prior character (e.g., "0") to ensure consistent GET call
> times.
> *Note:*
> Please refer to the attached [^HbaseGet.java], RegionserverFlameGraph and
> dataFlow for detailed steps to reproduce the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
