[
https://issues.apache.org/jira/browse/HBASE-30509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18136135#comment-18136135
]
mazhengxuan commented on HBASE-30509:
-------------------------------------
Hi [~wchevreuil], I've opened [PR
#172|https://github.com/apache/hbase-connectors/pull/172] to fix this in both
Spark 3 and Spark 4. Local unit and MiniCluster tests passed. Could you take a
look when you have a chance? Thanks!
> Deduplicate overlapping row-key scans and point lookups in Spark connectors
> ---------------------------------------------------------------------------
>
> Key: HBASE-30509
> URL: https://issues.apache.org/jira/browse/HBASE-30509
> Project: HBase
> Issue Type: Sub-task
> Components: hbase-connectors
> Reporter: mazhengxuan
> Assignee: mazhengxuan
> Priority: Major
> Labels: pull-request-available
>
> RowKeyFilter.mergeUnion can retain point lookups that are already covered by
> scan ranges.
> For a filter such as key < 'm' OR key = 'a', this can read the same row
> through both a Scan and a Get and produce duplicate results.
> We should deduplicate point lookups and remove those covered by scan ranges
> before executing the reads, with regression tests for overlapping OR
> conditions, repeated points and range boundaries in both Spark 3 and Spark 4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
