Sigma-Ma commented on code in PR #172:
URL: https://github.com/apache/hbase-connectors/pull/172#discussion_r4450613314
##########
spark4/hbase-spark4/src/main/scala/org/apache/hadoop/hbase/spark/datasources/ScanRange.scala:
##########
@@ -145,6 +145,31 @@ class RowKeyFilter(
if (currentRange != null) ranges += currentRange
if (currentPoint != null) points += currentPoint
+ // Normalize after the complete filter tree is built, before region pruning
and Get batching.
+ def normalize(): RowKeyFilter = {
+ val normalizedRanges = new ListBuffer[ScanRange]()
+ ranges
+ .sortWith((left, right) => Bytes.compareTo(left.lowerBound,
right.lowerBound) < 0)
+ .foreach {
+ range =>
+ normalizedRanges.lastOption match {
+ case Some(previous) =>
+ val cmp = previous.compareRange(previous.upperBound,
range.lowerBound)
+ if (cmp > 0 ||
+ (cmp == 0 && (previous.isUpperBoundEqualTo ||
range.isLowerBoundEqualTo))) {
+ previous.mergeUnion(range)
+ } else {
+ normalizedRanges += range
+ }
+ case None => normalizedRanges += range
+ }
+ }
+ ranges = normalizedRanges
+ val seen = new java.util.TreeSet[Array[Byte]](Bytes.BYTES_COMPARATOR)
+ points = points.filter(p => seen.add(p) &&
!ranges.exists(_.containsPoint(p)))
Review Comment:
Agreed. The coverage check is O(unique points × normalized ranges). I'll use
an array of the sorted, disjoint ranges and binary search for the candidate
range, while keeping the existing byte-content deduplication and point order.
I'll apply this to both connectors and add coverage for gaps and
inclusive/exclusive endpoints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
