LuciferYang opened a new issue, #9629: URL: https://github.com/apache/paimon/issues/9629
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Spark and Flink, on a data-evolution table. `DataEvolutionBatchScan.withFilter` extracts row-id ranges from the predicate and restricts the scan to them. ### Minimal reproduce step Filter `_ROW_ID` with an IN list that is not in ascending order and intersect it with a range: ```sql SELECT * FROM t WHERE _ROW_ID IN (25, 24, 23, ..., 5) AND _ROW_ID BETWEEN 1 AND 6; ``` Rows 5 and 6 are missing from the result. `RowIdPredicateVisitor` compresses the literals with `Range.toRanges`, which is a linear scan that merges consecutive ids and therefore assumes ascending input: ```java long rangeStart = iterator.next(); long rangeEnd = rangeStart; while (iterator.hasNext()) { long current = iterator.next(); if (current != rangeEnd + 1) { ranges.add(new Range(rangeStart, rangeEnd)); rangeStart = current; } rangeEnd = current; } ``` Descending literals turn into a descending list of one-element ranges, and `Range.and` intersects with two pointers that only move forward, so most of them are skipped. The result is then handed to `withRowRanges`, which limits which rows are read at all, so the rows that were dropped from the range list are never read and never filtered back in. The IN list has to be longer than 20 literals for `PredicateBuilder.in` to keep an `In` leaf; at or below that it expands into `Equal` predicates, one range each. Duplicates hit the same assumption: `IN (15, 15, ...)` produces `[15,15]` twice, so the list is no longer non-overlapping, which is the other precondition `Range.and` has. Two related problems in the same extraction path: `BETWEEN 10 AND 5` is legal SQL matching nothing, but the visitor builds `new Range(10, 5)`, violating the `from <= to` invariant the class asserts in its constructor. `Range.sortAndMergeOverlap` returns `Collections.emptyList()` for empty input, and the visitor's OR branch accumulates into whatever it returns: `rowIds.get().addAll(childList.get())` on the next child then throws `UnsupportedOperationException`. Reaching that needs an OR whose earlier children contribute nothing, for example `(_ROW_ID BETWEEN 1 AND 5 AND _ROW_ID BETWEEN 10 AND 20) OR _ROW_ID = 100`, where the inner AND intersects to empty. ### What doesn't meet your expectations? A row-id filter should not drop matching rows. `toRanges` and `and` have an ordering precondition that nothing in the visitor establishes, and the precondition is invisible from the signature: `toRanges` takes an `Iterable<Long>`. ### Anything else? `toRanges` has exactly one production caller today, so the missing precondition has not bitten anywhere else yet. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
