LuciferYang opened a new issue, #9623: URL: https://github.com/apache/paimon/issues/9623
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Spark. The engine-side filter conversion forwards a NULL inside an IN list unchanged, and the scan then hands it to the global index reader. ### Minimal reproduce step Query a column that carries a btree global index with an IN list holding more than 20 literals, one of them NULL: ```sql SELECT * FROM t WHERE k IN (1, 2, 3, ..., 20, NULL); ``` The scan fails with a `NullPointerException` out of `BTreeIndexReader.visitIn`. The list has to be longer than 20 because `PredicateBuilder.in` only builds an `In` leaf above that size; at or below it the list is expanded into `Equal` predicates, and an `Equal(null)` is pruned away by `SortedFileMetaSelector` before any reader sees it. `visitIn` loops the literals straight into a range query: ```java public Optional<GlobalIndexResult> visitIn(List<Object> literals) { return createResult( () -> { RoaringNavigableMap64 result = new RoaringNavigableMap64(); for (Object literal : literals) { result.or(rangeQuery(literal, literal, true, true)); } return result; }); } ``` `rangeQuery` serializes the key and compares it, so a null literal throws. The index file is selected in the first place because `SortedFileMetaSelector.visitIn` skips nulls and keeps the file if any other literal overlaps its min/max, which is correct on its own. ### What doesn't meet your expectations? `IN` never matches NULL in SQL, so a null literal in the list should be ignored, not turned into a failed scan. Every other reader in that package already does this. `globalindex/bitmap/BitmapIndexReader` null-checks the literal inside its own `in` loop, and its `equal`, `lessThan`, `greaterThan`, `between`, `notEqual` and `notIn` do the same; `SortedFileGlobalIndexReader.visitNotIn` returns an empty result for a null. The btree reader is the one leaf that does not. ### Anything else? I traced the Paimon side end to end: nothing between the Spark filter conversion and `BTreeIndexReader` removes the null. I did not run a Spark SQL query to confirm that Spark's optimizer keeps a NULL inside a 21-element IN list rather than folding it, so the "more than 20 literals" part of the reproduce path is from reading `PredicateBuilder`, not from an executed query. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
