spancer opened a new issue, #10332: URL: https://github.com/apache/paimon/issues/10332
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version 2.0.0 (paimon-bundle-2.0.0) ### Compute Engine None — reproduced with the Java API (`ReadBuilder.withFilter`), local filesystem catalog, Java 21. ### Minimal reproduce step Create an append table with a range-bitmap file index on `c`, write 5 rows where `c` is NULL in every row, then read with `c IS NULL`: ```java Identifier id = Identifier.create("db", "t_isnull"); catalog.createTable( id, Schema.newBuilder() .column("id", DataTypes.INT()) .column("c", DataTypes.INT()) .option("file-index.range-bitmap.columns", "c") .build(), false); Table table = catalog.getTable(id); List<GenericRow> rows = new ArrayList<>(); for (int i = 0; i < 5; i++) { rows.add(GenericRow.of(i, null)); } write(table, rows); Predicate isNull = new PredicateBuilder(table.rowType()).isNull(1); System.out.println(count(table, isNull)); // prints 2 System.out.println(count(table, null)); // prints 5 ``` Helper methods: ```java static void write(Table table, List<GenericRow> rows) throws Exception { BatchWriteBuilder builder = table.newBatchWriteBuilder(); try (BatchTableWrite write = builder.newWrite(); BatchTableCommit commit = builder.newCommit()) { for (GenericRow row : rows) write.write(row); commit.commit(write.prepareCommit()); } } static long count(Table table, Predicate filter) throws Exception { ReadBuilder builder = table.newReadBuilder(); if (filter != null) builder = builder.withFilter(filter); long[] count = new long[1]; try (RecordReader<InternalRow> reader = builder.newRead().createReader(builder.newScan().plan())) { reader.forEachRemaining(row -> count[0]++); } return count[0]; } ``` The same table with `file-index.bitmap.columns` or `file-index.bsi.columns` instead of `range-bitmap` returns 5. ### What doesn't meet your expectations? `c IS NULL` returns **2** rows; all **5** rows match. The rows returned are the first and the last row of the file; every row in between is dropped. The same table with `file-index.bitmap.columns` or `file-index.bsi.columns` instead of `range-bitmap` returns 5. ### Anything else? Root cause is in `org.apache.paimon.fileindex.rangebitmap.RangeBitmap#isNull`. When the column has no non-null value in the file (`cardinality <= 0`), it returns: ```java RoaringBitmap32.bitmapOf(0, rid - 1) ``` `bitmapOf(int...)` takes a list of positions, so this is the two-element set `{0, rid - 1}`, not the range `[0, rid)`. Any file with three or more rows where the indexed column is entirely NULL loses its middle rows for `IS NULL`. Suggested fix: return the full range, e.g. `RoaringBitmap32.bitmapOfRange(0, rid)`. `isNotNull` on the same reader is correct (it returns an empty bitmap for such a file), so the complement of `isNotNull` over `[0, rid)` gives the right answer too. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
