wombatu-kun opened a new pull request, #8735: URL: https://github.com/apache/paimon/pull/8735
### Purpose `BTreeIndexReader.allNonNullRows()` backs six predicate visitors (`IS NOT NULL`, `<>`, `NOT IN`, `LIKE`, `CONTAINS`, `ENDS WITH`) by range-scanning the whole btree index file to collect every non-null row id. For a column with few or no nulls this reads all keys and row-id lists across every data block just to rediscover almost the entire row set. This resolves the standing `TODO do not traverse all data if less null values` in `paimon-common/src/main/java/org/apache/paimon/globalindex/btree/BTreeIndexReader.java`. The btree writer assigns one dense local row id per logical row (a null row still advances the id and is recorded in the null bitmap), so the non-null rows are exactly the local id universe `[0, rowCount)` minus the null bitmap. The row count is already persisted as `IndexFileMeta.rowCount()`; it was simply not handed to the reader. This change plumbs it through `GlobalIndexIOMeta` (via a new overloaded constructor, so the many existing construction sites are untouched) and derives the complement with two bitmap operations (`addRange` + `andNot`) instead of a scan. When the row count is unknown (older metadata) the reader falls back to the previous range scan, so behavior is unchanged there. There is no on-disk format change. The complement is computed unconditionally rather than only when nulls are few, because deciding by null count would require `getLongCardinality()` on the shared null bitmap, which lazily mutates `Roaring64NavigableMap`'s cardinality cache and would race under the concurrent access exercised by `BTreeThreadSafetyTest`. ### Tests Added null-density test templates to `AbstractIndexReaderTest` (run across all 12 indexed data types and both readers): - `testIsNotNullAcrossNullDensities` verifies `visitIsNotNull` returns the exact non-null row ids across no, few (~1%), half, many (~99%) and all nulls. - `testNotEqualAndNotInWithNulls` verifies `visitNotEqual` and `visitNotIn` exclude null rows, a case not previously covered with nulls. `BTreeIndexReaderTest` now attaches the row count so the single-file reader exercises the new complement path; the multi-file `LazyFilteredBTreeIndexReaderTest` keeps the row count unknown and continues to cover the range-scan fallback. `BTreeThreadSafetyTest` (32-thread stress on one reader) guards the concurrency behavior of the changed method. All pass: 209 run, 0 failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
