LuciferYang opened a new issue, #9645: URL: https://github.com/apache/paimon/issues/9645
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Spark and Flink, on a table with `file-index.bsi.columns` covering a `TIMESTAMP(7)`, `TIMESTAMP(8)` or `TIMESTAMP(9)` column. ### Minimal reproduce step Write two rows whose timestamps differ only below microsecond resolution, with a BSI index on that column: ```sql CREATE TABLE t (id INT, ts TIMESTAMP(9)) WITH ('file-index.bsi.columns' = 'ts'); INSERT INTO t VALUES (1, TIMESTAMP '1970-01-01 00:00:01.000000000'), (2, TIMESTAMP '1970-01-01 00:00:01.000000500'); SELECT * FROM t WHERE ts <> TIMESTAMP '1970-01-01 00:00:01.000000000'; -- returns nothing ``` The index maps both rows to the same value. `BitSliceIndexBitmapFileIndex` truncates anything above millisecond precision to micros: ```java private Function<Object, Long> getTimeStampMapper(int precision) { return o -> { if (o == null) { return null; } else if (precision <= 3) { return ((Timestamp) o).getMillisecond(); } else { return ((Timestamp) o).toMicros(); } }; } ``` The data file itself keeps the nanoseconds: for precision above 6 the Parquet writer uses INT96 with nanos-of-day (`ParquetSchemaConverter`), and ORC stores nanos too. So the two rows really are different values, and only the index conflates them. This is not a lost pruning opportunity. `RawFileSplitRead` takes the BSI bitmap as the row selection: ```java if (fileIndexResult instanceof BitmapIndexResult) { selection = ((BitmapIndexResult) fileIndexResult).get(); } ... fileRecordReader = new ApplyBitmapIndexRecordReader(fileRecordReader, (BitmapIndexResult) fileIndexResult); ``` so rows the bitmap leaves out are never read. `<>` therefore drops row 2 as shown above, and strict `<` / `>` flip at a boundary that falls inside the truncated microsecond. Over-selection by `=` may be masked by a filter the engine keeps for itself; under-selection cannot be recovered by anything downstream. ### What doesn't meet your expectations? An index should not change which rows a query returns. Either the index has to carry the full precision of the column, or a sub-microsecond column should not get a BSI index at all, so the scan falls back to reading and filtering. ### Anything else? I have a patch that switches the mapper to nanos behind a new index version, and I am not sending it as a PR because it needs two decisions that belong with whoever owns this format: The version byte. Writing version 2 for every BSI column, which is the simple thing to do, makes an older reader reject index files for INT and BIGINT columns too, since the existing check is `if (version > VERSION_1) throw`. Bumping only for `TIMESTAMP(7-9)` keeps the rest readable, at the cost of a per-type condition in the writer. The range. `Timestamp.toMicros()` covers year 294247, while nanos since epoch overflow a long at 2262. Sentinel values like `TIMESTAMP '9999-12-31 23:59:59.999999999'` are common in SCD tables, and they index fine today. Mapping to nanos means such a value either fails the write, saturates (which reintroduces conflated values at the top of the range), or needs a per-precision unit (100ns at precision 7, 10ns at 8), which still leaves precision 9 capped at 2262. Happy to send whichever shape you prefer. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
