liwuhen opened a new pull request, #485:
URL: https://github.com/apache/paimon-rust/pull/485
### Purpose
Follow-up to a review comment from @JingsongLi on #430.
DataFusion temporal literal pushdown converts microsecond/nanosecond
timestamp
literals with euclidean division (`div_euclid` / `rem_euclid`), but the
BinaryRow write/extract path used truncating division. For pre-epoch,
sub-millisecond timestamps the two paths disagree:
- `-1us` (`1969-12-31 23:59:59.999999`) is stored as `(millis=0,
nanos=-1000)`
- the pushed literal becomes `(millis=-1, nanos=999_000)`
Because `Datum::Timestamp` compares lexicographically on `(millis, nanos)`, a
pushed equality/range predicate on such a value is false-negative and can
drop
matching rows during stats-based file pruning. This aligns the write path
with
the pushdown representation and with Paimon Java's canonical
`(millisecond, nanoOfMillisecond ∈ [0, 999999])` form.
### Brief change log
- Switch the three microsecond→`(millis, nanos)` conversions in
`crates/paimon/src/spec/binary_row.rs` from truncating `/ 1000`, `% 1000`
to
`div_euclid(1_000)` / `rem_euclid(1_000)`:
- `extract_datum_from_arrow` — `Timestamp`
- `extract_datum_from_arrow` — `LocalZonedTimestamp`
- `write_typed_value` — `TypedColumn::TimestampUs` (batch write path)
### Tests
- Unit test `test_negative_sub_millisecond_timestamp_uses_euclidean_parts`
(binary_row.rs): verifies the extraction and binary-row write paths both
normalize `-1us` to `(millis=-1, nanos=999_000)`. **Fails on the pre-fix
code.**
- Integration test
`test_negative_temporal_filter_pushdown_via_datafusion_scan`
(read_tables.rs): writes pre-epoch fractional timestamps in separate files
and
verifies pushed equality and range predicates return the correct rows
through a
DataFusion scan.
- Commands run locally:
- `cargo test -p paimon --lib spec::binary_row`
- `cargo test -p paimon-datafusion --test read_tables
test_negative_temporal_filter_pushdown_via_datafusion_scan`
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets --workspace --features
fulltext,vortex,mosaic -- -D warnings`
### API and Format
No change to the BinaryRow binary layout (still `i64` millis + `i32`
nano-of-milli). The *encoded values* for negative sub-millisecond timestamps
change to the canonical euclidean form; reads of both the old and new
encodings
reconstruct the same microsecond instant, so existing data remains readable.
### Documentation
No documentation changes required.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
