viirya opened a new pull request, #2069:
URL: https://github.com/apache/iceberg-rust/pull/2069
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
- Closes #.
## What changes are included in this PR?
<!--
Provide a summary of the modifications in this PR. List the main changes
such as new features, bug fixes, refactoring, or any other updates.
-->
Add support for converting DataFusion Timestamp scalar values to Iceberg
Datum for predicate pushdown. This enables timestamp literal comparisons to be
pushed down to the storage layer when DataFusion provides timestamp literals in
ScalarValue form, improving query performance.
Changes:
- Add conversion for all four timestamp time units (Second, Millisecond,
Microsecond, Nanosecond) to Iceberg's native microsecond representation
- Convert timestamps to Datum::timestamp_micros with appropriate scaling:
* TimestampSecond: multiply by 1,000,000
* TimestampMillisecond: multiply by 1,000
* TimestampMicrosecond: use directly
* TimestampNanosecond: divide by 1,000 (truncates sub-microsecond
precision)
- Add comprehensive unit tests covering all time units and edge cases
- Add sqllogictest (timestamp_predicate_pushdown.slt) to validate end-to-end
timestamp predicate filtering and pushdown behavior
- Create test_timestamp_table in sqllogictest engine setup
- Update show_tables.slt to include test_timestamp_table
Implementation details:
- Iceberg uses microseconds as the native timestamp representation
- Nanosecond timestamps lose sub-microsecond precision when converted
- Timezone information in ScalarValue is preserved but not used in
conversion (Iceberg timestamp type is timezone-agnostic)
- Currently DataFusion casts timestamp literals to strings in predicates,
but this implementation provides the foundation for direct timestamp literal
support when DataFusion evolves to provide timestamp ScalarValues
## Are these changes tested?
<!--
Specify what test covers (unit test, integration test, etc.).
If tests are not included in your PR, please explain why (for example, are
they covered by existing tests)?
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
