JingsongLi commented on code in PR #10601:
URL: https://github.com/apache/paimon/pull/10601#discussion_r4374882914
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -390,17 +428,17 @@ private Comparable<?> toParquetObject(Object value,
FieldRef fieldRef) {
if (value instanceof Timestamp) {
Timestamp timestamp = (Timestamp) value;
- timestampColumn(fieldRef, fileSchema, caseSensitive);
- int precision = getTimestampPrecision(type);
- if (precision <= 3) {
- // milliseconds
- return timestamp.getMillisecond();
- } else if (precision <= 6) {
- // microseconds
- return timestamp.toMicros();
+ switch (timestampUnit(
+ fieldRef, timestampColumn(fieldRef, fileSchema,
caseSensitive))) {
+ case MILLIS:
+ return timestamp.getMillisecond();
+ case MICROS:
+ return getTimestampPrecision(type) <= 3
+ ? firstMicroOfMilli(timestamp)
Review Comment:
[P1] Preserve finer timestamp literals when pushing less-than to MICROS
For a precision-3 field, this truncates a literal's submillisecond component
even though predicate evaluation preserves it. A read row at `...00.123`
satisfies `lessThan(ts, ...00.123456)`, but `visitLessThan` reuses this
conversion and emits `< ...123000` microseconds, excluding the matching
row/group. `PredicateBuilder` accepts the unmodified Timestamp literal and
`Timestamp.compareTo` includes its nanos; there is no contract that quantizes
the literal to the field precision.
Using actual head-written Iceberg MICROS files, I reproduced this for
TIMESTAMP(3) and TIMESTAMP_LTZ(3) at zero, negative, and positive epochs: an
unfiltered read returns one row and its residual predicate is true, while the
pushed read returns zero. Replacing only this filter class with exact
merge-base c9a36a81d7 retains the row from the same files through the same
reader, because the previous unit-mismatch fallback declines pushdown. The
added `testMillisTimestampOnMicrosFile` already supplies a 456000-nanosecond
component but asserts the truncated filter instead of checking row retention.
Please use operator-aware widening, or conservatively decline pushdown when the
literal is finer than the read field, and add full-reader assertions for these
boundaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]