JingsongLi commented on code in PR #8238:
URL: https://github.com/apache/paimon/pull/8238#discussion_r3457463895
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -299,11 +299,7 @@ private static Comparable<?> toParquetObject(
} else if (value instanceof Timestamp) {
Timestamp timestamp = (Timestamp) value;
int precision = getTimestampPrecision(type);
- if (precision <= 3) {
- // milliseconds
- return timestamp.getMillisecond();
- } else if (precision <= 6) {
- // microseconds
+ if (precision <= 6) {
return timestamp.toMicros();
Review Comment:
This predicate literal is now always epoch micros for TIMESTAMP(0..6), but
precision 0..3 files written by existing Paimon versions store epoch
milliseconds. The filter is created in ParquetFileFormat before
ParquetReaderFactory opens each file, so it cannot see whether a particular
file schema is TIMESTAMP_MILLIS or TIMESTAMP_MICROS. For old files, ts =
2024-01-01 becomes 1704067200000000 while the row-group stats/data are around
1704067200000, and Parquet can incorrectly drop matching row groups. We should
either build the timestamp filter after reading the file schema or
disable/avoid this pushdown for legacy low-precision timestamp files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
