JingsongLi commented on code in PR #8230:
URL: https://github.com/apache/paimon/pull/8230#discussion_r3457497363
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -299,11 +299,7 @@ private static Comparable<?> toParquetObject(
} else if (value instanceof Timestamp) {
Timestamp timestamp = (Timestamp) value;
int precision = getTimestampPrecision(type);
- if (precision <= 3) {
- // milliseconds
- return timestamp.getMillisecond();
- } else if (precision <= 6) {
- // microseconds
+ if (precision <= 6) {
return timestamp.toMicros();
Review Comment:
This filter literal is now always epoch micros for TIMESTAMP(0..6), but old
Paimon files with precision 0..3 still store epoch milliseconds. The filter is
built in ParquetFileFormat from the table type before ParquetReaderFactory
opens each file, so it cannot distinguish TIMESTAMP_MILLIS files from new
TIMESTAMP_MICROS files. For an old file, ts = 2024-01-01 becomes
1704067200000000 while the row-group stats/data are around 1704067200000, so
Parquet can incorrectly prune matching row groups. We need to build the
timestamp predicate after seeing the file schema, or otherwise avoid this
pushdown for legacy low-precision timestamp files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
