JingsongLi commented on code in PR #8238:
URL: https://github.com/apache/paimon/pull/8238#discussion_r3457462618
##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetSimpleStatsExtractor.java:
##########
@@ -223,8 +223,8 @@ private SimpleColStats toTimestampStats(Statistics<?>
stats, int precision) {
if (precision <= 3) {
LongStatistics longStats = (LongStatistics) stats;
return new SimpleColStats(
- Timestamp.fromEpochMillis(longStats.getMin()),
- Timestamp.fromEpochMillis(longStats.getMax()),
+ Timestamp.fromMicros(longStats.getMin()),
Review Comment:
After this change the extractor decodes TIMESTAMP(0..3) footer stats as
micros solely from the Paimon field precision. Existing Paimon Parquet files
written before this PR use TIMESTAMP_MILLIS and store min/max in epoch
milliseconds, so extracting stats for those files (for example during
migrate/clone or any metadata regeneration) would turn 2024-01-01T00:00:00.123
into a 1970 timestamp and write incorrect file stats. Can we derive the unit
from stats.type().getLogicalTypeAnnotation() / the column metadata, like the
reader does, and keep MILLIS for legacy files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
