JingsongLi commented on code in PR #10601:
URL: https://github.com/apache/paimon/pull/10601#discussion_r4492261459


##########
paimon-format/src/main/java/org/apache/paimon/format/parquet/writer/ParquetRowDataWriter.java:
##########
@@ -155,7 +155,17 @@ private FieldWriter createWriter(DataType t, Type type) {
         }
     }
 
-    private FieldWriter createTimestampWriter(int precision) {
+    private FieldWriter createTimestampWriter(int precision, Type parquetType) 
{
+        LogicalTypeAnnotation annotation = 
parquetType.getLogicalTypeAnnotation();
+        if (annotation instanceof 
LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) {
+            LogicalTypeAnnotation.TimeUnit unit =
+                    ((LogicalTypeAnnotation.TimestampLogicalTypeAnnotation) 
annotation).getUnit();
+            if (unit == LogicalTypeAnnotation.TimeUnit.MILLIS) {
+                return new TimestampMillsWriter(precision);
+            } else if (unit == LogicalTypeAnnotation.TimeUnit.MICROS) {
+                return new TimestampMicrosWriter(precision);

Review Comment:
   [P1] Keep millisecond precision when selecting the MICROS writer
   
   This new branch selects `TimestampMicrosWriter` for precision 3, but its 
`value.toMicros()` preserves fractions that the previous MILLIS writer 
discarded. A persisted `TIMESTAMP(3)` / `TIMESTAMP_LTZ(3)` table can receive 
these values through normal Spark SQL, because Spark's timestamp type does not 
retain the Paimon precision.
   
   I reproduced this with an ordinary append Parquet table (`bucket=-1`, 
`metadata.iceberg.storage=table-location`) and `INSERT ... TIMESTAMP 
'2024-01-01 00:00:00.123999'` plus a pre-epoch `.999999` row. Paimon reads 
`.123000` / `.999000`, while Iceberg fullscan reads `.123999` / `.999999`. 
Iceberg equality filters on the timestamp values returned by its own fullscan 
both return zero rows: the published Paimon p3 bounds have already been 
truncated to milliseconds and exclude those physical values. This is a 
supported SQL write path, not a hand-built row that violates an internal 
serializer invariant.
   
   Please truncate precision <=3 values to milliseconds before emitting their 
MICROS representation, while retaining the MICROS annotation and precision 4–6 
behavior. Changing only that writer operation makes the identical Spark SQL, 
independent Iceberg read, and equality-filter checks pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to