shangxinli commented on code in PR #18794:
URL: https://github.com/apache/hudi/pull/18794#discussion_r4484662590
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java:
##########
@@ -442,19 +454,32 @@ private ValueWriter makeWriter(HoodieSchema schema,
DataType dataType) {
} else if (dataType == DataTypes.LongType || dataType instanceof
DayTimeIntervalType) {
return (row, ordinal) -> recordConsumer.addLong(row.getLong(ordinal));
} else if (dataType == DataTypes.TimestampType) {
+ // Choose the encoding that matches outputTimestampType — same key
Spark's own
+ // ParquetWriteSupport uses to set the parquet schema, so writer and
schema agree.
+ // The Hudi avro writer schema only carries MICROS/MILLIS precision (no
INT96), so
+ // the previous version of this method drove the writer from schema
precision alone
+ // and silently ignored the user's outputTimestampType — see
apache/hudi#18752.
+ if ("INT96".equals(outputTimestampType)) {
+ return (row, ordinal) -> recordConsumer.addBinary(microsToInt96Binary(
+ (long) timestampRebaseFunction.apply(row.getLong(ordinal))));
Review Comment:
minor: the int96 path reuses timestampRebaseFunction built from the general
datetimeRebaseMode, so spark.sql.parquet.int96RebaseModeInWrite is effectively
ignored for this writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
