jonvex commented on code in PR #14120:
URL: https://github.com/apache/hudi/pull/14120#discussion_r2446150047
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieParquetFileFormatHelper.scala:
##########
@@ -140,6 +202,7 @@ object HoodieParquetFileFormatHelper {
Cast(expr, dec, if (needTimeZone) timeZoneId else None)
case (StringType, DateType) =>
Cast(expr, DateType, if (needTimeZone) timeZoneId else None)
+ case (LongType, TimestampNTZType) => expr // @ethan I think we just
want a no-op here?
Review Comment:
If you go to the codegen for casting, they don't support long->timestampntz
but spark has natural handling. But I got rid of this, because now we will
repair the data on this case. If we want to protect when repair is disabled, we
can actually make a better change:
org.apache.spark.sql.execution.datasources.parquet.HoodieParquetFileFormatHelper.isDataTypeEqual
we can add a case for (TimesatmpNTZType, LongType) => true, so that way we
won't even need to use the schema evolution paths for this col
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
