yihua opened a new pull request, #12941: URL: https://github.com/apache/hudi/pull/12941
### Change Logs In Spark, both parquet column types `TIMESTAMP(isAdjustedToUTC=true, unit=MILLIS)` and `TIMESTAMP(isAdjustedToUTC=true, unit=MICROS)` are converted `TimestampType` in `StructType` during schema conversion (https://github.com/apache/spark/blob/f40bf4dd7166e86ea5cc9962e8c5b68b88f8dcb5/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L276). However, `TimestampType` represents a time instant in microsecond precision. When trying to read the `TIMESTAMP(isAdjustedToUTC=true, unit=MILLIS)` as long (e.g., HMS provides the table schema and specifies the column type as long) in Hudi, `Spark33LegacyHoodieParquetFileFormat` falsely identifies that there is a type change or schema evolution (`val (implicitTypeChangeInfo, sparkRequestSchema) = HoodieParquetFileFormatHelper.buildImplicitSchemaChangeInfo(hadoopAttemptConf, footerFileMetaData, requiredSchema)`). This check is done by comparing the required schema (`LongType` for the column of interest) and the parquet file schema (`TIMESTAMP(isAdjustedToUTC=true, unit=MILLIS)` for the column of interest). To fix the issue, when comparing the field types in `HoodieParquetFileFormatHelper#isDataTypeEqual`, a special case of `(LongType, TimestampType)` is added so that we don't treat this case as schema evolution, as the timestamp logical type uses long type as the primitive type, so there is no need to do any conversion if the value needs to be read as long type from the column. ### Impact Fix bug ### Risk level none ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
