xiarixiaoyao commented on code in PR #5378:
URL: https://github.com/apache/hudi/pull/5378#discussion_r854745790
##########
hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark32HoodieParquetFileFormat.scala:
##########
@@ -225,6 +239,10 @@ class Spark32HoodieParquetFileFormat(private val
shouldAppendPartitionValues: Bo
if (enableVectorizedReader) {
val vectorizedReader =
if (shouldUseInternalSchema) {
+ val int96RebaseSpec =
+
DataSourceUtils.int96RebaseSpec(footerFileMetaData.getKeyValueMetaData.get,
int96RebaseModeInRead)
+ val datetimeRebaseSpec =
+
DataSourceUtils.datetimeRebaseSpec(footerFileMetaData.getKeyValueMetaData.get,
datetimeRebaseModeInRead)
Review Comment:
yes，
current schema evoluton work on spark3.1.x and >= spark3.2.1
for the subsequent evolution。
from spark3.2.x, schema evolution choose to rely on dsv2. since dsv2 only
work with spark version >= 3.2.1, so schema evolution cannot support
spark3.2.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
