voonhous commented on code in PR #17833:
URL: https://github.com/apache/hudi/pull/17833#discussion_r2736089878
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala:
##########
@@ -241,7 +252,15 @@ object HoodieSparkSchemaConverters {
}
}
- case other => throw new IncompatibleSchemaException(s"Unsupported
HoodieSchemaType: $other")
+ // VARIANT type (Spark >4.x only), which will be handled via SparkAdapter
Review Comment:
In the current implementation in Spark3.5, this is dead code as the variant
column will not have a logicalType of variant. It's a record instead.
The reason why it's dead code is because is:
1. The `tableSchema` is resolved from `schemaSpec.map(s =>
convertToHoodieSchema(s, tableName))` in `HoodieBaseHadoopFsRelationFactory`.
2. This converts a struct where the variant column is a struct of metadata
and value into HoodieSchema.
3. Hence, when the code flow reaches `HoodieSparkSchemaConverters`, the
variant column with HoodieSchema will not have the variant logical type to fall
into this code path. It will resolve to the `RECORD` path instead.
This might change if the table has an internalSchema though, which i think
we need to investigate. I'll create an issue for this!
#18021
Will inline the issue too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
