voonhous commented on code in PR #19583:
URL: https://github.com/apache/hudi/pull/19583#discussion_r3765205269
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/InternalRowToJsonStringConverter.scala:
##########
@@ -90,6 +92,11 @@ class InternalRowToJsonStringConverter(schema: StructType) {
structMap.toMap
case _ => value // fallback
}
+ case dt if SparkAdapterSupport.sparkAdapter.isVariantType(dt) =>
Review Comment:
Addressed. Reproduced it: 13 of the 14 cases error with
`ClassNotFoundException: org.apache.spark.sql.adapter.Spark4_2Adapter` (only
`emptyRow` survives, its schema has no field to match on). There is no version
adapter jar on this module's test classpath at all.
Matching on `dt.typeName == "variant"` now, which needs neither the adapter
nor the Spark 4 symbol, and the suite is 14/14 green -- including two new cases
for the JSON embedding and the fallback below.
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/InternalRowToJsonStringConverter.scala:
##########
@@ -90,6 +92,11 @@ class InternalRowToJsonStringConverter(schema: StructType) {
structMap.toMap
case _ => value // fallback
}
+ case dt if SparkAdapterSupport.sparkAdapter.isVariantType(dt) =>
+ // VariantVal.toString renders the variant as JSON; embed it as a
real JSON node so
+ // the image carries the variant's structure. Falling through to the
default would
+ // serialize the VariantVal bean, i.e. its raw value/metadata bytes
as base64.
+ mapper.readTree(value.toString)
Review Comment:
Addressed here rather than as a follow-up, since it is three lines:
`readTree` is wrapped and falls back to the raw rendering on
`JsonProcessingException`, so such a value degrades to a string image instead
of failing the query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
