voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3838207600
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +978,78 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works on this same table with the conf
off (pinned below)
+ // but breaks once it is on, because internal-schema resolution overrides
the user's DDL.
+ // Real support is #18285; until then Spark 3.x compat-mode readers must
keep
Review Comment:
Right - `DefaultSource` sets `userSchema = None` under `isUsingHiveCatalog`,
so the struct DDL never reaches the relation and HMS throws with the conf off
too. Scoped the note in 88545658: keeping the conf off is necessary but only
sufficient off a Hive catalog, and this test runs on the in-memory catalog so
it pins that half only. HMS stays out of scope for the pin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
