voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3838208163
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +978,78 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works on this same table with the conf
off (pinned below)
+ // but breaks once it is on, because internal-schema resolution overrides
the user's DDL.
+ // Real support is #18285; until then Spark 3.x compat-mode readers must
keep
+ // hoodie.schema.on.read.enable off.
+ assume(HoodieSparkUtils.isSpark3, "This test verifies Spark 3.x behavior
with schema-on-read")
+
+ withTempDir { tmpDir =>
+
HoodieTestUtils.extractZipToDirectory("variant_backward_compat/variant_schema_on_read_cow.zip",
tmpDir.toPath, getClass)
+ val tablePath =
tmpDir.toPath.resolve("variant_schema_on_read_cow").toString
+
+ // The schema-on-read legs assert the same exception as the plain leg,
so they would also
+ // pass if the internal schema silently failed to load and
HoodieBaseRelation fell back to
Review Comment:
Both correct. `HoodieBaseHadoopFsRelationFactory` swallows the load failure
(`case Failure(e) => None`) and falls through to `getTableSchema`, which throws
the same thing, so the assert pins the fixture and not the path. Rewritten in
88545658 to say exactly that, to name
`HoodieCopyOnWriteSnapshotHadoopFsRelationFactory` instead of
`HoodieBaseRelation`, and to credit the conf-off/conf-on catalog-DDL pair as
the leg that actually discriminates - with a DDL present the `None` fallback
resolves to the user schema and succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
