voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3841771931
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +978,107 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works on this same table with the conf
off (pinned below)
+ // but breaks once it is on, because internal-schema resolution overrides
the user's DDL.
+ // Real support is #18285; until then Spark 3.x compat-mode readers must
keep
+ // hoodie.schema.on.read.enable off - necessary, but only sufficient off a
Hive catalog:
+ // DefaultSource discards the user schema outright when
isUsingHiveCatalog, so under HMS the
+ // struct DDL below is never seen and the read throws with the conf off
too. This test runs
+ // on the in-memory catalog, so it pins the non-Hive half only.
+ assume(HoodieSparkUtils.isSpark3, "This test verifies Spark 3.x behavior
with schema-on-read")
+
+ withTempDir { tmpDir =>
+
HoodieTestUtils.extractZipToDirectory("variant_backward_compat/variant_schema_on_read_cow.zip",
tmpDir.toPath, getClass)
+ val tablePath =
tmpDir.toPath.resolve("variant_schema_on_read_cow").toString
+
+ // HoodieBaseHadoopFsRelationFactory (reached here via
+ // HoodieCopyOnWriteSnapshotHadoopFsRelationFactory, not
HoodieBaseRelation) swallows an
+ // internal-schema load failure into None and then falls back to
getTableSchema, which
+ // throws the same exception - so neither auto-resolve leg can, on its
own, prove the
+ // InternalSchema path ran. This assert only pins that the fixture still
carries a loadable
+ // internal schema; the conf-off/conf-on catalog-DDL pair below is what
discriminates,
+ // because with a DDL present the None fallback resolves to the user
schema and succeeds.
+ val schemaResolver = new TableSchemaResolver(createMetaClient(spark,
tablePath))
+ assert(schemaResolver.getTableInternalSchemaFromCommitMetadata.isPresent,
Review Comment:
Dropped. The description now states what the comment does - the
conf-off/conf-on catalog-DDL pair is the leg that discriminates, and the
fixture assert pins only that the table still carries a loadable internal
schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
