voonhous commented on code in PR #19691:
URL: https://github.com/apache/hudi/pull/19691#discussion_r3838208852
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestVariantDataType.scala:
##########
@@ -977,6 +978,78 @@ class TestVariantDataType extends HoodieSparkSqlTestBase {
}
}
+ test("Test Spark 3.x schema-on-read reads of a variant table with a
committed internal schema") {
+ // #18021: hoodie.schema.on.read.enable resolves the schema through the
InternalSchema round
+ // trip, whose sentinel detection restores the VARIANT logical type - the
exact input
+ // HoodieSparkSchemaConverters rejects on Spark 3.x. Verified 2026-08-20:
every leg fails
+ // LOUDLY with the same actionable error as the plain auto-resolve path;
there is no silent
+ // wrong data and no obscure secondary failure. Notably that includes the
documented
+ // struct-DDL compat mode, which works on this same table with the conf
off (pinned below)
+ // but breaks once it is on, because internal-schema resolution overrides
the user's DDL.
+ // Real support is #18285; until then Spark 3.x compat-mode readers must
keep
+ // hoodie.schema.on.read.enable off.
+ assume(HoodieSparkUtils.isSpark3, "This test verifies Spark 3.x behavior
with schema-on-read")
+
+ withTempDir { tmpDir =>
+
HoodieTestUtils.extractZipToDirectory("variant_backward_compat/variant_schema_on_read_cow.zip",
tmpDir.toPath, getClass)
+ val tablePath =
tmpDir.toPath.resolve("variant_schema_on_read_cow").toString
+
+ // The schema-on-read legs assert the same exception as the plain leg,
so they would also
+ // pass if the internal schema silently failed to load and
HoodieBaseRelation fell back to
+ // the commit-metadata schema. Pin that the fixture carries a loadable
internal schema, so
+ // those legs cannot pass without exercising the InternalSchema path.
+ val schemaResolver = new TableSchemaResolver(createMetaClient(spark,
tablePath))
+ assert(schemaResolver.getTableInternalSchemaFromCommitMetadata.isPresent,
+ "fixture must carry a committed internal schema; regenerate it per the
README")
+
+ def assertVariantRejected(leg: String)(f: => Unit): Unit = {
+ val ex = intercept[HoodieSchemaException](f)
+ assert(ex.getCause.getMessage.contains("VARIANT type is only supported
in Spark 4.0+"),
+ s"[$leg] expected the actionable variant rejection, got:
${ex.getCause}")
+ }
+
+ assertVariantRejected("auto-resolve, plain") {
+ spark.read.format("hudi").load(tablePath).collect()
+ }
+ assertVariantRejected("auto-resolve, schema-on-read") {
+ spark.read.format("hudi").option("hoodie.schema.on.read.enable",
"true").load(tablePath).collect()
+ }
+
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ |create table $tableName (
+ | id int,
+ | v struct<value: binary, metadata: binary>,
+ | ts long,
+ | note string
+ |) using hudi
+ |location '$tablePath'
+ |tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts'
+ |)
+ """.stripMargin)
+ try {
+ // With the conf off the compat struct DDL works even though the table
carries an
+ // internal schema - pinning that the conf, not the committed internal
schema itself,
+ // is what breaks compat mode.
+ val rows = spark.sql(s"select id, v, note from $tableName order by
id").collect()
+ assert(rows.map(r => (r.getInt(0), r.isNullAt(1),
r.getString(2))).toSeq
Review Comment:
Pinned in 88545658. Read the bytes off the fixture parquet rather than
deriving them: metadata `01 01 00 03 6B 65 79` (1-entry dictionary holding
`key`) for both rows, values `02 01 00 00 03 09 76 31` and `02 01 00 00 03 09
76 32` (`"v1"` / `"v2"`). The leg now asserts both per row instead of
`isNullAt`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
