voonhous commented on code in PR #19808:
URL: https://github.com/apache/hudi/pull/19808#discussion_r3911181843
##########
hudi-spark-datasource/hudi-spark-common/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/TestParquetSchemaEvolutionUtils.scala:
##########
@@ -199,6 +201,71 @@ class TestParquetSchemaEvolutionUtils {
private def footerOf(column: Type): FileMetaData =
new FileMetaData(Types.buildMessage().addField(column).named("test"), new
HashMap[String, String](), "test")
+ /**
+ * The Spark 3.x shape: no VariantType there, so a variant column is
declared as
+ * struct<value: binary, metadata: binary> (the shape Hive sync also
writes). Either member
+ * order is the same column, and the unshredded twin of the same file must
still read.
+ */
+ @Test
+ def testValidateNoShreddedVariantStructsRejectsTopLevelShreddedVariant():
Unit = {
Review Comment:
Added `Test Backward Compatibility: Shredded Variant Table Read Fails Fast
in Spark 3.x` to `TestVariantDataType`: the compat struct DDL pointed at
`variant_shredded_mixed_cow.zip`, asserting the guard's own message rather than
any failure naming the column, for both `select id, v` and the pruned `select
id, v.value`.
##########
hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark35ParquetReader.scala:
##########
@@ -117,6 +117,11 @@ class Spark35ParquetReader(enableVectorizedReader: Boolean,
}
val footerFileMetaData = fileFooter.getFileMetaData
+ // A variant column is declared as its unshredded struct shape on Spark
3.x (no VariantType);
+ // reject a file that shreds it here, before the reader is built, so the
read fails naming the
+ // column instead of projecting the group by name and returning a null
value for every
+ // shredded row.
+
ParquetSchemaEvolutionUtils.validateNoShreddedVariantStructs(requiredSchema,
footerFileMetaData)
Review Comment:
Confirmed - `HoodieSparkParquetReader.getUnsafeRowIterator` builds its
reader on whatever `SparkAdapter.createParquetReadSupport` returns, and Spark
3.x got the plain one, so a shredded variant in a log block met no guard.
Took the adapter route: `BaseSpark3Adapter` now returns
`Spark3HoodieParquetReadSupport`, whose `init` runs the guard against
`context.getFileSchema` and the catalyst request from
`SPARK_ROW_REQUESTED_SCHEMA`. The file schema rather than the requested one,
because a Spark 3.x request asks for the binary members alone, so typed_value
has already been clipped from the requested schema by that point.
`validateNoShreddedVariantStructs` now takes a `MessageType` so the reader call
sites and this one share the same walk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
