voonhous commented on code in PR #20159: URL: https://github.com/apache/hudi/pull/20159#discussion_r4152303242
########## website/releases/release-1.2.md: ########## @@ -50,15 +50,30 @@ The Trino-Hudi connector now lives in the Hudi repo (RFC-105, [#18837](https://g ## Known Issues -On Spark 4.1, `spark.sql.variant.pushVariantIntoScan` is on by default. With it on, snapshot reads of Merge-on-Read tables with `VARIANT` columns can go wrong when log files are merged: +These issues affect 1.2.0 and 1.2.1. Spark's `PushVariantIntoScan` rule is controlled by `spark.sql.variant.pushVariantIntoScan`, which is on by default in Spark 4.1 and off by default in Spark 4.0. + +On Spark 4.1 with the rule on, reads of `VARIANT` columns can go wrong. On Merge-on-Read tables, when a snapshot read merges log files: * A top-level `VARIANT` value set to null by an update is read back as a struct of nulls instead of `NULL`, so a filter such as `v IS NULL` misses the row. * A `VARIANT` nested inside a struct can crash the executor JVM. +* A query that extracts several paths from one `VARIANT` can fail with `NegativeArraySizeException` when rows lack some of those paths. -Copy-on-Write tables are not affected. Spark 4.0 is affected only if `spark.sql.variant.pushVariantIntoScan` is turned on, since it is off by default there. A fix is tracked in [#20039](https://github.com/apache/hudi/pull/20039). +On Copy-on-Write and Merge-on-Read tables: -:::tip -Until the fix is released, set `spark.sql.variant.pushVariantIntoScan=false` for Spark 4.1 reads of Merge-on-Read tables with `VARIANT` columns. +* A `VARIANT` column added with `ALTER TABLE ... ADD COLUMNS` cannot be read from files written before it was added. The read fails with `INVALID_VARIANT_SHREDDING_SCHEMA`. + +Reads with the rule off are not affected. The first two are fixed on the 1.2.x branch ([#20039](https://github.com/apache/hudi/pull/20039)); no release carries that fix yet. The other two are tracked in [#20040](https://github.com/apache/hudi/issues/20040) and [#20135](https://github.com/apache/hudi/issues/20135). + +On Spark 4.0 the rule is not supported. With it turned on, every read of a table that has a `VARIANT` column fails with `HoodieSchemaException: Illegal initial character: 0`, so leave it at its default there. Review Comment: Leaving this as is: no follow-up 1.2.x release is planned yet, so there is no target to name. Whoever cuts the next release that carries #20039 updates this line in its notes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
