voonhous opened a new pull request, #20159: URL: https://github.com/apache/hudi/pull/20159
### Describe the issue this Pull Request addresses The 1.2 release notes carry one known issue: on Spark 4.1, where `spark.sql.variant.pushVariantIntoScan` is on by default, Merge-on-Read reads of `VARIANT` columns lose null rows or crash (#20037). Three corrections: - It sits in the 1.2.1 section only. 1.2.0 has the same code. - 1.2.1 has two more failures that only happen with the rule on: a query extracting several paths through a log merge (`NegativeArraySizeException`, #20040) and a `VARIANT` column added by DDL read from older files (`INVALID_VARIANT_SHREDDING_SCHEMA`, #20135, Copy-on-Write too). Every read is correct with the rule off. - Spark 4.0 with the rule turned on does not hit those defects. Every read of a table with a `VARIANT` column fails with `Illegal initial character: 0`. ### Summary and Changelog - Release notes, Known Issues: the four symptoms, where each is fixed, the Spark 4.0 behaviour, and the workaround: turn the rule off with the `SET` statement when a read hits one of them. The 1.2.0 VARIANT section links to it. - SQL DDL page, versions 1.2.0 and 1.2.1: the same caution under the engine support table. - SQL DDL page, current: what the setting does and where Hudi supports it. <details> <summary>What the statements rest on</summary> The two added symptoms (Spark 4.1.1) and the Spark 4.0 sentence (Spark 4.0.2) were run on the `release-1.2.1` tag: COW and MOR, AVRO and SPARK record types, both arms of the conf. The first two symptoms are the #20037 evidence. 1.2.0 was not run; the code behind each symptom is the same in both tags. </details> ### Impact Docs only. ### Risk Level none ### Documentation Update This is the documentation update. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
