GitHub user voonhous created a discussion: Proposal: drop Spark 4.0 support in Hudi 1.3.0
## TL;DR Hudi master carries three Spark 4 minor versions on three different parquet-java lines. Spark 4.0 is the only one below parquet 1.16, which is the version that introduced the VARIANT logical type annotation. That single gap is the reason Hudi's variant support has a no-op seam in shared code, a forked `ReadSupport`, and a name/arity shape heuristic standing in for what should be a metadata lookup. Spark 4.0 reaches end of life on [2026-11-23](https://lists.apache.org/thread/cxyfz1zlqhltzd4dsk7wn9v4jdmljcd5). It currently has zero active CI coverage on master. Dropping it in 1.3.0 removes ~5,000 LOC, collapses the Spark 4 line onto a single variant capability floor, and lets the write path stop producing parquet files that do not self-identify as variant. # Parquet-java Version | Profile | Spark | parquet-java | avro | jackson | VARIANT annotation | |---|---|---|---|---|---| | `spark3.5` | 3.5.3 | 1.13.1 | 1.11.x | 2.15.x | n/a (no VariantType) | | `spark4.0` | 4.0.2 | **1.15.2** | 1.12.0 | 2.18.2 | **no** | | `spark4.1` | 4.1.1 | 1.16.0 | 1.12.1 | 2.20.0 | yes | | `spark4.2` | 4.2.0 | 1.17.0 | 1.12.1 | 2.21.2 | yes | The defensible argument is narrower and stronger: **dropping 4.0 makes every Spark 4 profile parquet >= 1.16, which means every Spark 4 profile can express and read the VARIANT logical type.** It is a capability floor, not a version bump. ## Why the 1.15 / 1.16 boundary specifically matters parquet-java 1.16.0 added `LogicalTypeAnnotation.VariantLogicalTypeAnnotation`. Per the parquet spec, a variant is a group annotated with the VARIANT logical type containing `metadata` and `value` binary fields. The annotation is what makes the file self-describing. On parquet 1.15.2 the annotation does not exist. Hudi therefore writes the group without it. Concretely, in `BaseSpark4Adapter` (`hudi-spark4-common`): ```scala // TODO(#18935) drop-spark4.0: when all remaining 4.x adapters are parquet 1.16+, // apply variantType() in this base and delete the no-op default plus the // Spark4_1Adapter override. protected def applyVariantLogicalType(builder: Types.GroupBuilder[GroupType]) = builder ``` The base default is a no-op so Spark 4.0 compiles; 4.1 and 4.2 each override it to actually apply the annotation. Two overrides and a dead default exist solely to keep 4.0 in the build. The read side pays a larger price. Because 4.0-written files carry no annotation, Hudi cannot ask parquet "is this a variant?" and instead infers it from field names and arity (`BaseSpark4Adapter.isDataTypeEqualForPhysicalSchema`): ```scala // TODO(voon) parquet-1.16: replace this name/arity shape heuristic with a // VariantLogicalTypeAnnotation check once all supported parquet versions are >= 1.16. def isVariantPhysicalSchema(structType: StructType): Boolean = { // 2 binary fields named metadata/value -> unshredded variant // 3 fields incl. typed_value -> shredded variant } ``` ## Correctness, not just tidiness `hoodie.parquet.variant.write.shredding.enabled` defaults to `true` (`HoodieStorageConfig`) and has **no Spark-version gate**. Spark 4.0 therefore writes shredded variant files. But `buildFullVariantReadSchema` is overridden only in the 4.1 and 4.2 adapters; 4.0 falls through to the base `None`. The in-tree comment states the situation plainly: > Spark 4.0 keeps the default None: the write-side methods above have no version > gate, so it does write shredded files, but its reader cannot rebuild them and > the projection shape would not help. The consequence is that Spark 4.0 produces files that (a) are not identifiable as variant by any other engine reading the parquet metadata, and (b) depend on Hudi-side compensating code to be read back. That compensating code has needed repeated repair: - #18674 "align Spark 4.1 MOR merge with PushVariantIntoScan and **restore Spark 4.0 reads**" - d42560fe (in flight) "detect shredded variant base files by shape so reconstruction engages" Every variant change now has to be validated against a version that cannot natively express the format it is writing. Additionally, Spark 4.0's `ParquetUnshreddedVariantConverter` builds its converter array in hardcoded `[value, metadata]` order and then indexes by schema position, so spec-ordered `[metadata, value]` files fail with `MALFORMED_VARIANT`. Hudi carries an entire forked read support, `Spark40HoodieParquetReadSupport.scala`, to reorder fields around this. SPARK-54410 fixed it in 4.1 by reading by name. That file exists for one bug in one version. ## Proposed motion Drop the `spark4.0` profile and delete `hudi-spark-datasource/hudi-spark4.0.x` in 1.3.0, subject to (1) the deprecation-policy answer and (2) no major runtime being stranded. Supported Spark matrix becomes 3.5, 4.1, 4.2 -- with every Spark 4 profile on parquet >= 1.16 and a single variant capability floor. ## References - Parquet 1.16.0 release notes (GH-3070, VARIANT logical type annotation): https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.16.0 - Parquet variant spec: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md - Spark versioning policy: https://spark.apache.org/versioning-policy.html - Spark EOL dates: https://endoflife.date/apache-spark - Hudi #18935 -- umbrella for Spark 4.0 workaround cleanup - Hudi #18334 -- Spark 4.0 variant field-order workaround - SPARK-54410 -- read variant fields by name (fixed in 4.1) GitHub link: https://github.com/apache/hudi/discussions/19585 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
