GitHub user wombatu-kun added a comment to the discussion: Proposal: drop Spark 4.0 support in Hudi 1.3.0
+1, though the load-bearing argument is narrower than the TL;DR: `hudi-spark4-common` compiles once per profile, so while `spark4.0` builds it the source cannot reference `VariantLogicalTypeAnnotation` at all. No config or runtime gate buys that back. Two corrections before this goes to the dev list. **Shredding is unreachable at default config.** `HoodieSparkSchemaConverters` always builds an unshredded `createVariant(...)`, and both write supports shred only when the schema already carries `typed_value` or the test-only `hoodie.parquet.variant.force.shredding.schema.for.test` is set. "Spark 4.0 writes shredded files its reader cannot rebuild" is not reachable today; the annotation gap is unconditional, so I would lead with that and drop the shredding paragraph. **The drop does not make variant files self-describing.** AVRO is the default write-path record type, and the avro path writes the variant group through `AvroSchemaConverterWithTimestampNTZ` as a bare `GroupType` on every profile; only `HoodieRowParquetWriteSupport` reaches `applyVariantLogicalType`. That converter sits in `hudi-hadoop-common`, which still compiles against parquet 1.13.1 under `spark3.5`, so the floor rises for `hudi-spark4-common` only and 4.1/4.2 files stay as opaque to other engines as 4.0 ones. Two consequences: annotating on the avro path is a follow-up worth filing alongside this, and `isVariantPhysicalSchema` has to stay as a fallback rather than being replaced by an annotation check. One practical note for the drop PR: `hudi_trino_e2e.yml` hardcodes `-Dspark4.0` and the `spark402` composes are the only ones carrying a `trinocoordinator` service, so that workflow needs moving onto the `spark411` compose in the same change. On gate (2): DBR 17.3 LTS ships Spark 4.0.0 with an LTS window into 2028, so that is the population to name in the deprecation note. GitHub link: https://github.com/apache/hudi/discussions/19585#discussioncomment-17985073 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
