voonhous commented on issue #18935: URL: https://github.com/apache/hudi/issues/18935#issuecomment-5449946832
### Status as of efe02e16 (master, 2026-08-28) Nothing functional is pending here. `variantType()` is already applied on every Spark line newer than 4.0; this stays open as a parked cleanup for when the `hudi-spark4.0.x` module is dropped. What the code looks like at that commit: - `SparkAdapter.applyVariantLogicalType` is an identity default ([SparkAdapter.scala#L494](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala#L494)), `BaseSpark4Adapter` re-declares it as a no-op with the `TODO(#18935)` marker ([BaseSpark4Adapter.scala#L288-L289](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark4Adapter.scala#L288-L289)), and `HoodieRowParquetWriteSupport` calls it when building the variant group ([HoodieRowParquetWriteSupport.java#L927](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowParquetWriteSupport.java#L927)). - `Spark4_1Adapter` and `Spark4_2Adapter` both override it with `builder.as(LogicalTypeAnnotation.variantType(1.toByte))` ([Spark4_1Adapter.scala#L288-L289](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_1Adapter.scala#L288-L289), [Spark4_2Adapter.scala#L288-L289](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_2Adapter.scala#L288-L289)). - Spark 4.0 cannot apply it. The `spark4.0` profile pins parquet 1.15.2 ([pom.xml#L2899](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/pom.xml#L2899)) while 4.1 and 4.2 pin 1.16.0 and 1.17.0, and `hudi-spark-bundle` shades only `parquet-avro` ([hudi-spark-bundle/pom.xml#L100](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/packaging/hudi-spark-bundle/pom.xml#L100)), so at runtime `LogicalTypeAnnotation` comes from the Spark distribution's own parquet-column jar. Bumping parquet in the 4.0 profile would not help, and an unconditional `variantType()` in the base would be a `NoSuchMethodError` on 4.0. - `Spark40HoodieParquetReadSupport` still exists with its `TODO(#18935)` ([Spark40HoodieParquetReadSupport.scala#L36](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark40HoodieParquetReadSupport.scala#L36)) and is wired in by `Spark4_0Adapter` ([Spark4_0Adapter.scala#L197](https://github.com/apache/hudi/blob/efe02e1621c17aa39a61716371a1857b4c086f79/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark4_0Adapter.scala#L197)). It works around the positional variant converter in Spark 4.0.x (#18334, closed), a separate reason that also only goes away with the 4.0 module. When 4.0 is dropped the work is: move the `variantType(1.toByte)` call into `BaseSpark4Adapter`, delete the two overrides, and delete `Spark40HoodieParquetReadSupport` along with the rest of `hudi-spark4.0.x`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
