deepakpanda93 commented on code in PR #19496: URL: https://github.com/apache/hudi/pull/19496#discussion_r3713676893
########## hudi-spark-datasource/SPARK_VERSION_UPGRADE.md: ########## @@ -0,0 +1,273 @@ +<!-- +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, software +* distributed under the License is distributed on an "AS IS" BASIS, +* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +* See the License for the specific language governing permissions and +--> + +# Adding Support for a New Spark Version + +This guide describes how to add support for a new Spark version to Hudi. It is written for Hudi +contributors, not for users upgrading the Spark version of an existing deployment. + +The work is spread across the build, the datasource modules, CI and the release tooling. Most of it +is mechanical, but the parts outside `hudi-spark-datasource` are easy to miss and are what usually +turns up later as a red CI job or a missing bundle at release time. + +## How Spark support is structured + +Hudi isolates version-specific Spark code behind a `SparkAdapter`, so that shared code can be +written once: + +``` +hudi-spark-common code shared by every Spark version +hudi-spark3-common shared by Spark 3.x, holds BaseSpark3Adapter +hudi-spark4-common shared by Spark 4.x, holds BaseSpark4Adapter +hudi-spark3.3.x Spark 3.3-specific adapter and overrides +hudi-spark3.4.x +hudi-spark3.5.x default profile +hudi-spark4.0.x +hudi-spark4.1.x +hudi-spark4.2.x +hudi-spark session extensions, procedures, SQL parser, logical plans +``` + +One version module is active per build, selected by a Maven profile. `SparkAdapterSupport` picks the +matching adapter at runtime from the Spark version actually on the classpath, so a bundle built for +one version fails fast rather than misbehaving on another. + +There is no per-version bundle module: `packaging/hudi-spark-bundle` derives its artifact id from +the `sparkbundle.version` property, so setting that property in the new profile is what produces +`hudi-spark<X.Y>-bundle_<scala>`. + +## Checklist + +Every file below was touched by both the Spark 4.1 (`20a01051ca6d`) and Spark 4.2 (`77f5851a5d53`) +additions, so treat it as the minimum set. + +**Build** + +- [ ] `pom.xml` — add the `sparkXY.version` property, a `spark<X.Y>` profile, and the modules it activates +- [ ] `hudi-spark-datasource/hudi-spark<X.Y>.x/pom.xml` — the new module + +**Version module** (`hudi-spark-datasource/hudi-spark<X.Y>.x/`) + +- [ ] `adapter/Spark<X>_<Y>Adapter.scala` extending `BaseSpark3Adapter` or `BaseSpark4Adapter` +- [ ] `HoodieSpark<XY>CatalystExpressionUtils`, `HoodieSpark<XY>CatalystPlanUtils`, `HoodieSpark<XY>SchemaUtils` +- [ ] Avro serializer and deserializer, plus the copies of Spark's own `AvroSerializer`/`AvroDeserializer` +- [ ] Parquet reader and legacy file format +- [ ] `parser/HoodieSpark<X>_<Y>ExtendedSqlAstBuilder.scala` and `...ExtendedSqlParser.scala` +- [ ] `antlr4/imports/SqlBase.g4` and `antlr4/.../HoodieSqlBase.g4`, copied from the matching Spark release +- [ ] Partition mapping and `HoodieInternalRow` implementations + +**Wiring** + +- [ ] `HoodieSparkUtils.scala` — `isSpark<X>_<Y>` and `gteqSpark<X>_<Y>` +- [ ] `SparkAdapterSupport.scala` — add the new version to the dispatch chain, newest first + +**CI** + +- [ ] `.github/workflows/bot.yml` — matrix entries for the java tests, scala tests, bundle validation and docker jobs +- [ ] `.asf.yaml` — the same jobs as required status checks, otherwise they do not block merges + +**Packaging and release** + +- [ ] `packaging/bundle-validation/base/build_<flink><hive><spark><scala>.sh` — base image for the new combination +- [ ] `packaging/bundle-validation/ci_run.sh` — component versions for the new `SPARK_RUNTIME`, and + the bundle artifact names for the new profile +- [ ] `packaging/bundle-validation/run_docker_java17.sh` +- [ ] `scripts/release/deploy_staging_jars_java17.sh` and `scripts/release/validate_staged_bundles.sh` + +**Docs** + +- [ ] `README.md` — the build-profile table +- [ ] `hudi-spark-datasource/README.md` — the module table and supported versions + +## Worked example: adding Spark 4.2 + +What follows is the shape of `77f5851a5d53 feat(spark): add Spark 4.2 support (#18621)`, which +touched 47 files. Snippets show the current state of those files rather than the exact original +commit, since Spark 4.2 has moved from a preview build to `4.2.0` since. Substitute your own Review Comment: Same situation as the other thread: the tree has moved past what this describes. ``` $ grep -n "spark42.version" pom.xml 181: <spark42.version>4.2.0</spark42.version> $ grep -n "spark4.2.0" packaging/bundle-validation/ci_run.sh 150:elif [[ ${SPARK_RUNTIME} == 'spark4.2.0' && ${SCALA_PROFILE} == 'scala-2.13' ]]; then ``` Neither says `4.2.0-preview4`. Both changed in `e5452c6ff38d feat(spark): upgrade Spark 4.2 dependency to GA 4.2.0 (#19365)` on 2026-07-24, the same commit that replaced the explicit version list in `validate.sh` with the globs. So the two observations have one shared cause, and the guide matches the tree as it stands. The premise sentence now names that commit instead of just asserting the move, so the claim can be checked without trusting it. Taking the underlying point seriously though, I went through every snippet in the guide against its source file rather than only the two flagged here, and that did turn up one genuine drift: the `ci_run.sh` block had `HADOOP_VERSION=3.4.3` where the file has `3.5.0`. Corrected. The pom property, the runtime branch string, `SPARK_VERSION`, the bundle name and the image tag all match the tree. Keeping the preview caveat at the end regardless, since it is about what to do when the next version is a preview rather than about 4.2 specifically. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
