deepakpanda93 opened a new pull request, #19496:
URL: https://github.com/apache/hudi/pull/19496
### Describe the issue this Pull Request addresses
Closes #16055
There is no documentation for adding support for a new Spark version. The
work spans the build, the
datasource modules, CI and the release tooling, and the parts outside
`hudi-spark-datasource` are
the ones that get missed: a CI job added to `bot.yml` but not to `.asf.yaml`
runs without ever
blocking a merge, and a missing entry in the bundle-validation table only
shows up at release time.
### Summary and Changelog
Adds `hudi-spark-datasource/SPARK_VERSION_UPGRADE.md`. A new file rather
than a section in the
existing README, since that one describes the module layout while this
describes a procedure.
The steps are not invented. They are derived from the two most recent real
additions, Spark 4.1
(`20a01051ca6d`) and Spark 4.2 (`77f5851a5d53`), which touched the same set
of cross-cutting files,
so the checklist reflects what those changes actually required.
Contents:
- how the adapter layering works, and why there is no per-version bundle
module
- a checklist grouped by build, version module, wiring, CI,
packaging/release and docs
- a worked Spark 4.2 example: the Maven profile and why it pins transitive
versions, the version
predicates, the adapter dispatch chain, the CI matrix and `.asf.yaml`
entries, and the
bundle-validation and release script changes
- verification commands, including rebuilding the versions you did not touch
- the mistakes that fail silently rather than loudly
Documentation only. No code changes.
### Impact
None on runtime behaviour.
### Risk Level
none
### Documentation Update
This PR is the documentation update. No Hudi website change needed.
### Verification
Since prose has no test, the guide was checked two ways against the tree.
**Audited against the existing Spark 4.2 support.** Each checklist item was
tested as a predicate
over the repository. 22 of 23 passed. The one failure was an error in the
guide, not a gap in the
repo: it said bundle names live in
`packaging/bundle-validation/validate.sh`, when they are in
`ci_run.sh`, and `validate.sh` needs no per-version edit at all because it
branches on
`[[ "$SPARK_VERSION" == 4.* ]]`. The Spark 4.2 commit did edit
`validate.sh`, but for explicit
version lists that have since been replaced by that glob, so the item was
right for the historical
commit and wrong for the current tree. Fixed, with a note for anyone reading
that commit as a
template.
**Ran the three verification commands the guide gives**, verbatim, on JDK 17:
| command | result |
| --- | --- |
| `mvn clean install -Dscala-2.13 -Dspark4.2 -DskipTests -pl
hudi-spark-datasource/hudi-spark4.2.x -am` | BUILD SUCCESS |
| `mvn test -Dscala-2.13 -Dspark4.2 -pl
hudi-spark-datasource/hudi-spark4.2.x` | 3 tests, 0 failures |
| `mvn clean package -Dscala-2.13 -Dspark4.2 -DskipTests -pl
packaging/hudi-spark-bundle -am` | BUILD SUCCESS, produced
`hudi-spark4.2-bundle_2.13-1.3.0-SNAPSHOT.jar` |
The third confirms the claim that `sparkbundle.version` alone produces the
correctly named bundle,
with no per-version bundle module.
Every file path and class name cited in the guide was checked to exist, and
both referenced commit
hashes resolve. `mvn apache-rat:check` passes.
**What this does not prove.** Auditing against a completed integration shows
the guide matches how
Spark 4.2 was done. It does not show the guide is sufficient to add a
genuinely new version, where
the hard parts are writing the adapter against a changed Catalyst API and
sourcing the right ANTLR
grammar. That can only be tested by a real integration.
### Note for reviewers
A guide naming specific files and versions goes stale silently, since no
test covers prose. It tells
readers to check the real files before copying, but reviewers may prefer
less version-specific
detail in exchange for a longer shelf life. Happy to trim if so.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
