nsivabalan opened a new pull request, #20352:
URL: https://github.com/apache/hudi/pull/20352
### Describe the issue this Pull Request addresses
Validating a release candidate before voting is a manual, multi-hour
checklist today. `scripts/release/validate_staged_release.sh` covers the source
release, but everything after it is done by hand on every RC: downloading the
staged bundles, running a quickstart, standing up a HoodieStreamer pipeline to
confirm commits, compaction and cleaning, building from source, and choosing
and running tests. That makes RC validation slow to start, hard to repeat, and
uneven across voters. Several environment traps also produce failures that look
like RC problems when they aren't:
- tests run on a different JDK than the release builds with
- MOR test tables so small that compaction never runs
- spark-shell scripts that hang when a line throws
### Summary and Changelog
Voters get one command that validates an RC at the depth they have time for,
against the artifacts actually being voted on, with a PASS/FAIL line per check
and a summary at the end.
```bash
scripts/release/validate_rc.sh --release=1.1.2 --rc_num=1
--scale=minimal|decent|large \
--staging_repo=<staging repo url> --signing_key=<key id>
--spark_home=<spark 3.5 dir>
```
| Scale | Steps | Time |
|---|---|---|
| minimal | source release checks (now including signing-key match);
quickstart on COW and MOR (datasource + Spark SQL, 21 checks) against the
staged spark bundle; continuous HoodieStreamer run on MOR | ~5-10 min |
| decent | minimal + build from the voted source tarball with the JDK its
pom declares + HoodieStreamer tests + a larger HoodieStreamer run | ~15-30 min |
| large | decent + `TestCOWDataSource`, `TestMORDataSource`,
`TestInsertTable`, `TestMergeIntoTable` + high-volume HoodieStreamer runs on
MOR and COW | ~2-4 h |
Changelog:
- `scripts/release/validate_rc.sh`: entry point. Picks steps from `--scale`;
`--steps` overrides. Each step runs isolated, so one failure doesn't hide the
others' results.
- `scripts/release/rc_validation/`:
- `common.sh`: picks the JDK from the source pom and downloads staged
bundles with sha1 checks. Its spark-shell runner treats compile errors and REPL
fall-through as failures.
- `quickstart.scala`: the COW and MOR quickstart checks.
- `run_streamer_e2e.sh`, `streamer_e2e_datagen.scala`,
`streamer_e2e_validate.scala`, `trips.avsc`: HoodieStreamer in continuous mode
with `ParquetDFSSource` and `FilebasedSchemaProvider`, fed by a timed batch
generator. Checks commit count, error records, ERROR log lines, record count
against the expected total, duplicate keys, arrival of the last batch,
completed async compaction (MOR) and files deleted by the cleaner.
- `run_tests.sh`: scale-based test selection. It fails if no tests ran, so
a selector that matches nothing doesn't produce a green build.
- `scripts/release/skills/hudi-rc-validation/SKILL.md`: an agent skill. It
maps a request like "quick sanity" or "thorough" to a scale, runs the script,
separates known environment failures from real RC problems, and drafts the vote
reply. Coding agents can use it; people can follow it as a runbook.
Verified against 1.1.2-rc1 (`orgapachehudi-1198`):
- minimal: all steps pass, ~8.5 min
- decent: all steps pass, ~28 min; 280k-record MOR run, build from source
tarball on JDK 8, 10/10 HoodieStreamer tests
- COW e2e at minimal size: passes
- large: not run end to end. Its pieces are the same scripts at higher
volume, plus suites that passed separately (243 datasource tests, 38 Spark SQL
tests).
While building this I found a pre-existing bug, which I'll file separately:
`CALL show_cleans(..., showArchived => true)` fails with `Not an Avro data
file` once cleans are archived to the LSM timeline. It affects the release as
well as master.
### Impact
New release tooling only, under `scripts/release/`. No change to any module,
public API, config or build.
### Risk Level
none. The scripts are run manually by release voters and nothing in the
build or CI calls them. All new files carry ASF headers and pass
`apache-rat:check`.
### Documentation Update
none. Usage is documented in the `validate_rc.sh` header (`--help`) and in
the skill. A follow-up can link it from the release guide on the website.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
