databird07 opened a new pull request, #20364:
URL: https://github.com/apache/hudi/pull/20364
### Describe the issue this Pull Request addresses
- Java CI on master (`bot.yml` + `java_ci_engines.yml`) uses about 680
runner job-minutes per push, counted as the sum of every job's duration.
- This PR cuts that total without dropping test areas or coverage.
- Whole workflow, every job counted: **682.1 → 484.3 job-minutes (−29%)**.
Master runs 40306329236 + 40306329230, PR runs 40374455400 + 40374455419.
- On the jobs this PR changes (all test jobs, plus its shared builds):
**550.1 → 348.6 (−37%)**.
- Bundle validation and integration-tests are unchanged by this PR and take
about 130 job-minutes on both sides.
### Summary and Changelog
**The total falls because each job does less work, not because the work is
spread over more jobs.**
- Job-minutes add up every job's duration. Splitting a job into more jobs
would leave the total unchanged.
- Most of the saving removes work: builds that ran twice, tests that ran
twice, sleeps, process forks, allocations and redundant matrix rows.
- One change is parallelism: two surefire JVMs on the same runner, using
cores that were idle. The runner is billed until the job ends, so a shorter job
uses fewer job-minutes. This was dropped for hudi-utilities, where it made the
job slower.
**Duplicate work removed**
- Build once: 3 shared build jobs upload `target/` and the Hudi `.m2`
artifacts. The test jobs restore them instead of each running `mvn clean
install -DskipTests`. The restore checks a manifest of the commit and profile
set.
- The JDK 11 artifact leaves out what its `mvn test` jobs never read: shaded
bundle jars (except `hudi-utilities-bundle`, which hudi-cli resolves from
`.m2`), the `hudi-examples-k8s` jar and `-sources.jar` files. 3.92 GB → 1.18
GB; restore time across its 4 jobs went from 59–95 to 17 job-minutes.
- `test-flink-2`: `-Dit.test=!ITTestHoodieDataSource` matched every class,
so the job also re-ran all 183 hudi-flink unit-test classes. Those already run
in `test-common-and-other-modules`. The filter is now
`IT*,!ITTestHoodieDataSource`; the job went from 42.7 to 14.0 min.
**Waiting removed**
- Test configs set `max_expected_clock_skew_ms=1` instead of 200 ms, in the
Flink tests and `HoodieSparkSqlTestBase`. That sleep runs on every instant
generated under a lock.
- ITTestDataStreamWrite checkpoints every 1 s instead of 4 s, and two
`sleep(2)` calls are gone.
- Heartbeat-expiry tests backdate the heartbeat file instead of sleeping
about 30 s.
- The streamer clustering-conflict retry is configurable:
`--clustering-conflict-retry-interval-seconds`, default 60 s as before. The
conflict test uses 2 s.
- TestWriterWithPartitionTTl now waits out its 1 s TTL explicitly. It used
to depend on the 200 ms sleeps adding up.
**Per-operation overhead removed**
- `NioPermissionLocalFileSystem` (test-only) sets file permissions through
`java.nio`. Without Hadoop native IO, as on CI, Hadoop starts a `chmod` process
per file; hudi-spark tests went from about 23k of those to 0. It is wired into
the utilities, Spark client and Scala SQL test bases.
- HFile whole-file reads use a `min(fileSize, defaultBufferSize)` buffer
instead of allocating 32 MB on every open (production code).
- `HadoopStorageConfiguration` Java serialization writes plain key/value
pairs instead of `Configuration#write`, which gzips every property's source
list (production code). Serialize and deserialize are 7–30x faster. Payloads in
the old format (e.g. Flink state from an older Hudi) are still read through
`Configuration#readFields`.
**Redundant test cases removed** (the full matrix still runs on the current
table version; v6 rows are kept)
- Older-version rows (v8/v9) were trimmed in TestMORDataSource,
TestColumnStatsIndex, the merge-mode SQL suites, and the
TestHoodieDeltaStreamer and SchemaEvolutionQuick matrices (pairwise).
- TestHoodieFileSystemViews runs MDT-on only, and each table is written once
for both view types.
- Read-only cases share one table fixture in TestColumnStatsIndexWithSQL and
TestHoodieMetadataTableValidator.
- TestBootstrapIndex runs 10 validations per thread instead of 50, still
with 20 concurrent threads.
**Fixes found along the way**
- KafkaTestUtils now waits for partition leaders and checks send acks. Lost
records used to cause a flaky 120 s timeout that reruns hid.
- TestColumnStatsIndexWithSQL now really writes v6/8/9 tables. It never
passed `tableVersion` to the writer before.
- RLI lookup-metrics tests shut down all live `Metrics` before clearing the
shared capturing reporter. A reporter left attached by an earlier test made
them flaky (also on master).
- `spark.ui.enable` is corrected to `spark.ui.enabled` in UtilitiesTestBase.
**Measured per job** (master runs 40306329236 + 40306329230 → PR runs
40374455400 + 40374455419, in minutes)
- Java part1 65.5 → 42.4, part2 70.3 → 42.1, part3 36.1 → 25.9
- Scala DML 62.3 → 29.6, Scala other 72.7 → 25.1
- Flink 1 34.4 → 24.2, Flink 2 39.5 → 14.2
- utilities 51.7 → 35.5, common + other modules 63.2 → 44.6
- spark-client + hadoop-common 39.8 → 34.9, hadoop-mr + java-client 14.6 →
9.6
- The three shared builds add 20.5.
- Not changed by this PR: bundle validation 96.2 → 99.4, integration-tests
35.8 → 36.2.
- Single-run job times vary by about 7–15 min per job.
**How it was verified**
- Each change was timed locally in 6–20 alternating before/after runs with
JaCoCo on, with the same test counts and 0 failures. Results are in each commit
message.
- JaCoCo diffs found no lost lines, apart from a few timing-dependent ones
that other tests still cover. Codecov on an earlier run went from 80.73% to
80.74%.
- Every changed production line in `HadoopStorageConfiguration` is covered
by `TestHadoopStorageConfiguration`, including the old-format and
unknown-version paths.
- The branch is rebased onto master c0ed52483f. The new build jobs use
`ubuntu-24.04`, matching master's runner pin (#20418). The workflow changes
were moved into the split `bot.yml` and `java_ci_engines.yml`. `build-flink`
lives in `java_ci_engines.yml`, because artifacts only download within the same
workflow run.
### Impact
- Mostly CI and test code. There are three production changes:
- The HFile read buffer size.
- The `HadoopStorageConfiguration` serialization format. Older payloads
still deserialize. An older Hudi reading the new format gets an empty
`Configuration` without any error. So mixing Hudi versions between driver and
executors, or downgrading a Flink job onto new state, isn't supported.
- The new streamer option, whose default keeps the old behaviour.
- One deliberate coverage tradeoff: no Spark-record StreamSync write to a v8
table. v8 still runs with Avro records.
- MDT-off file-system-view cases were removed. Those writer paths stay
covered by `TestHoodieBackedMetadata` and `TestHoodieTableFileSystemView`.
### Risk Level
medium
- Two surefire JVMs can expose shared-resource assumptions. Ports are per
fork, and the change was checked with 4 full-job runs and 60 repeated subset
runs in random order.
- Build once depends on restored outputs being treated as up to date.
Manifests are checked and mtimes are re-stamped.
- Each change is its own commit and can be reverted on its own.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
