voonhous commented on PR #18816: URL: https://github.com/apache/hudi/pull/18816#issuecomment-5497213179
Following up on the numbers above: the separate GHA job is gone. hudi-cli now runs inside `test-spark-client-and-hadoop-common`. Of the 18m10 that `test-hudi-cli` took, 7m14 was a build. hudi-cli's dependency closure is a superset of hudi-spark-client's, so the host job's existing `-am` build covers it (`-pl hudi-client/hudi-spark-client,hudi-cli`) and the only new cost is the tests: 18s UT + 10m10 FT. Job durations from the 12 most recent successful master runs of `bot.yml`, with what folding hudi-cli into each would cost: | job | median today | after the fold | worst-case slack under the critical path | |---|---|---|---| | critical path (`test-spark-java17-*`) | 70 | 70 | - | | test-common-and-other-modules | 54 | ~65 | +2.5 | | test-utilities | 50 | ~61 | +5.5 | | test-spark-client-and-hadoop-common | 27 | ~41 | +25.1 | | test-hudi-hadoop-mr-and-hudi-java-client | 15 | ~30 | +36.2 | The catch-all `test-common-and-other-modules` is the natural home and needs no extra build at all, but on its worst run of the 12 it lands within 2.5 min of the critical path, and it runs in Docker at `-Xmx4g`. `test-spark-client-and-hadoop-common` keeps 25+ min of headroom on every one of the 12 runs, is the same bare-runner JDK 11 / scala-2.12 / spark3.5 / flink2.2 shape the standalone job used, and already has both a UT and an FT step feeding one Codecov upload. `test-utilities` would be ~3 runner-minutes cheaper than that but sits 10 min from the critical path, which the spark4 matrix trimming would erase. Net effect: CI wall clock unchanged, the module costs +13.9 runner-minutes instead of +18.2, one fewer job in the matrix. Coverage rides the existing `spark-client-hadoop-common` flag, so the `hudi-cli` flag line added to `.codecov.yml` is reverted (the `hudi-cli` component was already there and is untouched). Host job timeout goes 60 -> 75, and hudi-cli stays excluded from `UT_MODULES`/`FT_MODULES` so it does not run twice. Azure is unchanged: hudi-cli is already inside the catch-all `UT_FT_10` list there, which is the +10 min measured above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
