voonhous opened a new pull request, #19925: URL: https://github.com/apache/hudi/pull/19925
### Describe the issue this Pull Request addresses Part of #19524, stacked on #19921 and #19923 (their commits are included until they merge). The Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork count, so the lever is running suites concurrently in one JVM. #19921 made the suites share one SparkContext behind a property and #19923 removed the cross-suite state; this PR turns the concurrency on for the DML shard and measures it. ### Summary and Changelog - Pom properties `hudi.scalatest.parallel` (default false) and `hudi.scalatest.threads` (default 0) mapped to the scalatest plugin's `parallel` and `threadCount`: suites run concurrently, tests within a suite stay sequential. Set to true and 2 on `test-spark-java17-scala-dml-tests` only. - Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's name prefix or as a session temp view (derived names such as `<generated>_pt` were dropped by other suites), and two helpers that sorted log file paths as strings now sort by the log file comparator (the write token precedes the instant in the name, so stage ids compared as text). The second is latent on master. Measurement, over three runs of the DML shard: scalatest "Run completed" wall clock against the same job on master, about 41 minutes on a fast runner and 52 to 56 on a slow one (seven recent runs). The scheduling bound at two threads is about half the serial time, so a gain lands below master's fast case in every run; a result inside master's range is no gain. The per-suite sum over the wall clock (about 1.8 locally) is reported too but measures thread utilisation, not speedup. | run | dml shard scalatest time | master reference | |---|---|---| | 1 | pending | 41 min fast runner, 52 to 56 min slow runner | | 2 | pending | | | 3 | pending | | <details> <summary>Local verification</summary> | check | result | |---|---| | Maven run with the two properties on (`-DlogForkedProcessCommand=true`) | forked ScalaTest command carries `-P2` next to the shared-session property; base session created on a pool thread | | whole dml tree at two threads, Spark 3.5 / Scala 2.12 / JDK 11, three passes | 24 suites; the first two passes exposed the two fixes above, the third was green (284 passed, 46 Spark-version cancels expected on 3.5) | | twenty-iteration loop of twelve interaction-heavy suites at two threads | in progress; iteration 1 green (93 tests, 329 s) | | Spark 4.2 / Scala 2.13 / JDK 17 test-compile of the stacked branches | green (phase 2 diff; phase 3 adds pom and workflow lines plus the two helper fixes) | Local two-thread wall clock for the whole dml tree was 20 to 21 minutes awake time on this laptop; earlier local runs also contained laptop-sleep gaps that expired write heartbeats, which is not reproducible on CI runners. </details> ### Impact CI configuration and test infrastructure only. ### Risk Level low. Off by default; on one shard, whose suites were prepared for it in #19921 and #19923 and which was looped locally at two threads before this PR. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
