voonhous opened a new pull request, #19928: URL: https://github.com/apache/hudi/pull/19928
### Describe the issue this Pull Request addresses Part of #19524, stacked on the shared-session and state-scoping PRs (successors of #19921 and #19923). Supersedes the proof of concept #19925. The Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork count, so the lever is running suites concurrently in one JVM. This PR turns that on for the three SQL-only shards and measures it. ### Summary and Changelog - Pom properties `hudi.scalatest.parallel` (default false) and `hudi.scalatest.threads` (default 0) mapped to the scalatest plugin's `parallel` and `threadCount`: suites run concurrently, tests within a suite stay sequential. Set to true and 2 on dml-1, dml-2 and other-1, next to the shared-session flag they carry. - Two fixes the local two-thread loop exposed: the per-test cleanup owns a table only by the suite's name prefix or as a session temp view (derived names such as `<generated>_pt` were dropped by other suites), and two helpers that sorted log file paths as strings now sort by the log file comparator (the write token precedes the instant in the name, so stage ids compared as text). The second is latent on master. Measurement, over three runs of each of the three shards: the scalatest "Run completed" wall clock against the same job serial. dml-1 and other-1 are the two jobs setting the run's critical runtime of about 41 minutes under #19892, so this is where the wall clock should move. A result inside the serial range is no gain. | shard | serial under #19892 and #19931 | run 1 | run 2 | dml-1 rerun (attempt 2) | full rerun (attempt 3) | full rerun (attempt 5) | |---|---|---|---|---|---|---| | dml-1 (dml.others) | 30:52, 30:28, [30:50](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268004) | [20:05](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487325) | [20:19](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474773) | [19:36](https://github.com/apache/hudi/actions/runs/34711686507/job/103608206970) | [20:13](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929537) | [20:10](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455068) | | dml-2 (dml.insert, dml.schema) | 19:15, 23:01, [18:46](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268040) | [16:31](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487358) | [13:27](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474810) | not rerun | [16:00](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929455) | [16:56](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455002) | | other-1 (procedure, ddl) | 31:52, 32:13, [33:06](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268051) | no data (ten failures, see below) | [22:30](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474695) | not rerun | [17:14](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929554) | [22:34](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455060) | | run's longest job | 41 to 44, [41](https://github.com/apache/hudi/actions/runs/34720098561) | [37, scala-other-2](https://github.com/apache/hudi/actions/runs/34708740840) | [36, scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507) | not a full run | [35, scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3) (a) | [37, scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5) | Serial times are the scalatest wall clocks of #19892's runs [34581330915](https://github.com/apache/hudi/actions/runs/34581330915) and [34499656524](https://github.com/apache/hudi/actions/runs/34499656524), plus #19931's first run, which has the same layout without this stack; every sample ran the same test names as those runs. One more two-thread sample after the four-thread experiment, on the revert commit ([run 34742475247](https://github.com/apache/hudi/actions/runs/34742475247)): dml-1 [15:25](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400514), dml-2 [16:21](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400544), other-1 [19:43](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400543), longest job 37 (ut-fat-and-ftc), all green with the same test names. Verdict after three or more samples per shard: dml-1 30 to 31 -> 15:25 to 20:19 (-34% to -50%), dml-2 19 to 23 -> 13:27 to 16:56 (-23%), other-1 32 to 33 -> 17:14 to 22:34 (-36%), every sample below every serial sample. The run's longest job moved from the Scala shards (39 to 44) to the serial scala-other-2 at 35 to 37, so the next step is that shard. **Phase 4a, four threads** (commit b8abd450c3c4, reverted by fe4e20376ae5; the PR stays at two threads): does the shard still scale on a 4 vCPU runner once two threads keep both busy 87 to 92 percent of the time? Same measurement, two samples (the second decided it). | shard | two threads (above) | sample 1 | sample 2 | |---|---|---|---| | dml-1 (dml.others) | 19:36 to 20:19 | [17:20](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440612) | [13:35](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069789) | | dml-2 (dml.insert, dml.schema) | 13:27 to 16:56 | [13:17](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440592) | [14:31](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069836) | | other-1 (procedure, ddl) | 17:14 to 22:34 | [20:04](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440550) | [18:43](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069804) | | run's longest job | 35 to 37 | [36, scala-other-2](https://github.com/apache/hudi/actions/runs/34737535334) | [36, three jobs tied](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2) | Four threads keep 3.6 of them busy on dml-1 and dml-2 and 3.0 on other-1, at the price of every suite running about 60 percent longer than at two threads (four sessions on four vCPUs). dml-1 gains on both samples; dml-2 and other-1 stay inside their two-thread spread. Test names identical to serial on all six shard runs. Decision: stay at two. The gain is on a shard that does not set the run's wall clock, and four-way contention on a 4 vCPU runner narrows every suite's timing margin; four threads is recorded here as a measured, safe setting for a shard that becomes the critical path. other-1's first run ([job 103593487295](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487295), 21:53) failed ten tests: a shared-session leak in the parquet reader, fixed in the shared-session PR, and a catalog-wide `show tables` assertion, fixed in the state-scoping PR. It is not counted; other-1's third sample comes from one more rerun after attempt 3. (a) On attempt 3 the Java j ob ut-other hung after finishing all its hudi-common test classes (27 to 34 min on eleven earlier runs); it was cancelled at 76 min and rerun alone as attempt 4 in 34 min, which is the figure used. The Scala shards are unaffected. Both threads are busy 91%, 92% and 87% of the wall clock on dml-1, dml-2 and other-1 (sum of suite spans over wall). <details> <summary>Evidence from the proof of concept (#19925)</summary> On master's layout the single dml job (dml-1 plus dml-2 together) at two threads: 27:58, 35:27 and 35:32 ([run 34697891947](https://github.com/apache/hudi/actions/runs/34697891947), attempts 1 to 3) against 41:22 on master's fastest runner and 52 to 56 on its typical one, the same 326 test names each time, both threads busy 96 to 98% of the time. Locally, twenty iterations of twelve interaction-heavy suites at two threads were green, and the whole dml tree at two threads passed in 22 minutes. </details> ### Impact CI configuration and test infrastructure only. ### Risk Level low. Off by default; on three shards whose suites were prepared for it in the two PRs below and looped locally at two threads. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
