voonhous opened a new pull request, #19928:
URL: https://github.com/apache/hudi/pull/19928

   ### Describe the issue this Pull Request addresses
   
   Part of #19524, stacked on the shared-session and state-scoping PRs 
(successors of #19921 and #19923). Supersedes the proof of concept #19925. The 
Java CI wall clock is set by the Scala SQL shards, and scalatest has no fork 
count, so the lever is running suites concurrently in one JVM. This PR turns 
that on for the three SQL-only shards and measures it.
   
   ### Summary and Changelog
   
   - Pom properties `hudi.scalatest.parallel` (default false) and 
`hudi.scalatest.threads` (default 0) mapped to the scalatest plugin's 
`parallel` and `threadCount`: suites run concurrently, tests within a suite 
stay sequential. Set to true and 2 on dml-1, dml-2 and other-1, next to the 
shared-session flag they carry.
   - Two fixes the local two-thread loop exposed: the per-test cleanup owns a 
table only by the suite's name prefix or as a session temp view (derived names 
such as `<generated>_pt` were dropped by other suites), and two helpers that 
sorted log file paths as strings now sort by the log file comparator (the write 
token precedes the instant in the name, so stage ids compared as text). The 
second is latent on master.
   
   Measurement, over three runs of each of the three shards: the scalatest "Run 
completed" wall clock against the same job serial. dml-1 and other-1 are the 
two jobs setting the run's critical runtime of about 41 minutes under #19892, 
so this is where the wall clock should move. A result inside the serial range 
is no gain.
   
   | shard | serial under #19892 and #19931 | run 1 | run 2 | dml-1 rerun 
(attempt 2) | full rerun (attempt 3) | full rerun (attempt 5) |
   |---|---|---|---|---|---|---|
   | dml-1 (dml.others) | 30:52, 30:28, 
[30:50](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268004)
 | 
[20:05](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487325)
 | 
[20:19](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474773)
 | 
[19:36](https://github.com/apache/hudi/actions/runs/34711686507/job/103608206970)
 | 
[20:13](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929537)
 | 
[20:10](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455068)
 |
   | dml-2 (dml.insert, dml.schema) | 19:15, 23:01, 
[18:46](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268040)
 | 
[16:31](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487358)
 | 
[13:27](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474810)
 | not rerun | 
[16:00](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929455)
 | 
[16:56](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455002)
 |
   | other-1 (procedure, ddl) | 31:52, 32:13, 
[33:06](https://github.com/apache/hudi/actions/runs/34720098561/job/103624268051)
 | no data (ten failures, see below) | 
[22:30](https://github.com/apache/hudi/actions/runs/34711686507/job/103601474695)
 | not rerun | 
[17:14](https://github.com/apache/hudi/actions/runs/34711686507/job/103611929554)
 | 
[22:34](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5/job/103666455060)
 |
   | run's longest job | 41 to 44, 
[41](https://github.com/apache/hudi/actions/runs/34720098561) | [37, 
scala-other-2](https://github.com/apache/hudi/actions/runs/34708740840) | [36, 
scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507) | not a 
full run | [35, 
scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507/attempts/3)
 (a) | [37, 
scala-other-2](https://github.com/apache/hudi/actions/runs/34711686507/attempts/5)
 |
   
   Serial times are the scalatest wall clocks of #19892's runs 
[34581330915](https://github.com/apache/hudi/actions/runs/34581330915) and 
[34499656524](https://github.com/apache/hudi/actions/runs/34499656524), plus 
#19931's first run, which has the same layout without this stack; every sample 
ran the same test names as those runs.
   
   One more two-thread sample after the four-thread experiment, on the revert 
commit ([run 
34742475247](https://github.com/apache/hudi/actions/runs/34742475247)): dml-1 
[15:25](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400514),
 dml-2 
[16:21](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400544),
 other-1 
[19:43](https://github.com/apache/hudi/actions/runs/34742475247/job/103684400543),
 longest job 37 (ut-fat-and-ftc), all green with the same test names.
   
   Verdict after three or more samples per shard: dml-1 30 to 31 -> 15:25 to 
20:19 (-34% to -50%), dml-2 19 to 23 -> 13:27 to 16:56 (-23%), other-1 32 to 33 
-> 17:14 to 22:34 (-36%), every sample below every serial sample. The run's 
longest job moved from the Scala shards (39 to 44) to the serial scala-other-2 
at 35 to 37, so the next step is that shard.
   
   **Phase 4a, four threads** (commit b8abd450c3c4, reverted by fe4e20376ae5; 
the PR stays at two threads): does the shard still scale on a 4 vCPU runner 
once two threads keep both busy 87 to 92 percent of the time? Same measurement, 
two samples (the second decided it).
   
   | shard | two threads (above) | sample 1 | sample 2 |
   |---|---|---|---|
   | dml-1 (dml.others) | 19:36 to 20:19 | 
[17:20](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440612)
 | 
[13:35](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069789)
 |
   | dml-2 (dml.insert, dml.schema) | 13:27 to 16:56 | 
[13:17](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440592)
 | 
[14:31](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069836)
 |
   | other-1 (procedure, ddl) | 17:14 to 22:34 | 
[20:04](https://github.com/apache/hudi/actions/runs/34737535334/job/103671440550)
 | 
[18:43](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2/job/103676069804)
 |
   | run's longest job | 35 to 37 | [36, 
scala-other-2](https://github.com/apache/hudi/actions/runs/34737535334) | [36, 
three jobs 
tied](https://github.com/apache/hudi/actions/runs/34737535334/attempts/2) |
   
   Four threads keep 3.6 of them busy on dml-1 and dml-2 and 3.0 on other-1, at 
the price of every suite running about 60 percent longer than at two threads 
(four sessions on four vCPUs). dml-1 gains on both samples; dml-2 and other-1 
stay inside their two-thread spread. Test names identical to serial on all six 
shard runs. Decision: stay at two. The gain is on a shard that does not set the 
run's wall clock, and four-way contention on a 4 vCPU runner narrows every 
suite's timing margin; four threads is recorded here as a measured, safe 
setting for a shard that becomes the critical path. other-1's first run ([job 
103593487295](https://github.com/apache/hudi/actions/runs/34708740840/job/103593487295),
 21:53) failed ten tests: a shared-session leak in the parquet reader, fixed in 
the shared-session PR, and a catalog-wide `show tables` assertion, fixed in the 
state-scoping PR. It is not counted; other-1's third sample comes from one more 
rerun after attempt 3. (a) On attempt 3 the Java j
 ob ut-other hung after finishing all its hudi-common test classes (27 to 34 
min on eleven earlier runs); it was cancelled at 76 min and rerun alone as 
attempt 4 in 34 min, which is the figure used. The Scala shards are unaffected. 
Both threads are busy 91%, 92% and 87% of the wall clock on dml-1, dml-2 and 
other-1 (sum of suite spans over wall).
   
   <details>
   <summary>Evidence from the proof of concept (#19925)</summary>
   
   On master's layout the single dml job (dml-1 plus dml-2 together) at two 
threads: 27:58, 35:27 and 35:32 ([run 
34697891947](https://github.com/apache/hudi/actions/runs/34697891947), attempts 
1 to 3) against 41:22 on master's fastest runner and 52 to 56 on its typical 
one, the same 326 test names each time, both threads busy 96 to 98% of the 
time. Locally, twenty iterations of twelve interaction-heavy suites at two 
threads were green, and the whole dml tree at two threads passed in 22 minutes.
   </details>
   
   ### Impact
   
   CI configuration and test infrastructure only.
   
   ### Risk Level
   
   low. Off by default; on three shards whose suites were prepared for it in 
the two PRs below and looped locally at two threads.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to