jamesfredley opened a new pull request, #16158: URL: https://github.com/apache/grails-core/pull/16158
<!-- markdownlint-disable MD033 MD041 --> **Draft** — opened as a draft so CI measures the change before review. See [Measuring this on CI](#measuring-this-on-ci). ## The problem A plain `./gradlew build` can leave a developer's workstation unusable. Observed on a 14-core / 20-thread machine: **23 JVMs, 12.4 GB**, with the desktop unresponsive. Three multipliers stack, and each looks reasonable on its own. **1. `availableProcessors()` counts logical processors.** SMT threads on x64, and on Apple silicon every efficiency core as well as every performance core. `* 3 / 4` of that is already measured against an inflated number — 15 forks on a 14-core host, and on an M-series Mac most of those forks land on efficiency cores. **2. `maxParallelForks` is per Test task.** With `org.gradle.parallel=true`, several test-bearing modules run concurrently, so the real ceiling is Gradle's worker-lease pool (`--max-workers`, default = `availableProcessors`), not any single task's fork count. **3. Each forked JVM sizes its own thread pools for the whole machine**, because no fork knows the others exist. Measured on JDK 21 at 20 visible processors: | Setting | Value | |---|---:| | `ParallelGCThreads` | 15 | | `ConcGCThreads` | 4 | | `CICompilerCount` | 12 | | **Total per fork** | **31** | That is 31 threads per fork *before a single test runs*. Gradle cannot see it: it charges each fork **one worker lease**, as though a fork were a single thread. Gradle's own default for `Test.maxParallelForks` is **1** for precisely this reason — raising it opts out of the protection Gradle already provides. 15 forks × 31 threads ≈ **465 threads contending for 20 hardware threads**, plus 15 × 1 GB heaps (2 GB in `grails-test-suite-persistence`) against a 5 GB daemon. That is the thrashing. ## The change Applied to all three builds in this repository — root, `grails-gradle` and `grails-forge` each have their own `settings.gradle`, so the duplication is unavoidable. 1. **Local test forks: 3/4 of the logical processors → half.** CI keeps its existing budget (`isCiBuild ? 4`, and `? 3` in `grails-gradle`), so **CI fork counts are unchanged**. 2. **Every test fork is told how many processors it may size its pools from**, as `availableProcessors / maxWorkerCount`, floored at 2. The denominator is the **build-wide** worker limit rather than any one task's `maxParallelForks`, because that is what actually bounds how many forks run at once. A floor of 2 keeps G1 rather than dropping to Serial GC. Supplied via `jvmArgumentProviders` rather than `jvmArgs`, because several modules assign `jvmArgs` wholesale and would discard it. Also fixes `grails-forge`, where `-PmaxTestParallel` was silently ignored. ## Measured result `:grails-core:test --rerun-tasks`, daemons stopped between runs, comparing second runs: | Order | Baseline | This branch | Change | |---|---:|---:|---:| | baseline first | 3m 25s | 2m 45s | **19.5% faster** | | treatment first | 3m 05s | 2m 24s | **22.2% faster** | Running this branch **first** rules out filesystem-cache ordering bias — the advantage held in both directions. | | Baseline | This branch | |---|---:|---:| | Forks | 15 | 10 | | `ActiveProcessorCount` | 20 (default) | 2 | | Threads per fork | 31 | 5 | | Projected total threads | ~465 | ~50 | | Peak JVMs | 29 | 24 | Thread figures are a computed projection from `java -XX:+PrintFlagsFinal -version`, not a live thread count. ## Second commit: `isolatedTestsTwo` was not actually isolated `grails-test-suite-uber/build.gradle` set `maxParallelForks = 1` on `isolatedTestsTwo`, but registered that block **before** the `tasks.withType(Test).configureEach` block in the same script. Gradle runs both as deferred configuration actions in registration order, so the later `configureEach` overwrote it and the task ran fully parallel. Its test patterns have been deliberately serialized since 2013 because they are order sensitive. Moving the override after the `configureEach` block fixes it. CI impact is negligible: the task filters three classes and sharding assigns it to a single shard. ## Measuring this on CI Baseline for `gradle.yml` on `8.0.x`: **median 2h 14m, p90 3h 52m**, with core build jobs at 1h 20m – 1h 40m. Draft PRs do trigger the workflow, so this PR's own run is the measurement. Since CI fork counts are unchanged, any CI movement comes from the per-fork processor cap alone. ## Follow-ups, deliberately not in this PR - **A memory budget.** An earlier revision budgeted forks against physical RAM, but a per-task budget cannot bind build-wide, so it was removed rather than shipped as a guarantee it could not keep. Bounding aggregate memory needs `org.gradle.workers.max` or a shared `BuildService`. - **A 5 GB Gradle daemon on a 7 GB macOS runner** (GitHub's macOS runners are 3 vCPU / 7 GB, versus 4 vCPU / 16 GB on Linux and Windows) is already marginal and worth revisiting separately. - **Testcontainers multiply with forks.** The Mongo, Redis and forked Geb suites hold containers in per-JVM statics with no `withReuse`, so N forks means N containers — and on macOS and Windows that memory comes from a Docker Desktop VM the JVM cannot see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
