This is an automated email from the ASF dual-hosted git repository. jamesfredley pushed a commit to branch fix/test-fork-oversubscription in repository https://gitbox.apache.org/repos/asf/grails-core.git
commit 13c964143fa3211d5df8b2787b0bdd159afbd7ce Author: James Fredley <[email protected]> AuthorDate: Sun Aug 16 01:01:00 2026 -0400 build(test): stop test forks oversubscribing the machine A plain `./gradlew build` could leave a developer's workstation unusable. Three multipliers stacked, each reasonable on its own: 1. `availableProcessors()` reports LOGICAL processors - SMT threads on x64, and on Apple silicon every efficiency core as well as every performance core. Taking 3/4 of that already overstates real capacity. 2. `maxParallelForks` is per Test task, and with `org.gradle.parallel=true` several test-bearing modules run at once, so the real ceiling is the worker-lease pool rather than any single task's fork count. 3. Every forked JVM sizes its own GC and JIT thread pools for the WHOLE machine, because no fork knows the others exist. On a 20-processor host that is 15 ParallelGCThreads + 4 ConcGCThreads + 12 CICompilerCount = 31 threads per fork before a single test runs. Gradle cannot see the third one: it charges each fork a single worker lease, as though a fork were one thread. Gradle's own default for maxParallelForks is 1 for exactly that reason; raising it opts out of that protection. Two changes, applied to all three builds in this repository (root, grails-gradle and grails-forge each have their own settings.gradle): - Local test forks drop from 3/4 of the logical processors to half. CI keeps its existing budget, so CI fork counts are unchanged. - Every test fork is told how many processors it may size its thread pools from, as availableProcessors / maxWorkerCount. The denominator is the BUILD-WIDE worker limit rather than any one task's maxParallelForks, because that is what actually bounds how many forks run concurrently. A floor of 2 keeps G1 rather than dropping to Serial GC. It is supplied through jvmArgumentProviders rather than jvmArgs because several modules assign jvmArgs wholesale, which would discard it. Measured on a 14-core/20-thread host with `:grails-core:test --rerun-tasks`, daemons stopped between runs, comparing second runs: baseline first: baseline 3m25s, treatment 2m45s (19.5% faster) treatment first: treatment 2m24s, baseline 3m05s (22.2% faster) Running the treatment first rules out filesystem-cache ordering bias. Peak JVM count fell from 29 to 24; projected per-fork JVM threads fell from 31 to 5, so projected total threads fell from roughly 465 to 50. `-PmaxTestParallel` still overrides the default, and now does so in grails-forge as well, where it was previously ignored. Assisted-by: claude-code:claude-opus-5 --- build.gradle | 49 +++++++++++++++++++++++++++++++++- grails-forge/build.gradle | 22 +++++++++++++-- grails-forge/gradle/test-config.gradle | 4 ++- grails-gradle/build.gradle | 28 ++++++++++++++++++- 4 files changed, 98 insertions(+), 5 deletions(-) diff --git a/build.gradle b/build.gradle index ce00bbd208..276d1f6e79 100644 --- a/build.gradle +++ b/build.gradle @@ -47,7 +47,14 @@ ext { // needing --rerun-tasks. Useful for repeatedly running the same test command while // chasing flaky tests across runs. doNotCacheTests = System.getenv('DO_NOT_CACHE_TESTS')?.toBoolean() - configuredTestParallel = findProperty('maxTestParallel') as Integer ?: (isCiBuild ? 4 : Runtime.runtime.availableProcessors() * 3 / 4 as int ?: 1) + // availableProcessors() reports LOGICAL processors: SMT threads on x64, and on Apple silicon + // every efficiency core as well as every performance core. Taking 3/4 of that therefore + // asked for more concurrent Grails/Spring test JVMs than the machine has real capacity for + // (each fork also holds a 1G heap, and the mongodb, redis and geb suites start a Docker + // container per fork). Half is the conservative fraction. CI keeps its existing budget, so + // this does not reduce CI fork counts - though CI forks do see fewer processors, below. + configuredTestParallel = findProperty('maxTestParallel') as Integer ?: + (isCiBuild ? 4 : Math.max(1, (Runtime.runtime.availableProcessors() / 2) as int)) excludeUnusedTransDeps = findProperty('excludeUnusedTransDeps') testProjectsStartWith = [ @@ -60,8 +67,48 @@ ext { profileProjects = [ /* Will be populated by subprojects loop below */] } +/** + * Tells a forked test JVM how many processors it may size its GC and JIT thread pools from. + * + * Every forked JVM otherwise sizes those pools for the WHOLE machine, because no fork knows the + * others exist. On a 20-processor host that is 15 ParallelGCThreads + 4 ConcGCThreads + 12 + * CICompilerCount = 31 threads per fork before a single test runs, so N concurrent forks become + * N*31 threads competing for the same cores. Gradle cannot see this: it charges each fork one + * worker lease, as though a fork were a single thread. + * + * The share is availableProcessors / maxWorkerCount, i.e. the BUILD-WIDE worker limit, because + * that - not any one task's maxParallelForks - is what bounds how many forks can run at once. + * With org.gradle.parallel=true, Test tasks from several projects execute concurrently, so a + * per-task share would still oversubscribe the machine. + * + * Supplied as an argument provider rather than through jvmArgs because several modules assign + * jvmArgs wholesale in their own build scripts, which would discard anything added here first. + */ +final class ActiveProcessorCountArgumentProvider implements CommandLineArgumentProvider { + + @Input + final int processorCount + + ActiveProcessorCountArgumentProvider(int availableProcessorCount, int maxWorkerCount) { + // Floor of 2: at ActiveProcessorCount=1 HotSpot's ergonomics select Serial GC, which + // costs more on a 1G test heap than the contention it saves. Two keeps G1 with a + // bounded worker count. + this.processorCount = Math.max(2, (availableProcessorCount / Math.max(1, maxWorkerCount)) as int) + } + + @Override + Iterable<String> asArguments() { + ["-XX:ActiveProcessorCount=$processorCount".toString()] + } +} + subprojects { + tasks.withType(Test).configureEach { testTask -> + testTask.jvmArgumentProviders.add(new ActiveProcessorCountArgumentProvider( + Runtime.runtime.availableProcessors(), gradle.startParameter.maxWorkerCount)) + } + for (String testPrefix : testProjectsStartWith) { if (name.startsWith(testPrefix)) { rootProject.ext['testProjects'] << name diff --git a/grails-forge/build.gradle b/grails-forge/build.gradle index d262f97fab..4ebe4a4f22 100644 --- a/grails-forge/build.gradle +++ b/grails-forge/build.gradle @@ -44,8 +44,24 @@ ext { doNotCacheTests = System.getenv('DO_NOT_CACHE_TESTS')?.toBoolean() } +/** Caps each test fork's view of the machine. See the root build.gradle for the rationale. */ +final class ActiveProcessorCountArgumentProvider implements CommandLineArgumentProvider { + + @Input + final int processorCount + + ActiveProcessorCountArgumentProvider(int availableProcessorCount, int maxWorkerCount) { + this.processorCount = Math.max(2, (availableProcessorCount / Math.max(1, maxWorkerCount)) as int) + } + + @Override + Iterable<String> asArguments() { + ["-XX:ActiveProcessorCount=$processorCount".toString()] + } +} + allprojects { - tasks.withType(Test).configureEach { Task testTask -> + tasks.withType(Test).configureEach { testTask -> testTask.dependsOn( gradle.includedBuild('grails-core').task(':publishAllPublicationsToTestCaseMavenRepoRepository'), gradle.includedBuild('grails-gradle').task(':publishAllPublicationsToTestCaseMavenRepoRepository') @@ -55,6 +71,8 @@ allprojects { // without --rerun-tasks (and without recompiling everything else). testTask.outputs.cacheIf { !doNotCacheTests } testTask.outputs.upToDateWhen { !doNotCacheTests } + testTask.jvmArgumentProviders.add(new ActiveProcessorCountArgumentProvider( + Runtime.runtime.availableProcessors(), gradle.startParameter.maxWorkerCount)) } } @@ -72,4 +90,4 @@ apply { // we must apply the publish configuration first or the docs config will not work from layout.projectDirectory.file('gradle/publish-root-config.gradle') from layout.projectDirectory.file('gradle/gradle-wrapper-root-config.gradle') -} \ No newline at end of file +} diff --git a/grails-forge/gradle/test-config.gradle b/grails-forge/gradle/test-config.gradle index 9d4d27ddf9..cf83eab2fc 100644 --- a/grails-forge/gradle/test-config.gradle +++ b/grails-forge/gradle/test-config.gradle @@ -49,7 +49,9 @@ tasks.withType(Test).configureEach { jvmArgs('-Duser.country=US', '-Duser.language=en', '--add-opens', 'java.base/java.lang=ALL-UNNAMED') forkEvery = 100 - maxParallelForks = Runtime.runtime.availableProcessors().intdiv(2) ?: 1 + // Honour -PmaxTestParallel here too, so the override behaves the same in all three builds. + maxParallelForks = findProperty('maxTestParallel') as Integer ?: + (Runtime.runtime.availableProcessors().intdiv(2) ?: 1) maxHeapSize = '2G' useJUnitPlatform() diff --git a/grails-gradle/build.gradle b/grails-gradle/build.gradle index 382c977102..48d0dec17a 100644 --- a/grails-gradle/build.gradle +++ b/grails-gradle/build.gradle @@ -47,7 +47,33 @@ ext { // needing --rerun-tasks. Useful for repeatedly running the same test command while // chasing flaky tests across runs. doNotCacheTests = System.getenv('DO_NOT_CACHE_TESTS')?.toBoolean() - configuredTestParallel = findProperty('maxTestParallel') as Integer ?: (isCiBuild ? 3 : Runtime.runtime.availableProcessors() * 3/4 as int ?: 1) + // Half the LOGICAL processors, not 3/4. This separate Gradle build mirrors the root + // build.gradle - see it for the full rationale. CI keeps its existing budget. + configuredTestParallel = findProperty('maxTestParallel') as Integer ?: + (isCiBuild ? 3 : Math.max(1, (Runtime.runtime.availableProcessors() / 2) as int)) +} + +/** Caps each test fork's view of the machine. See the root build.gradle for the rationale. */ +final class ActiveProcessorCountArgumentProvider implements CommandLineArgumentProvider { + + @Input + final int processorCount + + ActiveProcessorCountArgumentProvider(int availableProcessorCount, int maxWorkerCount) { + this.processorCount = Math.max(2, (availableProcessorCount / Math.max(1, maxWorkerCount)) as int) + } + + @Override + Iterable<String> asArguments() { + ["-XX:ActiveProcessorCount=$processorCount".toString()] + } +} + +subprojects { + tasks.withType(Test).configureEach { testTask -> + testTask.jvmArgumentProviders.add(new ActiveProcessorCountArgumentProvider( + Runtime.runtime.availableProcessors(), gradle.startParameter.maxWorkerCount)) + } } apply {
