This is an automated email from the ASF dual-hosted git repository. jamesfredley pushed a commit to branch build/test-fork-heap-budget in repository https://gitbox.apache.org/repos/asf/grails-core.git
commit dbe25519d14018837ac50d370b0dccdb83ef6f86 Author: James Fredley <[email protected]> AuthorDate: Tue Aug 18 17:07:15 2026 -0400 build(test): size the test fork heap from an explicit budget gradle/test-config.gradle hardcoded maxHeapSize to 768m on CI and 1024m locally. Those literals ignore the machine: every forked test JVM claimed a gigabyte regardless of how much memory the developer actually had, and nothing tied that number to how many forks could be alive at once. Forked test JVMs are children of the daemon, so org.gradle.jvmargs does not limit them. With org.gradle.parallel=true several Test tasks run concurrently, so the live fork count is bounded by the build-wide worker pool, not by any one task's maxParallelForks - the same bound ActiveProcessorCountArgumentProvider already uses for processor count a few lines below. Derive the per-fork heap from that bound: physical memory, minus the daemon's own max heap, halved to leave room for the OS, the compiler workers and the containers the mongodb, redis and geb suites start per fork, divided by the worker count, then clamped to [768, 1024]. The floor is 768 rather than something smaller because GrailsGradlePlugin gives every Grails test task minHeapSize = 768m when nothing else sets one, so a smaller maximum would produce -Xms768m -Xmx<less> and the JVM would refuse to start. Setting minHeapSize here instead was rejected: it would newly pin committed heap on plain Test tasks that currently leave it unset. For the same reason an explicit -PtestForkHeapMb below 768 now fails fast with a message pointing at -PmaxTestParallel and --max-workers, rather than being silently clamped to a value the caller did not ask for. The daemon heap is read from Runtime.maxMemory() rather than parsed out of org.gradle.jvmargs, since this script runs in the daemon; that sidesteps multiple -Xmx options and unit parsing entirely. CI keeps its measured 768m exactly, so this commit is CI-neutral by construction; the local overcommit is the only behaviour that changes. Both the override and isCiBuild settle the value before the memory probe runs, so the CI path does no fallible work. grails-test-suite-uber carried its own copy of the same literal and never applied the shared test configuration, which would have left the heaviest suite in the build as the one module exempt from the budget, so it reads the same property now. The mongodb configurations assign jvmArgs wholesale with their own -Xmx and grails-test-suite-persistence sets a deliberate 2048m; those keep their values and the boundary is documented where the budget is applied, because folding them in would change CI heaps. Reading physical memory degrades safely: a failed probe keeps the previous 1024m, while a successful probe on a machine whose RAM sits inside the daemon heap drops to the floor instead of pretending the probe failed. grails-gradle and grails-forge are separate Gradle builds and are left for a follow-up; porting this rule to grails-forge would move its CI forks from 2G to the budget, which is a CI change this commit deliberately avoids. Assisted-by: claude-code:claude-opus-5 --- build.gradle | 71 +++++++++++++++++++++++++++++++++++++ gradle/test-config.gradle | 11 +++++- grails-test-suite-uber/build.gradle | 6 +++- 3 files changed, 86 insertions(+), 2 deletions(-) diff --git a/build.gradle b/build.gradle index 61f25361f5..f69e736f25 100644 --- a/build.gradle +++ b/build.gradle @@ -55,6 +55,77 @@ ext { // this does not reduce CI fork counts - though CI forks do see fewer processors, below. configuredTestParallel = findProperty('maxTestParallel') as Integer ?: (isCiBuild ? 4 : Math.max(1, (Runtime.runtime.availableProcessors() / 2) as int)) + // Forked test JVMs are children of the daemon, so org.gradle.jvmargs does not limit their + // heaps - each one claimed a flat 1G locally no matter how much memory the machine had. + // Size them from a budget instead: physical memory, less the daemon's own heap, halved to + // leave room for the OS, the compiler workers and the Docker containers the mongodb, redis + // and geb suites start per fork, then split across the forks that can be alive at once. + // + // An explicit -PtestForkHeapMb wins everywhere, mirroring -PmaxTestParallel above, and CI + // keeps its measured 768m. Both of those settle the value WITHOUT running the probe below, + // so the CI path stays trivially neutral and does no fallible work. + def resolvedTestForkHeapMb = findProperty('testForkHeapMb') as Integer + if (resolvedTestForkHeapMb != null && resolvedTestForkHeapMb < 768) { + // Reject rather than silently clamp: the property promises an exact value, so quietly + // substituting a different one would be worse than refusing. Below 768 the forks that + // GrailsGradlePlugin gives minHeapSize = 768m would start as -Xms768m -Xmx<less>, which + // the JVM rejects outright - a confusing failure at test time instead of here. + throw new GradleException( + "-PtestForkHeapMb must be at least 768 (got ${resolvedTestForkHeapMb}). Grails test " + + 'tasks are given minHeapSize = 768m, so a smaller maximum produces an ' + + 'unstartable JVM. To use less memory overall, lower the fork count with ' + + '-PmaxTestParallel or --max-workers instead.') + } + if (resolvedTestForkHeapMb == null && isCiBuild) { + resolvedTestForkHeapMb = 768 + } + if (resolvedTestForkHeapMb == null) { + // This script runs IN the daemon, so the daemon's own max heap is simply its runtime + // value. Reading it beats parsing org.gradle.jvmargs, which may carry several -Xmx + // options (the JVM honours the LAST) or a unit a regex would miss. + long daemonMaxHeapMb = (Runtime.runtime.maxMemory() / (1024L * 1024L)) as long + long totalPhysicalMemoryMb = 0L + try { + def operatingSystemMxBean = java.lang.management.ManagementFactory.operatingSystemMXBean + if (operatingSystemMxBean instanceof com.sun.management.OperatingSystemMXBean) { + totalPhysicalMemoryMb = + ((operatingSystemMxBean as com.sun.management.OperatingSystemMXBean).totalMemorySize / (1024L * 1024L)) as long + } + } catch (Exception | LinkageError ignored) { + // A non-HotSpot JVM, a restricted management bean, or a module-access denial (which + // surfaces as an Error, not an Exception) leaves the previous fixed default below. + } + // Budget against the BUILD-WIDE worker count, never a per-task fork cap. With + // org.gradle.parallel=true several Test tasks each start forks until the shared worker + // pool is full, so a build with four tasks capped at two forks apiece still runs up to + // maxWorkerCount forks at once. Budgeting off the smaller number would hand every fork + // a heap the machine cannot actually honour - the same bound, and the same reasoning, + // as ActiveProcessorCountArgumentProvider below. + int concurrentTestForks = Math.max(1, gradle.startParameter.maxWorkerCount) + if (totalPhysicalMemoryMb <= 0L) { + // The probe did not run or told us nothing, so there is no budget to compute - + // keep the value this build used before rather than guess from a missing number. + resolvedTestForkHeapMb = 1024 + } else { + // A machine whose whole RAM is inside the daemon's own heap yields a non-positive + // remainder. That is a real reading, not a failed probe, so it must NOT fall back + // to the old 1G: give those forks the floor instead. + long forkBudgetMb = Math.max(0L, totalPhysicalMemoryMb - daemonMaxHeapMb) / 2L + // The floor is 768m, not something smaller, for two reasons. It is the value CI + // already runs, so it is known to carry these suites. And GrailsGradlePlugin gives + // every Grails Test task minHeapSize = 768m when nothing else has set one, so a + // smaller maximum here would hand those forks -Xms768m -Xmx<less>, which the JVM + // rejects outright before a single test runs. + // + // That floor wins whenever the computed share falls below it, so on a host with + // many workers and little memory the forks together still exceed this budget. Heap + // alone cannot fix that; the knob that has to come down there is the FORK COUNT + // (-PmaxTestParallel, or --max-workers which bounds them build-wide). This budget + // lowers memory pressure, it does not by itself prove a constrained machine fits. + resolvedTestForkHeapMb = Math.min(1024, Math.max(768, (forkBudgetMb / concurrentTestForks) as int)) + } + } + testForkHeapMb = resolvedTestForkHeapMb excludeUnusedTransDeps = findProperty('excludeUnusedTransDeps') testProjectsStartWith = [ diff --git a/gradle/test-config.gradle b/gradle/test-config.gradle index d6ea685c67..daadf9c7a8 100644 --- a/gradle/test-config.gradle +++ b/gradle/test-config.gradle @@ -84,7 +84,16 @@ tasks.withType(Test).configureEach { } excludes = ['**/*TestCase.class', '**/*$*.class'] maxParallelForks = rootProject.ext.configuredTestParallel - maxHeapSize = isCiBuild ? '768m' : '1024m' + // Test forks are child JVMs, not the Gradle daemon. The root build sizes their explicit + // per-fork memory budget from the worker-pool bound shared by parallel Test tasks. + // + // Not every Test task in this build honours it. gradle/mongodb-test-config.gradle and + // gradle/mongodb-forked-test-config.gradle ASSIGN jvmArgs wholesale with their own -Xmx, + // which beats maxHeapSize outright, and grails-test-suite-persistence sets a deliberate + // 2048m. Those are left exactly as they are here on purpose: folding them in would change + // CI heaps, and this change is deliberately CI-neutral. Unifying them is a follow-up that + // needs its own measurement. + maxHeapSize = "${rootProject.ext.testForkHeapMb}m" forkEvery = hasProperty('forkEveryUnitTest') ? getProperty('forkEveryUnitTest') as long : (isCiBuild ? 50 : 100) if (System.getProperty('debug.tests')) { jvmArgs += debugArguments diff --git a/grails-test-suite-uber/build.gradle b/grails-test-suite-uber/build.gradle index 74369b5e5b..05ab2ff2a6 100644 --- a/grails-test-suite-uber/build.gradle +++ b/grails-test-suite-uber/build.gradle @@ -132,7 +132,11 @@ tasks.withType(Test).configureEach { useJUnitPlatform() maxParallelForks = rootProject.ext.configuredTestParallel forkEvery = isCiBuild ? 50 : 100 - maxHeapSize = isCiBuild ? '768m' : '1024m' + // Same per-fork budget as the tasks using the shared test configuration (see the root + // build.gradle; the mongodb configs and persistence keep documented exceptions). This suite + // carried its own copy of the old 768m/1024m literal, which would have exempted the + // heaviest suite in the build from the budget meant to keep forks within the machine. + maxHeapSize = "${rootProject.ext.testForkHeapMb}m" jvmArgs('--add-opens=java.base/java.lang=ALL-UNNAMED', '--add-opens=java.base/java.util=ALL-UNNAMED') }
