[
https://issues.apache.org/jira/browse/GROOVY-12332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul King updated GROOVY-12332:
-------------------------------
Description:
h2. Summary
Since 6.0.0-beta-3 the compiler needs materially more heap to compile a single
very large class, enough to
cross Gradle's default worker heap: the same sources that build on beta-3 die
on a current snapshot with
{{java.lang.OutOfMemoryError: Java heap space}} about fifteen seconds in.
The useful part of the report is not the OOM but what does *not* OOM. Compiling
MORE total source succeeds,
provided it is spread over ordinary-sized classes. So this is not a leak
proportional to input size; the cost
tracks the size of the largest single class.
h2. The measurement
One project ([groovy-verify|https://github.com/paulk-asert/groovy-verify]), two
source sets, same compiler,
same 512m ceiling on a forked compiler JVM:
|| source set || total lines || largest class || 6.0.0-SNAPSHOT @ 512m ||
| {{src/test}} | 36,218 | 555 | *compiles* |
| {{src/main}} | 34,898 | *15,580* | *OOM* |
The larger body of source is the one that compiles. {{src/main}}'s
{{src/main/groovy/verification/VerifyChecker.groovy}} is a single 15,580-line
class; nothing in
{{src/test}} exceeds 555 lines.
Across Groovy versions, {{src/main}} unchanged:
|| Groovy || 512m || 640m ||
| 6.0.0-beta-3 | *OK* | OK |
| 6.0.0-SNAPSHOT | *OOM* | OK |
So the requirement moved from under 512m to about 640m for identical input —
modest in itself, but it
crosses the heap a default Gradle Groovy compile gets, which is why it presents
as a hard build failure
rather than as slower compilation.
h2. Reproduction
{code}
git clone https://github.com/paulk-asert/groovy-verify && cd groovy-verify
# with a 6.0.0 snapshot installed to ~/.m2
cat > /tmp/h.init.gradle <<'EOF'
allprojects {
repositories { mavenLocal() }
tasks.withType(GroovyCompile).configureEach {
groovyOptions.fork = true
groovyOptions.forkOptions.memoryMaximumSize = '512m'
}
}
EOF
./gradlew compileGroovy -PgroovyVersion=6.0.0-SNAPSHOT --init-script
/tmp/h.init.gradle # OOM
./gradlew compileGroovy -PgroovyVersion=6.0.0-beta-3 --init-script
/tmp/h.init.gradle # passes
{code}
Raising the ceiling to {{640m}} makes the snapshot pass. Compiling the test
source set at 512m (with the
main set already built, since it is a dependency) also passes on the snapshot.
Environment: JDK 25.0.3-tem, Gradle 9.7.1, macOS. At the default (unforked)
worker heap the failure is
reported as {{java.lang.OutOfMemoryError: Java heap space}} at
{{org.gradle.api.internal.tasks.compile.ApiGroovyCompiler.execute}}; at very
low ceilings it surfaces
instead as a bare {{parsing failed}} with no file or line, which is worth
knowing when triaging.
h2. What it is not
* *Not growth in the project being compiled.* An older commit of the same
project — 34,107 lines of
{{src/main}} against today's 34,898, i.e. 2.3% smaller — also OOMs at 512m on
the snapshot. The smaller,
older tree fails on the new compiler while the larger, current tree passes on
beta-3.
* *Not the parser DFA cache.* {{-Dgroovy.antlr4.cache.size=2000}}
(GROOVY-12318) still OOMs at 512m. That
same setting compiles the project fine at 6g, so the small cache is
functionally sound and the 512m
failure is memory elsewhere.
* *Not GroovyClassValue retention.* Neither {{-Dgroovy.use.classvalue=soft}}
(GROOVY-12281) nor
{{-Dgroovy.use.classvalue=false}} changes the outcome at 512m.
Every system property added since 6.0.0-beta-1 was enumerated and the
memory-related ones tested:
{{groovy.use.classvalue}}, {{groovy.antlr4.cache.size}},
{{groovy.antlr4.cache.threshold}},
{{groovy.errors.tolerance}}, {{groovy.indy.aot.link}},
{{groovy.hidden.classes.disable}},
{{groovy.grape.insecureProtocolPolicy}}. None of them addresses this.
h2. Suspected cause — NOT confirmed
GROOVY-12288 added {{CachingClassWriter}} in {{CompilationUnit}}, which
memoises {{getCommonSuperClass}} per
class:
{code:java}
private final Map<String, ClassNode> classNodeByInternalName = new HashMap<>(8);
private final Map<String, Map<String, String>> commonSuperByPair = new
HashMap<>(8);
{code}
{{commonSuperByPair}} is a map of maps over PAIRS of internal names, so it
grows with the distinct type
pairs met while writing one class, has no ceiling and no eviction, and lives
for the whole of that class's
generation. That shape — cost per class rather than per compilation unit,
superlinear in the types inside a
single class — is what the measurement above shows, and the commit post-dates
beta-3. It also fits the
absence of a knob: GROOVY-12288 introduced no property to bound or disable the
memo.
I have not confirmed this. Doing so needs a build of master with that commit
reverted, compared at 512m;
I did not want to overwrite the snapshot in the local repository to test it. If
the theory holds, a bound on
the memo (or clearing it per class rather than letting it live to the end of
generation) should restore the
earlier ceiling while keeping the speed-up the caching was added for.
h2. Impact and workaround
A project is affected in proportion to its largest class, not its size, so it
will look arbitrary from the
outside: most builds are unaffected and a few fail outright at the default
heap. The workaround is to fork
the Groovy compiler with a raised ceiling:
{code:groovy}
tasks.withType(GroovyCompile).configureEach {
groovyOptions.fork = true
groovyOptions.forkOptions.memoryMaximumSize = '2g'
}
{code}
Worth resolving before the next release, since the failure mode gives no hint
that class size is the
variable.
> A single large class OOMs at the default compiler heap since 6.0.0-beta-3
> -------------------------------------------------------------------------
>
> Key: GROOVY-12332
> URL: https://issues.apache.org/jira/browse/GROOVY-12332
> Project: Groovy
> Issue Type: Bug
> Reporter: Paul King
> Priority: Major
>
> h2. Summary
> Since 6.0.0-beta-3 the compiler needs materially more heap to compile a
> single very large class, enough to
> cross Gradle's default worker heap: the same sources that build on beta-3 die
> on a current snapshot with
> {{java.lang.OutOfMemoryError: Java heap space}} about fifteen seconds in.
> The useful part of the report is not the OOM but what does *not* OOM.
> Compiling MORE total source succeeds,
> provided it is spread over ordinary-sized classes. So this is not a leak
> proportional to input size; the cost
> tracks the size of the largest single class.
> h2. The measurement
> One project ([groovy-verify|https://github.com/paulk-asert/groovy-verify]),
> two source sets, same compiler,
> same 512m ceiling on a forked compiler JVM:
> || source set || total lines || largest class || 6.0.0-SNAPSHOT @ 512m ||
> | {{src/test}} | 36,218 | 555 | *compiles* |
> | {{src/main}} | 34,898 | *15,580* | *OOM* |
> The larger body of source is the one that compiles. {{src/main}}'s
> {{src/main/groovy/verification/VerifyChecker.groovy}} is a single 15,580-line
> class; nothing in
> {{src/test}} exceeds 555 lines.
> Across Groovy versions, {{src/main}} unchanged:
> || Groovy || 512m || 640m ||
> | 6.0.0-beta-3 | *OK* | OK |
> | 6.0.0-SNAPSHOT | *OOM* | OK |
> So the requirement moved from under 512m to about 640m for identical input —
> modest in itself, but it
> crosses the heap a default Gradle Groovy compile gets, which is why it
> presents as a hard build failure
> rather than as slower compilation.
> h2. Reproduction
> {code}
> git clone https://github.com/paulk-asert/groovy-verify && cd groovy-verify
> # with a 6.0.0 snapshot installed to ~/.m2
> cat > /tmp/h.init.gradle <<'EOF'
> allprojects {
> repositories { mavenLocal() }
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '512m'
> }
> }
> EOF
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-SNAPSHOT --init-script
> /tmp/h.init.gradle # OOM
> ./gradlew compileGroovy -PgroovyVersion=6.0.0-beta-3 --init-script
> /tmp/h.init.gradle # passes
> {code}
> Raising the ceiling to {{640m}} makes the snapshot pass. Compiling the test
> source set at 512m (with the
> main set already built, since it is a dependency) also passes on the snapshot.
> Environment: JDK 25.0.3-tem, Gradle 9.7.1, macOS. At the default (unforked)
> worker heap the failure is
> reported as {{java.lang.OutOfMemoryError: Java heap space}} at
> {{org.gradle.api.internal.tasks.compile.ApiGroovyCompiler.execute}}; at very
> low ceilings it surfaces
> instead as a bare {{parsing failed}} with no file or line, which is worth
> knowing when triaging.
> h2. What it is not
> * *Not growth in the project being compiled.* An older commit of the same
> project — 34,107 lines of
> {{src/main}} against today's 34,898, i.e. 2.3% smaller — also OOMs at 512m
> on the snapshot. The smaller,
> older tree fails on the new compiler while the larger, current tree passes
> on beta-3.
> * *Not the parser DFA cache.* {{-Dgroovy.antlr4.cache.size=2000}}
> (GROOVY-12318) still OOMs at 512m. That
> same setting compiles the project fine at 6g, so the small cache is
> functionally sound and the 512m
> failure is memory elsewhere.
> * *Not GroovyClassValue retention.* Neither {{-Dgroovy.use.classvalue=soft}}
> (GROOVY-12281) nor
> {{-Dgroovy.use.classvalue=false}} changes the outcome at 512m.
> Every system property added since 6.0.0-beta-1 was enumerated and the
> memory-related ones tested:
> {{groovy.use.classvalue}}, {{groovy.antlr4.cache.size}},
> {{groovy.antlr4.cache.threshold}},
> {{groovy.errors.tolerance}}, {{groovy.indy.aot.link}},
> {{groovy.hidden.classes.disable}},
> {{groovy.grape.insecureProtocolPolicy}}. None of them addresses this.
> h2. Suspected cause — NOT confirmed
> GROOVY-12288 added {{CachingClassWriter}} in {{CompilationUnit}}, which
> memoises {{getCommonSuperClass}} per
> class:
> {code:java}
> private final Map<String, ClassNode> classNodeByInternalName = new
> HashMap<>(8);
> private final Map<String, Map<String, String>> commonSuperByPair = new
> HashMap<>(8);
> {code}
> {{commonSuperByPair}} is a map of maps over PAIRS of internal names, so it
> grows with the distinct type
> pairs met while writing one class, has no ceiling and no eviction, and lives
> for the whole of that class's
> generation. That shape — cost per class rather than per compilation unit,
> superlinear in the types inside a
> single class — is what the measurement above shows, and the commit post-dates
> beta-3. It also fits the
> absence of a knob: GROOVY-12288 introduced no property to bound or disable
> the memo.
> I have not confirmed this. Doing so needs a build of master with that commit
> reverted, compared at 512m;
> I did not want to overwrite the snapshot in the local repository to test it.
> If the theory holds, a bound on
> the memo (or clearing it per class rather than letting it live to the end of
> generation) should restore the
> earlier ceiling while keeping the speed-up the caching was added for.
> h2. Impact and workaround
> A project is affected in proportion to its largest class, not its size, so it
> will look arbitrary from the
> outside: most builds are unaffected and a few fail outright at the default
> heap. The workaround is to fork
> the Groovy compiler with a raised ceiling:
> {code:groovy}
> tasks.withType(GroovyCompile).configureEach {
> groovyOptions.fork = true
> groovyOptions.forkOptions.memoryMaximumSize = '2g'
> }
> {code}
> Worth resolving before the next release, since the failure mode gives no hint
> that class size is the
> variable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
