[
https://issues.apache.org/jira/browse/GROOVY-12227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul King updated GROOVY-12227:
-------------------------------
Description:
Packed closures (GROOVY-12151, GEP-27) do not work under GraalVM native image.
A class compiled with {{groovy.target.closure.pack=true}} links its dispatch
tables on first closure creation, and that linkage fails inside a native image
— in two layers.
Without agent-recorded metadata, the bootstrap's method lookup fails:
{noformat}
java.lang.BootstrapMethodError: java.lang.NoSuchMethodException:
no such method: M.$packedDispatch$(int,Object[])Object/invokeStatic
{noformat}
With agent-recorded metadata the lookup succeeds, and the real wall appears —
the bootstrap spins hidden classes at run time, which a native image forbids:
{noformat}
Exception in thread "main" java.lang.BootstrapMethodError:
java.lang.InternalError:
com.oracle.svm.core.jdk.UnsupportedFeatureError: Classes cannot be defined at
runtime
by default when using ahead-of-time Native Image compilation. Tried to define
class:
M$$LambdadLbm0tKUrZ8kVMiSpulnyK
{noformat}
This matters because packing is otherwise a natural fit for native images: it
collapses per-closure inner classes into the hosting class (a 3-to-1 class
reduction in the repro below), which directly reduces image size and
reachability metadata, and its call site is a {{ConstantCallSite}} — no runtime
re-linking, unlike the mutable-call-site dispatch of regular indy.
h2. Reproducing
{code:java}
import groovy.transform.CompileStatic
import java.util.function.BiFunction
@CompileStatic
class M {
static void main(String[] args) {
BiFunction<Integer, Integer, Integer> lam = (Integer a, Integer b) -> a
+ b
BiFunction<Integer, Integer, Integer> coe = { Integer a, Integer b -> a
* b } as BiFunction<Integer, Integer, Integer>
var doubled = [1, 2, 3].collect { int n -> n * 2 }
println "lambda: ${lam.apply(20, 22)}"
println "coerced: ${coe.apply(6, 7)}"
println "collect: ${doubled}"
}
}
{code}
{noformat}
JAVA_OPTS="-Dgroovy.target.indy=false -Dgroovy.target.lambda.hoist=true
-Dgroovy.target.closure.pack=true" \
groovyc -cp groovy-callsite.jar M.groovy # emits a single class:
M.class
java -agentlib:native-image-agent=config-output-dir=cfg -cp
"groovy.jar:groovy-callsite.jar:." M
native-image --no-fallback -H:ConfigurationFileDirectories=cfg \
-H:IncludeResources='META-INF/dgminfo|META-INF/groovy/.*|META-INF/services/.*' \
-cp "groovy.jar:groovy-callsite.jar:." M
./m # fails as above
{noformat}
Reproduced with GraalVM CE 25.2.4 (native-image 25.0.4, the first release whose
Groovy substitution guard is fixed for Groovy 5+ — see oracle/graal#13096).
h2. Root cause
{{GeneratedDispatcher.bootstrap}} does two things a native image cannot support:
# {{caller.findStatic(host, "$packedDispatch$", ...)}} — a *runtime*
method-handle lookup, which under native image requires per-user-class
reflection metadata (unknowable to Groovy's own jar, so every user would need
an agent run).
# Three programmatic
{{LambdaMetafactory.metafactory(...).getTarget().invokeExact()}} calls — each
defines a hidden class at run time. Bytecode-level LMF invokedynamic is
pre-processed at image build time and works fine; *programmatic* metafactory
calls inside a bootstrap are runtime class definition, which is forbidden.
The surrounding design is already image-friendly: the result is wrapped in a
{{ConstantCallSite}} and never re-linked.
h2. Proposed fix (prototyped)
Two layered changes; JVM semantics unchanged.
*1. Constant bootstrap arguments (ClosureWriter).* Emit the three dispatch
tables as {{CONSTANT_MethodHandle}} bootstrap arguments, exactly as javac does
for LMF's {{implMethod}}:
{noformat}
invokedynamic packedDispatchers()Object
bsm = IndyInterface.packedDispatchers(Lookup, String, MethodType,
MethodHandle, MethodHandle,
MethodHandle)
args = [ MH(invokestatic host.$packedDispatch$), MH(...1$), MH(...2$) ]
{noformat}
Constant-pool method handles are resolved by the VM (and pre-resolved at image
build time), so linking needs no {{findStatic}} and no metadata. Verified: the
tracing agent records *zero* {{packedDispatch}} entries for the new bytecode.
The existing 3-arg bootstrap remains for class files emitted by earlier 6.0
snapshots.
*2. Method-handle bundle fallback (GeneratedDispatcher).* On a regular JVM the
{{LambdaMetafactory}} hidden-class adapters are kept — their interface call
inlines under the JIT, which is the documented rationale. Where classes cannot
be defined at run time, the tables are wrapped in method-handle-invoking
adapters instead: ordinary Java lambdas of {{GeneratedDispatcher}} itself,
whose bytecode-level LMF sites are AOT-compiled into the image, so no class
definition happens at run time. There is no JIT in such runtimes, so the
inlining rationale does not apply there.
Detection is
{{"runtime".equals(System.getProperty("org.graalvm.nativeimage.imagecode"))}},
evaluated *per link* rather than cached in a static — under native image the
class may be build-time initialized, where the property reports {{buildtime}},
and caching would bake the wrong answer into the image heap. A catch-based
fallback around the metafactory calls covers AOT runtimes the property probe
misses.
One subtlety: dispatch targets may throw checked exceptions the dispatcher
interfaces do not declare, and the hidden-class path propagates them
transparently. The wrapper path rethrows via {{UncheckedThrow}} to match, and
this is covered by a dedicated test.
h2. Opt-in / opt-out / permanent?
Permanent and automatic — no user-facing flag:
* JVM behaviour is byte-identical; the fallback engages only where the
hidden-class path *cannot work*. A flag would choose between "works" and
"crash".
* The gate already exists one level up: packing is itself opt-in
({{groovy.target.closure.pack}} / {{@PackedClosures}}) and experimental under
GEP-27.
* {{-Dgroovy.packed.dispatch.handles=true}} exists as a *diagnostic* knob only,
forcing the wrapper path on a JVM so CI can assert parity without a native
build.
h2. Verification
||check||result||
|{{PackedDispatcherHandleBundleTest}} (new: parity across all three dispatch
shapes + the GDK; undeclared checked-exception propagation)|2/2 pass|
|existing packed/closure suites ({{PackedClosuresTransformTest}},
{{ClosurePackCapabilityTest}}, {{PackedClosureBoundariesTest}},
{{PackedClosureMetaClassTest}}, {{PackedClosureDebugMetadataTest}},
{{ClosureAndInnerClassNodeStructureTest}}, {{ClosureCallMopGuardTest}})|57
tests green|
|repro, unpatched master, native|{{Classes cannot be defined at runtime}}|
|repro, patched, native|runs correctly: {{lambda: 42 / coerced: 42 / collect:
[2, 4, 6]}}, exit 0, no exceptions|
|agent metadata for the dispatcher|0 entries (was: per-class method
registrations)|
|footprint|single emitted class; 30 MB image; ~12 ms total run time|
The remaining agent configuration in the repro covers the general Groovy
runtime bootstrap (VMPluginFactory reflection, {{dgm$NNN}} classes,
{{META-INF/dgminfo}}) — app-independent, pre-existing, and out of scope here;
shipping that fixed set as reachability metadata in Groovy's own jar would be a
natural follow-up ticket.
> GeneratedDispatcher: avoid runtime class definition so packed closures work
> in native images
> ---------------------------------------------------------------------------------------------
>
> Key: GROOVY-12227
> URL: https://issues.apache.org/jira/browse/GROOVY-12227
> Project: Groovy
> Issue Type: Improvement
> Reporter: Paul King
> Assignee: Paul King
> Priority: Major
>
> Packed closures (GROOVY-12151, GEP-27) do not work under GraalVM native
> image. A class compiled with {{groovy.target.closure.pack=true}} links its
> dispatch tables on first closure creation, and that linkage fails inside a
> native image — in two layers.
> Without agent-recorded metadata, the bootstrap's method lookup fails:
> {noformat}
> java.lang.BootstrapMethodError: java.lang.NoSuchMethodException:
> no such method: M.$packedDispatch$(int,Object[])Object/invokeStatic
> {noformat}
> With agent-recorded metadata the lookup succeeds, and the real wall appears —
> the bootstrap spins hidden classes at run time, which a native image forbids:
> {noformat}
> Exception in thread "main" java.lang.BootstrapMethodError:
> java.lang.InternalError:
> com.oracle.svm.core.jdk.UnsupportedFeatureError: Classes cannot be defined at
> runtime
> by default when using ahead-of-time Native Image compilation. Tried to define
> class:
> M$$LambdadLbm0tKUrZ8kVMiSpulnyK
> {noformat}
> This matters because packing is otherwise a natural fit for native images: it
> collapses per-closure inner classes into the hosting class (a 3-to-1 class
> reduction in the repro below), which directly reduces image size and
> reachability metadata, and its call site is a {{ConstantCallSite}} — no
> runtime re-linking, unlike the mutable-call-site dispatch of regular indy.
> h2. Reproducing
> {code:java}
> import groovy.transform.CompileStatic
> import java.util.function.BiFunction
> @CompileStatic
> class M {
> static void main(String[] args) {
> BiFunction<Integer, Integer, Integer> lam = (Integer a, Integer b) ->
> a + b
> BiFunction<Integer, Integer, Integer> coe = { Integer a, Integer b ->
> a * b } as BiFunction<Integer, Integer, Integer>
> var doubled = [1, 2, 3].collect { int n -> n * 2 }
> println "lambda: ${lam.apply(20, 22)}"
> println "coerced: ${coe.apply(6, 7)}"
> println "collect: ${doubled}"
> }
> }
> {code}
> {noformat}
> JAVA_OPTS="-Dgroovy.target.indy=false -Dgroovy.target.lambda.hoist=true
> -Dgroovy.target.closure.pack=true" \
> groovyc -cp groovy-callsite.jar M.groovy # emits a single class:
> M.class
> java -agentlib:native-image-agent=config-output-dir=cfg -cp
> "groovy.jar:groovy-callsite.jar:." M
> native-image --no-fallback -H:ConfigurationFileDirectories=cfg \
>
> -H:IncludeResources='META-INF/dgminfo|META-INF/groovy/.*|META-INF/services/.*'
> \
> -cp "groovy.jar:groovy-callsite.jar:." M
> ./m # fails as above
> {noformat}
> Reproduced with GraalVM CE 25.2.4 (native-image 25.0.4, the first release
> whose Groovy substitution guard is fixed for Groovy 5+ — see
> oracle/graal#13096).
> h2. Root cause
> {{GeneratedDispatcher.bootstrap}} does two things a native image cannot
> support:
> # {{caller.findStatic(host, "$packedDispatch$", ...)}} — a *runtime*
> method-handle lookup, which under native image requires per-user-class
> reflection metadata (unknowable to Groovy's own jar, so every user would need
> an agent run).
> # Three programmatic
> {{LambdaMetafactory.metafactory(...).getTarget().invokeExact()}} calls — each
> defines a hidden class at run time. Bytecode-level LMF invokedynamic is
> pre-processed at image build time and works fine; *programmatic* metafactory
> calls inside a bootstrap are runtime class definition, which is forbidden.
> The surrounding design is already image-friendly: the result is wrapped in a
> {{ConstantCallSite}} and never re-linked.
> h2. Proposed fix (prototyped)
> Two layered changes; JVM semantics unchanged.
> *1. Constant bootstrap arguments (ClosureWriter).* Emit the three dispatch
> tables as {{CONSTANT_MethodHandle}} bootstrap arguments, exactly as javac
> does for LMF's {{implMethod}}:
> {noformat}
> invokedynamic packedDispatchers()Object
> bsm = IndyInterface.packedDispatchers(Lookup, String, MethodType,
> MethodHandle, MethodHandle,
> MethodHandle)
> args = [ MH(invokestatic host.$packedDispatch$), MH(...1$), MH(...2$) ]
> {noformat}
> Constant-pool method handles are resolved by the VM (and pre-resolved at
> image build time), so linking needs no {{findStatic}} and no metadata.
> Verified: the tracing agent records *zero* {{packedDispatch}} entries for the
> new bytecode. The existing 3-arg bootstrap remains for class files emitted by
> earlier 6.0 snapshots.
> *2. Method-handle bundle fallback (GeneratedDispatcher).* On a regular JVM
> the {{LambdaMetafactory}} hidden-class adapters are kept — their interface
> call inlines under the JIT, which is the documented rationale. Where classes
> cannot be defined at run time, the tables are wrapped in
> method-handle-invoking adapters instead: ordinary Java lambdas of
> {{GeneratedDispatcher}} itself, whose bytecode-level LMF sites are
> AOT-compiled into the image, so no class definition happens at run time.
> There is no JIT in such runtimes, so the inlining rationale does not apply
> there.
> Detection is
> {{"runtime".equals(System.getProperty("org.graalvm.nativeimage.imagecode"))}},
> evaluated *per link* rather than cached in a static — under native image the
> class may be build-time initialized, where the property reports
> {{buildtime}}, and caching would bake the wrong answer into the image heap. A
> catch-based fallback around the metafactory calls covers AOT runtimes the
> property probe misses.
> One subtlety: dispatch targets may throw checked exceptions the dispatcher
> interfaces do not declare, and the hidden-class path propagates them
> transparently. The wrapper path rethrows via {{UncheckedThrow}} to match, and
> this is covered by a dedicated test.
> h2. Opt-in / opt-out / permanent?
> Permanent and automatic — no user-facing flag:
> * JVM behaviour is byte-identical; the fallback engages only where the
> hidden-class path *cannot work*. A flag would choose between "works" and
> "crash".
> * The gate already exists one level up: packing is itself opt-in
> ({{groovy.target.closure.pack}} / {{@PackedClosures}}) and experimental under
> GEP-27.
> * {{-Dgroovy.packed.dispatch.handles=true}} exists as a *diagnostic* knob
> only, forcing the wrapper path on a JVM so CI can assert parity without a
> native build.
> h2. Verification
> ||check||result||
> |{{PackedDispatcherHandleBundleTest}} (new: parity across all three dispatch
> shapes + the GDK; undeclared checked-exception propagation)|2/2 pass|
> |existing packed/closure suites ({{PackedClosuresTransformTest}},
> {{ClosurePackCapabilityTest}}, {{PackedClosureBoundariesTest}},
> {{PackedClosureMetaClassTest}}, {{PackedClosureDebugMetadataTest}},
> {{ClosureAndInnerClassNodeStructureTest}}, {{ClosureCallMopGuardTest}})|57
> tests green|
> |repro, unpatched master, native|{{Classes cannot be defined at runtime}}|
> |repro, patched, native|runs correctly: {{lambda: 42 / coerced: 42 / collect:
> [2, 4, 6]}}, exit 0, no exceptions|
> |agent metadata for the dispatcher|0 entries (was: per-class method
> registrations)|
> |footprint|single emitted class; 30 MB image; ~12 ms total run time|
> The remaining agent configuration in the repro covers the general Groovy
> runtime bootstrap (VMPluginFactory reflection, {{dgm$NNN}} classes,
> {{META-INF/dgminfo}}) — app-independent, pre-existing, and out of scope here;
> shipping that fixed set as reachability metadata in Groovy's own jar would be
> a natural follow-up ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
