Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Wed, 2 Jun 2021 20:13:34 GMT, Maurizio Cimadamore
wrote:
>> This patch overhauls the library loading mechanism used by the Foreign
>> Linker API. We realized that, while handy, the *default* lookup abstraction
>> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms.
>>
>> This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> concern with library loading, only symbol lookup. For this reason, two
>> factories are added:
>>
>> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to
>> lookup symbols in libraries loaded by current loader
>> * `CLinker::systemLookup` - a more stable replacement for the *default*
>> lookup, which looks for symbols in libc.so (or its equivalent in other
>> platforms). The contents of this lookup are unspecified.
>>
>> Both factories are *restricted*, so they can only be called when
>> `--enable-native-access` is set.
>
>> _Mailing list message from [Chapman Flack](mailto:c...@anastigmatix.net) on
>> [security-dev](mailto:security-...@mail.openjdk.java.net):_
>>
>> On 06/02/21 13:30, Maurizio Cimadamore wrote:
>>
>> > This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> > abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> > concern with library loading, only symbol lookup. For this reason, two
>> > factories are added:
>>
>> Hi,
>>
>> While I am thinking about this, what will be the behavior when the JVM
>> itself has been dynamically loaded into an embedding application, and
>> launched with the JNI invocation API?
>>
>> Will there be a *Lookup flavor that is able to find any exported symbol
>> known in the embedding environment the JVM was loaded into? (This is the
>> sort of condition the Mac OS linker checks when given the -bundle_loader
>> option.)
>>
>> Regards,
>> Chapman Flack (maintainer of a project that happens to work that way)
>
> Hi,
> at the moment we don't have plans to add such a lookup, but I believe it
> should be possible to build such a lookup using `dlopen` and the linker API.
> I have provided an example elsewhere of how easy it easy to build a wrapper
> lookup around dlopen/dlsym:
>
> https://gist.github.com/mcimadamore/0883ea6f4836ae0c1d2a31c48197da1a
>
> Perhaps something like that could be useful in the use case you mention?
Hi @mcimadamore,
As you mentioned at
https://github.com/openjdk/jdk/pull/4316#issuecomment-853238872, the system
lookup is supported by the underlying `NativeLibraries` which also works on
OpenJDK16 to support `LibraryLookup::ofDefault`.
But my finding is that it `LibraryLookup::ofDefault` works good on AIX (with
`libc.a`) in OpenJDK16 but `CLinker.systemLookup()` ended up with
`NoSuchElementException` after changing the code in `SystemLookup.java` and
`CABI.java` as follows:
private static final SymbolLookup syslookup = switch (CABI.current()) {
case SysV, LinuxAArch64, MacOsAArch64, AIX -> libLookup(libs ->
libs.loadLibrary("syslookup"));
case Win64 -> makeWindowsLookup(); // out of line to workaround javac
crash
};
with a simple test & output:
import jdk.incubator.foreign.CLinker;
import static jdk.incubator.foreign.CLinker.*;
import jdk.incubator.foreign.SymbolLookup;
import jdk.incubator.foreign.Addressable;
public class Test {
private static CLinker clinker = CLinker.getInstance();
private static final SymbolLookup defaultLibLookup =
CLinker.systemLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign -Dforeign.restricted=permit Test
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.util.NoSuchElementException: No value present
<-
at java.base/java.util.Optional.get(Optional.java:143)
at Test.main(Test.java:11)
So I am wondering what happened to the system lookup in such case given there
should be no fundamental difference in leveraging `NativeLibraries` (I assume
the library loading in OpenJDK16 & 17 is the same at this point) unless there
is something else new in OpenJDK17 I am unaware of (e.g. the changes in
`Lib.gmk` or `lib-std.m4`, or a custom system lookup is required on AIX, etc).
Thanks.
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries [v7]
On Thu, 3 Jun 2021 20:49:44 GMT, Maurizio Cimadamore
wrote:
>> This patch overhauls the library loading mechanism used by the Foreign
>> Linker API. We realized that, while handy, the *default* lookup abstraction
>> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms.
>>
>> This patch replaces `LibraryLookup` with a simpler `SymbolLookup`
>> abstraction, a functional interface. Crucially, `SymbolLookup` does not
>> concern with library loading, only symbol lookup. For this reason, two
>> factories are added:
>>
>> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to
>> lookup symbols in libraries loaded by current loader
>> * `CLinker::systemLookup` - a more stable replacement for the *default*
>> lookup, which looks for symbols in libc.so (or its equivalent in other
>> platforms). The contents of this lookup are unspecified.
>>
>> Both factories are *restricted*, so they can only be called when
>> `--enable-native-access` is set.
>
> Maurizio Cimadamore has updated the pull request incrementally with one
> additional commit since the last revision:
>
> Address review comments on shim lib makefile
Just tried with `System.load()` but still ended up with pretty much the same
failure given both of them eventually invokes `ClassLoader.loadLibrary` to load
the library in which case there is no big difference at this point.
static {
System.load("/usr/lib/libc.a");
}
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load
/usr/lib/libc.a <
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1793)
at java.base/java.lang.System.load(System.java:672)
at StdLibTest.(StdLibTest.java:24)
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Tue, 12 Oct 2021 15:04:02 GMT, Maurizio Cimadamore
wrote:
> Is libc.a loadable on AIX (e.g. with System.loadLibrary) ?
I tried to load `libc.a` and `libc` this way but neither of them works on AIX.
e.g.
public class StdLibTest {
private static CLinker clinker = CLinker.getInstance();
static {
System.loadLibrary("libc.a"); <-
}
private static final SymbolLookup defaultLibLookup =
SymbolLookup.loaderLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
$ ./bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign
-Dforeign.restricted=permit --enable-native-access=ALL-UNNAMED StdLibTest
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load libc.a
<---
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
at java.base/java.lang.System.loadLibrary(System.java:694)
at StdLibTest.(StdLibTest.java:23)
and
public class StdLibTest {
private static CLinker clinker = CLinker.getInstance();
static {
System.loadLibrary("libc"); <---
}
private static final SymbolLookup defaultLibLookup =
SymbolLookup.loaderLookup();
public static void main(String args[]) throws Throwable {
Addressable strlenSymbol =
defaultLibLookup.lookup("strlen").get();
}
}
$ ./bin/java --enable-native-access=ALL-UNNAMED --add-modules
jdk.incubator.foreign
-Dforeign.restricted=permit --enable-native-access=ALL-UNNAMED StdLibTest
WARNING: Using incubator modules: jdk.incubator.foreign
Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load libc
<--
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
at java.base/java.lang.System.loadLibrary(System.java:694)
at StdLibTest.(StdLibTest.java:23)
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries
On Fri, 8 Oct 2021 21:29:19 GMT, Maurizio Cimadamore
wrote:
>> Hi @mcimadamore,
>>
>> As you mentioned at
>> https://github.com/openjdk/jdk/pull/4316#issuecomment-853238872, the system
>> lookup is supported by the underlying `NativeLibraries` which also works on
>> OpenJDK16 to support `LibraryLookup::ofDefault`.
>>
>> But my finding is that it `LibraryLookup::ofDefault` works good on AIX (with
>> `libc.a`) in OpenJDK16 but `CLinker.systemLookup()` ended up with
>> `NoSuchElementException` after changing the code in `SystemLookup.java` and
>> `CABI.java` as follows:
>>
>> private static final SymbolLookup syslookup = switch (CABI.current()) {
>> case SysV, LinuxAArch64, MacOsAArch64, AIX -> libLookup(libs ->
>> libs.loadLibrary("syslookup"));
>> case Win64 -> makeWindowsLookup(); // out of line to workaround
>> javac crash
>> };
>>
>> with a simple test & output:
>>
>> import jdk.incubator.foreign.CLinker;
>> import static jdk.incubator.foreign.CLinker.*;
>> import jdk.incubator.foreign.SymbolLookup;
>> import jdk.incubator.foreign.Addressable;
>>
>> public class Test {
>> private static CLinker clinker = CLinker.getInstance();
>> private static final SymbolLookup defaultLibLookup =
>> CLinker.systemLookup();
>>
>> public static void main(String args[]) throws Throwable {
>> Addressable strlenSymbol =
>> defaultLibLookup.lookup("strlen").get();
>> }
>> }
>>
>> bin/java --enable-native-access=ALL-UNNAMED --add-modules
>> jdk.incubator.foreign -Dforeign.restricted=permit Test
>> WARNING: Using incubator modules: jdk.incubator.foreign
>> Exception in thread "main" java.util.NoSuchElementException: No value
>> present <-
>> at java.base/java.util.Optional.get(Optional.java:143)
>> at Test.main(Test.java:11)
>>
>>
>> So I am wondering what happened to the system lookup in such case given
>> there should be no fundamental difference in leveraging `NativeLibraries` (I
>> assume the library loading in OpenJDK16 & 17 is the same at this point)
>> unless there is something else new in OpenJDK17 I am unaware of (e.g. the
>> changes in `Lib.gmk` or `lib-std.m4`, or a custom system lookup is required
>> on AIX, etc). Thanks.
>
>> So I am wondering what happened to the system lookup in such case given
>> there should be no fundamental difference in leveraging `NativeLibraries` (I
>> assume the library loading in OpenJDK16 & 17 is the same at this point)
>> unless there is something else new in OpenJDK17 I am unaware of (e.g. the
>> changes in `Lib.gmk` or `lib-std.m4`, or a custom system lookup is required
>> on AIX, etc). Thanks.
>
> In 17, SystemLookup loads a specific library that is generated at build time
> - which contains all the c stdlib symbols. That's what the Lib.gmk changes
> are for. What I suspect is that the library is not generated at all, or not
> correctly on AIX, which then causes the system lookup to misbehave.
>
> I would start by double checking that you have a file like this:
>
> /lib/libsyslookup.so
>
> And then, if the library exists, check that it has the right dependency; on
> my linux it's an empty library, but which depends on libc, libm and libdl:
>
>
> ldd lib/libsyslookup.so
> linux-vdso.so.1 (0x7fff2bdf7000)
> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f35f1def000)
> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x7f35f1ca)
> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f35f1c9a000)
Hi @mcimadamore,
Here's output of `libsyslookup.so` on AIX:
$ ldd ./lib/libsyslookup.so
./lib/libsyslookup.so needs: <--- nothing here
$ whereis libc
libc: /usr/lib/libc.a /usr/lib/libc128.a /usr/ccs/lib/libc.a
So it is high likely that there were no dependencies in this generated library.
> perhaps on AIX something similar to what we did for Windows [1] would be
> better.
That's what I thought to be the only way around but might need to figure out
the specifics on AIX.
-
PR: https://git.openjdk.java.net/jdk/pull/4316
Re: RFR: 8268129: LibraryLookup::ofDefault leaks symbols from loaded libraries [v7]
On Thu, 3 Jun 2021 20:49:44 GMT, Maurizio Cimadamore wrote: >> This patch overhauls the library loading mechanism used by the Foreign >> Linker API. We realized that, while handy, the *default* lookup abstraction >> (`LibraryLookup::ofDefault`) was behaving inconsistentlt across platforms. >> >> This patch replaces `LibraryLookup` with a simpler `SymbolLookup` >> abstraction, a functional interface. Crucially, `SymbolLookup` does not >> concern with library loading, only symbol lookup. For this reason, two >> factories are added: >> >> * `SymbolLookup::loaderLookup` - which obtains a lookup that can be used to >> lookup symbols in libraries loaded by current loader >> * `CLinker::systemLookup` - a more stable replacement for the *default* >> lookup, which looks for symbols in libc.so (or its equivalent in other >> platforms). The contents of this lookup are unspecified. >> >> Both factories are *restricted*, so they can only be called when >> `--enable-native-access` is set. > > Maurizio Cimadamore has updated the pull request incrementally with one > additional commit since the last revision: > > Address review comments on shim lib makefile If so, I am wondering whether there is anything else left (inherited from JDK16) in JDK17 we can leverage to support `libc.a` on AIX except the way similar to Windows. - PR: https://git.openjdk.java.net/jdk/pull/4316
RE: Incorrect assumption on the class name in compile(InputSource input, String name) at XSLTC.java
The issue for the XSLTC was raised at
https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-October/082767.html
:
The code in compile(InputSource input, String name) at
https://github.com/openjdk/jdk/blob/master/src/java.xml/share/classes/com/sun/org/apache/xalan/internal/xsltc/compiler/XSLTC.java
seems incorrect as follows:
public boolean compile(InputSource input, String name) {
try {
// Reset globals in case we're called by compile(ArrayList v);
reset();
// The systemId may not be set, so we'll have to check the URL
String systemId = null;
if (input != null) {
systemId = input.getSystemId();
}
// Set the translet class name if not already set
if (_className == null) {
if (name != null) {
setClassName(name);
}
else if (systemId != null && !systemId.equals("")) {
setClassName(Util.baseName(systemId)); <---
incorrect here
}
// Ensure we have a non-empty class name at this point
if (_className == null || _className.length() == 0) {
setClassName("GregorSamsa"); // default translet name
}
}
...
Specifically, the code above assumes that systemId is something like
"xxx:yyy" in which case the class name (_className) is
"die.verwandlung.yyy" ("die.verwandlung." is the default package name since
Java11) However,Util.baseName(systemId) returns null when systemId is
something like "xxx:" (empty after ":"), in which case the class name
(_className) ends up with "die.verwandlung."(an invalid class name inserted
in the constant pool of the generated bytecode).
>From the perspective of robustness, the code above should be updated to
handle the situation. e.g.
else if (systemId != null && !systemId.equals("")) {
String baseName = Util.baseName(systemId);
if ((baseName != null) && (baseName.length() > 0))
{ <--
setClassName(baseName);
}
which means it should check whether the returned base name from
Util.baseName(systemId) is empty before setting the class name; otherwise,
it should use the default class name ("GregorSamsa").
Thanks and Best Regards
Cheng Jin
J9 VM, IBM Ottawa Lab at Riverside, Ottawa
IBM Runtime Technologies
+1-613-356-5625
jinch...@ca.ibm.com
Re: Incorrect assumption on the class name in compile(InputSource input, String name) at XSLTC.java
Hi There, Hotspot developers already identified a bug in verification (failure to capture an invalid package name "die/verwandlung/" existing in the constant pool of bytecode) at https://bugs.openjdk.java.net/browse/JDK-8276241 as I raised via https://mail.openjdk.java.net/pipermail/hotspot-dev/2021-October/055618.html , which was associated with the issue here in XSLT transformation given the invalid package name "die/verwandlung/" was actually triggered by OpenJDK due to the incorrect result from setClassName(Util.baseName(systemId)) (systemId = "xxx:" in which case Util.baseName(systemId) returns null and setClassName(null) ended up with "die/verwandlung/" in generating the bytecode). So I expect someone in OpenJDK to take a look into this issue to see what really happened to the code there in such case. Thanks. Thanks and Best Regards Cheng Jin
Re: Incorrect assumption on the class name in compile(InputSource input, String name) at XSLTC.java
Hi There, Hotspot developers already identified a bug in verification (failure to capture an invalid package name "die/verwandlung/" existing in the constant pool of bytecode) at https://bugs.openjdk.java.net/browse/JDK-8276241 as I raised via https://mail.openjdk.java.net/pipermail/hotspot-dev/2021-October/055618.html , which was associated with the issue here in XSLT transformation given the invalid package name "die/verwandlung/" was actually triggered by OpenJDK due to the incorrect result from setClassName(Util.baseName(systemId)) (systemId = "xxx:" in which case Util.baseName(systemId) returns null and setClassName(null) ended up with "die/verwandlung/" in generating the bytecode). So I expect someone in OpenJDK to take a look into this issue to see what really happened to the code there in such case. Thanks. Thanks and Best Regards Cheng Jin
Incorrect assumption on the class name in compile(InputSource input, String name) at XSLTC.java
Hi there,
The code in compile(InputSource input, String name) at
https://github.com/openjdk/jdk/blob/master/src/java.xml/share/classes/com/sun/org/apache/xalan/internal/xsltc/compiler/XSLTC.java
seems incorrect as follows:
public boolean compile(InputSource input, String name) {
try {
// Reset globals in case we're called by compile(ArrayList v);
reset();
// The systemId may not be set, so we'll have to check the URL
String systemId = null;
if (input != null) {
systemId = input.getSystemId();
}
// Set the translet class name if not already set
if (_className == null) {
if (name != null) {
setClassName(name);
}
else if (systemId != null && !systemId.equals("")) {
setClassName(Util.baseName(systemId)); <---
incorrect here
}
// Ensure we have a non-empty class name at this point
if (_className == null || _className.length() == 0) {
setClassName("GregorSamsa"); // default translet name
}
}
...
Specifically, the code above assumes that systemId is something like
"xxx:yyy" in which case the class name (_className) is
"die.verwandlung.yyy" ("die.verwandlung." is the default package name since
Java11) However,Util.baseName(systemId) returns null when systemId is
something like "xxx:" (empty after ":"), in which case the class name
(_className) ends up with "die.verwandlung."(an invalid class name inserted
in the constant pool of the generated bytecode).
>From the perspective of robustness, the code above should be updated to
handle the situation. e.g.
else if (systemId != null && !systemId.equals("")) {
String baseName = Util.baseName(systemId);
if ((baseName != null) && (baseName.length() > 0))
{ <--
setClassName(baseName);
}
which means it should check whether the returned base name from
Util.baseName(systemId) is empty before setting the class name; otherwise,
it should use the default class name ("GregorSamsa").
Let me know if any other concern on this fix.
Thanks and Best Regards
Cheng Jin
Request for backport the OpenJDK fix in XSLTC.java to JDK11 & JDK17
Hi there,
As for the bug with the class name in XSLTC.java I previously raised at
https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-November/083135.html
,
I notice it has been resolved at
https://bugs.openjdk.java.net/browse/JDK-8276657 as follows:
https://github.com/openjdk/jdk/blob/a093cdddaf5ab88eb84a147e523db5c3e1be54be/src/java.xml/share/classes/com/sun/org/apache/xalan/internal/xsltc/compiler/XSLTC.java
public boolean compile(InputSource input, String name) {
try {
// Reset globals in case we're called by compile(ArrayList v);
reset();
// The systemId may not be set, so we'll have to check the URL
String systemId = null;
if (input != null) {
systemId = input.getSystemId();
}
// Set the translet class name if not already set
if (_className == null) {
if (name != null) {
setClassName(name);
}
else if (systemId != null && !systemId.isEmpty()) { <--
String clsName = Util.baseName(systemId);
if (clsName != null && !clsName.isEmpty()) {
setClassName(clsName);
}
}
With this fix at https://git.openjdk.java.net/jdk/pull/6620, would you
please backport it to JDK11 & JDK17 given the code there is at the same
place? Thanks in advance.
Thanks and Best Regards
Cheng Jin
Please help backport the fix with the XSLT compiler in JDK18 to JDK11 & JDK17
Hi there, I notice the issue with the XSLT compiler (https://github.com/openjdk/jdk/blob/master/src/java.xml/share/classes/com/sun/org/apache/xalan/internal/xsltc/compiler/XSLTC.java) I raised later last year at https://mail.openjdk.java.net/pipermail/core-libs-dev/2021-November/083135.html has been fixed in JDK18 via https://bugs.openjdk.java.net/browse/JDK-8276657. Could you help backport the fix to JDK11 and JDK17 give both of them share the same issue in code? Many thanks. Best Regards Cheng Jin
RE: When to initialize the method's class for MethodHandles.Lookup.findStatic()?
Hi Raffaello,
The code snippet I posted previously was not the original test I verified on
Java 17 (which was generated via asmtools to trigger verification)
Here's the pretty much the test I used:
Test_1.java
import java.lang.invoke.*;
public class Test_1 {
static MethodHandle mh;
static {
try {
mh = MethodHandles.lookup().findStatic(Test_2.class,
"testMethod", MethodType.methodType(int[].class));
} catch (NoSuchMethodException | IllegalAccessException e) {
e.printStackTrace();
}
}
public static void main(String[] args) throws Throwable {
//Test_1.mh.invoke();
}
}
Test_2.jasm
super public class Test_2
version 52:0
{
static Method "":"()V"
stack 5 locals 12
{
bipush -2;
istore 7;
iload 7;
sipush 997;
if_icmplt L127;
iconst_0;
istore 8;
L127: stack_frame_type full;
locals_map int, class "[B", class "[B", class
"[Ljava/lang/Class;", class "[I", class "[I", class "[I", int, bogus, bogus,
float, float;
iload 8;
istore 8;
return;
}
public Method "":"()V"
stack 1 locals 1
{
aload_0;
invokespecial Method java/lang/Object."":"()V";
return;
}
static Method testMethod:"()[I"
stack 6 locals 1
{
getstatic Field testArray:"[I";
areturn;
}
} // end Class Test_2
Jdk17/bin/java -jar asmtools.jar jasm Test_2.jasm // create the .class file
of Test_2
Jdk17/bin/java Test_1
java.lang.IllegalAccessException: no such method:
Test_2.testMethod()int[]/invokeStatic
at
java.base/java.lang.invoke.MemberName.makeAccessException(MemberName.java:972)
at
java.base/java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1117)
at
java.base/java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:3649)
at
java.base/java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:2588)
at Test_1.(Test_1.java:9)
Caused by: java.lang.VerifyError: Inconsistent stackmap frames at branch target
15
Exception Details:
Location:
Test_2.()V @15: iload
Reason:
Type top (current frame, locals[0]) is not assignable to integer (stack
map, locals[0])
Current Frame:
bci: @9
flags: { }
locals: { top, top, top, top, top, top, top, integer }
stack: { integer, integer }
Stackmap Frame:
bci: @15
flags: { }
locals: { integer, '[B', '[B', '[Ljava/lang/Class;', '[I', '[I', '[I',
integer, top, top, float, float }
stack: { }
Bytecode:
000: 10fe 3607 1507 1103 e5a1 0006 0336 0815
010: 0836 08b1
Stackmap Table:
full_frame(@15,{Integer,Object[#11],Object[#11],Object[#20],Object[#5],Object[#5],Object[#5],Integer,Top,Top,Float,Float},{})
at java.base/java.lang.invoke.MethodHandleNatives.resolve(Native Method)
at
java.base/java.lang.invoke.MemberName$Factory.resolve(MemberName.java:1085)
at
java.base/java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1114)
... 3 more
Test_2 was intentionally modified to throw VerifyError if initialized.
As you see above, mh.inovke() was commented out in Test_1.main() but the
Test_2 was still verified in the test.
So it is impossible to verify Test_2 if it is not initialized, which only means
Test_2 is initialized during the lookup.findstatic rather than mh.invoke().
Best Regards
Cheng
- Original Message -
> From: "Cheng Jin"
> To: "core-libs-dev"
> Sent: Thursday, March 17, 2022 5:42:57 PM
> Subject: When to initialize the method's class for
> MethodHandles.Lookup.findStatic()?
> Hi there,
>
> The document of
> INVALID URI REMOVED
> n_java_javase_17_docs_api_java.base_java_lang_invoke_MethodHandles.Loo
> kup.html-23findStatic-28java.lang.Class-2Cjava.lang.String-2Cjava.lang
> .invoke.MethodType-29=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=X90f3XIRXAH8
> hbNam6bIUlWfF_qUAezL9ue7M7bFuPQ=Xt-10pHYoPWnY6dByKowR4yeLtEs7kZkKUgt
> bxKUGvM=dPAMGMxphLLXT9N4ZdukiNvWyvRPAGkcGCBLTy_sm0c=
> in the Java API is ambiguous in terms of when to initialize the
> method's class as follows (the same description as in other OpenJDK
> versions)
>
> If the returned method handle is invoked, the method's class will be
> initialized, if it has not already been initialized.
>
>
> It occurs to me that the method's class should be initialized when
> invoking the method handle but OpenJDK actually chooses to do the
> initialization in
> lookup.findStatic() rather than in mh.
RE: When to initialize the method's class for MethodHandles.Lookup.findStatic()?
Hi David,
1) for the test with mh.invoke() in main(), the log shows:
[0.262s][info][class,init] Start class verification for: Test_1
[0.262s][info][class,init] End class verification for: Test_1
[0.263s][info][class,init] 282 Initializing 'Test_1' (0x000800c00800)
[0.263s][info][class,init] Start class verification for: Test_2
[0.263s][info][class,init] End class verification for: Test_2
[0.272s][info][class,init] 366 Initializing 'Test_2' (0x000800c00a08)
<--
2) for the test without mh.invoke() in main(), the log shows:
[0.296s][info][class,init] Start class verification for: Test_1
[0.296s][info][class,init] End class verification for: Test_1
[0.297s][info][class,init] 282 Initializing 'Test_1' (0x000800c00800)
[0.297s][info][class,init] Start class verification for: Test_2
[0.297s][info][class,init] End class verification for: Test_2
(Test_2 was verified but didn't get initialized)
The comparison above literally surprised me that the bytecode verification
happened prior to the class initialization, which means
the class got verified at first even without initialization coz I previously
thought the initialization should trigger the verification rather than in the
reversed order.
Could you explain a little more about why it goes in this way?
Best Regards
Cheng
-Original Message-
From: core-libs-dev On Behalf Of David
Holmes
Sent: March 17, 2022 7:46 PM
To: Raffaello Giulietti ;
core-libs-dev@openjdk.java.net
Subject: [EXTERNAL] Re: When to initialize the method's class for
MethodHandles.Lookup.findStatic()?
Run with -Xlog:class+init=info to see the classes that get initialized and in
what order.
David
On 18/03/2022 5:53 am, Raffaello Giulietti wrote:
> Hi again,
>
> here's code that shows that initialization doesn't happen during
> lookup but only upon invoking the method handle. (I'm on Java 17.)
>
> As long as the 2nd line in main() is commented, you don't see the
> message "Test_2 initialized", which shows that the lookup doesn't
> initialize Test_2.
> When you uncomment the line in main(), the message will appear.
>
> So, as advertised, it's the invocation of the method handle that can
> trigger initialization, not the lookup.
>
>
> HTH
> Raffaello
>
>
>
> import java.lang.invoke.*;
>
> public class Test_1 {
>
> static MethodHandle mh;
>
> static {
> try {
> mh = MethodHandles.lookup().findStatic(Test_2.class,
> "testMethod", MethodType.methodType(int.class, int.class));
> } catch (NoSuchMethodException | IllegalAccessException e) {
> e.printStackTrace();
> }
> }
>
> public static void main(String[] args) throws Throwable {
> System.out.println(mh);
> // System.out.println(Test_1.mh.invoke(0));
> }
>
> }
>
>
>
>
> public class Test_2 {
>
> static {
> System.out.println("Test_2 initialized");
> }
>
> static int testMethod(int value) { return (value + 1); }
>
> }
>
>
>
>
> On 2022-03-17 20:38, Raffaello Giulietti wrote:
>> Hi,
>>
>> as far as I can see, the code should not even compile, as there's a
>> static field in Test_1 which is initialized with an expression that
>> throws checked exceptions (findStatic(..)).
>>
>> In addition, it seems to me that there's nothing in your code that
>> reveals whether Test_2 has been initialized during the lookup. How
>> can you tell?
>>
>> Finally, the method handle invocation in Test_1 will throw, as you
>> don't pass any argument to a handle that expects one.
>>
>> Can you perhaps add more details?
>>
>>
>> Greetings
>> Raffaello
>>
>>
>>
>> On 2022-03-17 17:42, Cheng Jin wrote:
>>> Hi there,
>>>
>>> The document of
>>> INVALID URI REMOVED
>>> _en_java_javase_17_docs_api_java.base_java_lang_invoke_MethodHandles
>>> .Lookup.html-23findStatic-28java.lang.Class-2Cjava.lang.String-2Cjav
>>> a.lang.invoke.MethodType-29=DwIDaQ=jf_iaSHvJObTbx-siA1ZOg=X90f
>>> 3XIRXAH8hbNam6bIUlWfF_qUAezL9ue7M7bFuPQ=RvhguidNJ90V-HK-3Ctl-kUZE5
>>> cIfo_nt3_r8VZ0Fcc=tw_ph6oUkS0eCvzITWi9zEkarss5yNeHDrAIfvd3s3g=
>>> in the Java API is ambiguous in terms of when to initialize the
>>> method's class as follows (the same description as in other OpenJDK
>>> versions)
>>>
>>> If the returned method handle is invoked, the method's class will be
>>> initialized, if it has not already been initialized.
>>>
>>>
>>> It occurs to me that the method's class should be initialized when
>>> invoking t
RE: When to initialize the method's class for MethodHandles.Lookup.findStatic()?
Hi Raffaello,
My concern is why the verification happens even without initialization in the
test without mh.invoke() in the main(), which I don't think is covered or
explained in the JVM Spec.
Put it in another way, my understanding is, when the class gets loaded, it is
verified which doesn't necessarily lead to initialization, am I correct?
Best Regards
Cheng
-Original Message-
From: core-libs-dev On Behalf Of
Raffaello Giulietti
Sent: March 17, 2022 8:35 PM
To: core-libs-dev@openjdk.java.net
Subject: [EXTERNAL] Re: When to initialize the method's class for
MethodHandles.Lookup.findStatic()?
Cheng,
initialization is the last thing that happens because it's where user provided
code gets executed.
This has always been this way, as long as I can remember. See the JVMS for the
gory details.
Greetings
Raffaello
On 2022-03-18 01:21, Cheng Jin wrote:
> Hi David,
>
> 1) for the test with mh.invoke() in main(), the log shows:
> [0.262s][info][class,init] Start class verification for: Test_1
> [0.262s][info][class,init] End class verification for: Test_1
> [0.263s][info][class,init] 282 Initializing 'Test_1' (0x000800c00800)
> [0.263s][info][class,init] Start class verification for: Test_2
> [0.263s][info][class,init] End class verification for: Test_2
> [0.272s][info][class,init] 366 Initializing 'Test_2' (0x000800c00a08)
> <--
>
> 2) for the test without mh.invoke() in main(), the log shows:
> [0.296s][info][class,init] Start class verification for: Test_1
> [0.296s][info][class,init] End class verification for: Test_1
> [0.297s][info][class,init] 282 Initializing 'Test_1' (0x000800c00800)
> [0.297s][info][class,init] Start class verification for: Test_2
> [0.297s][info][class,init] End class verification for: Test_2
> (Test_2 was verified but didn't get initialized)
>
> The comparison above literally surprised me that the bytecode verification
> happened prior to the class initialization, which means
> the class got verified at first even without initialization coz I previously
> thought the initialization should trigger the verification rather than in the
> reversed order.
>
> Could you explain a little more about why it goes in this way?
>
> Best Regards
> Cheng
>
>
> -Original Message-
> From: core-libs-dev On Behalf Of David
> Holmes
> Sent: March 17, 2022 7:46 PM
> To: Raffaello Giulietti ;
> core-libs-dev@openjdk.java.net
> Subject: [EXTERNAL] Re: When to initialize the method's class for
> MethodHandles.Lookup.findStatic()?
>
> Run with -Xlog:class+init=info to see the classes that get initialized and in
> what order.
>
> David
>
> On 18/03/2022 5:53 am, Raffaello Giulietti wrote:
>> Hi again,
>>
>> here's code that shows that initialization doesn't happen during
>> lookup but only upon invoking the method handle. (I'm on Java 17.)
>>
>> As long as the 2nd line in main() is commented, you don't see the
>> message "Test_2 initialized", which shows that the lookup doesn't
>> initialize Test_2.
>> When you uncomment the line in main(), the message will appear.
>>
>> So, as advertised, it's the invocation of the method handle that can
>> trigger initialization, not the lookup.
>>
>>
>> HTH
>> Raffaello
>>
>>
>>
>> import java.lang.invoke.*;
>>
>> public class Test_1 {
>>
>> static MethodHandle mh;
>>
>> static {
>> try {
>> mh = MethodHandles.lookup().findStatic(Test_2.class,
>> "testMethod", MethodType.methodType(int.class, int.class));
>> } catch (NoSuchMethodException | IllegalAccessException e) {
>> e.printStackTrace();
>> }
>> }
>>
>> public static void main(String[] args) throws Throwable {
>> System.out.println(mh);
>> // System.out.println(Test_1.mh.invoke(0));
>> }
>>
>> }
>>
>>
>>
>>
>> public class Test_2 {
>>
>> static {
>> System.out.println("Test_2 initialized");
>> }
>>
>> static int testMethod(int value) { return (value + 1); }
>>
>> }
>>
>>
>>
>>
>> On 2022-03-17 20:38, Raffaello Giulietti wrote:
>>> Hi,
>>>
>>> as far as I can see, the code should not even compile, as there's a
>>> static field in Test_1 which is initialized with an expression that
>>> throws checked exceptions (findStatic(..)).
>>>
>>> In addition, it seems to me that there's nothing in your code that
>>> reveals whether Test_2 has
Incorrect return value for Java_jdk_internal_loader_NativeLibraries_load() in java.base/share/native/libjava/NativeLibraries.c
Hi there,
The return value from Java_jdk_internal_loader_NativeLibraries_load() at
https://github.com/openjdk/jdk/blob/master/src/java.base/share/native/libjava/NativeLibraries.c
seems incorrect according to the code logic in there.
Looking at the calling path from loadLibrary() to load() at
src/java.base/share/classes/jdk/internal/loader/NativeLibraries.java as follows:
public NativeLibrary loadLibrary(Class fromClass, String name) {
...
NativeLibrary lib = findFromPaths(LibraryPaths.SYS_PATHS, fromClass,
name); <--
if (lib == null && searchJavaLibraryPath) {
lib = findFromPaths(LibraryPaths.USER_PATHS, fromClass, name); <
}
return lib;
}
private NativeLibrary findFromPaths(String[] paths, Class fromClass,
String name) {
for (String path : paths) { < keep searching the
library paths till the library is located at the correct lib path
File libfile = new File(path, System.mapLibraryName(name));
NativeLibrary nl = loadLibrary(fromClass, libfile); ?
if (nl != null) {
return nl; <
}
libfile = ClassLoaderHelper.mapAlternativeName(libfile);
if (libfile != null) {
nl = loadLibrary(fromClass, libfile);
if (nl != null) {
return nl;
}
}
}
return null;
}
public NativeLibrary loadLibrary(Class fromClass, File file) {
// Check to see if we're attempting to access a static library
String name = findBuiltinLib(file.getName());
boolean isBuiltin = (name != null);
...
return loadLibrary(fromClass, name, isBuiltin); <
}
private NativeLibrary loadLibrary(Class fromClass, String name, boolean
isBuiltin) {
...
NativeLibraryImpl lib = new NativeLibraryImpl(fromClass, name,
isBuiltin, isJNI);
// load the native library
NativeLibraryContext.push(lib);
try {
if (!lib.open()) { <- call load()
return null;// fail to open the native library
}
...
// register the loaded native library
loadedLibraryNames.add(name);
libraries.put(name, lib);
return lib; <--- return an invalid lib as lib.open()
returns true
} finally {
releaseNativeLibraryLock(name);
}
...
}
/*
* Loads the named native library
*/
boolean open() {
...
return load(this, name, isBuiltin, isJNI, loadLibraryOnlyIfPresent);
}
private static native boolean load(NativeLibraryImpl impl, String name,
boolean isBuiltin, boolean isJNI, boolean throwExceptionIfFail);
and then native code in java.base/share/native/libjava/NativeLibraries.c
Java_jdk_internal_loader_NativeLibraries_load
(JNIEnv *env, jobject this, jobject lib, jstring name,
jboolean isBuiltin, jboolean isJNI, jboolean throwExceptionIfFail)
{
const char *cname;
jint jniVersion;
jthrowable cause;
void * handle;
jboolean loaded = JNI_FALSE;
...
handle = isBuiltin ? procHandle : JVM_LoadLibrary(cname,
throwExceptionIfFail);
if (isJNI) {
...
}
(*env)->SetLongField(env, lib, handleID, ptr_to_jlong(handle));
loaded = JNI_TRUE; <- always return true when isJNI is false and a
null handle
done:
JNU_ReleaseStringPlatformChars(env, name, cname);
return loaded;
}
Assuming there are multiple library paths existing in the jdk/lib which are
added to sun.boot.library.path and java.library.path
and the expected library (libnative.so) is only placed in jdk/lib. e.g.
lib/libA
lib/libB
lib/libC
lib/libnative.so
when the code in findFromPaths() searches the library paths set in
sun.boot.library.path and java.library.path starting from lib/LibA,
it will end up with an invalid value if load() returns true in the case of a
false isJNI and a null handle returned from JVM_LoadLibrary()
as the library doesn't exist in lib/libA passed to JVM_LoadLibrary().
To ensure findFromPaths() keeps searching the remaining lib paths, the simple
fix should look like:
Java_jdk_internal_loader_NativeLibraries_load( ...)
{
...
(*env)->SetLongField(env, lib, handleID, ptr_to_jlong(handle));
if (handle) { <- return true only for a non-null
handle
loaded = JNI_TRUE;
}
...
}
Please help check the code around there to see whether the fix is reasonable to
avoid the unexpected situation as explained above (the problem was spotted
since JDK17)
Best Regards
Cheng Jin
When to initialize the method's class for MethodHandles.Lookup.findStatic()?
Hi there,
The document of
https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/invoke/MethodHandles.Lookup.html#findStatic(java.lang.Class,java.lang.String,java.lang.invoke.MethodType)
in the Java API is ambiguous in terms of when to initialize the method's class
as follows (the same description as in other OpenJDK versions)
If the returned method handle is invoked, the method's class will be
initialized, if it has not already been initialized.
It occurs to me that the method's class should be initialized when invoking the
method handle but OpenJDK actually chooses to do the initialization in
lookup.findStatic() rather than in mh.invoke()
e.g.
import java.lang.invoke.*;
public class Test_1 {
static MethodHandle mh = MethodHandles.lookup().findStatic(Test_2.class,
"testMethod", MethodType.methodType(int.class, int.class)); <---
Test_2.class gets initialized and verified.
public static void main(String[] args) throws Throwable {
Test_1.mh.invoke();
}
}
public class Test_2 {
static int testMethod(int value) { return (value + 1); }
}
So there should be more clear explanation what is the correct or expected
behaviour at this point and why OpenJDK doesn't comply with the document to
delay the initialization of the method's class to mh.invoke().
Best Regards
Cheng Jin
