Re: Some remarks on panama libssl loading
Michael,
On 4/17/24 16:46, Michael Osipov wrote:
On 2024/04/17 14:21:06 Rainer Jung wrote:
Am 17.04.24 um 15:34 schrieb Michael Osipov:
Rainer, I do not fully understand the problem here. We use libtool to solve
exactly this problem with versioned SONAMEs. It will create symlinks to the
SONAME.
Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless it
is strictly needed?
E.g.:
lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@ ->
../../lib/libcrypto.so.111
lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
libssl.so.111
-r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
and so on...
Yes, I expect that! anyone is the JVM :(
The problem is, that the Java API does not care about these well thought
native traditions. You can not open libssl.so.3 using
System.loadlibrary(String name), because whatever you give it as "name"
parameter it will always try to open libname.so. It always prepends
"lib" to name and always suffixes it with plain ".so".
Yes, it might exist as the first in your list of symlinks, but on most
linux distributions this link is not installed by default, because it is
only needed when doing compilations. So it is only installed when you
install development packages for libs.
Ah, now I see your problem, but it looks like a downstream problem of your
distro of choice, no? I wonder how you compile then custom software if .so
isn't present and the linker cannot find it with -L? What if you install the
devel package to have .so link?
That works, but doesn't seem to be a reasonable requirement if you just
want to install Ubuntu and Tomcat and run a server.
-chris
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
чт, 18 апр. 2024 г. в 00:46, Rainer Jung :
>
> Hi Konstantin,
>
> Am 17.04.24 um 21:44 schrieb Konstantin Kolinko:
> > ср, 17 апр. 2024 г. в 17:21, Rainer Jung :
> >>
> >> Am 17.04.24 um 15:34 schrieb Michael Osipov:
> >>> Rainer, I do not fully understand the problem here. We use libtool to
> >>> solve exactly this problem with versioned SONAMEs. It will create
> >>> symlinks to the SONAME.
> >>> Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION}
> >>> unless it is strictly needed?
> >>>
> >>> E.g.:
> >>> lrwxr-xr-x 1 root wheel26 2024-03-22 10:20
> >>> /usr/lib/libcrypto.so@ -> ../../lib/libcrypto.so.111
> >>> lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@
> >>> -> libssl.so.111
> >>> -r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
> >>> and so on...
> >>
> >> Yes, I expect that! anyone is the JVM :(
> >>
> >> The problem is, that the Java API does not care about these well thought
> >> native traditions. You can not open libssl.so.3 using
> >> System.loadlibrary(String name), because whatever you give it as "name"
> >> parameter it will always try to open libname.so. It always prepends
> >> "lib" to name and always suffixes it with plain ".so".
> >>
> >> Yes, it might exist as the first in your list of symlinks, but on most
> >> linux distributions this link is not installed by default, because it is
> >> only needed when doing compilations. So it is only installed when you
> >> install development packages for libs.
> >
> > There are two methods,
> > System.loadLibrary(libname)
> > System.load(filename)
> > or the same methods in Runtime.
> >
> > The second method accepts an absolute path and apparently does no
> > manipulation on the name.
> >
> > There is also System.mapLibraryName().
> >
> > Note that o.a.t.jni.Library constructor uses all those three methods.
> > A code that was added several years ago uses mapLibraryName() and
> > load() to load a library from "catalina.home/bin". It then falls back
> > to the old algorithm that uses loadLibrary().
> >
> >
> > https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/System.html
> > https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/Runtime.html
>
> thanks for your valuable input as always. I am referring to
> java/org/apache/tomcat/util/openssl/openssl_h.java, which only contains
> which on Mac does
>
> System.loadLibrary("ssl");
> SYMBOL_LOOKUP =
> SymbolLookup.loaderLookup().or(Linker.nativeLinker().defaultLookup());
>
> and else
>
> SYMBOL_LOOKUP = SymbolLookup.libraryLookup(System.mapLibraryName("ssl"),
> LIBRARY_ARENA)
> .or(SymbolLookup.loaderLookup())
> .or(Linker.nativeLinker().defaultLookup());
>
> I *think* both attempts do not allow to use a versioned SONAME.
>
> I have not experimented with System.load(filename) and how to combine it
> with SymbolLookup. o.a.t.jni.Library does not use SymbolLookup. But it
> seems SymbolLookup is not restricted to the development library names.
> So there's hope we can improve!
>
> https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/foreign/SymbolLookup.html
Thank you for the references.
Those lines of code are not generated, but are written by Remy.
A comment referencing "https://github.com/sergot/openssl/issues/81; was a clue.
See the changes in February 2024 (starting with 2024-02-08).
SymbolLookup has two methods, just like the pair of methods in System/Runtime:
libraryLookup(String, Arena)
libraryLookup(Path, Arena).
The second method accepts a full path to a library file.
Interesting is that this new API allows unloading the library - that
is what the Arena argument is for.
As we do not plan to unload the library, I think it makes sense to use
the "good old" System.load() / loadLibrary() methods to load the
library, and use the SymbolLookup loaderLookup() factory method.
Best regards,
Konstantin Kolinko
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
Hi Konstantin,
Am 17.04.24 um 21:44 schrieb Konstantin Kolinko:
ср, 17 апр. 2024 г. в 17:21, Rainer Jung :
Am 17.04.24 um 15:34 schrieb Michael Osipov:
Rainer, I do not fully understand the problem here. We use libtool to solve
exactly this problem with versioned SONAMEs. It will create symlinks to the
SONAME.
Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless it
is strictly needed?
E.g.:
lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@ ->
../../lib/libcrypto.so.111
lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
libssl.so.111
-r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
and so on...
Yes, I expect that! anyone is the JVM :(
The problem is, that the Java API does not care about these well thought
native traditions. You can not open libssl.so.3 using
System.loadlibrary(String name), because whatever you give it as "name"
parameter it will always try to open libname.so. It always prepends
"lib" to name and always suffixes it with plain ".so".
Yes, it might exist as the first in your list of symlinks, but on most
linux distributions this link is not installed by default, because it is
only needed when doing compilations. So it is only installed when you
install development packages for libs.
There are two methods,
System.loadLibrary(libname)
System.load(filename)
or the same methods in Runtime.
The second method accepts an absolute path and apparently does no
manipulation on the name.
There is also System.mapLibraryName().
Note that o.a.t.jni.Library constructor uses all those three methods.
A code that was added several years ago uses mapLibraryName() and
load() to load a library from "catalina.home/bin". It then falls back
to the old algorithm that uses loadLibrary().
https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/System.html
https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/Runtime.html
thanks for your valuable input as always. I am referring to
java/org/apache/tomcat/util/openssl/openssl_h.java, which only contains
which on Mac does
System.loadLibrary("ssl");
SYMBOL_LOOKUP =
SymbolLookup.loaderLookup().or(Linker.nativeLinker().defaultLookup());
and else
SYMBOL_LOOKUP = SymbolLookup.libraryLookup(System.mapLibraryName("ssl"),
LIBRARY_ARENA)
.or(SymbolLookup.loaderLookup())
.or(Linker.nativeLinker().defaultLookup());
I *think* both attempts do not allow to use a versioned SONAME.
I have not experimented with System.load(filename) and how to combine it
with SymbolLookup. o.a.t.jni.Library does not use SymbolLookup. But it
seems SymbolLookup is not restricted to the development library names.
So there's hope we can improve!
https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/foreign/SymbolLookup.html
Best regards,
Rainer
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
Am 17.04.24 um 22:46 schrieb Michael Osipov:
On 2024/04/17 14:21:06 Rainer Jung wrote:
Am 17.04.24 um 15:34 schrieb Michael Osipov:
Rainer, I do not fully understand the problem here. We use libtool to solve
exactly this problem with versioned SONAMEs. It will create symlinks to the
SONAME.
Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless it
is strictly needed?
E.g.:
lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@ ->
../../lib/libcrypto.so.111
lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
libssl.so.111
-r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
and so on...
Yes, I expect that! anyone is the JVM :(
The problem is, that the Java API does not care about these well thought
native traditions. You can not open libssl.so.3 using
System.loadlibrary(String name), because whatever you give it as "name"
parameter it will always try to open libname.so. It always prepends
"lib" to name and always suffixes it with plain ".so".
Yes, it might exist as the first in your list of symlinks, but on most
linux distributions this link is not installed by default, because it is
only needed when doing compilations. So it is only installed when you
install development packages for libs.
Ah, now I see your problem, but it looks like a downstream problem of your
distro of choice, no? I wonder how you compile then custom software if .so
isn't present and the linker cannot find it with -L? What if you install the
devel package to have .so link?
I think it is not a distro problem, but a explicit decision I like. I
find it normal, that you separate development packages from production
packages and only install development packages on development systems,
not on production system. Here a development file is needed for a
production process :(
In old times for instance it was forbidden to install a compiler on a
production system. Yeah I know, once Java kicked in this rule was no
longer possible in the Java world. But in the native world it is often
still followed. So if there is no native compiler, you don't need the
devel version of the libraries as well.
Best regards,
Rainer
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
On 2024/04/17 14:21:06 Rainer Jung wrote:
> Am 17.04.24 um 15:34 schrieb Michael Osipov:
> > Rainer, I do not fully understand the problem here. We use libtool to solve
> > exactly this problem with versioned SONAMEs. It will create symlinks to the
> > SONAME.
> > Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless
> > it is strictly needed?
> >
> > E.g.:
> > lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@
> > -> ../../lib/libcrypto.so.111
> > lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
> > libssl.so.111
> > -r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
> > and so on...
>
> Yes, I expect that! anyone is the JVM :(
>
> The problem is, that the Java API does not care about these well thought
> native traditions. You can not open libssl.so.3 using
> System.loadlibrary(String name), because whatever you give it as "name"
> parameter it will always try to open libname.so. It always prepends
> "lib" to name and always suffixes it with plain ".so".
>
> Yes, it might exist as the first in your list of symlinks, but on most
> linux distributions this link is not installed by default, because it is
> only needed when doing compilations. So it is only installed when you
> install development packages for libs.
Ah, now I see your problem, but it looks like a downstream problem of your
distro of choice, no? I wonder how you compile then custom software if .so
isn't present and the linker cannot find it with -L? What if you install the
devel package to have .so link?
M
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
ср, 17 апр. 2024 г. в 17:21, Rainer Jung :
>
> Am 17.04.24 um 15:34 schrieb Michael Osipov:
> > Rainer, I do not fully understand the problem here. We use libtool to solve
> > exactly this problem with versioned SONAMEs. It will create symlinks to the
> > SONAME.
> > Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless
> > it is strictly needed?
> >
> > E.g.:
> > lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@
> > -> ../../lib/libcrypto.so.111
> > lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
> > libssl.so.111
> > -r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
> > and so on...
>
> Yes, I expect that! anyone is the JVM :(
>
> The problem is, that the Java API does not care about these well thought
> native traditions. You can not open libssl.so.3 using
> System.loadlibrary(String name), because whatever you give it as "name"
> parameter it will always try to open libname.so. It always prepends
> "lib" to name and always suffixes it with plain ".so".
>
> Yes, it might exist as the first in your list of symlinks, but on most
> linux distributions this link is not installed by default, because it is
> only needed when doing compilations. So it is only installed when you
> install development packages for libs.
There are two methods,
System.loadLibrary(libname)
System.load(filename)
or the same methods in Runtime.
The second method accepts an absolute path and apparently does no
manipulation on the name.
There is also System.mapLibraryName().
Note that o.a.t.jni.Library constructor uses all those three methods.
A code that was added several years ago uses mapLibraryName() and
load() to load a library from "catalina.home/bin". It then falls back
to the old algorithm that uses loadLibrary().
https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/System.html
https://docs.oracle.com/en/java/javase/22/docs/api/java.base/java/lang/Runtime.html
HTH
Best regards,
Konstantin Kolinko
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
Am 17.04.24 um 15:34 schrieb Michael Osipov:
Rainer, I do not fully understand the problem here. We use libtool to solve
exactly this problem with versioned SONAMEs. It will create symlinks to the
SONAME.
Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless it
is strictly needed?
E.g.:
lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@ ->
../../lib/libcrypto.so.111
lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
libssl.so.111
-r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
and so on...
Yes, I expect that! anyone is the JVM :(
The problem is, that the Java API does not care about these well thought
native traditions. You can not open libssl.so.3 using
System.loadlibrary(String name), because whatever you give it as "name"
parameter it will always try to open libname.so. It always prepends
"lib" to name and always suffixes it with plain ".so".
Yes, it might exist as the first in your list of symlinks, but on most
linux distributions this link is not installed by default, because it is
only needed when doing compilations. So it is only installed when you
install development packages for libs.
Best regards,
Rainer
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Re: Some remarks on panama libssl loading
Rainer, I do not fully understand the problem here. We use libtool to solve
exactly this problem with versioned SONAMEs. It will create symlinks to the
SONAME.
Do you expect anyone even with dlopen() to load libfoo.o.{SOVERSION} unless it
is strictly needed?
E.g.:
lrwxr-xr-x 1 root wheel26 2024-03-22 10:20 /usr/lib/libcrypto.so@ ->
../../lib/libcrypto.so.111
lrwxr-xr-x 1 root wheel 13 2024-03-22 10:20 /usr/lib/libssl.so@ ->
libssl.so.111
-r--r--r-- 1 root wheel 608008 2024-03-22 10:20 /usr/lib/libssl.so.111
and so on...
-
To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org
For additional commands, e-mail: dev-h...@tomcat.apache.org
Some remarks on panama libssl loading
Hi there, I did some small tests using OpenSSL/Panama with TC 10.1.23 for running the unit tests. First: they seem to work well using JDK 22 with OpenSSL 3.0.13, 3.1.5, 3.2.1 and 3.3.0. For JDK 23 the tests are still running, but also look good up to now. But some things around native library loading in the JVM seem to be nasty in principle: 1) Library file name First some (well-known) background info about shared library loading in Linux in general: When one *compiles* a binary or shared object against a library using "-lsomelib" the compile time linker looks for a file named libsomelib.so. It tries to resolve all symbols needed but not defined in the object to compile in these libs. If all goes well, it records the library dependency in the resulting object as NEEDED. This record does not refer to "libsomelib.so" as a name, but instead the so called SONAME of libsomelib.so. That's an internal name of the library often reflecting an API stability version. For example for OpenSSL 3.x the SONAME of libssl.so is libssl.so.3. Now during runtime of the compiled object the runtime linker notices the need for the recorded SONAME (NEEDED libssl.so.3). It then searches a file with *that* name. In the OpenSSL example it is libssl.so.3, not (!) libssl.so. On a "normal" linux system, libssl.so is therefore not installed, only libssl.so.3 (if OpenSSL 3 is installed at all). The file libssl.so (or better the symlink) is only part of the devel package typically installed on development systems, because it is only needed during compile time, not during runtime. Now back to the JVM: The Java API to load a native library (for example System.loadLibrary(String name)) gets the library name as a String parameter. What is this library name? Unfortunately the JVM devs did not do a good job. If you pass the API the String "somelib", they simply prepend it with "lib" and append ".so" and try to load that file. For example when you pass "ssl", they look for libssl.so. There is no way to tell Java to look for libssl.so.3 using this API. On a normal system, the file libssl.so does not exist and should not exist, because it is a development file. As a workaround one has to provide libssl.so, at least as a symlink to libssl.so.3. 2) Indirect dependencies Normally libssl.so(.3) has dependencies itself, e.g. for libcrypto.so.3 (NEEDED libcrypto.so.3). These dependencies are not longer loaded via the Java API but implicitly by the runtime linker. So now, you do not need a file libcrypto.so - you can't even make that work - you really need the file libcrypto.so.3. Very confusing for newbies and pretty inconsistent. 3) java.library.path The system property java.library.path is only used by the JVM when loading the libs requested via the Java API, so here libssl.so. The runtime linker looking then for libcrypto.so.3 is not set up to use java.library.path. So if the libs are not installed in a default system location, you need to set LD_LIBRARY_PATH to point to your libcrypto. If libssl is in the same directory (pretty much always), you then won't need java.library.path at all, because LD_LIBRARY_PATH also influences the search of libssl.so by the Java API. Again confusing. None of this is Tomcat's fault. They are all deficiencies of the (old) Java APIs which were IMHO misdesigned. It might be different on Windows or Macs. Just wanted to mention it in case others run into the same nits. We might want to document something though before raising more awareness for using OpenSSL via panama. Best regards, Rainer - To unsubscribe, e-mail: dev-unsubscr...@tomcat.apache.org For additional commands, e-mail: dev-h...@tomcat.apache.org
