I've started copying bits of the C code to create a standalone tool to load the dll and report some info.
No idea why yet, but it behaves differently from basically the same code when run via JNI. For example, the standalone code takes note of LD_LIBRARY_PATH when loading the dll, whereas Java apparently does not. On Thu, 30 Jun 2022 at 13:22, Alex Remily <alex.rem...@gmail.com> wrote: > > I also get the "Error looking up function 'ENGINE_load_rdrand'" when I run > the Jna test class from the command line. Interesting. dlopen finds the > LibreSSL when run from the CLI. > > On Thu, Jun 30, 2022 at 8:02 AM sebb <seb...@gmail.com> wrote: > > > That looks fine, however I don't see the same, and nor does GH. > > > > What version of macOS are you using? > > > > On Thu, 30 Jun 2022 at 12:52, Gary Gregory <garydgreg...@gmail.com> wrote: > > > > > > With the 1.1.0 tagm I get: > > > > > > garydgregory@xxx ~/git/commons-crypto ➦ 3b2561b java -cp > > > target/classes org.apache.commons.crypto.Crypto > > > Apache Commons Crypto 1.1.0 > > > Native code loaded OK: 1.1.0 > > > Native name: Apache Commons Crypto > > > Native built: Jun 30 2022 > > > OpenSSL library loaded OK, version: 0x1010107f > > > OpenSSL library info: OpenSSL 1.1.1g 21 Apr 2020 > > > Random instance created OK: > > > org.apache.commons.crypto.random.OpenSslCryptoRandom@266474c2 > > > Cipher AES/CTR/NoPadding instance created OK: > > > org.apache.commons.crypto.cipher.OpenSslCipher@66d3c617 > > > Additional OpenSSL_version(n) details: > > > 1: compiler: clang -fPIC -arch x86_64 -O3 -Wall -DL_ENDIAN > > > -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 > > > -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m > > > -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM > > > -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM > > > -DX25519_ASM -DPOLY1305_ASM -D_REENTRANT -DNDEBUG > > > 2: built on: Tue Apr 21 13:30:00 2020 UTC > > > 3: platform: darwin64-x86_64-cc > > > 4: OPENSSLDIR: "/usr/local/etc/openssl@1.1" > > > 5: ENGINESDIR: "/usr/local/Cellar/openssl@1.1/1.1.1g/lib/engines-1.1" > > > > > > and > > > > > > garydgregory@xxx ~/git/commons-crypto ➦ 3b2561b mvn -q exec:java > > > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > > isEnabled(): true > > > initialisationError(): null > > > > > > Gary > > > > > > On Thu, Jun 30, 2022 at 7:18 AM sebb <seb...@gmail.com> wrote: > > > > > > > > Yes, 1.1.0 builds OK on macOS, but the sample classes fail. > > > > > > > > As previously noted, please try: > > > > java -cp target/classes org.apache.commons.crypto.Crypto > > > > and > > > > mvn -q exec:java > > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > > > > > > > See the GH run I linked yesterday: > > > > https://github.com/apache/commons-crypto/actions/runs/2586011129 > > > > > > > > I just noticed also that 'openssl version' shows 1.1.1p but the Crypto > > > > class shows > > > > OpenSSL library info: LibreSSL 2.2.9 (on current code; the 1.11.0 code > > crashes) > > > > > > > > Something strange is going on; it looks like dlopen is finding a > > > > different version of the dynamic library from the default CLI. > > > > I have the same issue on my mac. > > > > > > > > On Thu, 30 Jun 2022 at 01:50, Gary Gregory <garydgreg...@gmail.com> > > wrote: > > > > > > > > > > FWIW, I just checked out the rel/commons-crypto-1.1.0 on macOS and > > ran > > > > > 'mvn clean verify' and everything built just fine. > > > > > > > > > > The Maven ant-run output was: > > > > > > > > > > [INFO] --- maven-antrun-plugin:1.8:run (make) @ commons-crypto --- > > > > > [INFO] Executing tasks > > > > > > > > > > make: > > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > > -classpath target/classes -o > > > > > > > target/jni-classes/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.h > > > > > org.apache.commons.crypto.random.OpenSslCryptoRandomNative > > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > > -I"/usr/local/Cellar/openjdk@8 > > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -c > > > > > > > src/main/native/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.c > > > > > -o target/commons-crypto-1.1.0-Mac-x86_64/OpenSslCryptoRandomNative.o > > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > > -classpath target/classes -o > > > > > target/jni-classes/org/apache/commons/crypto/cipher/OpenSslNative.h > > > > > org.apache.commons.crypto.cipher.OpenSslNative > > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > > -I"/usr/local/Cellar/openjdk@8 > > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -c > > > > > src/main/native/org/apache/commons/crypto/cipher/OpenSslNative.c -o > > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslNative.o > > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > > -classpath target/classes -o > > > > > target/jni-classes/org/apache/commons/crypto/OpenSslInfoNative.h > > > > > org.apache.commons.crypto.OpenSslInfoNative > > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > > -I"/usr/local/Cellar/openjdk@8 > > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > > -I"target/jni-classes/org/apache/commons/crypto/random" > > > > > -DVERSION='"1.1.0"' -DPROJECT_NAME='"Apache Commons Crypto"' > > > > > -I"target/jni-classes/org/apache/commons/crypto" -c > > > > > src/main/native/org/apache/commons/crypto/OpenSslInfoNative.c -o > > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslInfoNative.o > > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > > -I"/usr/local/Cellar/openjdk@8 > > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -o > > > > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslCryptoRandomNative.o > > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslNative.o > > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslInfoNative.o > > -dynamiclib > > > > > -L/usr/local/lib > > > > > [exec] strip -x > > > > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > [exec] cp > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > > > target/classes/org/apache/commons/crypto/native/Mac/x86_64/libcommons-crypto.jnilib > > > > > [exec] cp > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > > > target/classes/org/apache/commons/crypto/native/Mac/x86_64/libcommons-crypto.jnilib > > > > > > > > > > Gary > > > > > > > > > > On Wed, Jun 29, 2022 at 8:48 PM Gary Gregory <garydgreg...@gmail.com> > > wrote: > > > > > > > > > > > > I agree with Alex. > > > > > > > > > > > > Forget LibreSSL. Commons Crypto is for OpenSSL, nice, simple, and > > > > > > tight. Last time I checked I had an antique version of LibreSSL on > > my > > > > > > mac years ago, I just installed OpenSSL and never looked back. > > > > > > > > > > > > Gary > > > > > > > > > > > > On Wed, Jun 29, 2022 at 8:11 PM Alex Remily <alex.rem...@gmail.com> > > wrote: > > > > > > > > > > > > > > <The question is whether to do anything about the previously > > undetected > > > > > > > issues with JNA on macOS and Windows, and if so, whether this > > needs to > > > > > > > be done for the current or a later release. If we don't fix this > > > > > > > release, obviously it needs some mention in the release notes.> > > > > > > > > > > > > > > I wouldn't characterize the issues running against LibreSSL as > > > > > > > "undetected". I also don't see this as an issue with Mac or > > Windows, but > > > > > > > with LibreSSL. Install any supported OpenSSL library on any > > supported > > > > > > > architecture (to include Mac and Windows) and all commons crypto > > > > > > > functionality is available. I first encountered the rand issue > > you > > > > > > > describe years ago when I was becoming familiar with commons > > crypto. I did > > > > > > > a little research, discovered that I was running LibreSSL (and > > an old > > > > > > > version at that), installed a supported version of OpenSSL and > > forgot > > > > > > > all about it until this thread. I don't think it's unreasonable > > to expect > > > > > > > users to install a supported version of OpenSSL on a supported > > architecture > > > > > > > as a precondition of using commons crypto. I think it could > > become > > > > > > > cumbersome to try and support every vendor default *SSL > > install. That > > > > > > > said, I don't have a problem committing this particular "fix" to > > the > > > > > > > baseline, particularly since you have already done the work. I > > just don't > > > > > > > think that the project should obligate itself to do so or > > advertise > > > > > > > LibreSSL (or any other non-OpenSSL branch) support as such. I'm > > advocating > > > > > > > a "use at your own risk" approach to anything but OpenSSL > > proper. I agree > > > > > > > that we should update the documentation to reflect whatever we > > move forward > > > > > > > with. > > > > > > > > > > > > > > Anyway, that's my $0.02 worth. > > > > > > > > > > > > > > Alex > > > > > > > > > > > > > > On Wed, Jun 29, 2022 at 6:14 PM sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 18:06, Gary Gregory < > > garydgreg...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > We cannot remove support for Windows and macOS, that's silly. > > > > > > > > > > > > > > > > AFAICT that means we must support the different set of > > function names > > > > > > > > in LibreSSL. > > > > > > > > Note that we only currently use a small proportion of them. > > > > > > > > > > > > > > > > > I have not followed all the branches and commits, so I'm not > > sure what > > > > > > > > the > > > > > > > > > current problems are, but I know I was able to release all > > OSs last go > > > > > > > > > around. I don't see why we need to rip out anything as > > fundamental. > > > > > > > > > > > > > > > > Well, I have tried running the Crypto and OpenSslJna main > > classes on > > > > > > > > macOS and Windows, and they both fail with the 1.1.0 release. > > > > > > > > > > > > > > > > With current master, Crypto succeeds, but OpenSslJna fails to > > find > > > > > > > > ENGINE_load_rdrand on both macOS and Windows. > > > > > > > > (The job step succeeds, because the code catches the exception) > > > > > > > > > > > > > > > > It looks like the test which would have exposed at least one > > of the > > > > > > > > issues was never enabled because of a failures on macOS; this > > hid the > > > > > > > > same problem on Windows. > > > > > > > > > > > > > > > > I am not suggesting we rip out anything at this point. > > > > > > > > > > > > > > > > The question is whether to do anything about the previously > > undetected > > > > > > > > issues with JNA on macOS and Windows, and if so, whether this > > needs to > > > > > > > > be done for the current or a later release. If we don't fix > > this > > > > > > > > release, obviously it needs some mention in the release notes. > > > > > > > > > > > > > > > > Sebb > > > > > > > > > > https://github.com/apache/commons-crypto/actions/runs/2586011129 > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022, 12:00 sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 16:11, Alex Remily < > > alex.rem...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > I agree with Gary that we just don't support LibreSSL. > > To my > > > > > > > > knowledge > > > > > > > > > > > we've never advertised LibreSSL support, so I don't see > > it as an > > > > > > > > issue. > > > > > > > > > > > > > > > > > > > > In that case AFAICT we will have to drop *all* support for > > macOS and > > > > > > > > > > Windows, as they both seem to default to LibreSSL. > > > > > > > > > > > > > > > > > > > > Note that adding support for LibreSSL was much easier for > > JNI, as it > > > > > > > > > > uses far fewer methods. > > > > > > > > > > Rather than checking the version, I changed the code to > > try OpenSSL > > > > > > > > > > 1.1 names first, then a fallback. > > > > > > > > > > That happens to work for 1.0.x and 1.1.x and LibreSSL. > > > > > > > > > > > > > > > > > > > > If you want to try it out, compare 16345bc (old) with > > 3ee3f65 (new) > > > > > > > > > > macOS fails on 16345bc because it now uses LibreSSL which > > has a > > > > > > > > > > different mix of names. > > > > > > > > > > > > > > > > > > > > I think it's vital we support JNI as far as possible (and > > the code > > > > > > > > > > already does with commit 3ee3f65). > > > > > > > > > > > > > > > > > > > > However JNA is more of a backstop, so the fact that it has > > stopped > > > > > > > > > > working for macOS and Windows is less of an issue. > > > > > > > > > > > > > > > > > > > > But I don't think we can say we are not supporting > > LibreSSL at all. > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022 at 10:21 AM sebb <seb...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 14:17, Gary Gregory < > > garydgreg...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > The simple solution is to keep doing what we do now: > > only support > > > > > > > > > > OpenSSL > > > > > > > > > > > > > and not whatever Apple does with LibreSSL which may > > or may not > > > > > > > > be up > > > > > > > > > > to > > > > > > > > > > > > > date. > > > > > > > > > > > > > > > > > > > > > > > > I think this also affects Windows. > > > > > > > > > > > > > > > > > > > > > > > > I don't have Windows box at present, but I have seen > > this on GH > > > > > > > > > > Actions. > > > > > > > > > > > > > > > > > > > > > > > > If you have a WIndows build, perhaps you could try > > these tests: > > > > > > > > > > > > > > > > > > > > > > > > mvn -q exec:java > > > > > > > > -D"exec.mainClass=org.apache.commons.crypto.Crypto" > > > > > > > > > > > > mvn -q exec:java > > > > > > > > > > > > > > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > > > > > > > > > > > > > > > > > > > > > > > The first one should show the SSL details; on GH the > > output > > > > > > > > includes: > > > > > > > > > > > > > > > > > > > > > > > > OpenSSL library loaded OK, version: 0x20000000 > > > > > > > > > > > > OpenSSL library info: LibreSSL 3.0.2 > > > > > > > > > > > > Additional OpenSSL_version(n) details: > > > > > > > > > > > > 4: OPENSSLDIR: "C:/Windows/libressl/ssl" > > > > > > > > > > > > > > > > > > > > > > > > The second test crashes with: > > > > > > > > > > > > java.lang.UnsatisfiedLinkError: Error looking up > > function > > > > > > > > > > > > 'ENGINE_load_rdrand': The specified procedure could > > not be found. > > > > > > > > > > > > isEnabled(): false > > > > > > > > > > > > > > > > > > > > > > > > initialisationError(): java.lang.UnsatisfiedLinkError: > > Error > > > > > > > > looking > > > > > > > > > > > > up function 'ENGINE_load_rdrand': The specified > > procedure could > > > > > > > > not be > > > > > > > > > > > > found. > > > > > > > > > > > > at com.sun.jna.Function.<init>(Function.java:252) > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > > > It would certainly be easier to ignore the problem for > > now. > > > > > > > > > > > > > > > > > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022, 06:59 sebb <seb...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > It looks like macOS 10.5+ and Windows (latest) use > > LibreSSL by > > > > > > > > > > default > > > > > > > > > > > > > > rather than OpenSSL. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The LibreSSL API does not have the same functions > > as either > > > > > > > > 1.0.2 > > > > > > > > > > or > > > > > > > > > > > > > > 1.1.1, so needs its own JNA class. In particular > > it looks like > > > > > > > > > > > > > > ENGINE_load_rdrand is not present, nor is > > OpenSSL_version_num; > > > > > > > > > > 1.0.2 > > > > > > > > > > > > > > has the former only, and 1.1.1 has the latter only. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This makes it hard to support JNA with the current > > design. > > > > > > > > > > > > > > It would require another OpenSsl<version>NativeJna > > class, and > > > > > > > > the > > > > > > > > > > > > > > parent class OpenSslNativeJna would need to use > > yet another > > > > > > > > > > condition > > > > > > > > > > > > > > in each of its methods. > > > > > > > > > > > > > > > > > > > > > > > > > > > > This is quite tedious and error-prone. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Seems to me it would be better to use something > > like a set of > > > > > > > > > > > > > > singleton classes that implement the required > > methods. The > > > > > > > > > > appropriate > > > > > > > > > > > > > > one can then be initialised and used by > > OpenSslNativeJna to > > > > > > > > field > > > > > > > > > > the > > > > > > > > > > > > > > actual methods. i.e. replace the conditional logic > > with a > > > > > > > > static > > > > > > > > > > > > > > reference to the correct API interface instance. > > This should > > > > > > > > only > > > > > > > > > > > > > > affect non-public classes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any other suggestions? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sebb > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > > To unsubscribe, e-mail: > > dev-unsubscr...@commons.apache.org > > > > > > > > > > > > > > For additional commands, e-mail: > > dev-h...@commons.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > To unsubscribe, e-mail: > > dev-unsubscr...@commons.apache.org > > > > > > > > > > > > For additional commands, e-mail: > > dev-h...@commons.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > > > > > > > For additional commands, e-mail: > > dev-h...@commons.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > > For additional commands, e-mail: dev-h...@commons.apache.org > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
