I also get the "Error looking up function 'ENGINE_load_rdrand'" when I run the Jna test class from the command line. Interesting. dlopen finds the LibreSSL when run from the CLI.
On Thu, Jun 30, 2022 at 8:02 AM sebb <[email protected]> wrote: > That looks fine, however I don't see the same, and nor does GH. > > What version of macOS are you using? > > On Thu, 30 Jun 2022 at 12:52, Gary Gregory <[email protected]> wrote: > > > > With the 1.1.0 tagm I get: > > > > garydgregory@xxx ~/git/commons-crypto ➦ 3b2561b java -cp > > target/classes org.apache.commons.crypto.Crypto > > Apache Commons Crypto 1.1.0 > > Native code loaded OK: 1.1.0 > > Native name: Apache Commons Crypto > > Native built: Jun 30 2022 > > OpenSSL library loaded OK, version: 0x1010107f > > OpenSSL library info: OpenSSL 1.1.1g 21 Apr 2020 > > Random instance created OK: > > org.apache.commons.crypto.random.OpenSslCryptoRandom@266474c2 > > Cipher AES/CTR/NoPadding instance created OK: > > org.apache.commons.crypto.cipher.OpenSslCipher@66d3c617 > > Additional OpenSSL_version(n) details: > > 1: compiler: clang -fPIC -arch x86_64 -O3 -Wall -DL_ENDIAN > > -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 > > -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m > > -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM > > -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM > > -DX25519_ASM -DPOLY1305_ASM -D_REENTRANT -DNDEBUG > > 2: built on: Tue Apr 21 13:30:00 2020 UTC > > 3: platform: darwin64-x86_64-cc > > 4: OPENSSLDIR: "/usr/local/etc/[email protected]" > > 5: ENGINESDIR: "/usr/local/Cellar/[email protected]/1.1.1g/lib/engines-1.1" > > > > and > > > > garydgregory@xxx ~/git/commons-crypto ➦ 3b2561b mvn -q exec:java > > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > isEnabled(): true > > initialisationError(): null > > > > Gary > > > > On Thu, Jun 30, 2022 at 7:18 AM sebb <[email protected]> wrote: > > > > > > Yes, 1.1.0 builds OK on macOS, but the sample classes fail. > > > > > > As previously noted, please try: > > > java -cp target/classes org.apache.commons.crypto.Crypto > > > and > > > mvn -q exec:java > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > > > > > See the GH run I linked yesterday: > > > https://github.com/apache/commons-crypto/actions/runs/2586011129 > > > > > > I just noticed also that 'openssl version' shows 1.1.1p but the Crypto > > > class shows > > > OpenSSL library info: LibreSSL 2.2.9 (on current code; the 1.11.0 code > crashes) > > > > > > Something strange is going on; it looks like dlopen is finding a > > > different version of the dynamic library from the default CLI. > > > I have the same issue on my mac. > > > > > > On Thu, 30 Jun 2022 at 01:50, Gary Gregory <[email protected]> > wrote: > > > > > > > > FWIW, I just checked out the rel/commons-crypto-1.1.0 on macOS and > ran > > > > 'mvn clean verify' and everything built just fine. > > > > > > > > The Maven ant-run output was: > > > > > > > > [INFO] --- maven-antrun-plugin:1.8:run (make) @ commons-crypto --- > > > > [INFO] Executing tasks > > > > > > > > make: > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > -classpath target/classes -o > > > > > target/jni-classes/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.h > > > > org.apache.commons.crypto.random.OpenSslCryptoRandomNative > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > -I"/usr/local/Cellar/openjdk@8 > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -c > > > > > src/main/native/org/apache/commons/crypto/random/OpenSslCryptoRandomNative.c > > > > -o target/commons-crypto-1.1.0-Mac-x86_64/OpenSslCryptoRandomNative.o > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > -classpath target/classes -o > > > > target/jni-classes/org/apache/commons/crypto/cipher/OpenSslNative.h > > > > org.apache.commons.crypto.cipher.OpenSslNative > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > -I"/usr/local/Cellar/openjdk@8 > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -c > > > > src/main/native/org/apache/commons/crypto/cipher/OpenSslNative.c -o > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslNative.o > > > > [exec] "/usr/local/Cellar/openjdk@8/1.8.0+322/bin/javah" -force > > > > -classpath target/classes -o > > > > target/jni-classes/org/apache/commons/crypto/OpenSslInfoNative.h > > > > org.apache.commons.crypto.OpenSslInfoNative > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > -I"src/main/native/org/apache/commons/crypto/" > > > > -I"/usr/local/Cellar/openjdk@8 > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > -I"target/jni-classes/org/apache/commons/crypto/random" > > > > -DVERSION='"1.1.0"' -DPROJECT_NAME='"Apache Commons Crypto"' > > > > -I"target/jni-classes/org/apache/commons/crypto" -c > > > > src/main/native/org/apache/commons/crypto/OpenSslInfoNative.c -o > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslInfoNative.o > > > > [exec] gcc -arch x86_64 -Ilib/inc_mac > > > > -I"/usr/local/Cellar/openjdk@8/1.8.0+322/include" -O2 -fPIC > > > > -mmacosx-version-min=10.7 -fvisibility=hidden -I/usr/local/include > > > > -I/usr/local/opt/openssl/include -Ilib/include -I/usr/include > > > > -I"/usr/local/Cellar/openjdk@8 > /1.8.0+322/libexec/openjdk.jdk/Contents/Home/include/darwin" > > > > -I"target/jni-classes/org/apache/commons/crypto/cipher" > > > > -I"target/jni-classes/org/apache/commons/crypto/random" -o > > > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslCryptoRandomNative.o > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslNative.o > > > > target/commons-crypto-1.1.0-Mac-x86_64/OpenSslInfoNative.o > -dynamiclib > > > > -L/usr/local/lib > > > > [exec] strip -x > > > > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > [exec] cp > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > target/classes/org/apache/commons/crypto/native/Mac/x86_64/libcommons-crypto.jnilib > > > > [exec] cp > target/commons-crypto-1.1.0-Mac-x86_64/libcommons-crypto.jnilib > > > > > target/classes/org/apache/commons/crypto/native/Mac/x86_64/libcommons-crypto.jnilib > > > > > > > > Gary > > > > > > > > On Wed, Jun 29, 2022 at 8:48 PM Gary Gregory <[email protected]> > wrote: > > > > > > > > > > I agree with Alex. > > > > > > > > > > Forget LibreSSL. Commons Crypto is for OpenSSL, nice, simple, and > > > > > tight. Last time I checked I had an antique version of LibreSSL on > my > > > > > mac years ago, I just installed OpenSSL and never looked back. > > > > > > > > > > Gary > > > > > > > > > > On Wed, Jun 29, 2022 at 8:11 PM Alex Remily <[email protected]> > wrote: > > > > > > > > > > > > <The question is whether to do anything about the previously > undetected > > > > > > issues with JNA on macOS and Windows, and if so, whether this > needs to > > > > > > be done for the current or a later release. If we don't fix this > > > > > > release, obviously it needs some mention in the release notes.> > > > > > > > > > > > > I wouldn't characterize the issues running against LibreSSL as > > > > > > "undetected". I also don't see this as an issue with Mac or > Windows, but > > > > > > with LibreSSL. Install any supported OpenSSL library on any > supported > > > > > > architecture (to include Mac and Windows) and all commons crypto > > > > > > functionality is available. I first encountered the rand issue > you > > > > > > describe years ago when I was becoming familiar with commons > crypto. I did > > > > > > a little research, discovered that I was running LibreSSL (and > an old > > > > > > version at that), installed a supported version of OpenSSL and > forgot > > > > > > all about it until this thread. I don't think it's unreasonable > to expect > > > > > > users to install a supported version of OpenSSL on a supported > architecture > > > > > > as a precondition of using commons crypto. I think it could > become > > > > > > cumbersome to try and support every vendor default *SSL > install. That > > > > > > said, I don't have a problem committing this particular "fix" to > the > > > > > > baseline, particularly since you have already done the work. I > just don't > > > > > > think that the project should obligate itself to do so or > advertise > > > > > > LibreSSL (or any other non-OpenSSL branch) support as such. I'm > advocating > > > > > > a "use at your own risk" approach to anything but OpenSSL > proper. I agree > > > > > > that we should update the documentation to reflect whatever we > move forward > > > > > > with. > > > > > > > > > > > > Anyway, that's my $0.02 worth. > > > > > > > > > > > > Alex > > > > > > > > > > > > On Wed, Jun 29, 2022 at 6:14 PM sebb <[email protected]> wrote: > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 18:06, Gary Gregory < > [email protected]> wrote: > > > > > > > > > > > > > > > > We cannot remove support for Windows and macOS, that's silly. > > > > > > > > > > > > > > AFAICT that means we must support the different set of > function names > > > > > > > in LibreSSL. > > > > > > > Note that we only currently use a small proportion of them. > > > > > > > > > > > > > > > I have not followed all the branches and commits, so I'm not > sure what > > > > > > > the > > > > > > > > current problems are, but I know I was able to release all > OSs last go > > > > > > > > around. I don't see why we need to rip out anything as > fundamental. > > > > > > > > > > > > > > Well, I have tried running the Crypto and OpenSslJna main > classes on > > > > > > > macOS and Windows, and they both fail with the 1.1.0 release. > > > > > > > > > > > > > > With current master, Crypto succeeds, but OpenSslJna fails to > find > > > > > > > ENGINE_load_rdrand on both macOS and Windows. > > > > > > > (The job step succeeds, because the code catches the exception) > > > > > > > > > > > > > > It looks like the test which would have exposed at least one > of the > > > > > > > issues was never enabled because of a failures on macOS; this > hid the > > > > > > > same problem on Windows. > > > > > > > > > > > > > > I am not suggesting we rip out anything at this point. > > > > > > > > > > > > > > The question is whether to do anything about the previously > undetected > > > > > > > issues with JNA on macOS and Windows, and if so, whether this > needs to > > > > > > > be done for the current or a later release. If we don't fix > this > > > > > > > release, obviously it needs some mention in the release notes. > > > > > > > > > > > > > > Sebb > > > > > > > > https://github.com/apache/commons-crypto/actions/runs/2586011129 > > > > > > > > Gary > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022, 12:00 sebb <[email protected]> wrote: > > > > > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 16:11, Alex Remily < > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > I agree with Gary that we just don't support LibreSSL. > To my > > > > > > > knowledge > > > > > > > > > > we've never advertised LibreSSL support, so I don't see > it as an > > > > > > > issue. > > > > > > > > > > > > > > > > > > In that case AFAICT we will have to drop *all* support for > macOS and > > > > > > > > > Windows, as they both seem to default to LibreSSL. > > > > > > > > > > > > > > > > > > Note that adding support for LibreSSL was much easier for > JNI, as it > > > > > > > > > uses far fewer methods. > > > > > > > > > Rather than checking the version, I changed the code to > try OpenSSL > > > > > > > > > 1.1 names first, then a fallback. > > > > > > > > > That happens to work for 1.0.x and 1.1.x and LibreSSL. > > > > > > > > > > > > > > > > > > If you want to try it out, compare 16345bc (old) with > 3ee3f65 (new) > > > > > > > > > macOS fails on 16345bc because it now uses LibreSSL which > has a > > > > > > > > > different mix of names. > > > > > > > > > > > > > > > > > > I think it's vital we support JNI as far as possible (and > the code > > > > > > > > > already does with commit 3ee3f65). > > > > > > > > > > > > > > > > > > However JNA is more of a backstop, so the fact that it has > stopped > > > > > > > > > working for macOS and Windows is less of an issue. > > > > > > > > > > > > > > > > > > But I don't think we can say we are not supporting > LibreSSL at all. > > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022 at 10:21 AM sebb <[email protected]> > wrote: > > > > > > > > > > > > > > > > > > > > > On Wed, 29 Jun 2022 at 14:17, Gary Gregory < > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > The simple solution is to keep doing what we do now: > only support > > > > > > > > > OpenSSL > > > > > > > > > > > > and not whatever Apple does with LibreSSL which may > or may not > > > > > > > be up > > > > > > > > > to > > > > > > > > > > > > date. > > > > > > > > > > > > > > > > > > > > > > I think this also affects Windows. > > > > > > > > > > > > > > > > > > > > > > I don't have Windows box at present, but I have seen > this on GH > > > > > > > > > Actions. > > > > > > > > > > > > > > > > > > > > > > If you have a WIndows build, perhaps you could try > these tests: > > > > > > > > > > > > > > > > > > > > > > mvn -q exec:java > > > > > > > -D"exec.mainClass=org.apache.commons.crypto.Crypto" > > > > > > > > > > > mvn -q exec:java > > > > > > > > > > > > -D"exec.mainClass=org.apache.commons.crypto.jna.OpenSslJna" > > > > > > > > > > > > > > > > > > > > > > The first one should show the SSL details; on GH the > output > > > > > > > includes: > > > > > > > > > > > > > > > > > > > > > > OpenSSL library loaded OK, version: 0x20000000 > > > > > > > > > > > OpenSSL library info: LibreSSL 3.0.2 > > > > > > > > > > > Additional OpenSSL_version(n) details: > > > > > > > > > > > 4: OPENSSLDIR: "C:/Windows/libressl/ssl" > > > > > > > > > > > > > > > > > > > > > > The second test crashes with: > > > > > > > > > > > java.lang.UnsatisfiedLinkError: Error looking up > function > > > > > > > > > > > 'ENGINE_load_rdrand': The specified procedure could > not be found. > > > > > > > > > > > isEnabled(): false > > > > > > > > > > > > > > > > > > > > > > initialisationError(): java.lang.UnsatisfiedLinkError: > Error > > > > > > > looking > > > > > > > > > > > up function 'ENGINE_load_rdrand': The specified > procedure could > > > > > > > not be > > > > > > > > > > > found. > > > > > > > > > > > at com.sun.jna.Function.<init>(Function.java:252) > > > > > > > > > > > ... > > > > > > > > > > > > > > > > > > > > > > It would certainly be easier to ignore the problem for > now. > > > > > > > > > > > > > > > > > > > > > > > Gary > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Jun 29, 2022, 06:59 sebb <[email protected]> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > It looks like macOS 10.5+ and Windows (latest) use > LibreSSL by > > > > > > > > > default > > > > > > > > > > > > > rather than OpenSSL. > > > > > > > > > > > > > > > > > > > > > > > > > > The LibreSSL API does not have the same functions > as either > > > > > > > 1.0.2 > > > > > > > > > or > > > > > > > > > > > > > 1.1.1, so needs its own JNA class. In particular > it looks like > > > > > > > > > > > > > ENGINE_load_rdrand is not present, nor is > OpenSSL_version_num; > > > > > > > > > 1.0.2 > > > > > > > > > > > > > has the former only, and 1.1.1 has the latter only. > > > > > > > > > > > > > > > > > > > > > > > > > > This makes it hard to support JNA with the current > design. > > > > > > > > > > > > > It would require another OpenSsl<version>NativeJna > class, and > > > > > > > the > > > > > > > > > > > > > parent class OpenSslNativeJna would need to use > yet another > > > > > > > > > condition > > > > > > > > > > > > > in each of its methods. > > > > > > > > > > > > > > > > > > > > > > > > > > This is quite tedious and error-prone. > > > > > > > > > > > > > > > > > > > > > > > > > > Seems to me it would be better to use something > like a set of > > > > > > > > > > > > > singleton classes that implement the required > methods. The > > > > > > > > > appropriate > > > > > > > > > > > > > one can then be initialised and used by > OpenSslNativeJna to > > > > > > > field > > > > > > > > > the > > > > > > > > > > > > > actual methods. i.e. replace the conditional logic > with a > > > > > > > static > > > > > > > > > > > > > reference to the correct API interface instance. > This should > > > > > > > only > > > > > > > > > > > > > affect non-public classes. > > > > > > > > > > > > > > > > > > > > > > > > > > Any other suggestions? > > > > > > > > > > > > > > > > > > > > > > > > > > Sebb > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > > > To unsubscribe, e-mail: > [email protected] > > > > > > > > > > > > > For additional commands, e-mail: > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > > > To unsubscribe, e-mail: > [email protected] > > > > > > > > > > > For additional commands, e-mail: > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > > > For additional commands, e-mail: > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: [email protected] > > > > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
