RE: Can't link a static library with custom OpenSSL rsa engine

[Scott Neugroschl]

You need to put the static library at the END of your link command.  A static 
library is searched when it is encountered in the link stream, and only the 
items needed will be used from it.

Because you have it first, there are no undefined symbols, and no items will be 
used from it.

From: openssl-users  On Behalf Of Shariful 
Alam
Sent: Tuesday, November 17, 2020 12:40 PM
To: openssl-users@openssl.org
Subject: Can't link a static library with custom OpenSSL rsa engine

Hello,
I have a custom rsa engine. It builds and works fine. Later, I have added a 
static library with my custom engine code. My code compiles. However, when I 
try to load the custom engine it shows invalid engine "rsa-engine-new".  The 
full error is given below,
x@x:~/Downloads/x/x/x/rsa_engine$ openssl rsautl -decrypt -inkey private.pem 
-in msg.enc -engine rsa-engine-new
invalid engine "rsa-engine-new"
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not load 
the shared 
library:crypto/dso/dso_dlfcn.c:119:filename(/opt/openssl/lib/engines-1.1/rsa-engine-new.so):
 
/opt/openssl/lib/engines-1.1/rsa-engine-new.so:
 undefined symbol: dune_init
140112744122112:error:25070067:DSO support routines:DSO_load:could not load the 
shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not 
found:crypto/engine/eng_dyn.c:414:
140112744122112:error:2606A074:engine routines:ENGINE_by_id:no such 
engine:crypto/engine/eng_list.c:334:id=rsa-engine-new
140112744122112:error:25066067:DSO support routines:dlfcn_load:could not load 
the shared 
library:crypto/dso/dso_dlfcn.c:119:filename(librsa-engine-new.so):
 
librsa-engine-new.so:
 cannot open shared object file: No such file or directory
140112744122112:error:25070067:DSO support routines:DSO_load:could not load the 
shared library:crypto/dso/dso_lib.c:162:
140112744122112:error:260B6084:engine routines:dynamic_load:dso not 
found:crypto/engine/eng_dyn.c:414:

Now the error doesn't say much about the cause of invalid engine. However my 
guess is it is from the  "undefined symbol: dune_init". "dune_init" is from the 
static library. Therefire I believe my linking is not working. I use the 
following Makefile to compile the engine,

  1.  rsa-engine: rsa/rsa.c rsa/bignum.c rsa/aes.c rsa/x509parse.c rsa/pem.c
  2.  gcc -fPIC -o rsa/rsa.o -c rsa/rsa.c
  3.  gcc -fPIC -o rsa/bignum.o -c rsa/bignum.c
  4.  gcc -fPIC -o rsa/aes.o -c rsa/aes.c
  5.  gcc -fPIC -o rsa/x509parse.o -c rsa/x509parse.c
  6.  gcc -fPIC -o rsa/pem.o -c rsa/pem.c
  7.  gcc -fPIC -c rsa-engine.c
  8.  gcc -shared -o 
librsa_engine.so
 libdune/libdune.a -lcrypto rsa-engine.o rsa/rsa.o rsa/bignum.o rsa/aes.o 
rsa/x509parse.o rsa/pem.o
  9.  mv 
librsa_engine.so
 
rsa-engine-new.so
  10. sudo cp 
rsa-engine-new.so
 /opt/openssl/lib/engines-1.1/
  11. clean:
  12. rm -f *.o rsa/*.o *.so rsa-engine
So, can anyone please if my guess is correct or not? If my guess is correct, 
how can I fix my Makefile?

N.B: Static library

  *   libdune/libdune.a is in the same directory with the main rsa-engine.c
  *   libdune/libdune.a is compiled with -fPIC flag

Thanks,
Shariful

RE: Can't link a static library with custom OpenSSL rsa engine

[Heberlein, Kurt William]

You might try changing this:


8. gcc -shared -o librsa_engine.so libdune/libdune.a -lcrypto rsa-engine.o 
rsa/rsa.o rsa/bignum.o rsa/aes.o rsa/x509parse.o rsa/pem.o
to this:
   gcc –shared –o librsa_engine.so –L./libdune rsa_engine.o 
rsa/rsa.o rsa/bignum.o rsa/aes.o rsa/x509parse.o rsa/pem.o –Wl,-Bstatic 
–llibdune –Wl,-Bdynamic –lcrypto

just a guess.  cheers