>From my testing, it appears that EC keys that are managed by an engine (in my case, libp11 <https://github.com/OpenSC/libp11>) are not presently supported by OpenSSL when the engine is statically linked. This appears primarily to be due to the implementation of the `ecdsa_check` function in how it looks up the right method data to call.
(Context: I am currently writing a library that statically links OpenSSL and am trying to load libp11, statically linked against openssl too. I do not want to link libp11 statically due to licensing concerns. I have tested with OpenSSL 1.0.1t and 1.0.2h) Basically, when I call ECDSA_do_sign from my wrapper library, it calls `ecdsa_check` with function pointers to its own copies of `ecdsa_free`, `ecdsa_dup`. However, when the libp11 library loaded the private key and set its method data (with ECDSA_set_method) from its context, it has function pointers to its own copies of `ecdsa_free` and friends, and thus the `EC_KEY_get_key_method_data` function always returns NULL, resulting in openssl using its own routines to sign/decrypt/etc. rather than using the engine's routines. It seems to me that engines which statically link openssl will never work with EC keys. I have tested around with RSA keys as well, and they work fine as they do not have this `ecdsa_check` like function that filters based on function pointers. The method data there appears to be stored directly on the private key object. Questions: 1. Why is method data stored differently on EC keys than on RSA keys? 2. Was there a reason to do this filtering with function pointers this way? Was it only to distinguish ECDSA and ECDH function data? 3. If we were to patch openssl to store this method data directly on the EC key struct, is there anything we should be aware of breaking? Would a patch along these lines be accepted upstream? Thank you!
-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
