RE: Howto prevent cycles in engine invocation ?

[Fuchs, Andreas]

Hi Dimitry,

thanks, yes, that's why I'm doing inside the engine as well.

The question was regarding the library that my engine links against that in 
turn uses libcrypto again.

That linked library uses the EVP methods with the "default"-engine, which is 
however set from the surrounding application.

I guess it's a similar question to when an application links against liba.so 
and libb.so and both use openssl internally but set different default engines. 
How can you prevent them from interfering with each other ?

Thanks,
Andreas

________________________________
From: openssl-users [openssl-users-boun...@openssl.org] on behalf of Dmitry 
Belyavsky [beld...@gmail.com]
Sent: Tuesday, March 19, 2019 21:09
To: openssl-users@openssl.org
Subject: Re: Howto prevent cycles in engine invocation ?

Hello Andreas,

I used smth like
=======
RSA_METHOD my_rsa_method = {
    "My RSA method",
    0,                          /* pub_enc */
    0,                          /* pub_dec */
    0,          /* priv_enc */
    my_priv_dec,          /* priv_dec */
    0,                          /* rsa_mod_exp */
    0,                          /* bn_mod_exp */
    0,                          /* init */
    0,              /* finish */
    RSA_METHOD_FLAG_NO_CHECK|RSA_FLAG_SIGN_VER,          /* flags */
    NULL,                       /* app_data */
    my_rsa_sign,              /* rsa_sign */
    0                           /* rsa_verify */
};

static int my_priv_dec (int flen, const unsigned char *from,
    unsigned char *to, RSA *rsa, int padding)
{
  const RSA_METHOD *def_meth = RSA_PKCS1_SSLeay();
  if ((rsa->meth == &my_rsa_method) && RSA_get_ex_data(rsa, my_key_pos))
  {
    return my_op_rsa_decrypt(flen, from, to, rsa, padding);
  }

  return def_meth->rsa_priv_dec(flen, from, to, rsa, padding);
}
==============
But this code worked for 1.0.* branch

вт, 19 марта 2019 г., 19:52 Fuchs, Andreas 
<andreas.fu...@sit.fraunhofer.de<mailto:andreas.fu...@sit.fraunhofer.de>>:
Following scenario:

I have an engine that implements e.g. RSA decryption.
That engine links against a library that links against libcrypto to perform RSA 
decryption.

Now if I have an application that sets the default library to be my engine, 
won't I end up in an infinite loop ?

Or the other way around:
- Is there a way to change the default engine from within my engine before 
calling out to the library ?
- Can the library itself select "software-only" as an engine ?

I guess I don't have a concrete problem in my specific case, since I have 
app_data attached to any key and
if my engine receives a key without app_data, it will just call SSLeay() 
functions.

But what if that was not the case ?

Thanks a lot for any help,
Andreas