Re: [openssl-users] Loading engines recursively and crypto engine lock

[Krzysztof Konopko]

On 10 August 2016 at 16:19, Jakob Bohm  wrote:

> On 10/08/2016 15:49, Krzysztof Konopko wrote:
>
>> On 10 August 2016 at 15:31, Jakob Bohm  jb-open...@wisemo.com>>wrote:
>> ​
>> 1. Create a third engine3 which loads both engine1 and engine2
>>   internally (
>> ​​
>> without going through OpenSSL and its locks).
>>   So for example engine3->init calls both engine2->init and
>>   engine1->init.
>>
>>
>> ​I don't understand how engine3 could be initialised ​"
>> ​
>> without going through OpenSSL and its locks
>> ​" as it's OpenSSL taking the lock whenever initialising _every_ engine.
>> Also when I call `ENGINE_init()` (indirectly, somewhere deep inside
>> engine1), the implementation of
>> `ENGINE_init()`
>> ​ takes the engine lock. as well which is the source of the problem.
>>
> engine3 would call engine1 and engine2 without going through
> a call to ENGINE_init(), thus making OpenSSL itself see the
> engine1 and engine2 code as part of engine3 (even though
> engine3 really just calls the functions in engine1 and
> engine2).


​Oh, I see.  engine1 and engine2 would expose individual functions which
engine3 would bound as its engine function own (or use some plumbing to
forward calls to the original functions).  This means engine3 would either
link with engine1 and engine2 libraries or `dlopen()` them.  Or something
like that.  I think I get the idea now.
​


>
> ​
>>
>> 2. engine3 would export/provide all the methods from engine1
>>   and engine2 by forwarding or reexporting the calls.
>>
>> 3. OpenSSL itself is instructed to use only your engine3
>>   wrapper.
>>
>> 4. As a more ambitious project, someone could write a generic
>>   "engine3" which loads a list of actual engines from a config
>>   file.
>>
>> At the OpenSSL design level, the OpenSSL team might extend the
>> OPENSSL_SSL_CLIENT_ENGINE_AUTOvariable to accept a
>> colon-separatedlist of engines rather than just a single engine.
>>
>>
>>
>> ​That sounds interesting but engines in general (and specifically in my
>> case) are independent of each other and in different situations I may want
>> to load one but not the other (for example when testing).  But I guess that
>> would be a matter of moving the configuration control from where I have it
>> now into whatever mechanism OpenSSL could have (as proposed above).
>>
> The idea would be that "engine3" would be a workaround engine
> that simulates the (possibly missing) ability to specify
> multiple engines via the OPENSSL_SSL_CLIENT_ENGINE_AUTO
> variable.  This not-independent engine3 would do nothing but
> load other engines, and may or may not be configurable as to
> which real engines it loads.  By doing this, engine3 would also
> compensate for the fact that many other OpenSSL APIs seem to
> allow only a single engine reference as parameter.
>
> With engine3 responsible for initializing engine2 before engine1,
> engine1 would no longer contain code to load engine2, making
> engine1 more independent from engine2.
>
>
>
>
​OK, fair enough.  That makes sense.  In my case some re-factoring would be
required as the code that initialises engine2 does so explicitly with
`ENGINE_by_id()` and `ENGINE_init()` and it doesn't "know" it's called from
engine1.  And vice versa, engine1 does not know that the code it calls
loads and initialises an engine.  But I get the idea and it seems plausible.

Also it's important to me that this way or the other there's someone who
admits OpenSSL has a problem with loading engines recursively (or does not
support that intentionally) and I need to address that.

Thanks again!
Kris​
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] Loading engines recursively and crypto engine lock

[Jakob Bohm]

On 10/08/2016 15:49, Krzysztof Konopko wrote:
On 10 August 2016 at 15:31, Jakob Bohm >wrote:


I am not part of the OpenSSL team and have no idea what their
thinking or suggestions are.


​Thanks for responding!
​


However the following should be a generic workaround:

1. Create a third engine3 which loads both engine1 and engine2
  internally (
​​
without going through OpenSSL and its locks).
  So for example engine3->init calls both engine2->init and
  engine1->init.


​I don't understand how engine3 could be initialised ​"
​
without going through OpenSSL and its locks
​" as it's OpenSSL taking the lock whenever initialising _every_ 
engine.  Also when I call `ENGINE_init()` (indirectly, somewhere deep 
inside engine1), the implementation of

`ENGINE_init()`
​ takes the engine lock. as well which is the source of the problem.

engine3 would call engine1 and engine2 without going through
a call to ENGINE_init(), thus making OpenSSL itself see the
engine1 and engine2 code as part of engine3 (even though
engine3 really just calls the functions in engine1 and
engine2).

​

2. engine3 would export/provide all the methods from engine1
  and engine2 by forwarding or reexporting the calls.

3. OpenSSL itself is instructed to use only your engine3
  wrapper.

4. As a more ambitious project, someone could write a generic
  "engine3" which loads a list of actual engines from a config
  file.

At the OpenSSL design level, the OpenSSL team might extend the
OPENSSL_SSL_CLIENT_ENGINE_AUTOvariable to accept a
colon-separatedlist of engines rather than just a single engine.



​That sounds interesting but engines in general (and specifically in 
my case) are independent of each other and in different situations I 
may want to load one but not the other (for example when testing).  
But I guess that would be a matter of moving the configuration control 
from where I have it now into whatever mechanism OpenSSL could have 
(as proposed above).

The idea would be that "engine3" would be a workaround engine
that simulates the (possibly missing) ability to specify
multiple engines via the OPENSSL_SSL_CLIENT_ENGINE_AUTO
variable.  This not-independent engine3 would do nothing but
load other engines, and may or may not be configurable as to
which real engines it loads.  By doing this, engine3 would also
compensate for the fact that many other OpenSSL APIs seem to
allow only a single engine reference as parameter.

With engine3 responsible for initializing engine2 before engine1,
engine1 would no longer contain code to load engine2, making
engine1 more independent from engine2.

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] Loading engines recursively and crypto engine lock

[Krzysztof Konopko]

On 10 August 2016 at 15:31, Jakob Bohm  wrote:

> I am not part of the OpenSSL team and have no idea what their
> thinking or suggestions are.
>

​Thanks for responding!
​


>
> However the following should be a generic workaround:
>
> 1. Create a third engine3 which loads both engine1 and engine2
>   internally (
> ​​
> without going through OpenSSL and its locks).
>   So for example engine3->init calls both engine2->init and
>   engine1->init.
>
>
​I don't understand how engine3 could be initialised ​"
​
without going through OpenSSL and its locks
​" as it's OpenSSL taking the lock whenever initialising _every_ engine.
Also when I call `ENGINE_init()` (indirectly, somewhere deep inside
engine1), the implementation of
`ENGINE_init()`
​ takes the engine lock. as well which is the source of the problem.
​

> 2. engine3 would export/provide all the methods from engine1
>   and engine2 by forwarding or reexporting the calls.
>
> 3. OpenSSL itself is instructed to use only your engine3
>   wrapper.
>
> 4. As a more ambitious project, someone could write a generic
>   "engine3" which loads a list of actual engines from a config
>   file.
>
> At the OpenSSL design level, the OpenSSL team might extend the
> OPENSSL_SSL_CLIENT_ENGINE_AUTOvariable to accept a
> colon-separatedlist of engines rather than just a single engine.
>
>
>
​That sounds interesting but engines in general (and specifically in my
case) are independent of each other and in different situations I may want
to load one but not the other (for example when testing).  But I guess that
would be a matter of moving the configuration control from where I have it
now into whatever mechanism OpenSSL could have (as proposed above).

Thanks,
Kris​
-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users

Re: [openssl-users] Loading engines recursively and crypto engine lock

[Jakob Bohm]

On 08/08/2016 13:39, Krzysztof Konopko wrote:

Hi,

TL;DR;
Is it allowed to initialise engines recursively, ie. call 
`engine2->init` from `engine1->init`?


--

I have a solution in a consumer product based on OpenSSL 1.0.2 series 
that uses two engines: one (engine1) for selecting client certificate 
chain (TLS client auth) and another one (engine2) for RPC operations 
on associated private keys stored in H/W.  This works only if supplied 
(installed) locks are recursive as for each engine initialisation 
`CRYPTO_LOCK_ENGINE` is taken.


From what I see, OpenSSL 1.1.x onward, provides locking internally and 
it's non-recursive.  Also `lock_dbg_cb()` implementation in OpenSSL 
before 1.1.x suggests locks are not expected to be recursive.


Here's some more context of my use case.​

OpenSSL loads `engine1` for me automatically (`
​​
​
OPENSSL_SSL_CLIENT_ENGINE_AUTO` variable) which is convenient as I 
don't have control over application's `main()` function.  In my case 
it's proprietary code but equally it could be Python script (I do not 
fancy patching Python interpreter to get to its `main()` function and 
load/initialise engines explicitly).


So my _only_ entry point is `engine1->init`.  In that entry point I 
initialise engine2 which is a fairly slow operation (need to load 
certs from permanent storage) so definitely want to do this only once. 
Oh, and the app is heavily multi-threaded so I'm glad OpenSSL 
carefully takes crypto engine lock where needed.


But because engines are initialised recursively, the locking 
implementation I supply uses recursive mutex which works very well and 
makes perfect sense to me in this case (I know that the same thread 
calls locked functions recursively for a reason).  This works only 
before 1.1.x.


Alternatively I could lazy-initialise engine2 in certificate callback 
function but any initialisation failure here would be less meaningful 
and it would require another lock to protect engine2 handle.  In 
`engine1->init` I know a lock is already held so I thought it's safer 
to do more initialisation here.  Besides `engine2->init` is not called 
directly but through a layer of application logic so conceptually 
these two engines are orthogonal and know nothing about each other.


I guess initialising engines recursively does not work in OpenSSL 
1.1.x (it'd be a dead-lock) and I need to seek for a different place 
to initialise engine2, for example in certificate cb?  This would mean 
I "leak" some knowledge of engine2 existence into engine1, have 
guarantee that crypto engine lock is not held in certificate callabck 
function and need another lock to protect access to engine2 handle.


Please let me know what your views are and if the above makes sense.


I am not part of the OpenSSL team and have no idea what their
thinking or suggestions are.

However the following should be a generic workaround:

1. Create a third engine3 which loads both engine1 and engine2
  internally (without going through OpenSSL and its locks).
  So for example engine3->init calls both engine2->init and
  engine1->init.

2. engine3 would export/provide all the methods from engine1
  and engine2 by forwarding or reexporting the calls.

3. OpenSSL itself is instructed to use only your engine3
  wrapper.

4. As a more ambitious project, someone could write a generic
  "engine3" which loads a list of actual engines from a config
  file.

At the OpenSSL design level, the OpenSSL team might extend the
OPENSSL_SSL_CLIENT_ENGINE_AUTOvariable to accept a
colon-separatedlist of engines rather than just a single engine.


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

--
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users