RE: Safety of using OpenSSL from multiple components in the one process
We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL. That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc. I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case. Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well. Peter Peter Waltenberg Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 03/08/2005 10:26 PM Please respond to openssl-dev To openssl-dev@openssl.org cc Subject RE: Safety of using OpenSSL from multiple components in the one process Hi Peter, There is a requirement in my case for the OpenSSL API to be accessible to several components. In your case was your higher level component the only direct user of OpenSSL? If not, how did you make the API available to other components? Were the new functions accessed from one DLL with the OpenSSL functions accessed directly from the other DLL, or did you have redirecting stubs in your new DLL? Steven From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Waltenberg Sent: Wednesday, 3 August 2005 8:48 PM To: openssl-dev@openssl.org Cc: [EMAIL PROTECTED] Subject: Re: Safety of using OpenSSL from multiple components in the one process You are correct it is a problem in enterprise applications. The good news is that it can be resolved, the bad news is that we did that outside the OpenSSL codebase itself. We created new startup/shutdown functions in another library, added reference counting and load OpenSSL via that. It might be possible to do something simillar within the existing API's, but brute force did work. Peter Peter Waltenberg Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 03/08/2005 06:42 PM Please respond to openssl-dev To openssl-dev@openssl.org cc Subject Safety of using OpenSSL from multiple components in the one process Hi All, This is something that I think I've raised before but don't remember getting resolution on. OpenSSL maintains various global variables and structures, and there are cleanup functions that must be used to properly release the resources when finished. One example is the OID database managed by the add_all_algorithms function and it's associated release function, EVP_cleanup. All is good when the use of OpenSSL is fairly simple, such as a single component using it for the lifetime of the process. Where things get difficult/dangerous is when multiple seperate components in the one process, with no real knowledge of each other, make use of OpenSSL, and it's even worse if they dynamically load and unload OpenSSL using dlopen/LoadLibrary. With large enterprise applications this is a common situation since different teams develop components that the large product makes us of, and with the increasing use of plug-in architectures the dynamic loading/unloading is not uncommon. There seems to be no way offered by the OpenSSL API for these components to behave well. If they each do a dlopen - dlsym - ... - EVP_cleanup - dlclose sequence then it seems that they will trample on each other. If they take the extreme opposite and don't call EVP_cleanup then the process will leak until it falls over. This is a serious issue that I believe impacts the stability and therefore limits the usefulness of OpenSSL in large enterprise applications. Does anyone else have any thoughts on this? Regards, Steven
RE: Safety of using OpenSSL from multiple components in the one process
Ah, the namespacing issue we've discussed
previously.
The module that Imanage at CA
isa higher-level library that adds a
similar abstraction layeras yours, but there's now an internal requirement
to make the underlying OpenSSL interface also available to internal
developers. This was originally allowed for Open Source components that
use OpenSSL (OpenLDAP, Apachefor example) but has been extended to
internal code that is already using OpenSSL. Our higher-level library uses
extensive reference counting to provide a robust interface to callers. We
of course do an EVP_cleanup in our xxx_lib_release() function. However,
with developers now also able to get at OpenSSL anyone of them could call
EVP_cleanup at anytime. Mandating that EVP_cleanup not be called other
than by my module is ludicrous due to the resource leaks, particularly if our
higher-level library isn't also used in the same
process.
I stongly feel that for OpenSSL to advance to the next
level that usability in enterprise applications has to be treated as a
first-class requirement. I don't see how this can be done
withoutbreaking the current interface, but hey, that's why the version
number is 1.0. What I'd like to see
is:
1. A reference counted startup/shutdown function with
all functions that tear-down global structures controlled by these and otherwise
made unavailable to callers (ie. EVP_cleanup, CRYPTO_cleanup_all_ex_data,
ERR_free_strings, etc.). The init functions such as
xxx_add_all_algorithms() may be left as-isbut the associated cleanup
functions are dangerous while public.
2. Namespacing supported. I've implemented this
in my own builds largely using the C preprocessor to "#define
RSA_new CA1_RSA_new" for example with some other changes required in
the perl scripts and some assembler modules. The prefix can be specified
at config time so that, in your case Peter, you could use a prefix of
IBM_. A default OpenSSL build continues to have no prefix. This
means that I no longer have to fear some other OpenSSL build finding its way
into our process at runtime and subverting our FIPS
routines.
An alternative to 2 is to move all functions into a
structure of functions pointers, a la PKCS#11, and have a single function
exported from OpenSSL, eg. OpenSSL_get_functions(), that returns this.
However, I'm guessing that the extra pointer indirection would be considered too
heavy handed. As is evident in the Python world though, trading a few
cycles for increased developer productivity, and in this case determinism and
stability, is a wise move.
I'm happy to contribute the changes, but don't want to
do all of the work unless the OpenSSL core developers agree on the
solution.
Regards,
Steven
From: Peter Waltenberg
[mailto:[EMAIL PROTECTED] Sent: Thursday, 4 August 2005 8:09
AMTo: openssl-dev@openssl.orgCc:
[EMAIL PROTECTED]Subject: RE: Safety of using OpenSSL from multiple
components in the one process
We created new entry points. There's
a very shallow layer over the EVP layer which is mostly just a macro away from
native OpenSSL. That was done to satisfy
other requirements, abstracting the OpenSSL API's a little so we could isolate
IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so
we could keep IBM owned code separate from OpenSSL licenced code etc.
I'm not going to claim that it's impossible
to solve this in other ways, we just didn't have to try anything else since a
solution that mostly worked already was imposed by other considerations in our
case. Our problems have been more
with conflicts with other instances of OpenSSL in customer products. IBM
libraries are used by customers who integrate them with their code as
well. Peter Peter Waltenberg
"Steven Reddie"
[EMAIL PROTECTED] Sent by: [EMAIL PROTECTED]
03/08/2005 10:26 PM
Please respond
toopenssl-dev
To
openssl-dev@openssl.org
cc
Subject
RE: Safety of using
OpenSSL from multiple components in the one
process
Hi Peter, There is a requirement in my case for the OpenSSL
API to be accessible to several components. In your case was your higher
level component the only direct user of OpenSSL? If not, how did you make
the API available to other components? Were the new functions accessed
from one DLL with the OpenSSL functions accessed directly from the other DLL, or
did you have redirecting stubs in your new DLL? Steven
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Peter
WaltenbergSent: Wednesday, 3 August 2005 8:48 PMTo:
openssl-dev@openssl.orgCc: [EMAIL PROTECTED]Subject: Re:
Safety of using OpenSSL from multiple components in the one processYou are corr
RE: Safety of using OpenSSL from multiple components in the one process
Our problems and the solutions we've come up with appear be be near identical conceptually. I think if we were going to address those problems, we'd should also try to fix the current visibility of internal OpenSSL data types. I suspect that's more of a problem for the OpenSSL team who have to take extra care to avoid changes which break the ABI between releases. We'd also be happy to contribute code and effort, but as you've pointed out, the core team will have to want these changes made. Peter Peter Waltenberg Steven Reddie [EMAIL PROTECTED] 04/08/2005 09:01 AM To Peter Waltenberg/Australia/[EMAIL PROTECTED], openssl-dev@openssl.org cc Subject RE: Safety of using OpenSSL from multiple components in the one process Ah, the namespacing issue we've discussed previously. The module that I manage at CA is a higher-level library that adds a similar abstraction layer as yours, but there's now an internal requirement to make the underlying OpenSSL interface also available to internal developers. This was originally allowed for Open Source components that use OpenSSL (OpenLDAP, Apache for example) but has been extended to internal code that is already using OpenSSL. Our higher-level library uses extensive reference counting to provide a robust interface to callers. We of course do an EVP_cleanup in our xxx_lib_release() function. However, with developers now also able to get at OpenSSL anyone of them could call EVP_cleanup at anytime. Mandating that EVP_cleanup not be called other than by my module is ludicrous due to the resource leaks, particularly if our higher-level library isn't also used in the same process. I stongly feel that for OpenSSL to advance to the next level that usability in enterprise applications has to be treated as a first-class requirement. I don't see how this can be done without breaking the current interface, but hey, that's why the version number is 1.0. What I'd like to see is: 1. A reference counted startup/shutdown function with all functions that tear-down global structures controlled by these and otherwise made unavailable to callers (ie. EVP_cleanup, CRYPTO_cleanup_all_ex_data, ERR_free_strings, etc.). The init functions such as xxx_add_all_algorithms() may be left as-is but the associated cleanup functions are dangerous while public. 2. Namespacing supported. I've implemented this in my own builds largely using the C preprocessor to #define RSA_new CA1_RSA_new for example with some other changes required in the perl scripts and some assembler modules. The prefix can be specified at config time so that, in your case Peter, you could use a prefix of IBM_. A default OpenSSL build continues to have no prefix. This means that I no longer have to fear some other OpenSSL build finding its way into our process at runtime and subverting our FIPS routines. An alternative to 2 is to move all functions into a structure of functions pointers, a la PKCS#11, and have a single function exported from OpenSSL, eg. OpenSSL_get_functions(), that returns this. However, I'm guessing that the extra pointer indirection would be considered too heavy handed. As is evident in the Python world though, trading a few cycles for increased developer productivity, and in this case determinism and stability, is a wise move. I'm happy to contribute the changes, but don't want to do all of the work unless the OpenSSL core developers agree on the solution. Regards, Steven From: Peter Waltenberg [mailto:[EMAIL PROTECTED] Sent: Thursday, 4 August 2005 8:09 AM To: openssl-dev@openssl.org Cc: [EMAIL PROTECTED] Subject: RE: Safety of using OpenSSL from multiple components in the one process We created new entry points. There's a very shallow layer over the EVP layer which is mostly just a macro away from native OpenSSL. That was done to satisfy other requirements, abstracting the OpenSSL API's a little so we could isolate IBM apps. from changes in OpenSSL, so we could slide in the FIPS self tests, so we could keep IBM owned code separate from OpenSSL licenced code etc. I'm not going to claim that it's impossible to solve this in other ways, we just didn't have to try anything else since a solution that mostly worked already was imposed by other considerations in our case. Our problems have been more with conflicts with other instances of OpenSSL in customer products. IBM libraries are used by customers who integrate them with their code as well. Peter Peter Waltenberg Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 03/08/2005 10:26 PM Please respond to openssl-dev To openssl-dev@openssl.org cc Subject RE: Safety of using OpenSSL from multiple components in the one process Hi Peter, There is a requirement in my case for the OpenSSL API to be accessible to several components. In your case was your higher level component the only direct user of OpenSSL? If not, how did you make the API available to other components? Were the new
