Darryl Miles wrote:
David Schwartz wrote:
Mark Phalan wrote:
In this case, I presume 'pkinit' only supports one threading model
(or one
set of compatible threading models). So it can set the callbacks.
It can set the callbacks but it can't set them in a way which is safe
from races.
It can set them before it performs any OpenSSL operations. How is that not
safe from races?
Still don't get it. Lets use Linux as an example.
This is actually one of the easiest to solve. Of course the bigger problem is
that solutions here are all platform-dependent, and maintenance will be a pain.
ELF shared libraries support .init and .fini sections to contain code that
should be executed just after load and just before unload. Assuming you had a
default set of callbacks in the library, it would be simple to set them here.
Likewise any other shared library that manipulates callbacks can do its own
cleanup safely in their own .fini section.
The other feature you need, to make this reasonable, is support for weak
external symbols. I don't know of any compiler support for these but certainly
they're supported at the assembler and linker level. The default callbacks
should reference pthread functions as weak externs, and just no-op if the
externs are unresolved at runtime.
As an aside, it's probably still unsafe to assume any particular thread model,
on any OS. Third party thread libraries are still out there that run
cross-platform (e.g. GNU Pth).
You have a top level executable whose only runtime linked libraries are
"libc", "libdl", "libpthread". This library from 3 independent threads
decides it needs to dynamically load via the libdl API call "dlopen()"
the following libraries:
"libldap_r" (this in turn has a runtime library dependency on
"libcrypto" "libssl" amongst other DSOs).
"libcurl" (this in turn has a runtime library dependency on "libcrypto"
"libssl" amongst other DSOs).
"libstunnel" (this in turn has a runtime library dependency on
"libcrypto" "libssl" amongst other DSOs).
There are less contrived cases; Perl and Python built into executables
that in turn link with OpenSSL as well.
The top level application doesn't even know OpenSSL exists. It doesn't
care that is exists, since the API contract is has relates to some other
functionality.
The dynamic linker will de-duplicate the DSO being loaded, so when
"libcrypto" is loaded for the 2nd time it will realize it is already
loaded and reference count it up one and hand the same DSO address to
each of the 3 callers to share. Since that is the point of DSOs.
How does each of the 3 threads arbitrate in a thread-safe manner the
initialization of the OpenSSL related libraries. How does any one of
them know it is the "first user" ?
Once each of the 3 users has finished using their copy of OpenSSL they
can request the dynamic linker unload it "dlclose()". So it is also
necessary for OpenSSL to know when the last user has closed the library
and it needs to perform memory freeing and cleanup because there are no
users left. How does any one of them know it is the "last user" ?
I think these cases should have equal merit, if you are going to fix the
load case you must also fix the unload case.
Now you can re look at the problem and simplify it on the basis what can
be do if we know its a single thread. Then from that vantage point work
out what small areas of the process need to be protected.
If you ask me the dynamic linker should provide support for this issue
itself, the ability to embed a control block in the executable that
provides callbacks with the dynamic linker providing serialization of
running those callback and exposing the most basic locking primitives
(mutex) for use only from inside those callbacks.
Windows provides this infrastructure already, _DllMain symbol and the
fact that all platform provide access to the Windows API and the mutex
system calls are available. Maybe open source systems do as well but we
are just not aware of the hooking mechanism in this list.
The issue is we don't want the thread-aware code to manage this in the
OpenSSL DSOs since that will mess up the single-thread (thread unaware)
users of OpenSSL. This also means we don't want a runtime linker
dependancy on libpthread just for this purpose (libcrypto currently
doesn't have one, libssl does).
Since the thread locking primitives needs to be provided by the platform
and this is a linker caused issue, it is the dynamic link that should
provide the mechanism to clean up this mess.
This effectively becomes imposed baggage by the dynamic linker due to
_SOME_ users being multi-threaded users (as well as allowing the linker
freedom to load dependancies on a whim).
Perhaps I don't understand exactly what you mean by "threading model".
Are you referring to 1:1, n:1 etc?
That's part of it. It's also what threading library you are using.
You can still have a "default theading model" and still support all
"known threading models" via reconfiguration before first use. There is
no conflict here please stop implying there is one.
What I could do is say that when delivered as part of OpenSolaris
OpenSSL assumes the OS's threading model. If an application wants to
change that they the application should set its own callbacks.
If that's what you want, it's trivial to implement. Simply set callbacks
that are safe for the OS' threading model before you call any OpenSSL
functions.
This is exactly where the issue lies. We know we need to do this
(please stop repeating back to us that this is what we need to do).
The issue is we can not do this in a thread-safe manner since:
* The dynamic linker provides no arbitration (this is who I really blame)
* OpenSSL the library provides no arbitration (due to it having a
single thread / thread unaware use case).
Just to be totally clear. All I'd like to see is a compile-time option
(it can be disabled by default) to set a default set of locking
callbacks native to the OS it's compiling on. I'm not suggesting that
the ability to set locking callbacks be removed.
The real resistance to this Mark, is the fact you are now forcing a
dependency on libpthread for libcrypto which didn't exist before.
Yes I know the single threaded users (thread unaware users) can
recompile a special version for their own use but this still poses a
problem relating to the DSO name of the library and that fact you are
impacting their support issues in the future (for your gain).
Maybe what would be better is a libcrypto_r and a libssl_r de-facto
standard where multi-threaded support is a requirement and override-able
default implementation was already setup to go ?
This does not mean we should not fix the real issue, it is just a naming
work around until that point in time. This is more a packaging issue
than a OpenSSL developer issue.
It can do this but only by potentially introducing a race.
What is the race exactly? If they're already set, setting them again to
precisely the same callbacks is harmless.
Just because you have the "same threading model" doesn't mean you are
using the same threading primitive symbols.
Managing to publish and agree across many projects the exist symbols and
use of OpenSSL is a problem that is completely avoided by better
engineering. Trying to push out the solution to a "social" issue will
not guarantee a solution.
And if the problem is an
application setting them to different callbacks, OpenSSL can't help you --
if an application changes the callbacks from the default to unsafe
callbacks, you're still hosed.
This appears to be the point where we disagree. Many including mysely
are claiming that OpenSSL (or rather I am claiming the "dynamic linker")
_SHOULD_ be helping us.
You are correct in that currently OpenSSL can't help us, but this is why
we have a discussion here to resolve that.
Existing code that uses OpenSSL will still change those callbacks from the
defaults. New applications that don't use OpenSSL at all won't be affected
by the change. New libraries could just as easily be coded to always set
known safe callbacks.
There is only a race if some application either reverts to no callbacks or
sets unsafe callbacks. Both of those operations will still be fatal with the
suggested change.
Setting the callback(s) is unsafe. Since another thread might be using
them. Do you not get that ? There is no locking around the setting of
them so therefore they are thread-unsafe, the method to set locking is
not re-entrant. I harp back to my request for "re-entrant thread-safe
OpenSSL APIs" for setting up the threading callbacks.
Of course if the compiler generates code that does a single instruction
store (of the native platform bit-width) of the same value over and
over, no other thread will ever observe any change in that memory
location. The point of the matter is there is no guarantee's provided
by the compiler over how it goes about doing that so in order to get an
engineering guarantee as required to be thread-safe a mutex would
provide assurance.
This ignores other issues that are implementation details to outside
users of OpenSSL what else is going on inside those OpenSSL API calls
that isn't expected to be called from other threads at the same time.
The API could work like a hybrid sigaction() allowing
get-and-test-and-set in one atomic operation. This will require
libpthread runtime linkage in libcrypto.
Darryl
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Development Mailing List openssl-dev@openssl.org
Automated List Manager majord...@openssl.org
