RE: OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c)
I think all you'll be able to do is provide it as an option. It's quite hard to get right and it's probably not going to work everywhere OpenSSL is used. Having the code in the sources will save a lot of pain for everyone who wants this though - so from that point of view it's worth having it, it looks like all you need is an extra object and some extra link flags if you decide to go this way. The static/shared thing with FIPS is interesting, if you use shared libraries you have to be able to avoid inadvertant use of the wrong library, if you use static libraries it's harder to show where the certified code begins and ends. Peter Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 07/14/04 03:49 PM Please respond to openssl-dev To [EMAIL PROTECTED] cc Subject RE: OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c) Hi Peter, Those platforms, minus Mac OS/X, plus OSF Tru64 (Digital Unix), has been my experience also. I was hoping that init might be supported across the board. Thanks for the info. It's my understanding so far that the OpenSSL FIPS certification requires use of a static library build of OpenSSL. There was some discussion about ways to allow shared libraries to be used but I don't think I saw a definitive solution. I'll be happy if I just missed it. Do you think these issues should be addressed in OpenSSL, or are best left for the caller to handle? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Peter Waltenberg Sent: Wednesday, 14 July 2004 3:26 PM To: [EMAIL PROTECTED] Subject: Re: OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c) 1. It is safe to create/initialise synchronization objects (ie. a mutex) in DllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL mutex could be created to be used for on-demand (first-access) initialisation of subsystems. I don't recall the equivalent DLL entry point on Unix platforms or it's semantics, but as an aside does anyone know if such entry points are standard across all Unix implementations that support threading? Yes to all the new Unix platforms we support with our OpenSSL wrapper. (Solaris, Linux, AIX, HPUX, Mac OS/X). Support isn't uniform however, and is quirky on some older systems. If you want to follow this up I can probably provide example shared library entry point code and link options for quite a few platforms. 3. Point 1 does not address the situation where OpenSSL is built as a static library and included in some other DLL (and the FIPS support may require this to be the standard setup for many projects). Static libraries make it hard to draw the boundary around the FIPS certified code, which means you may end up having to certify the whole ball of wax, (expensive). Shared libraries will probably be more common. It would be the responsibility of the wrapping DLL to call whatever OpenSSL function initialises the master mutex from it's own DllMain(). This can get hairy when the static library using OpenSSL is also provided as a static library -- the responsibility has to be carried carefully up the chain. A hack we had to add recently to support legacy systems where the shared lib initialization code just doesn't get called. Peter Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 07/14/04 03:08 PM Please respond to openssl-dev To [EMAIL PROTECTED] cc Subject OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c) Hi Jeffrey, Come to think of it, maybe OpenSSL should simply perform a call to RAND_poll() as part of the DLL initialization. This would solve many problems. No, it would create some. RAND_poll() calls LoadLibrary() which is a no-no from DllMain(). malloc() is not safe to call from DllMain() when the C Runtime is being dynamically linked in, and since RAND_poll() is so complex it is possible that malloc() is being called or will be in the future without anyone realising it's a no-no. However, I know where you're going with this, and I agree that it's something that needs addressing. It's a general problem that has been troubling me on another project. I don't know that I have the greatest solution, but I'll do a quick dump of some of the options here. 1. It is safe to create/initialise synchronization objects (ie. a mutex) in DllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL mutex could be created to be used for on-demand (first-access) initialisation of subsystems. I don't recall the equivalent DLL entry point on Unix platforms or it's semantics, but as an aside does anyone know if such entry points are standard across all Unix implementations that support threading? 2. Alternatively, each subsystem could use it's own init-on-lock mutex (a la PTHREAD_MUTEX_INITIALIZER) but such a primitive might not be supported on all platforms and would introduce tighter coupling which is counter to OpenSSL's thread library agnostic approach
Re: possibly bug in crypto/rand/rand_win.c
David Schwartz wrote: Suggestion: This is a usage error. Be sure to initialize openssl when your application starts or when your DLL is loaded. Do not wait for the first thread to attempt to make a call to OpenSSL to initialize it. That is not always possible. The process may become multi-threaded long before it has any idea it's going to do anything related to SSL. Consider, for example, a DLL that extends Acrobat to be able to contain embedded interactive elements that communicate back to a server using a protocol layered on top of SSL. Acrobat may well be multi-threaded before the document is loaded and it is known that its content requires activating this external module. DS Thank you David. And thank everybody for the response. My program happens to be the kind David described. It's a shell extension loaded by explorer.exe. Aside from the way library get loaded, rand_win.c still seems to have some problem. Here's what MSDN says about CreateToolhelp32Snapshot: [QUOTE START] To enumerate the heap or module states for all processes, specify TH32CS_SNAPALL and set th32ProcessID to zero. Then, for each additional process in the snapshot, call CreateToolhelp32Snapshot again, specifying its process identifier and the TH32CS_SNAPHEAPLIST or TH32_SNAPMODULE value. [QUOTE END] I think it means one have to call snap N+1 times for N processes, while current implementation calls it only once. Yet changing 1 to N+1 did not fix the problem. My program still crashes from time to time on a slow and heavily loaded pc at heap_next. I'll post again if I make any progress. Cheers Lei __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
possibly bug in crypto/rand/rand_win.c
Hi,
Sorry if this message is sent twice.
I got problem running RAND_poll() in multi-threaded programs. The
function sometimes crashes at heap_next(hentry):
...
if (heaplist_first(handle, hlist))
do
{
RAND_add(hlist, hlist.dwSize, 3);
hentry.dwSize = sizeof(HEAPENTRY32);
if (heap_first(hentry,
hlist.th32ProcessID,
hlist.th32HeapID))
{
int entrycnt = 80;
do
RAND_add(hentry,
hentry.dwSize, 5);
while (heap_next(hentry)
^
*** this is where the problem is ***
--entrycnt 0);
}
} while (heaplist_next(handle,
hlist));
...
An article at
http://www.codeproject.com/threads/Thelp32ReadProcessMemory.asp revealed
that the function Heap32Next is not appropriate to be used in threaded
programs:
[QUOTE START]
The snapshot
The toolhelp functions make use of a snapshot to access process, thread,
module, and heap lists in the system. To quote MSDN:
The lists in system memory change when processes are started and ended,
threads are created and destroyed, executable modules are loaded and
unloaded from system memory, and heaps are created and destroyed. The
use of information from a snapshot prevents inconsistencies. Otherwise,
changes to a list could possibly cause a thread to incorrectly traverse
the list or cause an access violation (a GP fault). For example, if an
application traverses the thread list while other threads are created or
terminated, information that the application is using to traverse the
thread list might become outdated and could cause an error for the
application traversing the list.
[QUOTE END]
I guess that tells why. The most simple resolution will be to remove the
Heap32Next.(of course
this will trade off rand seed quality, too).
Any ideas?
regards
Lei
__
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
Re: possibly bug in crypto/rand/rand_win.c
Suggestion:
This is a usage error. Be sure to initialize openssl when your
application starts or
when your DLL is loaded. Do not wait for the first thread to attempt
to make
a call to OpenSSL to initialize it.
Come to think of it, maybe OpenSSL should simply perform a call to
RAND_poll()
as part of the DLL initialization. This would solve many problems.
Jeffrey Altman
Jiang Lei wrote:
Hi,
Sorry if this message is sent twice.
I got problem running RAND_poll() in multi-threaded programs. The
function sometimes crashes at heap_next(hentry):
...
if (heaplist_first(handle, hlist))
do
{
RAND_add(hlist, hlist.dwSize, 3);
hentry.dwSize = sizeof(HEAPENTRY32);
if (heap_first(hentry,
hlist.th32ProcessID,
hlist.th32HeapID))
{
int entrycnt = 80;
do
RAND_add(hentry,hentry.dwSize, 5);
while (heap_next(hentry)
^
*** this is where the problem is ***
--entrycnt 0);
}
} while (heaplist_next(handle,
hlist));
...
An article at
http://www.codeproject.com/threads/Thelp32ReadProcessMemory.asp revealed
that the function Heap32Next is not appropriate to be used in threaded
programs:
[QUOTE START]
The snapshot
The toolhelp functions make use of a snapshot to access process, thread,
module, and heap lists in the system. To quote MSDN:
The lists in system memory change when processes are started and ended,
threads are created and destroyed, executable modules are loaded and
unloaded from system memory, and heaps are created and destroyed. The
use of information from a snapshot prevents inconsistencies. Otherwise,
changes to a list could possibly cause a thread to incorrectly traverse
the list or cause an access violation (a GP fault). For example, if an
application traverses the thread list while other threads are created or
terminated, information that the application is using to traverse the
thread list might become outdated and could cause an error for the
application traversing the list.
[QUOTE END]
I guess that tells why. The most simple resolution will be to remove
the Heap32Next.(of course
this will trade off rand seed quality, too).
Any ideas?
regards
Lei
__
OpenSSL Project http://www.openssl.org
Development Mailing List [EMAIL PROTECTED]
Automated List Manager [EMAIL PROTECTED]
smime.p7s
Description: S/MIME Cryptographic Signature
RE: possibly bug in crypto/rand/rand_win.c
Suggestion: This is a usage error. Be sure to initialize openssl when your application starts or when your DLL is loaded. Do not wait for the first thread to attempt to make a call to OpenSSL to initialize it. That is not always possible. The process may become multi-threaded long before it has any idea it's going to do anything related to SSL. Consider, for example, a DLL that extends Acrobat to be able to contain embedded interactive elements that communicate back to a server using a protocol layered on top of SSL. Acrobat may well be multi-threaded before the document is loaded and it is known that its content requires activating this external module. DS __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c)
Hi Jeffrey,
Come to think of it, maybe OpenSSL should simply perform a call to
RAND_poll() as part of the DLL initialization. This would solve many
problems.
No, it would create some. RAND_poll() calls LoadLibrary() which is a no-no
from DllMain(). malloc() is not safe to call from DllMain() when the C
Runtime is being dynamically linked in, and since RAND_poll() is so complex
it is possible that malloc() is being called or will be in the future
without anyone realising it's a no-no.
However, I know where you're going with this, and I agree that it's
something that needs addressing. It's a general problem that has been
troubling me on another project. I don't know that I have the greatest
solution, but I'll do a quick dump of some of the options here.
1. It is safe to create/initialise synchronization objects (ie. a mutex) in
DllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL mutex could be
created to be used for on-demand (first-access) initialisation of
subsystems. I don't recall the equivalent DLL entry point on Unix platforms
or it's semantics, but as an aside does anyone know if such entry points are
standard across all Unix implementations that support threading?
2. Alternatively, each subsystem could use it's own init-on-lock mutex (a la
PTHREAD_MUTEX_INITIALIZER) but such a primitive might not be supported on
all platforms and would introduce tighter coupling which is counter to
OpenSSL's thread library agnostic approach.
3. Point 1 does not address the situation where OpenSSL is built as a static
library and included in some other DLL (and the FIPS support may require
this to be the standard setup for many projects). It would be the
responsibility of the wrapping DLL to call whatever OpenSSL function
initialises the master mutex from it's own DllMain(). This can get hairy
when the static library using OpenSSL is also provided as a static library
-- the responsibility has to be carried carefully up the chain.
4. In it's most basic form (OpenSSL as a DLL) Point 1 bypasses the thread
library agnostic approach that OpenSSL uses because there is no way for the
user to first setup the mutex initialisation callbacks. I can't think of a
good solution to this. Creating a wrapping DLL who's DllMain() sets up the
callbacks and then calls the OpenSSL master mutex initialisation function
would work, but then the ability for the caller of that DLL to customise the
mutex functions has been lost.
Regards,
Steven
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Jeffrey Altman
Sent: Tuesday, 13 July 2004 11:14 PM
To: [EMAIL PROTECTED]
Subject: Re: possibly bug in crypto/rand/rand_win.c
Suggestion:
This is a usage error. Be sure to initialize openssl when your
application starts or
when your DLL is loaded. Do not wait for the first thread to attempt
to make
a call to OpenSSL to initialize it.
Come to think of it, maybe OpenSSL should simply perform a call to
RAND_poll()
as part of the DLL initialization. This would solve many problems.
Jeffrey Altman
Jiang Lei wrote:
Hi,
Sorry if this message is sent twice.
I got problem running RAND_poll() in multi-threaded programs. The
function sometimes crashes at heap_next(hentry):
...
if (heaplist_first(handle, hlist))
do
{
RAND_add(hlist, hlist.dwSize, 3);
hentry.dwSize = sizeof(HEAPENTRY32);
if (heap_first(hentry,
hlist.th32ProcessID,
hlist.th32HeapID))
{
int entrycnt = 80;
do
RAND_add(hentry,hentry.dwSize, 5);
while (heap_next(hentry)
^
*** this is where the problem is ***
--entrycnt 0);
}
} while (heaplist_next(handle,
hlist));
...
An article at
http://www.codeproject.com/threads/Thelp32ReadProcessMemory.asp
revealed that the function Heap32Next is not appropriate to be used in
threaded
programs:
[QUOTE START]
The snapshot
The toolhelp functions make use of a snapshot to access process,
thread, module, and heap lists in the system. To quote MSDN:
The lists in system memory change when processes are started and
ended, threads are created and destroyed, executable modules are
loaded and unloaded from system memory, and heaps are created and
destroyed. The use of information from a snapshot prevents
inconsistencies. Otherwise, changes to a list could possibly cause a
thread to incorrectly traverse the list or cause an access violation
(a GP fault). For example, if an application traverses the thread
list while other threads are created or terminated, information that
the application is using to traverse the thread list might become
outdated and could cause an error for the application traversing
Re: OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c)
1. It is safe to create/initialise synchronization objects (ie. a mutex) in DllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL mutex could be created to be used for on-demand (first-access) initialisation of subsystems. I don't recall the equivalent DLL entry point on Unix platforms or it's semantics, but as an aside does anyone know if such entry points are standard across all Unix implementations that support threading? Yes to all the new Unix platforms we support with our OpenSSL wrapper. (Solaris, Linux, AIX, HPUX, Mac OS/X). Support isn't uniform however, and is quirky on some older systems. If you want to follow this up I can probably provide example shared library entry point code and link options for quite a few platforms. 3. Point 1 does not address the situation where OpenSSL is built as a static library and included in some other DLL (and the FIPS support may require this to be the standard setup for many projects). Static libraries make it hard to draw the boundary around the FIPS certified code, which means you may end up having to certify the whole ball of wax, (expensive). Shared libraries will probably be more common. It would be the responsibility of the wrapping DLL to call whatever OpenSSL function initialises the master mutex from it's own DllMain(). This can get hairy when the static library using OpenSSL is also provided as a static library -- the responsibility has to be carried carefully up the chain. A hack we had to add recently to support legacy systems where the shared lib initialization code just doesn't get called. Peter Steven Reddie [EMAIL PROTECTED] Sent by: [EMAIL PROTECTED] 07/14/04 03:08 PM Please respond to openssl-dev To [EMAIL PROTECTED] cc Subject OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c) Hi Jeffrey, Come to think of it, maybe OpenSSL should simply perform a call to RAND_poll() as part of the DLL initialization. This would solve many problems. No, it would create some. RAND_poll() calls LoadLibrary() which is a no-no from DllMain(). malloc() is not safe to call from DllMain() when the C Runtime is being dynamically linked in, and since RAND_poll() is so complex it is possible that malloc() is being called or will be in the future without anyone realising it's a no-no. However, I know where you're going with this, and I agree that it's something that needs addressing. It's a general problem that has been troubling me on another project. I don't know that I have the greatest solution, but I'll do a quick dump of some of the options here. 1. It is safe to create/initialise synchronization objects (ie. a mutex) in DllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL mutex could be created to be used for on-demand (first-access) initialisation of subsystems. I don't recall the equivalent DLL entry point on Unix platforms or it's semantics, but as an aside does anyone know if such entry points are standard across all Unix implementations that support threading? 2. Alternatively, each subsystem could use it's own init-on-lock mutex (a la PTHREAD_MUTEX_INITIALIZER) but such a primitive might not be supported on all platforms and would introduce tighter coupling which is counter to OpenSSL's thread library agnostic approach. 3. Point 1 does not address the situation where OpenSSL is built as a static library and included in some other DLL (and the FIPS support may require this to be the standard setup for many projects). It would be the responsibility of the wrapping DLL to call whatever OpenSSL function initialises the master mutex from it's own DllMain(). This can get hairy when the static library using OpenSSL is also provided as a static library -- the responsibility has to be carried carefully up the chain. 4. In it's most basic form (OpenSSL as a DLL) Point 1 bypasses the thread library agnostic approach that OpenSSL uses because there is no way for the user to first setup the mutex initialisation callbacks. I can't think of a good solution to this. Creating a wrapping DLL who's DllMain() sets up the callbacks and then calls the OpenSSL master mutex initialisation function would work, but then the ability for the caller of that DLL to customise the mutex functions has been lost. Regards, Steven -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeffrey Altman Sent: Tuesday, 13 July 2004 11:14 PM To: [EMAIL PROTECTED] Subject: Re: possibly bug in crypto/rand/rand_win.c Suggestion: This is a usage error. Be sure to initialize openssl when your application starts or when your DLL is loaded. Do not wait for the first thread to attempt to make a call to OpenSSL to initialize it. Come to think of it, maybe OpenSSL should simply perform a call to RAND_poll() as part of the DLL initialization. This would solve many problems. Jeffrey Altman Jiang Lei wrote: Hi, Sorry if this message is sent twice. I got problem running RAND_poll
RE: OpenSSL thread safety (was: possibly bug in crypto/rand/rand_win.c)
Title: Message
Hi
Peter,
Those
platforms, minus Mac OS/X, plus OSF Tru64 (Digital Unix), has been my experience
also. I was hoping that init might be supported across the board.
Thanks for the info.
It's
my understanding so far that the OpenSSL FIPS certification requires use of a
static library build of OpenSSL. There was some discussion about ways to
allow shared libraries to be used but I don't think I saw a definitive
solution. I'll be happy if I just missed it.
Do you
think these issues should be addressed in OpenSSL, or are best left for the
caller to handle?
-Original Message-From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On
Behalf Of Peter WaltenbergSent: Wednesday, 14 July 2004 3:26
PMTo: [EMAIL PROTECTED]Subject: Re: OpenSSL thread
safety (was: possibly bug in
crypto/rand/rand_win.c)1. It is
safe to create/initialise synchronization objects (ie. a mutex)
inDllMain() for DLL_PROCESS_ATTACH. A single master OpenSSL
mutex could becreated to be used for on-demand (first-access)
initialisation ofsubsystems. I don't recall the equivalent DLL
entry point on Unix platformsor it's semantics, but as an aside does
anyone know if such entry points arestandard across all Unix
implementations that support threading?Yes to all the new Unix platforms we support with our
OpenSSL wrapper. (Solaris, Linux, AIX, HPUX, Mac OS/X). Support isn't uniform however, and is quirky on some
older systems. If you want to follow
this up I can probably provide example shared library entry point code and
link options for quite a few platforms. 3.
Point 1 does not address the situation where OpenSSL is built as a
staticlibrary and included in some other DLL (and the FIPS support may
requirethis to be the standard setup for many projects).
Static libraries make it hard to
draw the boundary around the FIPS certified code, which means you may end up
having to certify the whole ball of wax, (expensive). Shared libraries will
probably be more common. It would be
theresponsibility of the wrapping DLL to call whatever OpenSSL
functioninitialises the master mutex from it's own DllMain().
This can get hairywhen the static library using OpenSSL is also
provided as a static library-- the responsibility has to be carried
carefully up the chain.A hack
we had to add recently to support legacy systems where the shared lib
initialization code just doesn't get called. Peter
"Steven Reddie"
[EMAIL PROTECTED] Sent by: [EMAIL PROTECTED]
07/14/04 03:08 PM
Please respond
toopenssl-dev
To
[EMAIL PROTECTED]
cc
Subject
OpenSSL thread safety
(was: possibly bug in
crypto/rand/rand_win.c)
Hi Jeffrey, Come to think of it, maybe OpenSSL should
simply perform a call to RAND_poll() as part of the DLL
initialization. This would solve manyproblems.No, it would
create some. RAND_poll() calls LoadLibrary() which is a no-nofrom
DllMain(). malloc() is not safe to call from DllMain() when the
CRuntime is being dynamically linked in, and since RAND_poll() is so
complexit is possible that malloc() is being called or will be in the
futurewithout anyone realising it's a no-no.However, I know where
you're going with this, and I agree that it'ssomething that needs
addressing. It's a general problem that has beentroubling me on
another project. I don't know that I have the greatestsolution, but
I'll do a quick dump of some of the options here.1. It is safe to
create/initialise synchronization objects (ie. a mutex) inDllMain() for
DLL_PROCESS_ATTACH. A single master OpenSSL mutex could becreated to
be used for on-demand (first-access) initialisation ofsubsystems. I
don't recall the equivalent DLL entry point on Unix platformsor it's
semantics, but as an aside does anyone know if such entry points
arestandard across all Unix implementations that support
threading?2. Alternatively, each subsystem could use it's own
init-on-lock mutex (a laPTHREAD_MUTEX_INITIALIZER) but such a primitive
might not be supported onall platforms and would introduce tighter
coupling which is counter toOpenSSL's thread library agnostic
approach.3. Point 1 does not address the situation where OpenSSL is
built as a staticlibrary and included in some other DLL (and the FIPS
support may requirethis to be the standard setup for many projects).
It would be theresponsibility of the wrapping DLL to call whatever
OpenSSL functioninitialises the master mutex from it's own DllMain().
This can get hairywhen the static library using OpenS
