[openssl-users] how to complete non-blocking IO with multiple worker threads ?
Environment: Linux (Fedora/RHEL) I have a client server implementation with the server waiting for connections via an epoll fd. Many worker threads epoll_wait( ) on the epoll fd. This implementation has the static thread-id and thread-locking callbacks registered to handle multiple threads to use the SSL API. However, the current implementation also spawns a dedicated thread to SSL_accept( )/SSL_connect( )/SSL_read( )/SSL_write( ) with SSL connections and works perfectly. I want to discard the dedicated thread implementation for SSL and use the worker threads waiting in epoll_wait( ) to service the non-blocking IO requests as is done for non-SSL communication, using the SSL API of course. Can this be done ? So, is there a way that such a multi-threaded non-blocking IO client/server application can complete the IO via various worker threads when a WANT_READ/WANT_WRITE is returned by SSL_get_error( ) Also, can WANT_READ/WANT_WRITE returned from SSL_accept( )/SSL_connect( ) also be serviced with non-blocking IO and via different worker threads ? I couldn't find anything pertaining to this in the archives. Please point me to one if I missed it. Regards, Milind -- openssl-users mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users
OpenSSL-1.0.1c non-blocking io int_thread_release, bad reference count
With OpenSSL-1.0.1c downloaded from openssl.org and compiled from source
on x86 Linux 2.6 (LinuxMint 12) using gcc-4.6.1, I get an abort with
following error
int_thread_release, bad reference count
Analyzing situation...
(gdb) bt
#0 0x00920416 in __kernel_vsyscall ()
#1 0x0037ac8f in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0x0037e2b5 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0x0020efae in int_thread_release (hash=0x42a4e18) at err.c:479
#4 0x0020f059 in int_thread_get_item (d=0x42a4e4c) at err.c:499
#5 0x002100de in ERR_get_state () at err.c:1014
#6 0x0020f9ff in get_error_values (inc=1, top=0, file=0x42a50b4,
line=0x42a50c0, data=0x42a50b8, flags=0x42a50bc) at err.c:798
#7 0x0020f88f in ERR_get_error_line_data (file=0x42a50b4, line=0x42a50c0,
data=0x42a50b8, flags=0x42a50bc) at err.c:766
#8 0x0808b3ca in zx_report_openssl_err (logkey=0x8142e19 SSL again read)
at zxsig.c:399
#9 0x08062d2e in hi_read (hit=0x42a5370, io=0xb67d8f44) at hiread.c:175
#10 0x08055245 in hi_in_out (hit=0x42a5370, io=0xb67d8f44) at hiios.c:294
#11 0x08056fa6 in hi_shuffle (hit=0x42a5370, shf=0x93b1d28) at hiios.c:348
#12 0x0804fdbf in thread_loop (_shf=0x93b1d28) at zxbusd.c:564
#13 0x007eed31 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#14 0x0042046e in clone () from /lib/i386-linux-gnu/libc.so.6
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb)
This is statistically reproducible, i.e. it does not happen every time,
but seems to happen about 20% of the time with 10 worker threads. With
one or two threads I have not seen it happen.
I am somewhat at loss about what would be the best way to debug this
or what would be the most useful input to give for you to debug it.
Peculiarities of my situation
1. I use nonblocking I/O with epoll(2) based I/O loop
2. I use late accept approach, calling SSL_set_accept_state(io-ssl)
at accept time and relying on SSL_read() to do the rest, i.e.
I never call SSL_accept().
3. I have multiple worker threads synchronized such that one thread
may be reading while other is writing, but never two threads
simultaneously reading or simultaneously writing. At accept time
there is no data on write queue so in reality only the read thread
will happen (I realize that in the bowels of SSL_read() a write(2)
system call may actually be made, but I do not think there is
any other intefering thread at this time).
4. I have multiple SSL connections, sharing SSL_CTX, i.e. the SSL_CTX
has multithreaded access.
5. I suspect the problem happens during accept processing as no data
seems to come through the connection.
6. This is somewhat of a Heisenbug as turning on debug prints,
such as SSL_CTX_set_info_callback() that would allow me to
see if it is in accept processing, makes the bug go away.
The openssl was configured and compiled as follows:
sampo-debug-linux-pentium, gcc:-DREF_CHECK -DCONF_DEBUG -DL_ENDIAN
-DTERMIO -g -march=pentium -Wall::-D_REENTRANT::-ldl:BN_LLONG ${x86_gcc_des}
${x86_gcc_opts}:${x86_elf_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
PATH=/usr/bin:/bin ./Configure --prefix=/apps/openssl/1.0.1c-gcc-4.6.1-debug
enable-rc5 enable-mdc2 no-dso no-hw no-engine zlib sampo-debug-linux-pentium
PATH=/usr/bin:/bin make install
Then when I compile my own code I get following link error
/usr/bin/cc -o zxid zxid.o -L. -lzxid -L/apps/openssl/std/lib -L/apps/lib
-pthread -lpthread -L/apps/lib -L/apps/openssl/std/lib -lcurl -lssl -lcrypto -lz
/usr/bin/ld: warning: type and size of dynamic symbol `DES_SPtrans' are not
defined
/usr/bin/ld: dynamic variable `DES_SPtrans' is zero size
/usr/bin/ld: /apps/openssl/std/lib/libcrypto.a(crypt586.o)(.text+0xa):
unresolvable R_386_32 relocation against symbol `DES_SPtrans'
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
make: *** [zxid] Error 1
Seems DES_SPtrans is referenced in des/crypt586.s and defined in des-586.s
Moving definition of fcrypt_body from crypt586.s to des-586.s seems to
solve the problem, though I doubt this is the right fix.
Perhaps the pentium assembly code has not been tested/used in ages?
Cheers,
--Sampo
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager majord...@openssl.org
Re: OpenSSL-1.0.1c non-blocking io int_thread_release, bad reference count
One more significant peculiarity of my situation I forgot to mention:
7. I am using ClientTLS authentication (using self signed cert)
Cheers,
--Sampo
sa...@zxid.org said:
With OpenSSL-1.0.1c downloaded from openssl.org and compiled from source
on x86 Linux 2.6 (LinuxMint 12) using gcc-4.6.1, I get an abort with
following error
int_thread_release, bad reference count
Analyzing situation...
(gdb) bt
#0 0x00920416 in __kernel_vsyscall ()
#1 0x0037ac8f in raise () from /lib/i386-linux-gnu/libc.so.6
#2 0x0037e2b5 in abort () from /lib/i386-linux-gnu/libc.so.6
#3 0x0020efae in int_thread_release (hash=0x42a4e18) at err.c:479
#4 0x0020f059 in int_thread_get_item (d=0x42a4e4c) at err.c:499
#5 0x002100de in ERR_get_state () at err.c:1014
#6 0x0020f9ff in get_error_values (inc=1, top=0, file=0x42a50b4,
line=0x42a50c0, data=0x42a50b8, flags=0x42a50bc) at err.c:798
#7 0x0020f88f in ERR_get_error_line_data (file=0x42a50b4, line=0x42a50c0,
data=0x42a50b8, flags=0x42a50bc) at err.c:766
#8 0x0808b3ca in zx_report_openssl_err (logkey=0x8142e19 SSL again read)
at zxsig.c:399
#9 0x08062d2e in hi_read (hit=0x42a5370, io=0xb67d8f44) at hiread.c:175
#10 0x08055245 in hi_in_out (hit=0x42a5370, io=0xb67d8f44) at hiios.c:294
#11 0x08056fa6 in hi_shuffle (hit=0x42a5370, shf=0x93b1d28) at hiios.c:348
#12 0x0804fdbf in thread_loop (_shf=0x93b1d28) at zxbusd.c:564
#13 0x007eed31 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#14 0x0042046e in clone () from /lib/i386-linux-gnu/libc.so.6
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb)
This is statistically reproducible, i.e. it does not happen every time,
but seems to happen about 20% of the time with 10 worker threads. With
one or two threads I have not seen it happen.
I am somewhat at loss about what would be the best way to debug this
or what would be the most useful input to give for you to debug it.
Peculiarities of my situation
1. I use nonblocking I/O with epoll(2) based I/O loop
2. I use late accept approach, calling SSL_set_accept_state(io-ssl)
at accept time and relying on SSL_read() to do the rest, i.e.
I never call SSL_accept().
3. I have multiple worker threads synchronized such that one thread
may be reading while other is writing, but never two threads
simultaneously reading or simultaneously writing. At accept time
there is no data on write queue so in reality only the read thread
will happen (I realize that in the bowels of SSL_read() a write(2)
system call may actually be made, but I do not think there is
any other intefering thread at this time).
4. I have multiple SSL connections, sharing SSL_CTX, i.e. the SSL_CTX
has multithreaded access.
5. I suspect the problem happens during accept processing as no data
seems to come through the connection.
6. This is somewhat of a Heisenbug as turning on debug prints,
such as SSL_CTX_set_info_callback() that would allow me to
see if it is in accept processing, makes the bug go away.
The openssl was configured and compiled as follows:
sampo-debug-linux-pentium, gcc:-DREF_CHECK -DCONF_DEBUG -DL_ENDIAN
-DTERMIO -g -march=pentium -Wall::-D_REENTRANT::-ldl:BN_LLONG ${x86_gcc_des}
${x86_gcc_opts}:${x86_elf_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
PATH=/usr/bin:/bin ./Configure --prefix=/apps/openssl/1.0.1c-gcc-4.6.1-debug
enable-rc5 enable-mdc2 no-dso no-hw no-engine zlib sampo-debug-linux-pentium
PATH=/usr/bin:/bin make install
Then when I compile my own code I get following link error
/usr/bin/cc -o zxid zxid.o -L. -lzxid -L/apps/openssl/std/lib -L/apps/lib
-pthread -lpthread -L/apps/lib -L/apps/openssl/std/lib -lcurl -lssl -lcrypto
-lz
/usr/bin/ld: warning: type and size of dynamic symbol `DES_SPtrans' are not
defined
/usr/bin/ld: dynamic variable `DES_SPtrans' is zero size
/usr/bin/ld: /apps/openssl/std/lib/libcrypto.a(crypt586.o)(.text+0xa):
unresolvable R_386_32 relocation against symbol `DES_SPtrans'
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
make: *** [zxid] Error 1
Seems DES_SPtrans is referenced in des/crypt586.s and defined in des-586.s
Moving definition of fcrypt_body from crypt586.s to des-586.s seems to
solve the problem, though I doubt this is the right fix.
Perhaps the pentium assembly code has not been tested/used in ages?
Cheers,
--Sampo
__
OpenSSL Project http://www.openssl.org
User Support Mailing Listopenssl-users@openssl.org
Automated List Manager majord...@openssl.org
__
OpenSSL Project http://www.openssl.org
User Support Mailing List
Re: OpenSSL-1.0.1c non-blocking io int_thread_release, bad reference count
On 2012-09-25 16:48 +0200 (Tue), sa...@zxid.org wrote: ...I get an abort with following error int_thread_release, bad reference count 3. I have multiple worker threads synchronized such that one thread may be reading while other is writing, but never two threads simultaneously reading or simultaneously writing. According to the OpenSSL FAQ: 1. Is OpenSSL thread-safe? Yes (with limitations: an SSL connection may not concurrently be used by multiple threads). I may be wrong here, but it seems to me that if you have one thread reading and a different thread writing, that's two threads concurrently using one connection. And underneath the covers, as it were, both threads may be both reading and writing, since a read or write to an OpenSSL handle can translate to both reads and writes on the underlying TCP handle. cjs -- Curt Sampson c...@cynic.net +81 90 7737 2974 It is easier to write an incorrect program than understand a correct one. --Alan Perlis, Epigrams on Programming (#7) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: OpenSSL-1.0.1c non-blocking io int_thread_release, bad reference count
Curt Sampson c...@cynic.net said: On 2012-09-25 16:48 +0200 (Tue), sa...@zxid.org wrote: ...I get an abort with following error int_thread_release, bad reference count 3. I have multiple worker threads synchronized such that one thread may be reading while other is writing, but never two threads simultaneously reading or simultaneously writing. According to the OpenSSL FAQ: 1. Is OpenSSL thread-safe? Yes (with limitations: an SSL connection may not concurrently be used by multiple threads). I may be wrong here, but it seems to me that if you have one thread reading and a different thread writing, that's two threads concurrently using one connection. And underneath the covers, as it were, both threads may be both reading and writing, since a read or write to an OpenSSL handle can translate to both reads and writes on the underlying TCP handle. I have in past, say 2008 vintage OpenSSL, used OpenSSL in this way without problem. Basically the logic is that the server will not try to write payload data before it gets request from client, therefore during the accept processing there is effectively only one thread, the reader, that will do both reading and writing. Once you are past the accept handshake, the reading and writing are unidirectional per thread and this should be possible to make work. But of course it is possible that a new handshake happens in midconnection. I wonder if there is any way to prevent that. I'll try adding light debug prints and more locks to see if your theory is right (problem with debug prints is the heisenbug effect). Cheers, --Sampo cjs -- Curt Sampson c...@cynic.net +81 90 7737 2974 It is easier to write an incorrect program than understand a correct one. --Alan Perlis, Epigrams on Programming (#7) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: OpenSSL-1.0.1c non-blocking io int_thread_release, bad reference count
On 2012-09-27 07:19 +0200 (Thu), sa...@zxid.org wrote: Once you are past the accept handshake, the reading and writing are unidirectional per thread and this should be possible to make work. But of course it is possible that a new handshake happens in midconnection. As you point out, in the event of a handshake, it's not unidirectional per thread. But the other thing to point out is that even if you never have a read on an OpenSSL handle produce a write on the underlying handle, or vice versa, it may still be the case that OpenSSL's internal locking of data structures won't properly handle a simultaneous read on one thread and write from a different thread. cjs -- Curt Sampson c...@cynic.net +81 90 7737 2974 It is easier to write an incorrect program than understand a correct one. --Alan Perlis, Epigrams on Programming (#7) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
using non-blocking IO
I am using an IMAP library, which supports only blocking SSL IO. I am planning to convert my that to do non-blocking SSL IO. Which is the best place to set the flag to non-blocking? These are my concerns: 1. What areas of the program will have to be converted because of this? re SSL_read and SSL_write the only functions which will get affected? 2. Which is the most suitable place to set the underlying IO to non-blocking? Is it after the connection is setup and certificates are validated, or before those parts? If we set the non-blocking before certificate validation and negotiation, will that make any difference in those part? 4. How non-blocking may affect the renegotiation? -- Sabyasachi __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
non-blocking IO and revocation checking
From a brief look at the cert validation code, it appears that there's no way to do non-blocking IO while performing on-line revocation checks, e.g. querying a CDP or an OCSP responder. Is that correct? Specifically, I see in ssl_cert.c a call is made to X509_verify_cert(), or a user-supplied verification callback. It looks like X509_verify_cert() only checks CRLs, and only CRLs that have been previously stored in the context. So, no CDP lookups. Further, it looks like if a user-supplied callback is provided that does a CDP lookup, there's no way for the callback to return a would-block error, or for the handshake to be retried once the revocation data is available. Is that correct? b.c. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: non-blocking IO and revocation checking
On Mon, Jun 04, 2007, Brian Craft wrote: From a brief look at the cert validation code, it appears that there's no way to do non-blocking IO while performing on-line revocation checks, e.g. querying a CDP or an OCSP responder. Is that correct? Specifically, I see in ssl_cert.c a call is made to X509_verify_cert(), or a user-supplied verification callback. It looks like X509_verify_cert() only checks CRLs, and only CRLs that have been previously stored in the context. So, no CDP lookups. Further, it looks like if a user-supplied callback is provided that does a CDP lookup, there's no way for the callback to return a would-block error, or for the handshake to be retried once the revocation data is available. Is that correct? Currently no. Adding support via special return codes is not that hard to do but would cause problems for existing applications. The reason is that they would not know how to handle the condition. It could behave in a manner similar to the existing non-blocking I/O but it would be via a different socket (e.g. the one downloading the CRL or performing OCSP). Steve. -- Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage OpenSSL project core developer and freelance consultant. Funding needed! Details on homepage. Homepage: http://www.drh-consultancy.demon.co.uk __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: Non-blocking IO
Perry L. Jones wrote: can some one please point me to an example of non-blocking IO reading with openssl. Thanks, Perry __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] Yes, The source code for the O'Reilly book on OpenSSL is freely downloadable from ora.com you will have to search for the books title in it, is an example on how to use non-blocking ssl and managing the various states of connection using ssl_read and ssl_write. Hope That Helps Joe S. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Non-blocking IO
can some one please point me to an example of non-blocking IO reading with openssl. Thanks, Perry __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Non-blocking IO
Hi, You could have a look at apps/s_client.c code looking for nbio option. It seems that there is two way of doing such thing: - craete nbio BIO and then connect, etc.. - connect (a socket for example) and then set non blocking IO (with the socket utilities) hope it could help Fred -Original Message- From: Perry L. Jones [mailto:[EMAIL PROTECTED] Sent: Tue 11/22/2005 2:25 PM To: openssl-users@openssl.org Cc: Subject:Non-blocking IO can some one please point me to an example of non-blocking IO reading with openssl. Thanks, Perry __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
hardware accelerator/non-blocking IO
Hi Does openssl provide an api for non-blocking IO over an SSL hardware accelerator? Cheers Paul This message contains confidential information and is intended only for the named individual and may not be disseminated without prior permission. If you are not the named addressee, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this message in error and delete this e-message from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, delayed in transmission, incomplete, or may contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this Message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. This message is provided for informational purposes and should not be construed as a solicitation or offer to buy or sell any software or services. This email has been scanned for all viruses by the MessageLabs SkyScan service. http://www.messagelabs.com
new non-blocking IO / state-machine demo
Hi there, Given the recent wave of threads on the openssl lists, I thought I'd get round to doing something I meant to do some time ago - and hack up a quick demo. Anyway, I've just committed it into the openssl CVS repository, so it should be present in the next nightly snapshot (downloadable from www.openssl.org) and it's inside the openssl/demos/tunala/ directory. The reason for "tunala" can be found in the header file. :-) Anyway - this was a very fast attempt to get an SSL tunneler up and running to illustrate the idea of having an abstract memory-based state-machine with the application-specifics of network-IO and what-not placed around the outside of it. I've tried to explain where things go and how it works inside the README in that directory as well as sporadic source comments wherever I felt the need or motivation. Now it's there, I'm sure I'll get round to actually testing a few things only to find it is full of bugs, but the principle should be sound. I've based it conceptually around something much more substantial I've been working on lately that itself would be too bulky to stick in as a "demo" and has dependencies on loads of other stuff - but the principle is working fine for me elsewhere so if there's bugs in "tunala", they're probably fixable reasonably straightforwardly. Well, take a look if it is of interest. There's been a lot of discussion lately about a wide array of relevant issues; async-IO as an alternative to threading/forking etc, non-blocking logic and avoiding "deadlocks" with SSL, using OpenSSL in a purely data-only capacity, etc. For some time I thought it would be handy to put my thoughts down in code rather than emails, so please take a fish around inside it if you're curious. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List [EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
Non-blocking IO examples?
Hi all, Does anyone have any examples of Non-blocking IO using OpenSSL? A simple client and a simple server perhaps? Thanks, Ben --- perl -wle '$_ = 1; (1 x $_) !~ /^(11+)\1+$/ print while $_++' __ OpenSSL Project http://www.openssl.org User Support Mailing List[EMAIL PROTECTED] Automated List Manager [EMAIL PROTECTED]
