RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Thanks for the info. One last question :) So if I am using blocking sockets, than would I ever get a WANT_WRITE error? I'm guessing no? No, it should just block until it gets some application data or can send the application data. But if I am using BIO pairs, and blocking sockets, is it possible to get a WANT_WRITE error? Yes. Though why you would use BIO pairs with blocking sockets, I can't imagine. I'm guessing you meant non-blocking sockets. With BIO pairs, you get these errors all the time. The do BIO pairs properly, you need to do the four things I mentioned in my first response, and it works best if you make no assumptions about the relationships between those four things. The simplest approach is a loop that tries to do all four of those things and returns as soon as one pass through the loop makes no forward progress. Any time something changes, you call the loop. A change could be the socket getting a 'read' hit on select, or new application data to be encrypted when there was none before. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Thanks for the info. One last question :) So if I am using blocking sockets, than would I ever get a WANT_WRITE error? I'm guessing no? But if I am using BIO pairs, and blocking sockets, is it possible to get a WANT_WRITE error? Sorry, that was two more questions. Ed -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Sunday, April 17, 2005 11:01 PM To: openssl-users@openssl.org Subject: Re: Confusion about SSL_ERROR_WANT_READ/WRITE If all that was sent was the protocol data that the write was waiting for to satisfy the ssl state machine, and no application data was sent, would SSL_read return the number of bytes actually read off the socket (which is just protocol data), or would it read that transparently and return 0 indicating that no application data was read? Ah. Key question! SSL_read will return a positive number indicating the number of APPLICATION DATA bytes written into your buffer. A ZERO indicates a closed connection. A negative result indicates an error (or rather, that your request could not be satisfied). In the case of a WANT_READ or WANT_WRITE, that some action in the BIO needs to occur to satisfy the request. The important thing to keep in mind is that the SSL objects are not inherently tied to sockets. You might be trying to read SSL decrypted data from your own internal buffer. In which case, a WANT_READ means that you need to move a few more bytes into the BIO's buffer. Check out the man page for the SSL_get_error function yet once again. Skip down to the section titled SSL_ERROR_WANT_READ, SSL_ERROR_WANT_WRITE in the context of these discussions, keeping in mind the idea that an SSL object might not be necessarily connected to a socket. (What really frustrated me when I was learning how this worked was that the examples and discussions in the O'Reilly OpenSSL book were wrong on this topic.) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
If all that was sent was the protocol data that the write was waiting for to satisfy the ssl state machine, and no application data was sent, would SSL_read return the number of bytes actually read off the socket (which is just protocol data), or would it read that transparently and return 0 indicating that no application data was read? Ah. Key question! SSL_read will return a positive number indicating the number of APPLICATION DATA bytes written into your buffer. A ZERO indicates a closed connection. A negative result indicates an error (or rather, that your request could not be satisfied). In the case of a WANT_READ or WANT_WRITE, that some action in the BIO needs to occur to satisfy the request. The important thing to keep in mind is that the SSL objects are not inherently tied to sockets. You might be trying to read SSL decrypted data from your own internal buffer. In which case, a WANT_READ means that you need to move a few more bytes into the BIO's buffer. Check out the man page for the SSL_get_error function yet once again. Skip down to the section titled SSL_ERROR_WANT_READ, SSL_ERROR_WANT_WRITE in the context of these discussions, keeping in mind the idea that an SSL object might not be necessarily connected to a socket. (What really frustrated me when I was learning how this worked was that the examples and discussions in the O'Reilly OpenSSL book were wrong on this topic.) smime.p7s Description: S/MIME cryptographic signature
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Yes, I think I understand what you are saying. If I get a WANT_READ from a call to SSL_write, that means I need to read some data before I can send. Not quite, it means the OpenSSL engine must read some data (from the socket) before you can perform the 'write' logical operation on the connection state machine. But like you said, there may not be any data to read since the other end may not have sent anything. There may not be any application data, but there should be data sent over the SSL connection. But I think my problem was that I was thinking in terms of application data. What I failed to realize was that there may not be any application data to read, but if the other end is a valid ssl client, there should have been some ssl protocol data that was sent, that my end needs to read before my call to SSL_write will succeed. Does that sound right? If by your end, you mean your end of the SSL connection, yes. If by your end, you mean the application, no. The purpose of the SSL_read function is to read application data from the SSL connection state machine. You should call it if and only if that is what you want to do. And since an SSL_read may write as well as read, and SSL_write may read as well as write, then either of these calls would read the required protocol data such that a retry of the call that resulted in the error should now succeed. There you go. Since you're using socket BIOs, the state machine will access the socket when it needs to, so you just need to retry the operation later. If you want, you can use 'select' to tell when it's enough later. So eventhough my call to SSL_write resulted in the WANT_READ error, if my read thread happened to do an SSL_read first, it still would have read the protocol data, and my retry of SSL_write should succeed. Am I right? Close? Way off? If either an SSL_write or an SSL_read results in a WANT_READ error, it means that neither call can progress until some data is read from the socket. You can retry the operation later, try another operation, or whatever you want to do. You can take the hint that 'select'ing on the socket for readability will likely tell you when the operation is going to succeed. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Yes, I think I understand what you are saying. If I get a WANT_READ from a call to SSL_write, that means I need to read some data before I can send. Not quite, it means the OpenSSL engine must read some data (from the socket) before you can perform the 'write' logical operation on the connection state machine. But like you said, there may not be any data to read since the other end may not have sent anything. There may not be any application data, but there should be data sent over the SSL connection. Protocol data? Like an ack for some previous data sent? But I think my problem was that I was thinking in terms of application data. What I failed to realize was that there may not be any application data to read, but if the other end is a valid ssl client, there should have been some ssl protocol data that was sent, that my end needs to read before my call to SSL_write will succeed. Does that sound right? If by your end, you mean your end of the SSL connection, yes. If by your end, you mean the application, no. The purpose of the SSL_read function is to read application data from the SSL connection state machine. You should call it if and only if that is what you want to do. Since I'm using socket BIO, I am letting SSL_read/SSL_write handle all my socket I/O. So I am not explicitly reading from the socket and feeding it to OpenSSL (I just call SSL_read). So when I say your end, I mean the other end of the socket. And since an SSL_read may write as well as read, and SSL_write may read as well as write, then either of these calls would read the required protocol data such that a retry of the call that resulted in the error should now succeed. There you go. Since you're using socket BIOs, the state machine will access the socket when it needs to, so you just need to retry the operation later. If you want, you can use 'select' to tell when it's enough later. So eventhough my call to SSL_write resulted in the WANT_READ error, if my read thread happened to do an SSL_read first, it still would have read the protocol data, and my retry of SSL_write should succeed. Am I right? Close? Way off? If either an SSL_write or an SSL_read results in a WANT_READ error, it means that neither call can progress until some data is read from the socket. You can retry the operation later, try another operation, or whatever you want to do. You can take the hint that 'select'ing on the socket for readability will likely tell you when the operation is going to succeed. I do select on the socket. Basically, I have a thread pool that I use for I/O. Writes are synchronous, so I expect to finish writing all the data before I exit my write function. But since I don't want to tie up a thread blocking on the read waiting for data to arrive (since I have no idea when data will arrive), I add it to a list of sockets that I am select'ing on. Since my write is synchronous, and if I get a WANT_READ error, then that means I need to read some ssl data before I can continue. So I will select on the socket until data arrives. I'm assuming that the data WILL arrive. There is no chance that I could be blocked here indefinitely is there? I'm assuming that the data is some SSL protocol data that is SHOULD have been sent by the other end of the connection (assuming it is a valid SSL client). Now, I also have a read thread that was select'ing on the socket waiting for data to arrive. So either of these 2 threads may read data. Both threads are select'ing on the socket. So if the read thread wakes up first and acquires the lock, then it will do an SSL_read before the write thread wakes up and retries an SSL_write (which was the operation that caused the WANT_READ error in the first place). So my question is, is this ok? If it was an SSL_write that caused the WANT_READ error, do I HAVE to retry the SSL_write before I can do an SSL_read? The SSL_read should read whatever data the ssl state machine was expecting, and the next try of SSL_write should then succeed right? __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
There may not be any application data, but there should be data sent over the SSL connection. Protocol data? Like an ack for some previous data sent? Well, remember no data at all can be sent until a key is negotiated. So if you immediately call SSL_write, it will be unable to do anything. If either an SSL_write or an SSL_read results in a WANT_READ error, it means that neither call can progress until some data is read from the socket. You can retry the operation later, try another operation, or whatever you want to do. You can take the hint that 'select'ing on the socket for readability will likely tell you when the operation is going to succeed. I do select on the socket. Basically, I have a thread pool that I use for I/O. Writes are synchronous, so I expect to finish writing all the data before I exit my write function. But since I don't want to tie up a thread blocking on the read waiting for data to arrive (since I have no idea when data will arrive), I add it to a list of sockets that I am select'ing on. Since my write is synchronous, and if I get a WANT_READ error, then that means I need to read some ssl data before I can continue. So I will select on the socket until data arrives. I'm assuming that the data WILL arrive. There is no chance that I could be blocked here indefinitely is there? I'm assuming that the data is some SSL protocol data that is SHOULD have been sent by the other end of the connection (assuming it is a valid SSL client). You can impose timeouts if you want. You have this same issue for TCP. If the other side doesn't read any data, eventually your 'write' will block forever. You have to handle this yourself. Now, I also have a read thread that was select'ing on the socket waiting for data to arrive. So either of these 2 threads may read data. Both threads are select'ing on the socket. So if the read thread wakes up first and acquires the lock, then it will do an SSL_read before the write thread wakes up and retries an SSL_write (which was the operation that caused the WANT_READ error in the first place). So my question is, is this ok? Yes. Just understand that it's not unusual to see data on the socket (in a call to 'select') and then not get any *application* data from SSL_read. If it was an SSL_write that caused the WANT_READ error, do I HAVE to retry the SSL_write before I can do an SSL_read? No. You will likely need to enable SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER and will probably want to enable SSL_MODE_ENABLE_PARTIAL_WRITE as well. Read up on these and make sure you understand the implications and especially the very unusual default of net accepting moving write buffers! The SSL_read should read whatever data the ssl state machine was expecting, and the next try of SSL_write should then succeed right? Yes. Just remember that there are two weird cases that could happen -- expect them. 1) You get a read hit from 'select', but before you can call SSL_read, your write thread calls SSL_write, which reads the data. So now when you call SSL_read, nothing at all happens. 2) You get a read hit from 'select', but it's all protocol data, no application data. So you call SSL_read and no application data is returned. And, of course, remember that you need a mutex for the connection to prevent a concurrent SSL_read and SSL_write. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
First of all, I want to thank everyone for all the information, especially David with his excellent explanations. I know this thread is getting long, but I really think I'm getting it now :) Just a few more questions and comments... There may not be any application data, but there should be data sent over the SSL connection. Protocol data? Like an ack for some previous data sent? Well, remember no data at all can be sent until a key is negotiated. So if you immediately call SSL_write, it will be unable to do anything. Of course :) If either an SSL_write or an SSL_read results in a WANT_READ error, it means that neither call can progress until some data is read from the socket. You can retry the operation later, try another operation, or whatever you want to do. You can take the hint that 'select'ing on the socket for readability will likely tell you when the operation is going to succeed. I do select on the socket. Basically, I have a thread pool that I use for I/O. Writes are synchronous, so I expect to finish writing all the data before I exit my write function. But since I don't want to tie up a thread blocking on the read waiting for data to arrive (since I have no idea when data will arrive), I add it to a list of sockets that I am select'ing on. Since my write is synchronous, and if I get a WANT_READ error, then that means I need to read some ssl data before I can continue. So I will select on the socket until data arrives. I'm assuming that the data WILL arrive. There is no chance that I could be blocked here indefinitely is there? I'm assuming that the data is some SSL protocol data that is SHOULD have been sent by the other end of the connection (assuming it is a valid SSL client). You can impose timeouts if you want. You have this same issue for TCP. If the other side doesn't read any data, eventually your 'write' will block forever. You have to handle this yourself. Of course. But what I mean is, if I get a WANT_READ from an SSL_write, than I assume that means I am waiting for some protocol data to satisfy the ssl state machine, right? After all, SSL_write should not be waiting for any application data. So if that is the case, does that mean that the protocol data that I am waiting for SHOULD have been sent by the other end of the connection? Now, I also have a read thread that was select'ing on the socket waiting for data to arrive. So either of these 2 threads may read data. Both threads are select'ing on the socket. So if the read thread wakes up first and acquires the lock, then it will do an SSL_read before the write thread wakes up and retries an SSL_write (which was the operation that caused the WANT_READ error in the first place). So my question is, is this ok? Yes. Just understand that it's not unusual to see data on the socket (in a call to 'select') and then not get any *application* data from SSL_read. If it was an SSL_write that caused the WANT_READ error, do I HAVE to retry the SSL_write before I can do an SSL_read? No. You will likely need to enable SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER and will probably want to enable SSL_MODE_ENABLE_PARTIAL_WRITE as well. Read up on these and make sure you understand the implications and especially the very unusual default of net accepting moving write buffers! The SSL_read should read whatever data the ssl state machine was expecting, and the next try of SSL_write should then succeed right? Yes. Just remember that there are two weird cases that could happen -- expect them. 1) You get a read hit from 'select', but before you can call SSL_read, your write thread calls SSL_write, which reads the data. So now when you call SSL_read, nothing at all happens. Will SSL_read return 0 bytes read, or will I get a WANT_READ error indicating there was nothing to be read since the data was already read off the socket? 2) You get a read hit from 'select', but it's all protocol data, no application data. So you call SSL_read and no application data is returned. Does SSL_read always return the number of bytes of application data read? If so, that means that SSL_read could return 0, and that this should not be construed as an error. And, of course, remember that you need a mutex for the connection to prevent a concurrent SSL_read and SSL_write. Of course. That was my first mistake. But I know better now :) DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
A return result of 0 typically means the other side closed the connection. Here is the section from SSL_read's man page with regards to a 0 return: 0 The read operation was not successful. The reason may either be a clean shutdown due to a close notify alert sent by the peer (in which case the SSL_RECEIVED_SHUTDOWN flag in the ssl shutdown state is set (see SSL_shutdown(3), SSL_set_shutdown(3)). It is also pos- sible, that the peer simply shut down the underlying transport and the shutdown is incomplete. Call SSL_get_error() with the return value ret to find out, whether an error occurred or the connection was shut down cleanly (SSL_ERROR_ZERO_RETURN). -Joe On Apr 17, 2005, at 9:12 PM, Edward Chan wrote: Does SSL_read always return the number of bytes of application data read? If so, that means that SSL_read could return 0, and that this should not be construed as an error. smime.p7s Description: S/MIME cryptographic signature
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Right, but let's say I'm doing an SSL_write, and I get a WANT_READ error. I then select on the socket until data is available for reading. I then call SSL_read. If all that was sent was the protocol data that the write was waiting for to satisfy the ssl state machine, and no application data was sent, would SSL_read return the number of bytes actually read off the socket (which is just protocol data), or would it read that transparently and return 0 indicating that no application data was read? Or would it just read the required protocol data and return an error of WANT_READ to indicate that I should retry the SSL_read when more data arrives? Now that I think about it, I'm guessing the latter. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Sunday, April 17, 2005 9:48 PM To: openssl-users@openssl.org Subject: Re: Confusion about SSL_ERROR_WANT_READ/WRITE A return result of 0 typically means the other side closed the connection. Here is the section from SSL_read's man page with regards to a 0 return: 0 The read operation was not successful. The reason may either be a clean shutdown due to a close notify alert sent by the peer (in which case the SSL_RECEIVED_SHUTDOWN flag in the ssl shutdown state is set (see SSL_shutdown(3), SSL_set_shutdown(3)). It is also pos- sible, that the peer simply shut down the underlying transport and the shutdown is incomplete. Call SSL_get_error() with the return value ret to find out, whether an error occurred or the connection was shut down cleanly (SSL_ERROR_ZERO_RETURN). -Joe On Apr 17, 2005, at 9:12 PM, Edward Chan wrote: Does SSL_read always return the number of bytes of application data read? If so, that means that SSL_read could return 0, and that this should not be construed as an error. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
You're right -- the latter. Another thing to think about is that at any time, the remote peer might request a re-negotiation. During such time, the session key will be re-established requiring a few round-trips during the DH process. This will all be handled behind the scenes as you attempt to move application data through the system. If the SSL state machine is waiting on some remote data to be received during the re-negotiation, you will get a WANT_READ in response to an SSL_write (or an SSL_read). At this point, you could call select() waiting for data to arrive. When select() indicates that the socket has data ready, you can just call SSL_write (or SSL_read) again so that the state machine can work its way through the protocol. In my program's case, I had to periodically call SSL_read() on a non-blocking socket so that I could detect whenever the remote peer closed the connection, even though I was not expecting any application data to arrive. On Apr 17, 2005, at 10:20 PM, Edward Chan wrote: Right, but let's say I'm doing an SSL_write, and I get a WANT_READ error. I then select on the socket until data is available for reading. I then call SSL_read. If all that was sent was the protocol data that the write was waiting for to satisfy the ssl state machine, and no application data was sent, would SSL_read return the number of bytes actually read off the socket (which is just protocol data), or would it read that transparently and return 0 indicating that no application data was read? Or would it just read the required protocol data and return an error of WANT_READ to indicate that I should retry the SSL_read when more data arrives? Now that I think about it, I'm guessing the latter. smime.p7s Description: S/MIME cryptographic signature
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Thanks for this explanation. As I read more, I think I am getting a better understanding of this. So unlike normal tcp connections, where a read juts reads, and a write just writes, SSL_read may write, and SSL_write may read. This is all done under the hood, so I don't need to be concerned with that, except to reissue the call when I get a WANT_READ or WANT_WRITE error. And when I get one of these, I basically just have to wait (select/poll or whatever) until the socket is readable/writable, then reissue the call. Does that sound right? Yes, that's it. If you use socket BIOs, then it all takes place under the hood. You don't have to worry about it, but you do have to know that the semantics of SSL_read and SSL_write are not the same as read and write. And regarding the use of multiple threads, if I protect the SSL object with a lock, that should be fine right? But it sounds like a single thread for both read and writes is the norm. Is this true? And if so, other than the fact that I need to co-ordinate access to the SSL obj with a mutex, is there any draw back to using multiple threads? Neither is the norm. Some I/O strategies use a single thread both reading and writing, where that thread may handle only one connection or dozens. Some I/O strategies use one thread for all reads to all connections and one for all writes to all connections. Some use a pool of threads, any one of which may do a read or write to any connection at any time. What is best depends upon the specifics of a given project, primarily the scalability requirements and the complexity that can be tolerated. One common I/O strategy called 'speculative write' allows whatever thread generated data for a connection to try to write it immediately. If the write fails with a 'would block' error, then the connection is added to a poll or select set to try the write later from an I/O thread. In this case, you would need a lock because one thread might try to write to the connection while an I/O thread is reading from it. The SSL state machine is not protected against concurrent accesses to the same connection. So if you have a situation where you might try to access the same connection from two threads (the typical case being a read and a write, but one could imagine others), you will need to associate a mutex with the connection. Semantically, an SSL connection is a single engine and SSL_read and SSL_write are entry points to that single engine. This is different from a TCP connection which is semantically two unrelated byte streams, one in each direction. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Ok, this is getting much clearer. Last question (hopefully)...so if an SSL_write gets a WANT_READ, is it ok for the read thread to do an SSL_read before I retry the SSL_write? Does it matter who does the requested operation as long as it is done? Or does the read thread have to wait until the write thread retries the SSL_write when there is data available before it can read anymore data? And similarly, if the read thread gets a WANT_WRITE, can the write thread do an SSL_write before the read thread retries the SSL_read? If the write thread does an SSL_write before the read thread retries the SSL_read (assuming socket is writable), will it have written whatever data the SSL_read needed to have written? In other words, can the operation specified the WANT_READ/WRITE have to be done by retrying the operation that caused it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Saturday, April 16, 2005 3:02 AM To: openssl-users@openssl.org Subject: RE: Confusion about SSL_ERROR_WANT_READ/WRITE Thanks for this explanation. As I read more, I think I am getting a better understanding of this. So unlike normal tcp connections, where a read juts reads, and a write just writes, SSL_read may write, and SSL_write may read. This is all done under the hood, so I don't need to be concerned with that, except to reissue the call when I get a WANT_READ or WANT_WRITE error. And when I get one of these, I basically just have to wait (select/poll or whatever) until the socket is readable/writable, then reissue the call. Does that sound right? Yes, that's it. If you use socket BIOs, then it all takes place under the hood. You don't have to worry about it, but you do have to know that the semantics of SSL_read and SSL_write are not the same as read and write. And regarding the use of multiple threads, if I protect the SSL object with a lock, that should be fine right? But it sounds like a single thread for both read and writes is the norm. Is this true? And if so, other than the fact that I need to co-ordinate access to the SSL obj with a mutex, is there any draw back to using multiple threads? Neither is the norm. Some I/O strategies use a single thread both reading and writing, where that thread may handle only one connection or dozens. Some I/O strategies use one thread for all reads to all connections and one for all writes to all connections. Some use a pool of threads, any one of which may do a read or write to any connection at any time. What is best depends upon the specifics of a given project, primarily the scalability requirements and the complexity that can be tolerated. One common I/O strategy called 'speculative write' allows whatever thread generated data for a connection to try to write it immediately. If the write fails with a 'would block' error, then the connection is added to a poll or select set to try the write later from an I/O thread. In this case, you would need a lock because one thread might try to write to the connection while an I/O thread is reading from it. The SSL state machine is not protected against concurrent accesses to the same connection. So if you have a situation where you might try to access the same connection from two threads (the typical case being a read and a write, but one could imagine others), you will need to associate a mutex with the connection. Semantically, an SSL connection is a single engine and SSL_read and SSL_write are entry points to that single engine. This is different from a TCP connection which is semantically two unrelated byte streams, one in each direction. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
You're on the money. This confused me, too. I had a program that needed to see if there was incoming data, and so I performed an SSL_read(). I received back a WANT_READ, because there was no data yet to read. (I'm using non-blocking I/O). But then some time later I needed to send data. The logic of the program was such that I could expect nothing on the READ side anyway until I had sent something first (query/response). At first, I thought I was stuck having to endlessly perform only the SSL_read even though there was no data available before I would be able to perform my SSL_write. I realized that when you receive a WANT_READ or a WANT_WRITE, you just need to perform the same operation again with the same parameters, but that does not exclude you from performing the other operation elsewhere. Just make sure that two threads aren't trying to do this at the same time on the same connection. On Apr 16, 2005, at 10:22 AM, Edward Chan wrote: Ok, this is getting much clearer. Last question (hopefully)...so if an SSL_write gets a WANT_READ, is it ok for the read thread to do an SSL_read before I retry the SSL_write? Does it matter who does the requested operation as long as it is done? Or does the read thread have to wait until the write thread retries the SSL_write when there is data available before it can read anymore data? And similarly, if the read thread gets a WANT_WRITE, can the write thread do an SSL_write before the read thread retries the SSL_read? If the write thread does an SSL_write before the read thread retries the SSL_read (assuming socket is writable), will it have written whatever data the SSL_read needed to have written? In other words, can the operation specified the WANT_READ/WRITE have to be done by retrying the operation that caused it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Saturday, April 16, 2005 3:02 AM To: openssl-users@openssl.org Subject: RE: Confusion about SSL_ERROR_WANT_READ/WRITE Thanks for this explanation. As I read more, I think I am getting a better understanding of this. So unlike normal tcp connections, where a read juts reads, and a write just writes, SSL_read may write, and SSL_write may read. This is all done under the hood, so I don't need to be concerned with that, except to reissue the call when I get a WANT_READ or WANT_WRITE error. And when I get one of these, I basically just have to wait (select/poll or whatever) until the socket is readable/writable, then reissue the call. Does that sound right? Yes, that's it. If you use socket BIOs, then it all takes place under the hood. You don't have to worry about it, but you do have to know that the semantics of SSL_read and SSL_write are not the same as read and write. And regarding the use of multiple threads, if I protect the SSL object with a lock, that should be fine right? But it sounds like a single thread for both read and writes is the norm. Is this true? And if so, other than the fact that I need to co-ordinate access to the SSL obj with a mutex, is there any draw back to using multiple threads? Neither is the norm. Some I/O strategies use a single thread both reading and writing, where that thread may handle only one connection or dozens. Some I/O strategies use one thread for all reads to all connections and one for all writes to all connections. Some use a pool of threads, any one of which may do a read or write to any connection at any time. What is best depends upon the specifics of a given project, primarily the scalability requirements and the complexity that can be tolerated. One common I/O strategy called 'speculative write' allows whatever thread generated data for a connection to try to write it immediately. If the write fails with a 'would block' error, then the connection is added to a poll or select set to try the write later from an I/O thread. In this case, you would need a lock because one thread might try to write to the connection while an I/O thread is reading from it. The SSL state machine is not protected against concurrent accesses to the same connection. So if you have a situation where you might try to access the same connection from two threads (the typical case being a read and a write, but one could imagine others), you will need to associate a mutex with the connection. Semantically, an SSL connection is a single engine and SSL_read and SSL_write are entry points to that single engine. This is different from a TCP connection which is semantically two unrelated byte streams, one in each direction. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Yes, I think I understand what you are saying. If I get a WANT_READ from a call to SSL_write, that means I need to read some data before I can send. But like you said, there may not be any data to read since the other end may not have sent anything. But I think my problem was that I was thinking in terms of application data. What I failed to realize was that there may not be any application data to read, but if the other end is a valid ssl client, there should have been some ssl protocol data that was sent, that my end needs to read before my call to SSL_write will succeed. Does that sound right? And since an SSL_read may write as well as read, and SSL_write may read as well as write, then either of these calls would read the required protocol data such that a retry of the call that resulted in the error should now succeed. So eventhough my call to SSL_write resulted in the WANT_READ error, if my read thread happened to do an SSL_read first, it still would have read the protocol data, and my retry of SSL_write should succeed. Am I right? Close? Way off? Ed -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Saturday, April 16, 2005 10:43 AM To: openssl-users@openssl.org Subject: Re: Confusion about SSL_ERROR_WANT_READ/WRITE You're on the money. This confused me, too. I had a program that needed to see if there was incoming data, and so I performed an SSL_read(). I received back a WANT_READ, because there was no data yet to read. (I'm using non-blocking I/O). But then some time later I needed to send data. The logic of the program was such that I could expect nothing on the READ side anyway until I had sent something first (query/response). At first, I thought I was stuck having to endlessly perform only the SSL_read even though there was no data available before I would be able to perform my SSL_write. I realized that when you receive a WANT_READ or a WANT_WRITE, you just need to perform the same operation again with the same parameters, but that does not exclude you from performing the other operation elsewhere. Just make sure that two threads aren't trying to do this at the same time on the same connection. On Apr 16, 2005, at 10:22 AM, Edward Chan wrote: Ok, this is getting much clearer. Last question (hopefully)...so if an SSL_write gets a WANT_READ, is it ok for the read thread to do an SSL_read before I retry the SSL_write? Does it matter who does the requested operation as long as it is done? Or does the read thread have to wait until the write thread retries the SSL_write when there is data available before it can read anymore data? And similarly, if the read thread gets a WANT_WRITE, can the write thread do an SSL_write before the read thread retries the SSL_read? If the write thread does an SSL_write before the read thread retries the SSL_read (assuming socket is writable), will it have written whatever data the SSL_read needed to have written? In other words, can the operation specified the WANT_READ/WRITE have to be done by retrying the operation that caused it? -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Saturday, April 16, 2005 3:02 AM To: openssl-users@openssl.org Subject: RE: Confusion about SSL_ERROR_WANT_READ/WRITE Thanks for this explanation. As I read more, I think I am getting a better understanding of this. So unlike normal tcp connections, where a read juts reads, and a write just writes, SSL_read may write, and SSL_write may read. This is all done under the hood, so I don't need to be concerned with that, except to reissue the call when I get a WANT_READ or WANT_WRITE error. And when I get one of these, I basically just have to wait (select/poll or whatever) until the socket is readable/writable, then reissue the call. Does that sound right? Yes, that's it. If you use socket BIOs, then it all takes place under the hood. You don't have to worry about it, but you do have to know that the semantics of SSL_read and SSL_write are not the same as read and write. And regarding the use of multiple threads, if I protect the SSL object with a lock, that should be fine right? But it sounds like a single thread for both read and writes is the norm. Is this true? And if so, other than the fact that I need to co-ordinate access to the SSL obj with a mutex, is there any draw back to using multiple threads? Neither is the norm. Some I/O strategies use a single thread both reading and writing, where that thread may handle only one connection or dozens. Some I/O strategies use one thread for all reads to all connections and one for all writes to all connections. Some use a pool of threads, any one of which may do a read or write to any connection at any time. What is best depends upon
Confusion about SSL_ERROR_WANT_READ/WRITE
Title: Confusion about SSL_ERROR_WANT_READ/WRITE I have an app where reads and writes happen from different threads. Now, ideally, one would envision that I just replace the reads/writes with SSL_read/SSL_write. Now I know it is not as simple as that. What exactly is the meaning of the SSL_ERROR_WANT_READ/WRITE errors? If I get I get a WANT_READ from an SSL_read, I guess that means I need to read more data but no data is available on the socket yet. So I basically poll/select waiting for more data to arrive before I can reissue the SSL_read. Is that right? If I get a WANT_WRITE from an SSL_read, does that mean that I need to do an SSL_write before I can reissue the SSL_read? And if so, what if I have no data to write? Can I just do an SSL_write() with an empty buffer? I'm assuming it is not application data that needs to be written, but some data contained in some internal ssl buffer? If I get a WANT_WRITE from an SSL_write, does that mean the network buffer is full and I cannot write to the socket yet? In which case, I would just poll/select until the socket becomes writable and retry the SSL_write? What if I get a WANT_READ from an SSL_write? Does that mean that I need to do an SSL_read before I can reissue the SSL_write? Again, I'm assuming the data that needs to be read is not application data. So can I just do an SSL_read giving it a 0 length buffer? Will this cause whatever data that is stored in the ssl buffers to be processed without reading data off the network (since I have a separate thread for reading and processing application data)? Ed
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
Whats wrong with select? Or are you using java that does not support select() On Fri, 2005-04-15 at 18:24, Edward Chan wrote: I have an app where reads and writes happen from different threads. Now, ideally, one would envision that I just replace the reads/writes with SSL_read/SSL_write. Now I know it is not as simple as that. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: Confusion about SSL_ERROR_WANT_READ/WRITE
On Fri, Apr 15, 2005, Edward Chan wrote: I have an app where reads and writes happen from different threads. Now, ideally, one would envision that I just replace the reads/writes with SSL_read/SSL_write. Now I know it is not as simple as that. If you read and write the same stream in different threads that's a problem as this isn't supported. Some people have reported success by using appropriate locking. What exactly is the meaning of the SSL_ERROR_WANT_READ/WRITE errors? Irrespective of the call that returned that error it has the same meaning: the underlying transport (socket) has indicated that data cannot be currently read or written and that the call should be retried when that is possible. So if you get SSL_ERROR_WANT_READ you might typically select on the socket until data can be read. If you get SSL_ERROR_WANT_WRITE you do the same but until data can be written: due to internal buffering you wont often see SSL_ERROR_WANT_WRITE. Steve. -- Dr Stephen N. Henson. Email, S/MIME and PGP keys: see homepage OpenSSL project core developer and freelance consultant. Funding needed! Details on homepage. Homepage: http://www.drh-consultancy.demon.co.uk __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
I have an app where reads and writes happen from different threads. Now, ideally, one would envision that I just replace the reads/writes with SSL_read/SSL_write. Now I know it is not as simple as that. You need to wrap each SSL connection with a lock and hold that lock when you call SSL_read or SSL_write. This will prevent concurrent accesses to the same connection from different threads, which is not supported. What exactly is the meaning of the SSL_ERROR_WANT_READ/WRITE errors? [snip] The OpenSSL connection does not have exactly the same semantics as a TCP connection. Say you try to send data before the handshaking is finished. OpenSSL cannot send any data over the socket until it reads the handshake from the other side.So a 'WANT_READ' error means that OpenSSL needs to read some encrypted data from the other side before it can write the application data you want to send. The way you deal with these is just by not doing the thing that the error stops you from doing until you've made some forward progress. There are four things you are happening: 1) If the application wants to send some plaintext, that plaintext has to go OpenSSL to encrypt. 2) If OpenSSL has some decrypted data, it need to get to the application. 3) If some encrypted data is (ready on / received on) the socket, it needs to get to OpenSSL. 4) If OpenSSL has some encrypted data to send, and the socket is ready to receive, the data needs to b sent. These operations inter-relate. Sometimes it's obvious, for example, you can't receive any decrypted data until the encrypted data is ready on the socket. However, sometimes it's not obvious. So say you go to send some data using SSL_write and you get 'WANT_READ'. That means OpenSSL wants to read some encrypted data from the other side before it can do the send. So you could, for example, 'select' on the socket and when there's data to read, call OpenSSL again. It will then do step 3 itself. If you go to receive some data using SSL_read and get 'WANT_WRITE, that means OpenSSL can't receive any data because it has to send some data to the other side first. So you could 'select' for write to wait for the socket buffer to drain and then call OpenSSL again. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: Confusion about SSL_ERROR_WANT_READ/WRITE
Thanks for this explanation. As I read more, I think I am getting a better
understanding of this. So unlike normal tcp connections, where a read juts
reads, and a write just writes, SSL_read may write, and SSL_write may read.
This is all done under the hood, so I don't need to be concerned with that,
except to reissue the call when I get a WANT_READ or WANT_WRITE error. And
when I get one of these, I basically just have to wait (select/poll or
whatever) until the socket is readable/writable, then reissue the call.
Does that sound right?
And regarding the use of multiple threads, if I protect the SSL object with
a lock, that should be fine right? But it sounds like a single thread for
both read and writes is the norm. Is this true? And if so, other than the
fact that I need to co-ordinate access to the SSL obj with a mutex, is there
any draw back to using multiple threads?
So if I had the following:
/* Read thread */
bool ok = false;
while (!ok)
{
mutex.lock(); // protect ssl
int ret = SSL_read(ssl, buf, len);
int err = SSL_get_error(ssl, ret);
mutex.unlock();
if (err == SSL_ERROR_NONE)
{
ok = true;
}
else if (err == SSL_ERROR_WANT_READ)
{
fd_set read_fds;
FD_ZERO(read_fds);
FD_SET(m_sock, read_fds);
// wait for socket to be readable
if (select(1, read_fds, 0, 0, 0) = 0)
return 0; // error
continue; // re-issue the read
}
else if (err == SSL_ERROR_WANT_WRITE)
{
fd_set write_fds;
FD_ZERO(write_fds);
FD_SET(m_sock, write_fds);
// wait for socket to be wriable
if (select(1, 0, write_fds, 0, 0) = 0)
return 0; // error
continue; // re-issue the read
}
else
{
return 0; // error
}
}
/* write thread */
int offset = 0;
while (len)
{
mutex.lock();
int ret = SSL_write(ssl, buf+offset, len);
int err = SSL_get_error(ssl, ret);
mutex.unlock();
if (err == SSL_ERROR_NONE)
{
offset += ret;
len -= ret;
}
else if (err == SSL_ERROR_WANT_READ)
{
fd_set read_fds;
FD_ZERO(read_fds);
FD_SET(m_sock, read_fds);
// wait for socket to be readable
if (select(1, read_fds, 0, 0, 0) = 0)
return 0; // error
continue; // re-issue the write
}
else if (err == SSL_ERROR_WANT_WRITE)
{
fd_set write_fds;
FD_ZERO(write_fds);
FD_SET(m_sock, write_fds);
// wait for socket to be writable
if (select(1, 0, write_fds, 0, 0) = 0)
return 0; // error
continue; // re-issue the write
}
else
{
return 0; // error
}
}
Does that look ok?
Since these the read and writes may be done in different threads, than it
could happen that the write thread got a WANT_READ and was waiting for data
to arrive. But the read thread may also be waiting for data to arrive. One
of these threads will wake up first. If the read thread wakes up, it will
do SSL_read. If the write thread wakes up, it will try a SSL_write. Only
one will happen first because they are protected by a lock. But if the read
thread was able to read first. Then when the write thread acquires the lock
and retries the SSL_write, it will still succeed because whatever data it
was waiting to read was read by the read thread. Does that make sense?
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
Sent: Friday, April 15, 2005 4:58 PM
To: openssl-users@openssl.org
Subject: RE: Confusion about SSL_ERROR_WANT_READ/WRITE
I have an app where reads and writes happen from different threads.
Now, ideally, one would envision that I just replace the
reads/writes
with SSL_read/SSL_write. Now I know it is not as simple as that.
You need to wrap each SSL connection with a lock and
hold that lock when you call SSL_read or SSL_write. This will
prevent concurrent accesses to the same connection from
different threads, which is not supported.
What exactly is the meaning of the SSL_ERROR_WANT_READ/WRITE errors?
[snip]
The OpenSSL connection does not have exactly the same
semantics as a TCP connection. Say you try to send data
before the handshaking is finished.
OpenSSL cannot send any data over the socket until it reads
the handshake from the other side.So a 'WANT_READ' error
means that OpenSSL needs to read some encrypted data from the
other side before it can write the application data you want to send
