Re: Blocking on a non-blocking socket?
- Original Message - > From: "Wiebe Cazemier" > To: [email protected] > Sent: Thursday, 23 May, 2024 12:22:31 > Subject: Blocking on a non-blocking socket? > > Hi List, > > I have a very obscure problem with an application using O_NONBLOCK still > blocking. Over the course of a year of running with hundreds of thousands of > clients, it has happened twice over the last month that a worker thread froze. > It's a long story, but I'm pretty sure it's not a deadlock or spinning event > loop or something, primarily because the application recovers after about 20 > minutes with a client errorring out with ETIMEDOUT. Coincidentally, that 20 > minutes matches the timeout description of the tcp man page [1]. > > It really looks like a non-blocking socket is still blocking. I found > something > with a similar problem ([2]), but what they think of SSL_MODE_AUTO_RETRY does > not match the documentation. > > So, is there indeed any way an application that has SSL_MODE_AUTO_RETRY on > (which is default since 1.1.1) can block? Looking at the source code, I don't > see any calls to fcntl() that removes the O_NONBLOCK. > > My IO method is SSL_read() and SSL_write() with an SSL object given to > SSL_set_fd(). > > The only SSL modes I change from the default is that I set > SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER. > > There are two primary deployments of this application, one with OpenSSL 1.1.1 > and one with 3.0.0. Only 1.1.1 has shown this problem, but it may be a > coincidence. > > Side question, is it a problem to set SSL_set_fd() before using fcntl to set > the > fd to O_NONBLOCK? I ask, because the docs say "The BIO and hence the SSL > engine > inherit the behaviour of fd. If fd is non-blocking, the ssl will also have > non-blocking behaviour.". The 'inherit' may be a key word here; not sure when > it's done. > > Regards, > > Wiebe Cazemier As a follow-up, the fault did turn out to be my own... As I imagine [1] is. They describe SSL_MODE_AUTO_RETRY 'attempts to renegotiate a broken SSL connection', but all SSL_MODE_AUTO_RETRY indeed really does is read multiple records at a time, without returning from read. Despite what I thought before, my code actually did have an unfortunate edge case where there was a while loop spinning on SSL_write() when there was no room in the socket. This would eventually fail with ETIMEDOUT. Well, it was educational at least... [1] https://github.com/alanxz/rabbitmq-c/issues/586
Re: Blocking on a non-blocking socket?
On 24/05/2024 02:30, Wiebe Cazemier wrote: Can you show me in the code where that is? It's here: https://github.com/openssl/openssl/blob/b9e084f139c53ce133e66aba2f523c680141c0e6/ssl/record/rec_layer_s3.c#L1038-L1054 The "retry" codepath occurs where we hit the "goto start". My main concern is, if it would get an EWOULDBLOCK, there is (almost) no sense in retrying because in the 100 microseconds or so that passed, there is likely still no data. This situation does not occur. The "auto-retry" only occurs in the case where we have *successfully* read a non-application data record. If we get an EWOULDBLOCK then this is always propagated back to the application. > Wouldn't the option then have to be called 'read more than one record at a time'? To me, 'retry' is a bit of a misnomer in that description. The "retry" here is the normal meaning of the English word, and does not refer to a "network" retry, i.e. we tried to read application data but actually got something else, so retry that attempt. Matt
Re: Blocking on a non-blocking socket?
Hi Detlef, - Original Message - > From: "Detlef Vollmann" > To: [email protected] > Sent: Friday, 24 May, 2024 12:02:37 > Subject: Re: Blocking on a non-blocking socket? > > That's correct, but if I understand Matt correctly, this isn't the case. > The idea of SSL_MODE_AUTO_RETRY is that if there's data, but it isn't > application data but some kind of handshake data, then SSL_read doesn't > return (after handling the handshake data), but immediately retries. > If this retry fails with EWOULDBLOCK (or actually BIO_read returns 0), > then SSL_read returns with 0 and SSL_WANT_READ. Wouldn't the option then have to be called 'read more than one record at a time'? To me, 'retry' is a bit of a misnomer in that description. Tracing the code, the retry seems to be considered based on BIO_fd_non_fatal_error(), which looks at EWOULDBLOCK. See [1] and [2]. Wiebe [1] https://github.com/openssl/openssl/blob/b9e084f139c53ce133e66aba2f523c680141c0e6/crypto/bio/bss_fd.c#L226 [2] https://github.com/openssl/openssl/blob/b9e084f139c53ce133e66aba2f523c680141c0e6/crypto/bio/bss_fd.c#L113
Re: Blocking on a non-blocking socket?
On 5/24/24 03:30, Wiebe Cazemier via openssl-users wrote: Hi Matt, - Original Message - From: "Matt Caswell" To: [email protected] Sent: Friday, 24 May, 2024 00:26:28 Subject: Re: Blocking on a non-blocking socket? Not quite. When you call SSL_read() it is because you are hoping to read application data. OpenSSL will go ahead and attempt to read a record from the socket. If there is no data (and you are using a non-blocking socket), or only a partial record available then the SSL_read() call will fail and indicate SSL_ERROR_WANT_READ. If a full record is available it will process it. If the record contains application data then the SSL_read() call will return successfully and provide the application data to the application. If the record contains non-application data (i.e. some TLS protocol message like a key update, or new session ticket) then, with SSL_MODE_AUTO_RETRY on it will automatically try and read another record (and the above process repeats). Can you show me in the code where that is? It seems the callers of BIO_read() [1] are responsible for doing the retry, because the reader functions abort when retry is set. Those are many callers, for x509, evp, b64, etc. But, the code is kind of hard to trace, because it's all calls to bio_method_st.bread function pointers. My main concern is, if it would get an EWOULDBLOCK, there is (almost) no sense in retrying That's correct, but if I understand Matt correctly, this isn't the case. The idea of SSL_MODE_AUTO_RETRY is that if there's data, but it isn't application data but some kind of handshake data, then SSL_read doesn't return (after handling the handshake data), but immediately retries. If this retry fails with EWOULDBLOCK (or actually BIO_read returns 0), then SSL_read returns with 0 and SSL_WANT_READ. Without SSL_MODE_AUTO_RETRY SSL_read would return after handling the handshake data with 0 and SSL_WANT_READ. I'm not sure how useful this is, but this is how I understand Matt. Detlef
Re: Blocking on a non-blocking socket?
Hi Matt, - Original Message - > From: "Matt Caswell" > To: [email protected] > Sent: Friday, 24 May, 2024 00:26:28 > Subject: Re: Blocking on a non-blocking socket? > Not quite. > > When you call SSL_read() it is because you are hoping to read > application data. > > OpenSSL will go ahead and attempt to read a record from the socket. If > there is no data (and you are using a non-blocking socket), or only a > partial record available then the SSL_read() call will fail and indicate > SSL_ERROR_WANT_READ. > > If a full record is available it will process it. If the record contains > application data then the SSL_read() call will return successfully and > provide the application data to the application. > > If the record contains non-application data (i.e. some TLS protocol > message like a key update, or new session ticket) then, with > SSL_MODE_AUTO_RETRY on it will automatically try and read another record > (and the above process repeats). Can you show me in the code where that is? It seems the callers of BIO_read() [1] are responsible for doing the retry, because the reader functions abort when retry is set. Those are many callers, for x509, evp, b64, etc. But, the code is kind of hard to trace, because it's all calls to bio_method_st.bread function pointers. My main concern is, if it would get an EWOULDBLOCK, there is (almost) no sense in retrying because in the 100 microseconds or so that passed, there is likely still no data. Plus, is there a limit on how often it's retried? If the connection is broken (packet loss, so nobody is aware) in the middle of rekeying, it can retry all it wants, but nothing will ever come. If it does that, then at some point, reads on the socket would fail with ETIMEDOUT, which is what I'm seeing. [1] https://github.com/openssl/openssl/blob/b9e084f139c53ce133e66aba2f523c680141c0e6/crypto/bio/bio_lib.c#L303
RE: Blocking on a non-blocking socket?
On Thursday, May 23, 2024 10:26 AM Matt Caswell wrote: >On 23/05/2024 15:08, [email protected] wrote: >> On Thursday, May 23, 2024 9:56 AM, Wiebe Cazemier wrote: From: "Neil Horman" from: [ https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html | https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html ] >>> SSL_MODE_AUTO_RETRY in non-blocking mode should cause SSL_reaa/SSL_write to return -1 with an error code of WANT_READ/WANT_WRITE until such time as the re-negotiation has completed. I need to confirm thats the case in the code, but it seems to be. If the underlying socket is in non-blocking mode, there should be no way for calls to block in SSL_read/SSL_write on the socket read/write system >>> call. >>> >>> I still don't really see what the difference is between >>> SSL_MODE_AUTO_RETRY on or off in non-blocking mode? >>> >>> The person at [1] seems to have had a similar issue, and was >>> convinced clearing SSL_MODE_AUTO_RETRY fixed it. But I agree, I don't know >how it could be. >>> OpenSSL would have to remove the O_NONBLOCK, or do select/poll, and I >>> can't find it doing that. >>> >>> I hope it happens again soon and I'm around to attach a debugger. >> >> I may be incorrect here, but my interpretation is as follows: >> >> SSL_MODE_AUTO_RETRY on - if there is a packet ready to read on the socket, >> the >packet is retrieved. Same for write. If not ready, because EWOULDBLOCK, the >operation is retried automatically by OpenSSL. >> >> SSL_MODE_AUTO_RETRY off - if there is a packet ready to read on the socket, >> the >packet is retrieved. Same for write. If not ready, the OpenSSL operation >reports an >error. > >Not quite. > >When you call SSL_read() it is because you are hoping to read application data. > >OpenSSL will go ahead and attempt to read a record from the socket. If there >is no >data (and you are using a non-blocking socket), or only a partial record >available >then the SSL_read() call will fail and indicate SSL_ERROR_WANT_READ. > >If a full record is available it will process it. If the record contains >application data >then the SSL_read() call will return successfully and provide the application >data to >the application. > >If the record contains non-application data (i.e. some TLS protocol message >like a >key update, or new session ticket) then, with SSL_MODE_AUTO_RETRY on it will >automatically try and read another record (and the above process repeats). If >SSL_MODE_AUTO_RETRY off it will not attempt to retry and the SSL_read() call >will >fail and indicate SSL_ERROR_WANT_READ. > > From an application perspective, if SSL_MODE_AUTO_RETRY is off with a non- >blocking socket, it is not possible to tell the difference between "no >record/only >partial record is available" and "we tried to read application data but got a >non- >application data record". They both result in SSL_read() failing and indicating >SSL_ERROR_WANT_READ. > >For non-blocking mode it really doesn't make much difference to the >application. >Either way it should not cause it to block. Thanks Matt. That helps with my own understanding.
Re: Blocking on a non-blocking socket?
On 23/05/2024 15:08, [email protected] wrote: On Thursday, May 23, 2024 9:56 AM, Wiebe Cazemier wrote: From: "Neil Horman" from: [ https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html | https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html ] SSL_MODE_AUTO_RETRY in non-blocking mode should cause SSL_reaa/SSL_write to return -1 with an error code of WANT_READ/WANT_WRITE until such time as the re-negotiation has completed. I need to confirm thats the case in the code, but it seems to be. If the underlying socket is in non-blocking mode, there should be no way for calls to block in SSL_read/SSL_write on the socket read/write system call. I still don't really see what the difference is between SSL_MODE_AUTO_RETRY on or off in non-blocking mode? The person at [1] seems to have had a similar issue, and was convinced clearing SSL_MODE_AUTO_RETRY fixed it. But I agree, I don't know how it could be. OpenSSL would have to remove the O_NONBLOCK, or do select/poll, and I can't find it doing that. I hope it happens again soon and I'm around to attach a debugger. I may be incorrect here, but my interpretation is as follows: SSL_MODE_AUTO_RETRY on - if there is a packet ready to read on the socket, the packet is retrieved. Same for write. If not ready, because EWOULDBLOCK, the operation is retried automatically by OpenSSL. SSL_MODE_AUTO_RETRY off - if there is a packet ready to read on the socket, the packet is retrieved. Same for write. If not ready, the OpenSSL operation reports an error. Not quite. When you call SSL_read() it is because you are hoping to read application data. OpenSSL will go ahead and attempt to read a record from the socket. If there is no data (and you are using a non-blocking socket), or only a partial record available then the SSL_read() call will fail and indicate SSL_ERROR_WANT_READ. If a full record is available it will process it. If the record contains application data then the SSL_read() call will return successfully and provide the application data to the application. If the record contains non-application data (i.e. some TLS protocol message like a key update, or new session ticket) then, with SSL_MODE_AUTO_RETRY on it will automatically try and read another record (and the above process repeats). If SSL_MODE_AUTO_RETRY off it will not attempt to retry and the SSL_read() call will fail and indicate SSL_ERROR_WANT_READ. From an application perspective, if SSL_MODE_AUTO_RETRY is off with a non-blocking socket, it is not possible to tell the difference between "no record/only partial record is available" and "we tried to read application data but got a non-application data record". They both result in SSL_read() failing and indicating SSL_ERROR_WANT_READ. For non-blocking mode it really doesn't make much difference to the application. Either way it should not cause it to block. Matt
RE: Blocking on a non-blocking socket?
On Thursday, May 23, 2024 9:56 AM, Wiebe Cazemier wrote: >> From: "Neil Horman" >> from: >> [ https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html | >> https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html ] > >> SSL_MODE_AUTO_RETRY in non-blocking mode should cause >> SSL_reaa/SSL_write to return -1 with an error code of >> WANT_READ/WANT_WRITE until such time as the re-negotiation has >> completed. I need to confirm thats the case in the code, but it seems >> to be. If the underlying socket is in non-blocking mode, there should >> be no way for calls to block in SSL_read/SSL_write on the socket read/write >> system >call. > >I still don't really see what the difference is between SSL_MODE_AUTO_RETRY on >or >off in non-blocking mode? > >The person at [1] seems to have had a similar issue, and was convinced clearing >SSL_MODE_AUTO_RETRY fixed it. But I agree, I don't know how it could be. >OpenSSL would have to remove the O_NONBLOCK, or do select/poll, and I can't >find it doing that. > >I hope it happens again soon and I'm around to attach a debugger. I may be incorrect here, but my interpretation is as follows: SSL_MODE_AUTO_RETRY on - if there is a packet ready to read on the socket, the packet is retrieved. Same for write. If not ready, because EWOULDBLOCK, the operation is retried automatically by OpenSSL. SSL_MODE_AUTO_RETRY off - if there is a packet ready to read on the socket, the packet is retrieved. Same for write. If not ready, the OpenSSL operation reports an error. Randall
Re: Blocking on a non-blocking socket?
Hi Neil, - Original Message - > From: "Neil Horman" > To: "Wiebe Cazemier" > Cc: "udhayakumar" , [email protected] > Sent: Thursday, 23 May, 2024 23:42:18 > Subject: Re: Blocking on a non-blocking socket? > from: > [ https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html | > https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html ] > SSL_MODE_AUTO_RETRY in non-blocking mode should cause SSL_reaa/SSL_write to > return -1 with an error code of WANT_READ/WANT_WRITE until such time as the > re-negotiation has completed. I need to confirm thats the case in the code, > but > it seems to be. If the underlying socket is in non-blocking mode, there should > be no way for calls to block in SSL_read/SSL_write on the socket read/write > system call. I still don't really see what the difference is between SSL_MODE_AUTO_RETRY on or off in non-blocking mode? The person at [1] seems to have had a similar issue, and was convinced clearing SSL_MODE_AUTO_RETRY fixed it. But I agree, I don't know how it could be. OpenSSL would have to remove the O_NONBLOCK, or do select/poll, and I can't find it doing that. I hope it happens again soon and I'm around to attach a debugger. Regards, Wiebe [1] https://github.com/alanxz/rabbitmq-c/issues/586
Re: Blocking on a non-blocking socket?
from: https://www.openssl.org/docs/man1.0.2/man3/SSL_CTX_set_mode.html SSL_MODE_AUTO_RETRY in non-blocking mode should cause SSL_reaa/SSL_write to return -1 with an error code of WANT_READ/WANT_WRITE until such time as the re-negotiation has completed. I need to confirm thats the case in the code, but it seems to be. If the underlying socket is in non-blocking mode, there should be no way for calls to block in SSL_read/SSL_write on the socket read/write system call. On Thu, May 23, 2024 at 8:15 AM Wiebe Cazemier wrote: > - Original Message - > > From: "Neil Horman" > > To: "udhayakumar" > > Cc: "Wiebe Cazemier" , [email protected] > > Sent: Thursday, 23 May, 2024 22:05:22 > > Subject: Re: Blocking on a non-blocking socket? > > > do you have a stack trace of the thread hung in this state? That would > confirm > > whats going on here > > Neil > > Hi Neil, > > No, I don't. I wasn't there to attach a debugger. It recovered before I > could do that. And despite a lot of effort, I can't reproduce it either. > > But in general, what does SSL_MODE_AUTO_RETRY on/off change in > non-blocking mode? The documentation is too vague for me. It says: > > > Setting SSL_MODE_AUTO_RETRY for a nonblocking BIO will process > non-application data records until either no more data is available or an > application data record has been processed. > > But how is that different from disabling SSL_MODE_AUTO_RETRY? > > Regards, > > Wiebe >
Re: Blocking on a non-blocking socket?
- Original Message - > From: "Neil Horman" > To: "udhayakumar" > Cc: "Wiebe Cazemier" , [email protected] > Sent: Thursday, 23 May, 2024 22:05:22 > Subject: Re: Blocking on a non-blocking socket? > do you have a stack trace of the thread hung in this state? That would confirm > whats going on here > Neil Hi Neil, No, I don't. I wasn't there to attach a debugger. It recovered before I could do that. And despite a lot of effort, I can't reproduce it either. But in general, what does SSL_MODE_AUTO_RETRY on/off change in non-blocking mode? The documentation is too vague for me. It says: > Setting SSL_MODE_AUTO_RETRY for a nonblocking BIO will process > non-application data records until either no more data is available or an > application data record has been processed. But how is that different from disabling SSL_MODE_AUTO_RETRY? Regards, Wiebe
Re: Blocking on a non-blocking socket?
do you have a stack trace of the thread hung in this state? That would confirm whats going on here Neil On Wed, May 22, 2024 at 11:12 PM udhayakumar wrote: > hi Wiebe Cazemier, > > The flag SSL_MODE_AUTO_RETRY will cause read/write operations to only > return after the handshake and successful completion. if cable is unplugged > connection got broken until re-authentication it's holding read/write > operations i think. > > > / udhay!. > > On 5/23/24 7:52 AM, Wiebe Cazemier via openssl-users wrote: > > e only SSL modes I change from the default is that I >
Re: Blocking on a non-blocking socket?
hi Wiebe Cazemier, The flag SSL_MODE_AUTO_RETRY will cause read/write operations to only return after the handshake and successful completion. if cable is unplugged connection got broken until re-authentication it's holding read/write operations i think. / udhay!. On 5/23/24 7:52 AM, Wiebe Cazemier via openssl-users wrote: e only SSL modes I change from the default is that I
