RE: SSL_read empty -> close?
> From: Felipe Gasper
> Sent: Thursday, 3 November, 2022 10:43
> >
> > And your description looks wrong anyway: shutdown(SHUT_RD) has
> > implementation-defined behavior for TCP sockets (because TCP does not
> > announce the read side of half-close to the peer), and on Linux causes
> > blocked receives and subsequent receives to return 0 (according to
> > references
>
> perl -MSocket -MIO::Socket::INET -e'my $s = IO::Socket::INET->new( Server =>
> 1, Listen => 1 ) or die; my $port = $s->sockport(); my $c = IO::Socket::INET-
> >new("localhost:$port") or die; syswrite $c, "hello"; my $sc = $s->accept();
> shutdown($sc, SHUT_RD); sysread $sc, my $buf, 512 or die $!; print $buf'
>
> ^^ The above, I believe, demonstrates to the contrary: the read buffer is
> populated prior to shutdown and drained afterward.
As I noted, I hadn't tested it. The Linux man page is ambiguous:
If how is SHUT_RD, further receptions will be disallowed.
It doesn't define "receptions". It's entirely possible that SHUT_RD will cause
the stack to reject further application data (i.e. packets that increment the
sequence number for anything other than ACK) from the peer, but permit the
socket owner to continue to receive already-buffered data. That's arguably a
poor implementation, and not what the man page appears to imply. And it looks
to be in conflict with the Single UNIX Specification Issue 7 (not that Linux
claims to be UNIX-conformant), which states that SHUT_SD "Disables further
receive operations"; "operations" certainly seems to refer to actions taken by
the caller, not by the peer.
There is a fair bit of debate about this online, and a number of people opine
that the Linux behavior is correct, and SUS (they often refer to "POSIX", but
POSIX has been superseded by SUS) is wrong. Others disagree.
The Linux kernel does take some action for a TCP socket that has SHUT_RD
requested for it, but the behavior is not simple. (One SO comment mentions it
causes it to exit the read loop in tcp_splice_read(), for example.) I'd be
leery about relying on it.
I'm not sure how shutdown(SHUT_RD) is useful in the case of a TCP socket being
used for TLS, to be perfectly honest. If the application protocol delimits
messages properly and is half-duplex (request/response), then one side should
know that no more data is expected and the other can detect incomplete
messages, so there's likely no issue. If not, there's no way to guarantee you
haven't encountered an incomplete message in bounded time (FPL Theorem
applies). SHUT_RD does not signal the peer, so the peer can still get a RST if
it continues to send. Perhaps I'm missing something, but I don't see what
failure mode is being avoided by using SHUT_RD.
--
Michael Wojcik
Re: SSL_read empty -> close?
> On Nov 3, 2022, at 11:37, Michael Wojcik via openssl-users
> wrote:
>
>> It’s a rare
>> issue, but when it does it’s a head-scratcher. To avoid that, it’s necessary
>> to shutdown(SHUT_RD) then drain the read buffer before close().
>
> Well, it's not *necessary* to do a half-close. Applications often know when
> they've received all the data the peer intends to send, thanks to
> record-delimiting mechanisms in the application protocol.
>
> And your description looks wrong anyway: shutdown(SHUT_RD) has
> implementation-defined behavior for TCP sockets (because TCP does not
> announce the read side of half-close to the peer), and on Linux causes
> blocked receives and subsequent receives to return 0 (according to references
> -- I have't tested it), which means after shutdown(SHUT_RD) you *can't* drain
> the receive buffer. shutdown(SHUT_WR) would work, since it sends a FIN,
> telling the peer you won't be sending any more data, and still allows you to
> receive.
perl -MSocket -MIO::Socket::INET -e'my $s = IO::Socket::INET->new( Server => 1,
Listen => 1 ) or die; my $port = $s->sockport(); my $c =
IO::Socket::INET->new("localhost:$port") or die; syswrite $c, "hello"; my $sc =
$s->accept(); shutdown($sc, SHUT_RD); sysread $sc, my $buf, 512 or die $!;
print $buf'
^^ The above, I believe, demonstrates to the contrary: the read buffer is
populated prior to shutdown and drained afterward.
>> I would guess that many don’t and just don’t see the
>> RST thing frequently enough to worry about it. Regardless, the documentation
>> is already pretty voluminous, so if this doesn’t bite many folks, then hey.
>
> Yes, but wiki articles are always appreciated.
I’ll see if I can whip something up.
-FG
RE: SSL_read empty -> close?
> From: Felipe Gasper > Sent: Thursday, 3 November, 2022 08:51 > > You probably know this, but: On Linux, at least, if a TCP socket close()s > with a non-empty read buffer, the kernel sends TCP RST to the peer. Yes, that's a conditional-compliance (SHOULD) requirement from the Host Requirements RFC. See RFC 1122, 4.2.2.13. > Some > applications “panic” when they receive the RST and discard data. Well, applications do a lot of things. Receiving an RST informs the peer that some of the data they sent was not successfully processed by the local application, so treating that as an error condition is not inappropriate. But generally it's better if the application protocol imposes its own record structure and control information on top of TCP's very basic stream. > It’s a rare > issue, but when it does it’s a head-scratcher. To avoid that, it’s necessary > to shutdown(SHUT_RD) then drain the read buffer before close(). Well, it's not *necessary* to do a half-close. Applications often know when they've received all the data the peer intends to send, thanks to record-delimiting mechanisms in the application protocol. And your description looks wrong anyway: shutdown(SHUT_RD) has implementation-defined behavior for TCP sockets (because TCP does not announce the read side of half-close to the peer), and on Linux causes blocked receives and subsequent receives to return 0 (according to references -- I have't tested it), which means after shutdown(SHUT_RD) you *can't* drain the receive buffer. shutdown(SHUT_WR) would work, since it sends a FIN, telling the peer you won't be sending any more data, and still allows you to receive. > So it seems like this *shouldn’t* be obscure, if applications do the > shutdown/drain thing. It's obscure in the sense that a great many people trying to use TLS get much more basic things wrong. More generally, the OpenSSL documentation mostly covers the OpenSSL APIs, and leaves networking up to the OpenSSL consumer to figure out. The OpenSSL wiki covers topics that people have written, and those are going to focus on common questions and areas of particular interest for someone. If the interactions among the OpenSSL API, the TLS protocol (in its various versions), and the shutdown system call haven't historically been a problem for many people, then it's "obscure" in the literal sense of not having attracted much notice. And in practice, the majority of TLS use is with HTTP, and HTTP does a fairly good job of determining when more data is expected, and handling cases where it isn't. An HTTP client that receives a complete response and then attempts to use the conversation for its next request, and gets an RST on that, for example, will just open a new conversation; it doesn't care that the old one was terminated. HTTP servers are simliarly tolerant because interactive user agents in particular cancel requests by closing (or, unfortunately, aborting) the connection all the time. > I would guess that many don’t and just don’t see the > RST thing frequently enough to worry about it. Regardless, the documentation > is already pretty voluminous, so if this doesn’t bite many folks, then hey. Yes, but wiki articles are always appreciated. -- Michael Wojcik
Re: SSL_read empty -> close?
> On Nov 3, 2022, at 10:17, Michael Wojcik via openssl-users > wrote: > >> Does OpenSSL’s documentation mention that? (I’m not exhaustively >> familiar with it, but I don’t remember having seen such.) > > I doubt it. I don't see anything on the wiki, and this is a pretty obscure > issue, all things considered. You probably know this, but: On Linux, at least, if a TCP socket close()s with a non-empty read buffer, the kernel sends TCP RST to the peer. Some applications “panic” when they receive the RST and discard data. It’s a rare issue, but when it does it’s a head-scratcher. To avoid that, it’s necessary to shutdown(SHUT_RD) then drain the read buffer before close(). So it seems like this *shouldn’t* be obscure, if applications do the shutdown/drain thing. I would guess that many don’t and just don’t see the RST thing frequently enough to worry about it. Regardless, the documentation is already pretty voluminous, so if this doesn’t bite many folks, then hey. Thank you! -F
RE: SSL_read empty -> close?
> From: Felipe Gasper > Sent: Thursday, 3 November, 2022 07:42 > > It sounds, then like shutdown() (i.e., TCP half-close) is a no-no during a > TLS session. Um, maybe. Might generally be OK in practice, particularly with TLSv1.3, which got rid of some of the less-well-considered ideas of earlier TLS versions. Honestly I'd have to spend some time digging through chapter & verse of the RFCs to arrive at any reliable opinion on the matter, though. Someone else here may have already considered it. > Does OpenSSL’s documentation mention that? (I’m not exhaustively > familiar with it, but I don’t remember having seen such.) I doubt it. I don't see anything on the wiki, and this is a pretty obscure issue, all things considered. > It almost seems like, given that TLS notify-close then TCP close() (i.e., > without awaiting the peer’s TLS notify-close) is legitimate, OpenSSL could > gainfully tolerate/hide the EPIPE that that close() likely produces, and have > SSL_read() et al just return empty-string. Well, it could, but OpenSSL generally doesn't try to provide that type of abstraction. Also note this paragraph from the wiki page on TLSv1.3 (https://wiki.openssl.org/index.php/TLS1.3): If a client sends it's [sic] data and directly sends the close notify request and closes the connection, the server will still try to send tickets if configured to do so. Since the connection is already closed by the client, this might result in a write error and receiving the SIGPIPE signal. The write error will be ignored if it's a session ticket. But server applications can still get SIGPIPE they didn't get before. So session tickets can also be a source of EPIPE when a client closes the connection. > It surprises me that notify-close then close() is considered legitimate use. There are so many TLS implementations and TLS-using applications out there that interoperability would be hugely compromised if we didn't allow a large helping of Postel's Interoperability Principle. So most applications try to be accommodating. There's even an OpenSSL flag to ignore the case where a peer closes without sending a close-notify, in case you run into one of those and want to suppress the error. -- Michael Wojcik
Re: SSL_read empty -> close?
> On Nov 2, 2022, at 16:36, Michael Wojcik via openssl-users > wrote: > >> From: Felipe Gasper >> Sent: Wednesday, 2 November, 2022 12:46 >> >> I wouldn’t normally expect EPIPE from a read operation. I get why it happens; >> it just seems odd. Given that it’s legitimate for a TLS peer to send the >> close_notify and then immediately do TCP close, it also seems like EPIPE is a >> “fact of life” here. > > Yeah. That's because an OpenSSL "read" operation can do sends under the > covers, and an OpenSSL "send" can do receives, in order to satisfy the > requirements of TLS. Depending on the TLS version and cipher suite being > used, it might need to do that for renegotiation or the like. Or if the > socket is non-blocking you can get WANT_READ from a send and WANT_WRITE from > a receive. > > In your example it was actually a sendmsg that produced the EPIPE, but within > the logical "read" operation. > > The original idea of SSL was "just be a duplex bytestream service for the > application", i.e. be socket-like; but that abstraction proved to be rather > leaky. Much as sockets themselves are a leaky abstraction once you try to do > anything non-trivial. It sounds, then like shutdown() (i.e., TCP half-close) is a no-no during a TLS session. Does OpenSSL’s documentation mention that? (I’m not exhaustively familiar with it, but I don’t remember having seen such.) It almost seems like, given that TLS notify-close then TCP close() (i.e., without awaiting the peer’s TLS notify-close) is legitimate, OpenSSL could gainfully tolerate/hide the EPIPE that that close() likely produces, and have SSL_read() et al just return empty-string. I don’t recall having seen it in 1.1 or earlier OpenSSLs. Then again, maybe some application wants to know it shouldn’t send a notify-close. It surprises me that notify-close then close() is considered legitimate use. WebSocket and SCTP both have similar shutdown workflows, but I think they expect the peers to await each other’s “farewell” message. -FG
RE: SSL_read empty -> close?
> From: Felipe Gasper > Sent: Wednesday, 2 November, 2022 12:46 > > I wouldn’t normally expect EPIPE from a read operation. I get why it happens; > it just seems odd. Given that it’s legitimate for a TLS peer to send the > close_notify and then immediately do TCP close, it also seems like EPIPE is a > “fact of life” here. Yeah. That's because an OpenSSL "read" operation can do sends under the covers, and an OpenSSL "send" can do receives, in order to satisfy the requirements of TLS. Depending on the TLS version and cipher suite being used, it might need to do that for renegotiation or the like. Or if the socket is non-blocking you can get WANT_READ from a send and WANT_WRITE from a receive. In your example it was actually a sendmsg that produced the EPIPE, but within the logical "read" operation. The original idea of SSL was "just be a duplex bytestream service for the application", i.e. be socket-like; but that abstraction proved to be rather leaky. Much as sockets themselves are a leaky abstraction once you try to do anything non-trivial. -- Michael Wojcik
Re: SSL_read empty -> close?
> On Oct 26, 2022, at 13:34, Michael Wojcik via openssl-users
> wrote:
>
>> From: openssl-users On Behalf Of Felipe
>> Gasper
>> Sent: Wednesday, 26 October, 2022 11:15
>>
>> I’m seeing that OpenSSL 3, when it reads empty on a socket, sends some
>> sort of response, e.g.:
>>
>> - before read
>> [pid 42417] read(7276781]>, "", 5) = 0
>> [pid 42417] sendmsg(7276781]>, {msg_name=NULL, msg_namelen=0,
>> msg_iov=[{iov_base="\0022", iov_len=2}], msg_iovlen=1,
>> msg_control=[{cmsg_len=17, cmsg_level=SOL_TLS, cmsg_type=0x1}],
>> msg_controllen=17, msg_flags=0}, 0) = -1 EPIPE (Broken pipe)
>> - after read
>>
>> What is that being sent after the read()? Is there a way to disable
>> it?
>
> I'd guess it's a TLS Alert Close_notify.
>
> When read/recv on a TCP stream socket returns 0, it means a TCP FIN has been
> received from the peer (or possibly some interfering middleman, such as a
> firewall). This indicates the peer will no longer be sending any application
> data, only at most ACKs and perhaps a RST if conversation does not go quietly
> into that good night. Since TLS requires bidirectional communications, that
> means the TLS conversation is effectively open, and the local end needs to be
> closed; and TLS requires sending a close_notify so the peer knows the
> conversation has not been truncated.
>
> Now, the most common cause of a FIN is the peer calling close(), which means
> it can't receive that close_notify. But TCP supports half-close, and the peer
> *could have* called shutdown(, SD_SEND), indicating that it was done sending
> but still wanted to be able to receive data. So the local side has no way of
> knowing, at the point where it gets a 0 from read(), that the peer definitely
> can't see the close_notify; and thus it's still obligated by the TLS
> specification (I believe) to send it.
>
> At any rate, that's my understanding of the requirement for sending
> close_notify - I haven't confirmed that in the RFC - and what I suspect
> OpenSSL is doing there. I could well be wrong.
>
> If the peer *has* called close, then EPIPE is what you'd expect. Note that on
> UNIXy systems this means you should have set the disposition of SIGPIPE to
> SIG_IGN to avoid being signaled, but all well-written UNIX programs should do
> that anyway. (SIGPIPE, as Dennis Ritchie noted many years ago, was always
> intended as a failsafe for poorly-written programs that fail to check for
> errors when writing.)
I wouldn’t normally expect EPIPE from a read operation. I get why it happens;
it just seems odd. Given that it’s legitimate for a TLS peer to send the
close_notify and then immediately do TCP close, it also seems like EPIPE is a
“fact of life” here.
-FG
RE: SSL_read empty -> close?
> From: openssl-users On Behalf Of Felipe
> Gasper
> Sent: Wednesday, 26 October, 2022 11:15
>
> I’m seeing that OpenSSL 3, when it reads empty on a socket, sends some
> sort of response, e.g.:
>
> - before read
> [pid 42417] read(7276781]>, "", 5) = 0
> [pid 42417] sendmsg(7276781]>, {msg_name=NULL, msg_namelen=0,
> msg_iov=[{iov_base="\0022", iov_len=2}], msg_iovlen=1,
> msg_control=[{cmsg_len=17, cmsg_level=SOL_TLS, cmsg_type=0x1}],
> msg_controllen=17, msg_flags=0}, 0) = -1 EPIPE (Broken pipe)
> - after read
>
> What is that being sent after the read()? Is there a way to disable
> it?
I'd guess it's a TLS Alert Close_notify.
When read/recv on a TCP stream socket returns 0, it means a TCP FIN has been
received from the peer (or possibly some interfering middleman, such as a
firewall). This indicates the peer will no longer be sending any application
data, only at most ACKs and perhaps a RST if conversation does not go quietly
into that good night. Since TLS requires bidirectional communications, that
means the TLS conversation is effectively open, and the local end needs to be
closed; and TLS requires sending a close_notify so the peer knows the
conversation has not been truncated.
Now, the most common cause of a FIN is the peer calling close(), which means it
can't receive that close_notify. But TCP supports half-close, and the peer
*could have* called shutdown(, SD_SEND), indicating that it was done sending
but still wanted to be able to receive data. So the local side has no way of
knowing, at the point where it gets a 0 from read(), that the peer definitely
can't see the close_notify; and thus it's still obligated by the TLS
specification (I believe) to send it.
At any rate, that's my understanding of the requirement for sending
close_notify - I haven't confirmed that in the RFC - and what I suspect OpenSSL
is doing there. I could well be wrong.
If the peer *has* called close, then EPIPE is what you'd expect. Note that on
UNIXy systems this means you should have set the disposition of SIGPIPE to
SIG_IGN to avoid being signaled, but all well-written UNIX programs should do
that anyway. (SIGPIPE, as Dennis Ritchie noted many years ago, was always
intended as a failsafe for poorly-written programs that fail to check for
errors when writing.)
--
Michael Wojcik
