Re: h2 + text/event-stream: closed on both sides by FIN/ACK?
On Mon, 24 Sep 2018 at 16:36, Willy Tarreau wrote: > > On Mon, Sep 24, 2018 at 02:30:35PM +, Pierre Cheynier wrote: > > OK, I conclude this SSE pattern is not working out-of-the-box when using h2 > > as of > > now. Is it still true even if setting the user set the proper connection > > headers on > > server side? > > Yes, it's irrelevant to the headers, it's related to the fact that each > request from an H2 connection is a different stream and that the server- > side idle connection is attached to a stream. So streams are short-lived > and the server-side connection is closed for now. But hopefully it won't > anymore for 1.9 ;-) Just to be clear though; Content-Length or chunked transfer-encoding are required if you want to use keep-alive on the backend, even with HTTP/1.1 (or with other products). It's just that it won't work in H2 either way currently, but even if you just use HTTP/1.1 you'd need it. Lukas
Re: h2 + text/event-stream: closed on both sides by FIN/ACK?
On Mon, Sep 24, 2018 at 02:30:35PM +, Pierre Cheynier wrote: > OK, I conclude this SSE pattern is not working out-of-the-box when using h2 > as of > now. Is it still true even if setting the user set the proper connection > headers on > server side? Yes, it's irrelevant to the headers, it's related to the fact that each request from an H2 connection is a different stream and that the server- side idle connection is attached to a stream. So streams are short-lived and the server-side connection is closed for now. But hopefully it won't anymore for 1.9 ;-) Willy
RE: h2 + text/event-stream: closed on both sides by FIN/ACK?
> Hi Pierre, Hi Willy, > The close on the server side is expected, that's a limitation of the current > design that we're addressing for 1.9 and which is much harder than initially >expected. The reason is that streams are independent in H2 while in H1 the > same stream remains idle and recycled for a new request, allowing us to keep > the server-side connection alive. Thus in H2 we can't benefit from the > keep-alive mechanisms we have in H1. But we're currently working on > addressing this. As a side effect, it should end up considerably simplifying > the H1 code as well, but for now it's a nightmare, too many changes at once... OK, I conclude this SSE pattern is not working out-of-the-box when using h2 as of now. Is it still true even if setting the user set the proper connection headers on server side? Thanks, Pierre
Re: h2 + text/event-stream: closed on both sides by FIN/ACK?
Hi Pierre, On Mon, Sep 24, 2018 at 02:10:21PM +, Pierre Cheynier wrote: > > You'll notice that in the HTTP/2 case, the stream is closed as you mentioned > > (DATA len=0 + ES=1) then HAProxy immediately send FIN-ACK to the server. > > Same for the client just after it forwarded the headers. It never wait for > > any > > SSE frame. > > EDIT: in fact, analyzing my capture, I see that my workstation (curl) may be > the > originator, since it sends a close at TLS level (the close-notify).. > > $ curl --version > curl 7.61.0 (x86_64-pc-linux-gnu) libcurl/7.61.0 OpenSSL/1.1.0h zlib/1.2.11 > libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.4) libssh2/1.8.0 nghttp2/1.32.0 > librtmp/2.3 > Release-Date: 2018-07-11 > Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 > pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp > Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB > SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL > > curl or haproxy issue? what do you think? In my experience when fed with a single request, curl closes right after receiving a complete response. You can try to pass it two requests on the same line to see if it only closes at the end. The close on the server side is expected, that's a limitation of the current design that we're addressing for 1.9 and which is much harder than initially expected. The reason is that streams are independent in H2 while in H1 the same stream remains idle and recycled for a new request, allowing us to keep the server-side connection alive. Thus in H2 we can't benefit from the keep-alive mechanisms we have in H1. But we're currently working on addressing this. As a side effect, it should end up considerably simplifying the H1 code as well, but for now it's a nightmare, too many changes at once... Cheers, Willy
RE: h2 + text/event-stream: closed on both sides by FIN/ACK?
> You'll notice that in the HTTP/2 case, the stream is closed as you mentioned > (DATA len=0 + ES=1) then HAProxy immediately send FIN-ACK to the server. > Same for the client just after it forwarded the headers. It never wait for > any > SSE frame. EDIT: in fact, analyzing my capture, I see that my workstation (curl) may be the originator, since it sends a close at TLS level (the close-notify).. $ curl --version curl 7.61.0 (x86_64-pc-linux-gnu) libcurl/7.61.0 OpenSSL/1.1.0h zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.4) libssh2/1.8.0 nghttp2/1.32.0 librtmp/2.3 Release-Date: 2018-07-11 Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL curl or haproxy issue? what do you think? Pierre
Re: h2 + text/event-stream: closed on both sides by FIN/ACK?
Hi Lukas,
On Mon, Sep 24, 2018 at 01:46:57AM +0200, Lukas Tribus wrote:
> Hello,
>
>
> On Fri, 21 Sep 2018 at 15:45, Pierre Cheynier wrote:
> > Let me know if you see something obvious here, or if this is candidate to a
> > bug.
> >
> > We have a service using SSE through text/event-stream content-type.
> >
> > In HTTP/1.1 we have a normal stream as expected :
> > < HTTP/1.1 200 OK
> > < Content-Type: text/event-stream
> > data: {"a": "b"}
> >
> > data: {"a": "b"}
> >
> > data: {"a": "b"}
> > (...)
> >
> > HAProxy on its side adds the `Connection: close` header.
> >
> > When adding 'alpn h2,http/1.1' to the bind directive, we observe the
> > following: after the first 200OK, the connection is closed by haproxy both
> > on server and client side by sending a FIN/ACK.
>
> The backend server is not providing neither Content-Length, nor using
> chunked Transfer-Encoding in the response. This makes using keep-alive
> impossible, regardless of the HTTP version.
>
> Theoretically the frontend connection could be kept up in this
> situation as far as I can tell, but that is an optimization that will
> require more work in haproxy (as the http layer and error handling
> becomes more http version agnostic - currently many transaction based
> problems affect the entire H2 mux).
In fact not, it should work as we close streams and not connections.
So I conclude we have a bug there that I need to explore further.
Pierre, it would be interesting to know if the connection ends up on
a timeout or after the server closes. If it's a timeout it might be
expected that it's the same on both sides and that we simply close
when it strikes. If the server closed, we should only close the
stream (empty DATA frame with ES=1) but not the connection. It is
possible that we do this and face a protocol error later leading
to the connection being closed, so that's why I'm interested in the
exact sequence.
Cheers,
Willy
Re: h2 + text/event-stream: closed on both sides by FIN/ACK?
Hello,
On Fri, 21 Sep 2018 at 15:45, Pierre Cheynier wrote:
> Let me know if you see something obvious here, or if this is candidate to a
> bug.
>
> We have a service using SSE through text/event-stream content-type.
>
> In HTTP/1.1 we have a normal stream as expected :
> < HTTP/1.1 200 OK
> < Content-Type: text/event-stream
> data: {"a": "b"}
>
> data: {"a": "b"}
>
> data: {"a": "b"}
> (...)
>
> HAProxy on its side adds the `Connection: close` header.
>
> When adding 'alpn h2,http/1.1' to the bind directive, we observe the
> following: after the first 200OK, the connection is closed by haproxy both on
> server and client side by sending a FIN/ACK.
The backend server is not providing neither Content-Length, nor using
chunked Transfer-Encoding in the response. This makes using keep-alive
impossible, regardless of the HTTP version.
Theoretically the frontend connection could be kept up in this
situation as far as I can tell, but that is an optimization that will
require more work in haproxy (as the http layer and error handling
becomes more http version agnostic - currently many transaction based
problems affect the entire H2 mux).
The easiest way to fix this problem is to make your backend server
keep-alive aware. Future haproxy (major) releases will likely handle
this case better.
Correct, Willy?
Regards,
Lukas
