Re: Quick update on pending HTTP/2 issues

[Willy Tarreau]

Hi again guys,

so another quick update on the subject :
  - the currently known POST issues have been resolved a few days ago,
requiring a significant number of changes which make the code better
anyway so it was not bad in the end ;

  - the abortonclose case has been solved as well. The issue was that
in HTTP/1 there's no way for a browser to mention that it's aborting
so the best it can do is to close, resulting in a read0 being received
and the option is there to indicate that haproxy must consider this
read0 as an abort. In HTTP/2 we have RST_STREAM to explicitly abort,
and the read0 is sent at the end of all requests to indicate that the
request is complete, thus creates the confusion with the option above.
The fix consists in making abortonclose useless on read0 for H2, and
now it's OK.

I'd have liked to issue 1.8.2 with these fixes, but apparently a few other
issues are pending. Let's say that we'll release before the end of the week
with or without the fixes.

(PS: yes I'm still thinking about issuing 1.7.10 ASAP, but the 1.8 issues
 have been diverting everyone *a lot* and it's not finished).

Willy

Re: Quick update on pending HTTP/2 issues

[Willy Tarreau]

Hi Lukas!

On Mon, Dec 11, 2017 at 11:54:07AM +0100, Lukas Tribus wrote:
> Ok, the POST issue is obviously more important, but just to provide a
> complete picture we also have those 2 minor issues left in H2, which
> can easily be worked around though:

Well, I could finally get rid of all the issues affecting the POST, and
some of them required quite complex setups just to get the proper time
race to trigger them! So I could get back to these two parts :

> - the abortonclose incompatibility:
> https://www.mail-archive.com/haproxy@formilux.org/msg28186.html

I'm currently estimating what it would cost to mirror the flag into the
stream instead of checking it in the proxy, and to always clear it from
H2 streams.

> - improper close (no GOAWAY) when "timeout server" < "timeout client"
> https://www.mail-archive.com/haproxy@formilux.org/msg27990.html

I totally failed to reproduce this one for now. I properly get the
504 timeout as a HEADERS frame, then later I'm seeing the GOAWAY(0)
frame indicating a connection close on idle. I also tried with the
master prior to the POST fixes and it did the same, though maybe
something changed since you last saw this. I'm interested in getting
a double-check on this in the development branch.

Thanks!
Willy

Re: Quick update on pending HTTP/2 issues

[Willy Tarreau]

Hi Lukas,

On Mon, Dec 11, 2017 at 11:54:07AM +0100, Lukas Tribus wrote:
> Hello Willy,
> 
> 
> 
> 2017-12-07 19:55 GMT+01:00 Willy Tarreau :
> > Guys,
> >
> > just to warn you, there's currently an issue affecting HTTP/2 with POST
> > payloads to "slow" servers.
> 
> Ok, the POST issue is obviously more important, but just to provide a
> complete picture we also have those 2 minor issues left in H2, which
> can easily be worked around though:
> 
> - the abortonclose incompatibility:
> https://www.mail-archive.com/haproxy@formilux.org/msg28186.html

Yes I'm well aware of the abortonclose issue indeed, and as you're
saying, at the moment it's less important for me since there is a
workaround.

> - improper close (no GOAWAY) when "timeout server" < "timeout client"
> https://www.mail-archive.com/haproxy@formilux.org/msg27990.html

Thanks for the reminder, I already forgot about this one :-)

> Also, a quick reminder about 1.7.10; I can see a lot of patches are
> already backported, is there still something missing?

I don't know yet. From what I can see in 1.8, part of the truncated
objects are caused by H2 itself, and other parts are caused by a bad
behaviour of the HTTP forwarding code when facing a SHUTR (it's
considered as an abort even in cases where we can't move due to a
buffer full condition). So we may end up having to backport a fix
there. Also I'd like to re-run a number of non-reg tests on 1.7.10
before we release it.

Thanks!
Willy

Re: Quick update on pending HTTP/2 issues

[Lukas Tribus]

Hello Willy,



2017-12-07 19:55 GMT+01:00 Willy Tarreau :
> Guys,
>
> just to warn you, there's currently an issue affecting HTTP/2 with POST
> payloads to "slow" servers.

Ok, the POST issue is obviously more important, but just to provide a
complete picture we also have those 2 minor issues left in H2, which
can easily be worked around though:

- the abortonclose incompatibility:
https://www.mail-archive.com/haproxy@formilux.org/msg28186.html

- improper close (no GOAWAY) when "timeout server" < "timeout client"
https://www.mail-archive.com/haproxy@formilux.org/msg27990.html



Also, a quick reminder about 1.7.10; I can see a lot of patches are
already backported, is there still something missing?




Thanks,

Lukas

Quick update on pending HTTP/2 issues

[Willy Tarreau]

Guys,

just to warn you, there's currently an issue affecting HTTP/2 with POST
payloads to "slow" servers. It's a bit difficult to explain but in short
if haproxy's buffers fill up during the transfer, there's a risk that the
wakeup event to restart decoding once the buffer is released cannot be
processed immediately, and is not detected later, causing the end of data
never to be sent and the request to time out.

I now manage to reproduce it reasonably easily with curl. As long as there
is some other activity on the connection, things will continue to flow and
there will not be any problem but in this particular case we end up with a
black hole trying to process something we can't do.

This one is a bit annoying because after turning it into any direction,
it seems the only way to address it is to revisit the now very old stream
interface API, and given that each and every timeout bug we faced during
1.7 has some of its roots in there, I'd rather be extremely cautious
before trying to be creative.

I'm still working on this as time permits. In the mean time I'd suggest
that those having to support "large" POSTs (those that don't fit in
haproxy's + kernel's buffers, ie more than a few hundreds of kB) better
disable HTTP/2 for now.

I'll keep you updated once an acceptable solution is found, and now it's
getting pretty clear that we need to kill all this aging mess for 1.9!

Cheers,
Willy