Re: H2 Protocol Errors in HTX Mode (1.9.4 & 1.9.4-dev)

[Willy Tarreau]

Hi Luke,

On Sat, Mar 23, 2019 at 02:52:26PM +0100, Luke Seelenbinder wrote:
> Hi Willy,
> 
> I just upgraded to 1.9.5, and this bug is still present (but seems to be
> somewhat diminished). On 1.9.4, approximately 5 of these images failed to
> load, on 1.9.5, it's usually 1 or 2. So overall it seems there is
> improvement, but something is still a bit wonky. :)

Christopher spotted issues around abortonclose and managed to address
them in an elegant way. We'll merge them into 2.0-dev and keep an eye
on them before backporting them to the next 1.9. I think we're getting
closer to having all these issues addressed. Additionally Olivier also
fixed some nasty issues affecting the streams ordering on H2 which can
make some streams starve for a while as long as there is activity on
others. I don't know if that's your case, but it could be possible that
it contributes to the issue. I'll let you know once all this stuff is
backported to 1.9. If you're interested in testing 2.0-dev, I'll tell
you once everything is merged there so that you don't waste your time
testing code that doesn't contain the fix.

Cheers,
Willy

Re: H2 Protocol Errors in HTX Mode (1.9.4 & 1.9.4-dev)

[Luke Seelenbinder]

Hi Willy,

I just upgraded to 1.9.5, and this bug is still present (but seems to be 
somewhat diminished). On 1.9.4, approximately 5 of these images failed to load, 
on 1.9.5, it's usually 1 or 2. So overall it seems there is improvement, but 
something is still a bit wonky. :)

Best,
Luke

—
Luke Seelenbinder
SermonAudio.com  | Senior Software Engineer


> On Mar 1, 2019, at 05:35, Willy Tarreau  wrote:
> 
> On Fri, Feb 22, 2019 at 01:35:19PM +0100, Luke Seelenbinder wrote:
>> Hi List,
>> 
>> We recently started using HAProxy to act as a first point of entry for most
>> of our traffic. We initially set it up with H2 + HTX frontend and H1.1
>> backend; however, this led to some strange behavior consistently reproducible
>> on one page.
>> 
>> Whenever we loaded this page
>> (https://www.sermonaudio.com/search.asp?currsection=new=MP4) a we
>> would get a series of H2 protocol errors (SPDY_PROTOCOL_ERROR, as reported by
>> Chrome) for images stored on media-cloud.sermonaudio.com (running on the same
>> server / ip as www.semronaudio.com). Disabling HTX fixed the problem. Our
>> configuration is quite simple, just a few ACLs for redirection and forcing
>> SSL. I initially thought the issue could be related to the number of objects
>> being loaded (just over 100), so I tuned h2.max-concurrent-streams, but that
>> had no effect. This is observable in both 1.9.4 and the latest (as of two
>> days ago) 1.9 git branch.
> 
> That's a bit weird. I don't see what could cause a decoding error in HTX
> that doesn't happen without. I'll check if I can spot different code paths
> between HTX and legacy that could cause such errors to be emitted. One
> thing that could happen would be if in one of the htx decoding functions
> we end up reading a wrong amount of data sometimes, causing the stream to
> be desynchronized, but I really don't see where that would happen.
> 
>> I did not observe the issue on any other page but my search was not
>> exhaustive. It also never occurs on an individual request when made with
>> curl, for example.
> 
> OK that's useful info, it avois the usual "strange, works for me" after the
> first succcessful test :-)
> 
>> I would be able to stand up a server that has the same configuration with HTX
>> enabled for testing, if that would be helpful, you would have to provide the
>> /etc/hosts entries, though. :)
> 
> Might be. Let me take a look at the code first. Maybe I'll ask you to
> retry with a different version or to try a patch.
> 
> Thanks,
> Willy

Re: H2 Protocol Errors in HTX Mode (1.9.4 & 1.9.4-dev)

[Willy Tarreau]

On Fri, Feb 22, 2019 at 01:35:19PM +0100, Luke Seelenbinder wrote:
> Hi List,
> 
> We recently started using HAProxy to act as a first point of entry for most
> of our traffic. We initially set it up with H2 + HTX frontend and H1.1
> backend; however, this led to some strange behavior consistently reproducible
> on one page.
> 
> Whenever we loaded this page
> (https://www.sermonaudio.com/search.asp?currsection=new=MP4) a we
> would get a series of H2 protocol errors (SPDY_PROTOCOL_ERROR, as reported by
> Chrome) for images stored on media-cloud.sermonaudio.com (running on the same
> server / ip as www.semronaudio.com). Disabling HTX fixed the problem. Our
> configuration is quite simple, just a few ACLs for redirection and forcing
> SSL. I initially thought the issue could be related to the number of objects
> being loaded (just over 100), so I tuned h2.max-concurrent-streams, but that
> had no effect. This is observable in both 1.9.4 and the latest (as of two
> days ago) 1.9 git branch.

That's a bit weird. I don't see what could cause a decoding error in HTX
that doesn't happen without. I'll check if I can spot different code paths
between HTX and legacy that could cause such errors to be emitted. One
thing that could happen would be if in one of the htx decoding functions
we end up reading a wrong amount of data sometimes, causing the stream to
be desynchronized, but I really don't see where that would happen.

> I did not observe the issue on any other page but my search was not
> exhaustive. It also never occurs on an individual request when made with
> curl, for example.

OK that's useful info, it avois the usual "strange, works for me" after the
first succcessful test :-)

> I would be able to stand up a server that has the same configuration with HTX
> enabled for testing, if that would be helpful, you would have to provide the
> /etc/hosts entries, though. :)

Might be. Let me take a look at the code first. Maybe I'll ask you to
retry with a different version or to try a patch.

Thanks,
Willy

H2 Protocol Errors in HTX Mode (1.9.4 & 1.9.4-dev)

[Luke Seelenbinder]

Hi List,

We recently started using HAProxy to act as a first point of entry for most of 
our traffic. We initially set it up with H2 + HTX frontend and H1.1 backend; 
however, this led to some strange behavior consistently reproducible on one 
page.

Whenever we loaded this page 
(https://www.sermonaudio.com/search.asp?currsection=new=MP4) a we 
would get a series of H2 protocol errors (SPDY_PROTOCOL_ERROR, as reported by 
Chrome) for images stored on media-cloud.sermonaudio.com (running on the same 
server / ip as www.semronaudio.com). Disabling HTX fixed the problem. Our 
configuration is quite simple, just a few ACLs for redirection and forcing SSL. 
I initially thought the issue could be related to the number of objects being 
loaded (just over 100), so I tuned h2.max-concurrent-streams, but that had no 
effect. This is observable in both 1.9.4 and the latest (as of two days ago) 
1.9 git branch.

I did not observe the issue on any other page but my search was not exhaustive. 
It also never occurs on an individual request when made with curl, for example.

I would be able to stand up a server that has the same configuration with HTX 
enabled for testing, if that would be helpful, you would have to provide the 
/etc/hosts entries, though. :)

Thanks for any help!

Best,
Luke

—
Luke Seelenbinder
SermonAudio.com  | Senior Software Engineer