Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-02-01 Thread Willy Tarreau
On Fri, Feb 01, 2019 at 07:50:23PM +, Luke Seelenbinder wrote: > Hi Willy, > > I just had a chance to check this--I haven't run every test I could think of > yet, but it works! Not a single server-side error or disconnection! Excellent, many thanks! > I'll definitely be rolling this out

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-02-01 Thread Luke Seelenbinder
Hi Willy, I just had a chance to check this—I haven't run every test I could think of yet, but it works! Not a single server-side error or disconnection! I'll definitely be rolling this out when 1.9.4 lands. I will let you know if anything else comes up during any remaining testing. Best,

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-02-01 Thread Luke Seelenbinder
Hi Willy, Great news! Thank you again for your work to clean this all up. > http://git.haproxy.org/?p=haproxy-1.9.git;a=snapshot;h=HEAD;sf=tgz I'll pull this and test as soon as I get the chance. That is likely to be Monday given how my Friday is going. :-) Based on what you've said given

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-02-01 Thread Willy Tarreau
Hi Luke, On Wed, Jan 30, 2019 at 07:43:06PM +0100, Willy Tarreau wrote: > I've also found a few erroneous state transitions and an issue affecting > trailers which could also break the connection. For now it's only in > 2.0-dev because it'll take me a while to collect all pending patches for >

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-31 Thread Willy Tarreau
Hi again, On Wed, Jan 30, 2019 at 07:43:06PM +0100, Willy Tarreau wrote: > It was indeed caused by late frames triggering GOAWAY, while we were > missing some info to determine if the frame was violating the protocol > or just a late response. I found another cause of GOAWAY that appeared in

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Willy Tarreau
Hi Luke, I finally managed to kill them all, and everything works fine for me. It was indeed caused by late frames triggering GOAWAY, while we were missing some info to determine if the frame was violating the protocol or just a late response. I've also found a few erroneous state transitions

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Willy Tarreau
On Wed, Jan 30, 2019 at 10:18:03AM +, Luke Seelenbinder wrote: (...) > I don't know if any of this rings any bells, but that's what I've been able > to determine so far. Overall the patch improves things significantly, but > some latent issues are still around. No bell yet but that's detailed

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Luke Seelenbinder
Hi Willy, I've done a good bit more digging. Here's my notes so far. 1. On occasion the old behavior (CD-- then SD--) still exhibits. It's a bit too consistent for me to push to our production platform yet. 2. It seems Chrome and Firefox's behavior is different enough to change the frequency

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Luke Seelenbinder
Hi Willy, > 403 is strange, it's forbidden. 400 I could understand. What might happen > is the following scenario : > I'm really not thinking of any case resulting in this but I'll take a > second look. This…is actually a configuration issue on my side, I think. Sorry for the wild-goose

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Willy Tarreau
Hi Luke, On Wed, Jan 30, 2019 at 08:41:03AM +, Luke Seelenbinder wrote: > It works! I'm seeing very, very few CD-- -> SD-- chains now. I did see a few > in h2<->h2 mode, but precious few, so I'm very happy to say the bug as > previous manifested is remedied! Thanks for digging so wide and

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-30 Thread Luke Seelenbinder
Hi Willy, > I've merged the attached patch that fixes the problem for me, please try > it, it should apply cleanly on top of 1.9.3. It works! I'm seeing very, very few CD-- -> SD-- chains now. I did see a few in h2<->h2 mode, but precious few, so I'm very happy to say the bug as previous

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-29 Thread Willy Tarreau
Hi Luke, On Tue, Jan 29, 2019 at 10:52:15AM +, Luke Seelenbinder wrote: > Hi Willy, > > > By the way, how do you manage to cancel a single stream in the browser ? > > Pressing Esc might break all of them I guess ? Thus I'm uncertain how to > > achieve this. > > So we're in a very specific

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-29 Thread Luke Seelenbinder
Hi Willy, > By the way, how do you manage to cancel a single stream in the browser ? > Pressing Esc might break all of them I guess ? Thus I'm uncertain how to > achieve this. So we're in a very specific use-case of delivering map tiles, which are predominately requested via Leaflet.js.

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-29 Thread Willy Tarreau
On Tue, Jan 29, 2019 at 11:39:32AM +0100, Willy Tarreau wrote: > By the way, how do you manage to cancel a single stream in the browser ? > Pressing Esc might break all of them I guess ? Thus I'm uncertain how to > achieve this. I think I found a solution for this, I open two tabs in the browser

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-29 Thread Willy Tarreau
Hi Luke, On Tue, Jan 29, 2019 at 10:06:03AM +, Luke Seelenbinder wrote: > I just pulled, compiled, and tested the newly minted 1.9.3, and I'm > experiencing the same issue with alpn h2 on the backend definition. Ah sh*t :-( > I also > strongly suspect it's not related to maximum streams per

Re: H2 Server Connection Resets (1.9.2 & 1.9.3)

2019-01-29 Thread Luke Seelenbinder
Hi Willy, > As you like. My first rule is never to make people take risks they're not > willing to take. It's perfectly OK to me if you don't feel confident with > 2.0-dev in prod. I'm going to perform the 1.9 backports. If you're > interested in testing them from the branch before I release it

Re: H2 Server Connection Resets (1.9.2)

2019-01-25 Thread Willy Tarreau
Hi Luke, On Fri, Jan 25, 2019 at 08:08:22AM +, Luke Seelenbinder wrote: > Hi Willy, > > > OK so instead of sending you a boring series, I can propose you to run > > a test on 2.0-dev, which contains all the fixes I had to go through > > because of tiny issues everywhere related to this. If

Re: H2 Server Connection Resets (1.9.2)

2019-01-25 Thread Luke Seelenbinder
Hi Willy, > OK so instead of sending you a boring series, I can propose you to run > a test on 2.0-dev, which contains all the fixes I had to go through > because of tiny issues everywhere related to this. If you're using git, > just clone the master and checkout commit f7a259d46f8. > you can

Re: H2 Server Connection Resets (1.9.2)

2019-01-24 Thread Willy Tarreau
Hi Luke, On Wed, Jan 23, 2019 at 05:16:04PM +, Luke Seelenbinder wrote: > Hi Willy, > > This is all very good to hear. I'm glad you were able to get to the bottom of > it all! > > Feel free to send along patches if you want me to test before the 1.9.3 > release. I'm more than happy to do

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Luke Seelenbinder
Hi Willy, This is all very good to hear. I'm glad you were able to get to the bottom of it all! Feel free to send along patches if you want me to test before the 1.9.3 release. I'm more than happy to do so. Best, Luke — Luke Seelenbinder Stadia Maps | Founder stadiamaps.com ‐‐‐

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Willy Tarreau
Hi Luke, On Wed, Jan 23, 2019 at 10:47:33AM +, Luke Seelenbinder wrote: > We were using http-reuse always and experiencing this > issue (as well as getting 80+% connection reuse). When I scaled it back to > http-reuse safe, the frequency of this issue seemed to be much lower. > (Perhaps

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Luke Seelenbinder
Hi Willy, > When using "http-reuse always" the issue disappears and I > can never get any issue at all. Now that I've fixed this, I'm seeing the > issue with the SD flags. Now that's interesting. We were using http-reuse always and experiencing this issue (as well as getting 80+% connection

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Willy Tarreau
On Wed, Jan 23, 2019 at 11:09:53AM +0100, Willy Tarreau wrote: > On Wed, Jan 23, 2019 at 09:24:19AM +, Luke Seelenbinder wrote: > > > I've place an nginx instance after my local haproxy dev config, and > > > found something which might explain what you're observing : the process > > >

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Willy Tarreau
On Wed, Jan 23, 2019 at 09:24:19AM +, Luke Seelenbinder wrote: > > I've place an nginx instance after my local haproxy dev config, and > > found something which might explain what you're observing : the process > > apparently leaks FDs and fails once in a while, causing 500 to be returned : >

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Aleksandar Lazic
Hi Lukas. Am 23.01.2019 um 10:24 schrieb Luke Seelenbinder: > Hi Willy, > > Thanks for continuing to look into this. > >> > >> I've place an nginx instance after my local haproxy dev config, and >> found something which might explain what you're observing : the process >> apparently leaks FDs

Re: H2 Server Connection Resets (1.9.2)

2019-01-23 Thread Luke Seelenbinder
Hi Willy, Thanks for continuing to look into this. > > I've place an nginx instance after my local haproxy dev config, and > found something which might explain what you're observing : the process > apparently leaks FDs and fails once in a while, causing 500 to be returned : That's

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Willy Tarreau
Hi Luke, I've place an nginx instance after my local haproxy dev config, and found something which might explain what you're observing : the process apparently leaks FDs and fails once in a while, causing 500 to be returned : 2019/01/23 08:22:13 [crit] 25508#0: *36705 open()

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Luke Seelenbinder
Hi Aleksandar, Thanks for your tips. > Do you have such a info in the nginx log? > > "http2 flood detected" I did not find this in any of the logs from when the buggy configuration was deployed. > Can you try to set some timeout values for`timeout http-keep-alive` I do have this set

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Willy Tarreau
On Tue, Jan 22, 2019 at 02:57:23PM +, Luke Seelenbinder wrote: > There is a strong correlation between client connections canceling requests > (resulting in a HTTP log string of CD--) and then a whole string of requests > immediately after resulting in server connection resets (resulting in

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Luke Seelenbinder
Hi Willy, I just confirmed the other patchset works, so I will start going down this road. :-) While testing the other issue, I discovered something fascinating. Our application is typically used by clients that cancel requests with reasonable frequency (5+%). When zooming in and out of maps

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Willy Tarreau
On Tue, Jan 22, 2019 at 09:42:53AM +, Luke Seelenbinder wrote: > Hi Willy, Aleks, > > I will try the things suggested this afternoon (hopefully) or tomorrow and > get back to you. > > > At least if nginx does this it should send a GOAWAY > > frame indicating that it will stop after stream

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Luke Seelenbinder
Hi Willy, Aleks, I will try the things suggested this afternoon (hopefully) or tomorrow and get back to you. > At least if nginx does this it should send a GOAWAY > frame indicating that it will stop after stream #2001. That's my understanding as well (and the docs say as much). I assumed

Re: H2 Server Connection Resets (1.9.2)

2019-01-22 Thread Willy Tarreau
Hi Luke, On Mon, Jan 21, 2019 at 09:30:39AM +, Luke Seelenbinder wrote: > After enabling h2 backends (technically `server ... alpn h2,http/1.1`), we > began seeing a high number of backend /server/ connection resets. A > reasonable number of client-side connection resets due to timeouts,

Re: H2 Server Connection Resets (1.9.2)

2019-01-21 Thread Aleksandar Lazic
Hi Luke. Am 21.01.2019 um 10:30 schrieb Luke Seelenbinder: > Hi all, > > One more bug (or configuration hole) from our transition to 1.9.x using > end-to-end h2 connections. > > After enabling h2 backends (technically `server … alpn h2,http/1.1`), we > began seeing a high number of backend

H2 Server Connection Resets (1.9.2)

2019-01-21 Thread Luke Seelenbinder
Hi all, One more bug (or configuration hole) from our transition to 1.9.x using end-to-end h2 connections. After enabling h2 backends (technically `server … alpn h2,http/1.1`), we began seeing a high number of backend /server/ connection resets. A reasonable number of client-side connection