On Wed, Jan 23, 2019 at 09:24:19AM +0000, Luke Seelenbinder wrote: > > I've place an nginx instance after my local haproxy dev config, and > > found something which might explain what you're observing : the process > > apparently leaks FDs and fails once in a while, causing 500 to be returned : > > That's fascinating. I would have thought nginx would have had a bit better > care given to things like that. . .
Well, it's possible I'm hitting a corner case. I don't want to blame nginx for such situations, we all have our share of crap when it comes to error handling :-) > Oddly enough, I cannot find any log entries that approximate this. However, > it's possible since we're primarily (99+%) using nginx as a reverse-proxy > that the fd issues wouldn't appear for us. OK, I just deployed it with the default config and added "http2" at the end of the "listen :443 ssl" line. > My next thought is to try tcpdump to try to determine what's on the wire when > the CD-- and SD-- pairs appear, but since our stack is SSL e2e, that might > prove difficult. Any suggestions? For me it's a pain when there's SSL in the mix. Some here on the list are used to know how to extract the master key and use it to decipher the traffic, but I don't know how to do this. As an alternative, nginx supports H2 in clear, so if there's a path where you can disable SSL (e.g. on hosts where haproxy and nginx are on the same machine), then you can communicate in clear by having "proto h2" on the haproxy's server line and "http2" in the nginx config, both without the "ssl" keyword. > One more interesting piece of data: if we use htx without h2 on the backends, > we only see CD-- entries consistently (with a very, very few SD-- entries). > Thus, it would seem whatever is causing the issue is directly related to h2 > backends. I further think we can safely say it is directly related to h2 > streams breaking (due to client-side request cancellations) resulting in the > whole connection breaking in HAProxy or nginx (though determining which will > be the trick). I'm pretty sure you found an issue in haproxy related to the way these requests are aborted. It's just that trying to reproduce this, I'm first hitting other issues in the way related to the limitations above :-) > There's also a strong possibility we replace nginx with HAProxy entirely for > our SSL + H2 setup as we overhaul the backends, so this problem will probably > be resolved by removing the problematic interaction. One more reason for me to speed up figuring what's happening before it becomes too hard to reproduce it! > I'm still working on running h2load against our nginx servers to see if that > turns anything up. Great, thanks! > > And at this point the connection is closed and reopened for new requests. > > There's never any GOAWAY sent. > > If I'm understanding this correctly, that implies as long as nginx sends > GOAWAY properly, HAProxy will not attempt to reuse the connection? I've discovered that it's not the case contrary to what I thought (I have the patch for this, still just testing it). That's how I ended up finding all this mess, because my nginx never sends me a GOAWAY and sees failures before. > > I managed to work around the problem by limiting the number of total > > requests per connection. I find this extremely dirty but if it helps... > > I just need to figure how to best do it, so that we can use it as well > > for H2 as for H1. > > We're pretty satisfied with our h2 fe <-> be h1.1 setup right now, so we will > probably stick with that for now, since we don't want to have any more > operational issues from bleeding-edge bugs. (Not a comment on HAProxy, per > se, just a business reality. :-) ) I'm more than happy to try out anything > you turn up on our staging setup! You're absolutely right on this and don't need to justify your choices. For us having H2 on the backend is only a matter of completeness. While it does make sense for those deploying CDNs for example, or those dealing with APIs, on the local network it doesn't bring any real benefit and further increases the risk of head-of-line blocking due to the shared connection. And it indeed increases the risk of facing early bugs in products. Both haproxy's and nginx's HTTP/1 stacks are proven and rock solid, so you're clearly taking less risks with this. Regards, Willy
