Hi Frank,
On Fri, Feb 23, 2018 at 12:10:13PM +0000, Frank Schreuder wrote:
> > Well, at least you don't use threads nor lua nor caching nor HTTP/2 so
> > it cannot come from any of those we have identified. It could still come
> > from openssl however.
>
> There are some bugfixes marked as medium in the haproxy 1.8 repository
> related to SSL. Would it be possible that they are related to the crashes I'm
> seeing?
Unfortunately no, I'm not seeing anything which could explain this. I'm
currently working on an issue affecting 1.8 with H2 where a wrongly sized
error message could cause a leak of struct stream+struct buffer, but I'm
not seeing this impact non-H2 traffic, so that cannot be your issue either
(it only happens when the advertised content-length in the response is
lower than the actual data sent).
> > > I'm using systemd to reload haproxy for new SSL certificates every few
> > > minutes.
> >
> > OK. I'm seeing that you load certs from a directory in your config. Do you
> > have a high number of certs ? I'm asking because we've already seen some
> > configs eating multiple gigs of RAM with the certs because there were a lot.
>
> Yes I have around 40k SSL certificates in this directory and is growing over
> time.
OK then that makes sense. I would love to find a way to report the amount
of RAM used by certs, it could be helpful to troubleshoot such issues.
> > In your case they're loaded twice (one for the IPv4 bind line, one for the
> > IPv6). William planned to work on a way to merge all identical certs and
> > have
> > a single instance of them when loaded multiple times, which should already
> > reduce the amount of memory consumed by this.
>
> I can bind ipv4 and ipv6 in the same line with:
> bind ipv4@:443,ipv6@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt
> /etc/haproxy/ssl/ crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534
>
> This would also solve the "double load" issue right?
Sure! I totally forgot about this syntax ;-)
> > > frontend fe_http
> > > bind ipv4@:80 backlog 65534
> > > bind ipv6@:80 backlog 65534
> > > bind ipv4@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt
> > >/etc/haproxy/ssl/ crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534
> > > bind ipv6@:443 ssl crt /etc/haproxy/ssl/invalid.pem crt
> > >/etc/haproxy/ssl/ crt /etc/haproxy/customer-ssl/ strict-sni backlog 65534
> > > bind-process 1-7
> > > tcp-request inspect-delay 5s
> > > tcp-request content accept if { req_ssl_hello_type 1 }
> >
> > This one is particularly strange. I suspect it's a leftover from an old
> > configuration dating from the days where haproxy didn't support SSL,
> > because it's looking for SSL messages inside the HTTP traffic, which
> > will never be present. You can safely remove tose two lines.
>
> We use this to guard against some attacks we have seen in the past. Setting
> up connections without ssl handshake to use all available connections. I will
> remove them if you are sure this no longer works.
It doesn't do what you seem to believe it does (it has never worked). It
works if you're *not* decrypting SSL. Typically you'd have a TCP listener
passing the traffic to the next layer, and enforcing this control. Then it
would work. But here it's placed the other way around. SSL is unstripped
at the edge, and the clear text extracted from SSL stream is fed through
the TCP rules. Thus these rules can never match, unless of course you're
doing a double encapsulation of SSL, which is not really the purpose here :-)
> > Aside this I really see nothing suspicious in your configuration that could
> > justify a problem. Let's hope you can at least either catch a core or attach
> > a gdb to one of these processes.
>
> I will let you know as soon as I'm able to get a backtrace. In the meanwhile
> I will improve and test my new configuration changes.
OK.
Cheers,
Willy
