Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 6:09 AM Igor Cicimov wrote: > On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov < > ig...@encompasscorporation.com> wrote: > >> Hi, >> >> I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL >> terminating Haproxy-1.8.18. They share same storage and state via lock file >> and there is only one active AMQ at any given time. I'm testing this now >> with dynamic backend using Consul DNS resolution: >> >> # dig +short @127.0.0.1 -p 8600 activemq.service.consul >> 10.140.4.122 >> 10.140.3.171 >> >> # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV >> 1 1 61616 ip-10-140-4-122.node.dc1.consul. >> 1 1 61616 ip-10-140-3-171.node.dc1.consul. >> >> The backends status, the current "master": >> >> root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep >> java >> tcp0 0 0.0.0.0:81610.0.0.0:* >> LISTEN 5031374919617256/java >> tcp0 0 0.0.0.0:6161 0.0.0.0:* >> LISTEN 5031374919317256/java >> >> and the "slave": >> >> root@ip-10-140-4-122:~# netstat -tuplen | grep java >> >> So the service ports are not available on the second one. >> >> This is the relevant part of the HAP config that I think might be of >> interest: >> >> global >> server-state-base /var/lib/haproxy >> server-state-file hap_state >> >> defaults >> load-server-state-from-file global >> default-server init-addrlast,libc,none >> >> listen amq >> bind ... ssl crt ... >> mode http >> >> option prefer-last-server >> >> # when this is on the backend is down >> #option tcp-check >> >> default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s >> maxconn 25 maxqueue 256 weight 100 >> >> # working but both show as up >> server-template amqs 2 activemq.service.consul:8161 check >> >> # working old static setup >> #server ip-10-140-3-171 10.140.3.171:8161 check >> #server ip-10-140-4-122 10.140.4.122:8161 check >> >> This is working but the thing is I see both servers as UP in the HAP >> console: >> [image: amqs.png] >> Is this normal for this kind of setup or I'm doing something wrong? >> >> Another observation, when I have tcp check enabled like: >> >> option tcp-check >> >> the way I had it with the static lines like: >> >> server ip-10-140-3-171 10.140.3.171:8161 check >> server ip-10-140-4-122 10.140.4.122:8161 check >> >> then both servers show as down. >> Thanks in advance for any kind of input. >> Igor >> >> Ok, the state has changed now, I have correct state on one haproxy: > > [image: amqs_hap1.png] > but on the second the whole backend is down: > > [image: amqs_hap2.png] > I confirmed via telnet that I can connect to port 8161 to the running amq > server from both haproxy servers. > > Hi Igor, You're using the libc resolver function at startup time to resolve your backend, this is not recommended integration with Consul. You will find some good explanations in this blog article: https://www.haproxy.com/fr/blog/haproxy-and-consul-with-dns-for-service-discovery/ Basically, you should first create a "resolvers" section, in order to allow HAProxy to perform DNS resolution at runtime too. resolvers consul nameserver consul 127.0.0.1:8600 accepted_payload_size 8192 Then, you need to adjust your server-template line, like this: server-template amqs 10 _activemq._tcp.service.consul resolvers consul resolve-prefer ipv4 check In the example above, I am using on purpose the SRV records, because HAProxy supports it and it will use all information available in the response to update server's IP, weight and port. I hope this will help you. Baptiste
Re: Using server-template for DNS resolution
On Fri, Feb 8, 2019 at 2:29 PM Igor Cicimov wrote: > Hi, > > I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL > terminating Haproxy-1.8.18. They share same storage and state via lock file > and there is only one active AMQ at any given time. I'm testing this now > with dynamic backend using Consul DNS resolution: > > # dig +short @127.0.0.1 -p 8600 activemq.service.consul > 10.140.4.122 > 10.140.3.171 > > # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV > 1 1 61616 ip-10-140-4-122.node.dc1.consul. > 1 1 61616 ip-10-140-3-171.node.dc1.consul. > > The backends status, the current "master": > > root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep > java > tcp0 0 0.0.0.0:81610.0.0.0:* > LISTEN 5031374919617256/java > tcp0 0 0.0.0.0:6161 0.0.0.0:* > LISTEN 5031374919317256/java > > and the "slave": > > root@ip-10-140-4-122:~# netstat -tuplen | grep java > > So the service ports are not available on the second one. > > This is the relevant part of the HAP config that I think might be of > interest: > > global > server-state-base /var/lib/haproxy > server-state-file hap_state > > defaults > load-server-state-from-file global > default-server init-addrlast,libc,none > > listen amq > bind ... ssl crt ... > mode http > > option prefer-last-server > > # when this is on the backend is down > #option tcp-check > > default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s > maxconn 25 maxqueue 256 weight 100 > > # working but both show as up > server-template amqs 2 activemq.service.consul:8161 check > > # working old static setup > #server ip-10-140-3-171 10.140.3.171:8161 check > #server ip-10-140-4-122 10.140.4.122:8161 check > > This is working but the thing is I see both servers as UP in the HAP > console: > [image: amqs.png] > Is this normal for this kind of setup or I'm doing something wrong? > > Another observation, when I have tcp check enabled like: > > option tcp-check > > the way I had it with the static lines like: > > server ip-10-140-3-171 10.140.3.171:8161 check > server ip-10-140-4-122 10.140.4.122:8161 check > > then both servers show as down. > Thanks in advance for any kind of input. > Igor > > Ok, the state has changed now, I have correct state on one haproxy: [image: amqs_hap1.png] but on the second the whole backend is down: [image: amqs_hap2.png] I confirmed via telnet that I can connect to port 8161 to the running amq server from both haproxy servers.
Using server-template for DNS resolution
Hi, I have a Jetty frontend exposed for couple of ActiveMQ servers behind SSL terminating Haproxy-1.8.18. They share same storage and state via lock file and there is only one active AMQ at any given time. I'm testing this now with dynamic backend using Consul DNS resolution: # dig +short @127.0.0.1 -p 8600 activemq.service.consul 10.140.4.122 10.140.3.171 # dig +short @127.0.0.1 -p 8600 _activemq._tcp.service.consul SRV 1 1 61616 ip-10-140-4-122.node.dc1.consul. 1 1 61616 ip-10-140-3-171.node.dc1.consul. The backends status, the current "master": root@ip-10-140-3-171:~/configuration-management# netstat -tuplen | grep java tcp0 0 0.0.0.0:81610.0.0.0:* LISTEN 5031374919617256/java tcp0 0 0.0.0.0:6161 0.0.0.0:* LISTEN 5031374919317256/java and the "slave": root@ip-10-140-4-122:~# netstat -tuplen | grep java So the service ports are not available on the second one. This is the relevant part of the HAP config that I think might be of interest: global server-state-base /var/lib/haproxy server-state-file hap_state defaults load-server-state-from-file global default-server init-addrlast,libc,none listen amq bind ... ssl crt ... mode http option prefer-last-server # when this is on the backend is down #option tcp-check default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 25 maxqueue 256 weight 100 # working but both show as up server-template amqs 2 activemq.service.consul:8161 check # working old static setup #server ip-10-140-3-171 10.140.3.171:8161 check #server ip-10-140-4-122 10.140.4.122:8161 check This is working but the thing is I see both servers as UP in the HAP console: [image: amqs.png] Is this normal for this kind of setup or I'm doing something wrong? Another observation, when I have tcp check enabled like: option tcp-check the way I had it with the static lines like: server ip-10-140-3-171 10.140.3.171:8161 check server ip-10-140-4-122 10.140.4.122:8161 check then both servers show as down. Thanks in advance for any kind of input. Igor
Re: H2: interoperability issue due to lack of CONTINUATION frame support
Hello,
On Sat, 1 Sep 2018 at 20:02, Lukas Tribus wrote:
>
> Hi Willy,
>
>
> haproxy is currently unable to handle CONTINUATION [1] frames (see
> commit 61290ec77 - [2]).
>
> If a client emits a CONTINUATION frame, we will break the connection
> and send GOAWAY due to INTERNAL_ERROR. This of course leads to
> interoperability issues.
>
> Notably, older Chrome/Chromium releases (50.0.2645.4 and older),
> lacking Chromium commit a8bff211e ("Increase sent control frame
> fragment size to 16 kB") [3], will cap the maximum control frame size
> at 1024 bytes, triggering CONTINUATION, when large headers are used
> (long URIs, or large cookies).
For the record/archives:
CONTINUATION is supported since v1.9.2 and it does fix this issue.
lukas
[PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy
Hi,
This patch adds a new component in contrib. It is a Prometheus exporter
for HAProxy. It is developed as a "service", i.e. an applet which can be
called from a "use-service" rule. To build HAProxy with this component,
you should use the Makefile variable "EXTRA_OBJS":
$> make TARGET=linux2628
EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
To be used, it must be enabled in the configuration with an
"http-request" rule and the corresponding HTTP proxy must enable the HTX
support. For instance:
frontend test
mode http
...
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
...
More details in the README.
I'm not especially a Prometheus expert. And I must admit I never use it.
So if anyone have comments or suggestions, he is welcome.
Thanks
--
Christopher
>From d7e241786cc7056f28dad8bdd8fd72171c7a4e78 Mon Sep 17 00:00:00 2001
From: Christopher Faulet
Date: Thu, 7 Feb 2019 15:38:42 +0100
Subject: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus
exporter for HAProxy
It has been developped as a service applet. Internally, it is called
"promex". To build HAProxy with the promex service, you should use the Makefile
variable "EXTRA_OBJS". To be used, it must be enabled in the configuration with
an "http-request" rule and the corresponding HTTP proxy must enable the HTX
support. For instance:
frontend test
mode http
...
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
...
See contrib/prometheus-exporter/README for details.
---
contrib/prometheus-exporter/README| 242 ++
.../prometheus-exporter/service-prometheus.c | 2284 +
2 files changed, 2526 insertions(+)
create mode 100644 contrib/prometheus-exporter/README
create mode 100644 contrib/prometheus-exporter/service-prometheus.c
diff --git a/contrib/prometheus-exporter/README b/contrib/prometheus-exporter/README
new file mode 100644
index 0..f7fda6f07
--- /dev/null
+++ b/contrib/prometheus-exporter/README
@@ -0,0 +1,242 @@
+PROMEX: A Prometheus exporter for HAProxy
+---
+
+Prometheus is a monitoring and alerting system. More and more people use it to
+monitor their environment (this is written February 2019). It collects metrics
+from monitored targets by scraping metrics HTTP endpoints on these targets. For
+HAProxy, The Prometheus team offically supports an exporter written in Go
+(https://github.com/prometheus/haproxy_exporter). But it requires an extra
+software to deploy and monitor. PROMEX, on its side, is a built-in Prometheus
+exporter for HAProxy. It was developed as a service and is directly available in
+HAProxy, like the stats applet.
+
+However, PROMEX is not built by default with HAProxy. It is provided as an extra
+component for everyone want to use it. So you need to explicity build HAProxy
+with the PROMEX service, using the Makefile variable "EXTRA_OBJS". For instance:
+
+> make TARGET=linux2628 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
+
+if HAProxy provides the PROMEX service, the following build option will be
+reported by the command "haproxy -vv":
+
+Built with the Prometheus exporter as a service
+
+To be used, it must be enabled in the configuration with an "http-request" rule
+and the corresponding HTTP proxy must enable the HTX support. For instance:
+
+frontend test
+mode http
+...
+option http-use-htx
+http-request use-service prometheus-exporter if { path /metrics }
+...
+
+
+This service has been developed as a third-party component because it could
+become obsolete, depending on how much time Prometheus will remain heavily
+used. This is said with no ulterior motive of course. Prometheus is a great
+software and I hope all the well for it. But we involve in a environment moving
+quickly and a solution may be obvious today could be deprecated the next
+year. And because PROMEX is not integrated by default into the HAProxy codebase,
+it will need some interest to be actively supported. All contribution of any
+kind are welcome.
+
+You must also be careful if you use with huge configurations. Unlike the stats
+applet, all metrics are not grouped by service (proxy, listener or server). With
+PROMEX, all lines for a given metric are provided as one single group. So
+instead of collecting all metrics for a proxy before moving to the next one, we
+must loop on all proxies for each metric. Same for the servers. Thus, it will
+spend much more ressources to produce the Prometheus metrics than the CSV export
+through the stats page. To give a comparison order, quick benchmarks shown that
+a PROMEX dump is 5x slower and 20x more verbose than a CSV export.
+
+Exported metrics
+--
+
+* Globals metrics
+
Re: possible use of unitialized value in v2.0-dev0-274-g1a0fe3be
On Wed, Feb 06, 2019 at 07:12:31PM +0100, Tim Düsterhus wrote: (...) > Thus I believe this is a false-positive. I should have read the whole thread, it would have saved me a reply :-) Willy
Re: possible use of unitialized value in v2.0-dev0-274-g1a0fe3be
Hello,
On Wed, Feb 06, 2019 at 02:28:27PM -0200, Ricardo Nabinger Sanchez wrote:
> Hello,
>
> scan-build found a 28-step path where an unitialized value could be used in
> h2s_htx_bck_make_req_headers().
>
> Here is a shortened version:
>
> 4378 idx = htx_get_head(htx); // returns the SL that we skip
> 4379 while ((idx = htx_get_next(htx, idx)) != -1) {
> 4380 blk = htx_get_blk(htx, idx);
> 4381 type = htx_get_blk_type(blk);
> 4382
> 4383 if (type == HTX_BLK_UNUSED)
> 4384 continue;
> 4385
> 4386 if (type != HTX_BLK_HDR)
> // (here, assume condition is true, so control leaves the loop...)
> 4387 break;
> 4388
> 4389 if (unlikely(hdr >= sizeof(list)/sizeof(list[0]) - 1))
> 4390 goto fail;
> 4391
> // (... and list will not be initialized.)
> 4392 list[hdr].n = htx_get_blk_name(htx, blk);
> 4393 list[hdr].v = htx_get_blk_value(htx, blk);
> 4394 hdr++;
> 4395 }
>
> ...
The most important piece was skipped here :
/* marker for end of headers */
list[hdr].n = ist("");
So the list is *always* terminated.
> 4450 /* look for the Host header and place it in :authority */
> 4451 auth = ist2(NULL, 0);
> 4452 for (hdr = 0; hdr < sizeof(list)/sizeof(list[0]); hdr++)
> {
> 4453 if (isteq(list[hdr].n, ist("")))
> // (here, assume the condition is false, so control keeps in this block...)
> 4454 break; // end
Here the only way to reach this point is to have met an initialized
element, since the first one after the last initialized one sets the
name to the empty string and causes the break to happen.
> 4455
> 4456 if (isteq(list[hdr].n, ist("host"))) {
> 4457 auth = list[hdr].v;
> // (... auth receives an uninitialized value from list ...)
Thus no by definition.
> 4458 break;
> 4459 }
> 4460 }
> 4461 }
(...)
> While this feels like a convoluted or unlikely scenario, the path leading
> to the use of uninitialized value seems to be correctly unearthed by
> scan-build.
Fortunately no, it's a classical false positive instead.
Thanks!
Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
On Thu, Feb 07, 2019 at 06:37:28PM +0100, Willy Tarreau wrote: > > I'll try with h2c and see if I can put it between client and haproxy. > > Then I suspect that you'll see haproxy either emit RST_STREAM or emit > too large a frame and this frame get rejected. So after checking the code, I can confirm that you'll get the huge frame at once. I see how it would be possible to refragment this into one or several CONTINUATION frames with a limited extra cost. It will not be very pretty nor easy but I'm seeing a solution. Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
On Thu, Feb 07, 2019 at 06:44:01PM +0200, Jarno Huuskonen wrote: > At least on my test case haproxy listens http2 and uses http/1.1 > to backend server > (example config and example backend server (in go) are in earlier > mail: https://www.mail-archive.com/haproxy@formilux.org/msg32727.html > (just increase the header size (l in the go server) > 16309 and the > http2 connection between client <-> haproxy fails with frame error). OK, I initially misunderstood the setup then. It would be possible that the client never advertised support for a larger frame and that we don't have a solution to deliver something that large. In fact the only option would be to fragment the output and build a CONTINUATION frame. I don't know how difficult it could be in this case :-/ > So the test setup is something like: > client(curl/chrome) -> http2/haproxy -> http/1.1/go server port 8081 > > I'll try with h2c and see if I can put it between client and haproxy. Then I suspect that you'll see haproxy either emit RST_STREAM or emit too large a frame and this frame get rejected. Thanks for the explanation! Willy
Re: http-use-htx and IIS
Il 2019-02-07 17:50 Marco Corte ha scritto: A configuration similar to the following lets the GETs work properly, but the POST fails after the server timeout (session state "SD" in haproxy logs): Sorry. I was wrong. It is a capital "S" S : the TCP session was unexpectedly aborted by the server, or the server explicitly refused it. Anyway it happens _many seconds_ after the request. .marcoc
http-use-htx and IIS
Hello! I am testing haproxy version 1.9.4 on Ubuntu 18.04. With the "option http-use-htx", haproxy shows a strange behaviour when the real server is IIS and if the users' browsers try to do a POST. A configuration similar to the following lets the GETs work properly, but the POST fails after the server timeout (session state "SD" in haproxy logs): listen tesi-http bind 10.64.44.112:443 name HTTPS ssl crt XXX option http-use-htx server vxws142a-82 11.22.33.44:82 If the connection to the real server is closed, all GETs and POSTs work listen tesi-http bind 10.64.44.112:443 name HTTPS ssl crt XXX option http-use-htx option http-server-close server vxws142a-82 11.22.33.44:82 In other words, it seems that the "option http-use-htx" only works together with "http-server-close" with this real server. Notes: . The behaviour does not change if "alpn h2,http/1.1" is added/removed from the "bind" line. . The behaviour does not change if the browser is firefox or IE . With tcpdump on the haproxy node I was not able to understand who is waiting for whom :-) . I am not sure that I can install wireshark on the real server. Can someone please suggest me what to look for in the IIS or in the haproxy configuration? Thank you .marcoc
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, Willy Tarreau wrote: > On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > > Hi, > > > > On Thu, Feb 07, Steve GIRAUD wrote: > > > Thanks Jarno for the investigation. > > > > No problem. > > > > > The large header is only on response and there is only one large header > > > (18k). > > > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > > > Did you check with curl or chrome if you get the same framing error > > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > > request is ok. > > > > I'm CC:ing Willy, in case this is something that a config option can fix > > or possibly a incompatability/bug with http2 implementation. > > I might have an idea. The default H2 max-frame-size is 16kB (by the > spec). It is possible that your server ignores the setting and tries > to push a frame size that is larger than the agreed limit, which > becomes a protocol violation. Or it is possible as well that the > server doesn't know how to send such a large header with this frame > size and simply aborts the response. At least on my test case haproxy listens http2 and uses http/1.1 to backend server (example config and example backend server (in go) are in earlier mail: https://www.mail-archive.com/haproxy@formilux.org/msg32727.html (just increase the header size (l in the go server) > 16309 and the http2 connection between client <-> haproxy fails with frame error). So the test setup is something like: client(curl/chrome) -> http2/haproxy -> http/1.1/go server port 8081 I'll try with h2c and see if I can put it between client and haproxy. -Jarno > You could install h2c between haproxy and your server, in wiretap mode, > it's very convenient to see what is exchanged : > >h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: > > Then you configure haproxy to communicate to 127.0.0.1: to join the > server while your server in fact listens on :. > > Depending on what you see, we may have the possibility to work around > it by advertising a larger max-frame-size in the settings frame when > the buffers are larger. > > Regards, > Willy > -- Jarno Huuskonen
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Thanks for your precious answer, I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. For information, we use IIS10 on windows2016 for the backend server. And Jarno reproduce the same problem with his go http server implementation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. We have dump the exchange beetween haproxy and the backend. The response seems not to be aborted by the backend. We are installing h2c to investigate. Regards, Steve De : Willy Tarreau Envoyé : jeudi 7 février 2019 17:03 À : Jarno Huuskonen; Steve GIRAUD; haproxy@formilux.org Objet : Re: HAProxy returns a 502 error when ssl offload and response has a large header Hi, On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > Hi, > > On Thu, Feb 07, Steve GIRAUD wrote: > > Thanks Jarno for the investigation. > > No problem. > > > The large header is only on response and there is only one large header > > (18k). > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > Did you check with curl or chrome if you get the same framing error > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > request is ok. > > I'm CC:ing Willy, in case this is something that a config option can fix > or possibly a incompatability/bug with http2 implementation. I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. You could install h2c between haproxy and your server, in wiretap mode, it's very convenient to see what is exchanged : h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: Then you configure haproxy to communicate to 127.0.0.1: to join the server while your server in fact listens on :. Depending on what you see, we may have the possibility to work around it by advertising a larger max-frame-size in the settings frame when the buffers are larger. Regards, Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, 2019 at 04:50:12PM +0200, Jarno Huuskonen wrote: > Hi, > > On Thu, Feb 07, Steve GIRAUD wrote: > > Thanks Jarno for the investigation. > > No problem. > > > The large header is only on response and there is only one large header > > (18k). > > > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails > > Did you check with curl or chrome if you get the same framing error > that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > > > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > > > If I request my backend directly in h2 + ssl but without haproxy, the > > request is ok. > > I'm CC:ing Willy, in case this is something that a config option can fix > or possibly a incompatability/bug with http2 implementation. I might have an idea. The default H2 max-frame-size is 16kB (by the spec). It is possible that your server ignores the setting and tries to push a frame size that is larger than the agreed limit, which becomes a protocol violation. Or it is possible as well that the server doesn't know how to send such a large header with this frame size and simply aborts the response. You could install h2c between haproxy and your server, in wiretap mode, it's very convenient to see what is exchanged : h2c_linux_amd64 wiretap 127.0.0.1: 127.0.0.1: Then you configure haproxy to communicate to 127.0.0.1: to join the server while your server in fact listens on :. Depending on what you see, we may have the possibility to work around it by advertising a larger max-frame-size in the settings frame when the buffers are larger. Regards, Willy
Re: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, On Thu, Feb 07, Steve GIRAUD wrote: > Thanks Jarno for the investigation. No problem. > The large header is only on response and there is only one large header (18k). > > haproxy + ssl + http2+ tune.bufsize:32768 --> request fails Did you check with curl or chrome if you get the same framing error that I got (Error in the HTTP2 framing layer / ERR_SPDY_FRAME_SIZE_ERROR))? > haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok > > If I request my backend directly in h2 + ssl but without haproxy, the request > is ok. I'm CC:ing Willy, in case this is something that a config option can fix or possibly a incompatability/bug with http2 implementation. -Jarno > Hi, > > On Wed, Feb 06, Steve GIRAUD wrote: > > Effectively, the header size is 17 556 bytes. > > Is the large header(s) only on response (and not on request) ? > (Is it one large header 17k header ?) > > > If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the > > request failed. > > For me (tested with current 2.0dev) increasing global tune.bufsize to > 32768 allowed larger response header. With my limited testing http/https on > frontend didn't make difference. > (Does my test config work for you (you'll need to comment option htx > with haprox-1.8) ?) > > But if I use curl --http2 to haproxy+ssl frontend and my silly > httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails > with curl: (16) Error in the HTTP2 framing layer > (chrome reports ERR_SPDY_FRAME_SIZE_ERROR). > > Is haproxy trying / sending a larger http2 frame than clients are > willing to receive (SETTINGS_MAX_FRAME_SIZE?) ? > > (Same request with --http1.1 to haproxy+ssl frontend works). > > I'm attaching my test config and the httpsrv.go that I used as a > backend server. > Maybe http2 gurus can take a look and see if the frame size error is > expected or not ? > > -Jarno > > > De : Jarno Huuskonen > > Envoyé : mercredi 6 février 2019 09:36 > > À : Steve GIRAUD > > Cc : haproxy@formilux.org > > Objet : Re: HAProxy returns a 502 error when ssl offload and response has a > > large header > > > > Hi, > > > > On Wed, Feb 06, Steve GIRAUD wrote: > > > Hello everybody, > > > Has anyone ever found that HAProxy returns a 502 error when ssl offload > > > is enabled and the http response contains a very long header. > > > If I turn off SSL offload , all is OK with the same header. > > > > What's the size of the (very long) headers (how many bytes) ? > > Is it by any chance larger than the bufsize or maxrewrite ? > > > > > Default settings : > > > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 -- Jarno Huuskonen
RE: RSA Conference 2019 Attendees List
Hi, Just following up. I know you're busy. Just give me a 1,2 or 3 1. Send data counts and pricing. 2. I could potentially use your solution, but in future. 3. Leave me alone, stalker. Regards, Nile Smith Demand Generation Executive From: Nile Smith [mailto:nile.sm...@clienthubmarketing.com] Sent: Wednesday, February 06, 2019 12:42 PM To: 'haproxy@formilux.org' Subject: RSA Conference 2019 Attendees List Importance: High Hi, I am contacting you to know if you would be interested in purchasing RSA Conference 2019 Attendees List which includes complete contact details and verified email address. Attendees Profile: . CEO . COO . Directors . Vice Presidents, . Marketing Head . Key Decision Makers . And Much More Each record in the list contains - Contact Name, Job Title, Company/Business Name, Complete Mailing Address, email, Telephone/Fax Number, Website/URL, Revenue, Employee Size, SIC Code, Industry. If you're open to learning more, I'm happy to send over a pricing and data counts details. Looking forward to seeing if you agree it's a good fit for you guys. Thanks and Regards! Nile Smith Demand Generation Executive. If you do not wish to receive future emails from us, please reply as 'leave out'
RE: HAProxy returns a 502 error when ssl offload and response has a large header
Hi, Thanks Jarno for the investigation. The large header is only on response and there is only one large header (18k). haproxy + ssl + http2+ tune.bufsize:32768 --> request fails haproxy + ssl + http1.1 + tune.bufsize:32768 --> request ok If I request my backend directly in h2 + ssl but without haproxy, the request is ok. Regards, -- Steve Giraud Hi, On Wed, Feb 06, Steve GIRAUD wrote: > Effectively, the header size is 17 556 bytes. Is the large header(s) only on response (and not on request) ? (Is it one large header 17k header ?) > If I increase the bufsize to 40 000 bytes and the maxrewrite to 20 000 the > request failed. For me (tested with current 2.0dev) increasing global tune.bufsize to 32768 allowed larger response header. With my limited testing http/https on frontend didn't make difference. (Does my test config work for you (you'll need to comment option htx with haprox-1.8) ?) But if I use curl --http2 to haproxy+ssl frontend and my silly httpsrv.go sends x-dummy larger than 16309 then curl --http2 fails with curl: (16) Error in the HTTP2 framing layer (chrome reports ERR_SPDY_FRAME_SIZE_ERROR). Is haproxy trying / sending a larger http2 frame than clients are willing to receive (SETTINGS_MAX_FRAME_SIZE?) ? (Same request with --http1.1 to haproxy+ssl frontend works). I'm attaching my test config and the httpsrv.go that I used as a backend server. Maybe http2 gurus can take a look and see if the frame size error is expected or not ? -Jarno > De : Jarno Huuskonen > Envoyé : mercredi 6 février 2019 09:36 > À : Steve GIRAUD > Cc : haproxy@formilux.org > Objet : Re: HAProxy returns a 502 error when ssl offload and response has a > large header > > Hi, > > On Wed, Feb 06, Steve GIRAUD wrote: > > Hello everybody, > > Has anyone ever found that HAProxy returns a 502 error when ssl offload is > > enabled and the http response contains a very long header. > > If I turn off SSL offload , all is OK with the same header. > > What's the size of the (very long) headers (how many bytes) ? > Is it by any chance larger than the bufsize or maxrewrite ? > > > Default settings : > > maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 > > -Jarno > > -- > Jarno Huuskonen -- Jarno Huuskonen
Re: [ANNOUNCE] haproxy-1.9.4
Am 06.02.2019 um 17:19 schrieb Willy Tarreau: > Hi Aleks, > > On Wed, Feb 06, 2019 at 05:16:58PM +0100, Aleksandar Lazic wrote: >> Maybe this patch was to late for 1.9.4 please can you consider to add it >> to 2.0 and later 1.9.5, thanks. >> >> https://www.mail-archive.com/haproxy@formilux.org/msg32693.html > > I wanted to check it with Christopher first but I know he's busy working > on some extremely boring stuff, and don't want to risk trading his stuff > for a review :-) ;-) > I'll also have to correct a number of spelling mistakes so better be sure > before doing this. Ah cool. thanks. BTW: the openssl reg-tests was passed without errors https://gitlab.com/aleks001/haproxy19-centos/-/jobs/157330203 ## Starting vtest ## Testing with haproxy version: 1.9.4 0 tests failed, 0 tests skipped, 35 tests passed the boringssl reg-tests passed with errors. https://gitlab.com/aleks001/haproxy-19-boringssl/-/jobs/157330626 ## Starting vtest ## Testing with haproxy version: 1.9.4 #top TEST ./reg-tests/connection/b0.vtc FAILED (8.790) exit=2 1 tests failed, 0 tests skipped, 34 tests passed ## Gathering results ## > Thanks! > Willy Regards Aleks
