Issue with parsing DNS from AWS
We have a setup with ECS and AWS's Service Discovery being load balanced by HAProxy in order to support sticky sessions for WebSocket handshakes, and we're working on making it more efficient by upgrading to 1.8.9 and taking advantage of seamless reloads and DNS service discovery. We have a solution almost working, however, we're seeing an issue during scaling when the DNS response crosses a certain size. We're using the following config (anonymized): https://gist.github.com/jredville/523de951d5ab6b60a0d345516bcf46d4 What we're seeing is: * if we bring up 3 target servers, they come up as healthy, and traffic is routed appropriately. If we restart haproxy, it comes up healthy * if we then scale to 4 or more servers, the 4th and additional are never recognized, however, the first 3 stay healthy * if we restart haproxy with 4 or more servers, no servers come up healthy We've attempted to modify the init-addr setting, accepted_payload_size, check options, and we've tried with and without a server-template and this is the behavior we consistently see. If we run strace over haproxy, we see it making the DNS requests but never updating the state of the servers. At this point we're not sure if we have something wrong in config or if there is a bug in how haproxy parses responses from AWS. Johnathan (cc'd) has pcap's if that would be helpful as well. Thanks, Jim
stable-bot: WARNING: 29 bug fixes in queue for next release
Hi, This is a friendly bot that watches fixes pending for the next haproxy-stable release! One such e-mail is sent every week once patches are waiting in the last maintenance branch, and an ideal release date is computed based on the severity of these fixes and their merge date. Responses to this mail must be sent to the mailing list. Last release 1.8.9 was issued on 2018/05/18. There are currently 29 patches in the queue cut down this way: - 2 BUILD, first one merged on 2018/05/23 - 2 MAJOR, first one merged on 2018/06/06 - 15 MEDIUM, first one merged on 2018/05/23 - 10 MINOR, first one merged on 2018/05/23 Thus the computed ideal release date for 1.8.10 would be 2018/06/20, which was within the last week. The current list of patches in the queue is: - BUILD : threads: unbreak build without threads - BUILD : fd: fix typo causing a warning when threads are disabled - MAJOR : map: fix a segfault when using http-request set-map - MAJOR : lua: Dead lock with sockets - MEDIUM : spoe: Return an error when the wrong ACK is received in sync mode - MEDIUM : lua/socket: Sheduling error on write: may dead-lock - MEDIUM : cache: don't cache when an Authorization header is present - MEDIUM : threads: handle signal queue only in thread 0 - MEDIUM : contrib/modsecurity: Use network order to encode/decode flags - MEDIUM : contrib/mod_defender: Use network order to encode/decode flags - MEDIUM : fd: Only check update_mask against all_threads_mask. - MEDIUM : spoe: Flags are not encoded in network order - MEDIUM : stick-tables: Decrement ref_cnt in table_* converters - MEDIUM : lua/socket: wrong scheduling for sockets - MEDIUM : dns: Delay the attempt to run a DNS resolution on check failure. - MEDIUM : lua/socket: Length required read doesn't work - MEDIUM : lua/socket: Notification error - MEDIUM : lua/socket: Buffer error, may segfault - MEDIUM : servers: Add srv_addr default placeholder to the state file - MINOR : contrib/mod_defender: update pointer on the end of the frame - MINOR : lua: Socket.send threw runtime error: 'close' needs 1 arguments. - MINOR : unix: Make sure we can transfer abns sockets on seamless reload. - MINOR : contrib/mod_defender: Don't reset the status code during disconnect - MINOR : don't ignore SIG{BUS,FPE,ILL,SEGV} during signal processing - MINOR : contrib/modsecurity: Don't reset the status code during disconnect - MINOR : signals: ha_sigmask macro for multithreading - MINOR : contrib/modsecurity: update pointer on the end of the frame - MINOR : ssl/lua: prevent lua from affecting automatic maxconn computation - MINOR : contrib/spoa_example: Don't reset the status code during disconnect --- The haproxy stable-bot is freely provided by HAProxy Technologies to help improve the quality of each HAProxy release. If you have any issue with these emails or if you want to suggest some improvements, please post them on the list so that the solutions suiting the most users can be found.
Re: remaining process after (seamless) reload
On Tue, Jun 19, 2018 at 4:30 PM William Lallemand wrote: > That's interesting, we can suppose that this bug is not related anymore to the > signal problem we had previously. > Looks like it's blocking in the thread sync point. > Are you able to do a backtrace with gdb? that could help a lot. yes, sorry, forgot to paste it. #0 0x56269318f0fb in thread_sync_barrier (barrier=0x5626933fa528 ) at src/hathreads.c:112 #1 thread_enter_sync () at src/hathreads.c:125 #2 0x5626931377a2 in sync_poll_loop () at src/haproxy.c:2376 #3 run_poll_loop () at src/haproxy.c:2433 #4 run_thread_poll_loop (data=data@entry=0x5626a7aa1000) at src/haproxy.c:2463 #5 0x5626930b3856 in main (argc=, argv=) at src/haproxy.c:3065 -- William
Re: remaining process after (seamless) reload
On Tue, Jun 19, 2018 at 04:09:51PM +0200, William Dauchy wrote: > Hello William, > > Not much progress on my side, apart from the fact I forgot to mention > where the process are now stuck using all the cpu, in > src/hathreads.c:112 > while (*barrier != all_threads_mask) > pl_cpu_relax(); > That's interesting, we can suppose that this bug is not related anymore to the signal problem we had previously. Looks like it's blocking in the thread sync point. Are you able to do a backtrace with gdb? that could help a lot. Thanks, -- William Lallemand
Re: [PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash
> Le 18 juin 2018 à 15:54, Thierry Fournier a > écrit : > > I don’t known. In fact it works, so it is not a bug. But, when I use the > reservation for an ex_data slot, it returns the slot 0, and this slot is > used for the compatibility layer and can be crush some data. I conclude > that is a bug in Openssl. The reservation function must give a slot > starting to 1. > Indeed, it’s unexpected. > Maybe, the recommendation is to not mix the compatibility functions > like set_app_data() with *ex_data*() functions. But, I don’t see > anything about this. > Neither do I... In any case it's cleaner with *_get_ex_new_index. ++ Manu
Re: remaining process after (seamless) reload
Hello William, Not much progress on my side, apart from the fact I forgot to mention where the process are now stuck using all the cpu, in src/hathreads.c:112 while (*barrier != all_threads_mask) pl_cpu_relax(); -- William
Re: [PATCH] MINOR: reg-tests: Add reg-tests/README file.
On Tue, Jun 19, 2018 at 02:24:23PM +0200, Frederic Lecaille wrote: > Here is a new patch to add reg-tests/README file and update CONTRIBUTING > file. Applied, thanks. Willy
[PATCH] MINOR: reg-tests: Add reg-tests/README file.
On 06/19/2018 10:18 AM, Willy Tarreau wrote: On Mon, Jun 18, 2018 at 07:50:38PM +0200, Frederic Lecaille wrote: Hello, Here is a simple patch to add a Makefile target to run all "*.vtc" regression testing files found in 'reg-tests' directory. (...) Thank you very much for this, Fred! I hope this will ignite a long series of such tests. It may be useful to add a README in the reg-tests directory indicating how to install varnishtest (especially the version compatible with haproxy), and probably suggest contributing new tests in the CONTRIBUTING file to encourage the practice. Here is a new patch to add reg-tests/README file and update CONTRIBUTING file. Fred. >From cc4ea839d4783de2e38389801d469cd908f3a469 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= Date: Tue, 19 Jun 2018 14:06:07 +0200 Subject: [PATCH] MINOR: reg-tests: Add reg-tests/README file. Add reg-tests/README file about how to compile and use varnishtest, and how to produce patches to add regression testing files to HAProxy sources. Also update CONTRIBUTING file to encourage the contributors to write regression testing files. --- CONTRIBUTING | 9 reg-tests/README | 63 2 files changed, 72 insertions(+) create mode 100644 reg-tests/README diff --git a/CONTRIBUTING b/CONTRIBUTING index b2c2b49..9e0d625 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -467,6 +467,12 @@ do not think about them anymore after a few patches. harm when nobody is able to tell whether or not the patch needs to be backported or can be reverted in case of regression. + When fixing a bug which is reproducible, if possible, the contributors are + strongly encouraged to write a regression testing VTC file for varnishtest + to add to reg-tests directory. More information about varnishtest may be + found in README file of reg-tests directory and in doc/regression-testing.txt + file. + 12) Discuss on the mailing list When submitting changes, please always CC the mailing list address so that @@ -615,6 +621,9 @@ part is being touched. The following tags are suggested but not limitative : - tests regression test files. No code is affected, no need to upgrade. + - reg-tests regression test files for varnishtest. No code is affected, no + need to upgrade. + - init initialization code, arguments parsing, etc... - configconfiguration parser, mostly used when adding new config keywords diff --git a/reg-tests/README b/reg-tests/README new file mode 100644 index 000..5427ad6 --- /dev/null +++ b/reg-tests/README @@ -0,0 +1,63 @@ + * Regression testing for HAProxy with varnishtest * + + +This little README file is about how to compile and run varnishtest test case files (VTC files) to test HAProxy for any regression. + +To do so, you will have to compile varnishtest program sources which comes with +Varnish cache application. varnishtest is a very useful program which has been +developed to test Varnish cache application. varnishtest has been modified in +collaboration with Varnish cache conceptor Poul-Henning Kamp to support HAProxy in +addition to Varnish cache. + + +* varnishtest compilation * + + - Clone recent Varnish cache sources which may be found in this GitHub repository +https://github.com/varnishcache/varnish-cache: + +$ git clone https://github.com/varnishcache/varnish-cache + + - Compile Varnish cache sources: + +$ cd varnish-cache && ./autogen.sh && ./configure && make + + Then varnishtest program may be found in bin/varnishtest directory. + VTC files for Varnish cache may be found in bin/varnishtest/tests directory. + The Varnish cache manuals are located in 'man' directory. You will have to + have a look at varnishtest(7) and vtc(7) manuals. + + Some information may also be found in doc/regression-testing.txt in HAProxy + sources. + + +* varnishtest execution * + + You must set HAPROXY_PROGRAM environment variable to give the location + of the HAProxy program to test to varnishtest: + +$ HAPROXY_PROGRAM= varnishtest ... + + The HAProxy VTC files found in HAProxy sources may be run with the reg-tests + Makefile target. You must set the VARNISHTEST_PROGRAM environment variable to + give the location of the varnishtest program which has been previously compiled. + +$ VARNISHTEST_PROGRAM= HAPROXY_PROGRAM= make reg-tests + + Note that varnishtest is run with -t5 and -l option. -l option is to keep + keep varnishtest temporary directory in case of failed test cases. core files + may be found in this directory (if enabled by ulimit). + + +* varnishtest patches for HAProxy VTC files * + + When producing a patch to add a VCT regression testing file to reg-tests directory, + please follow these simple rules: + +- Add the commit number on the first line. +- Then, copy and paste the commit log. +- If your VTC file needs others files,
Re: [PATCH] MINOR: tests: First regression testing file.
On Mon, Jun 18, 2018 at 07:50:38PM +0200, Frederic Lecaille wrote: > Hello, > > Here is a simple patch to add a Makefile target to run all "*.vtc" > regression testing files found in 'reg-tests' directory. (...) Thank you very much for this, Fred! I hope this will ignite a long series of such tests. It may be useful to add a README in the reg-tests directory indicating how to install varnishtest (especially the version compatible with haproxy), and probably suggest contributing new tests in the CONTRIBUTING file to encourage the practice. Willy