Issue with parsing DNS from AWS

2018-06-19 Thread Jim Deville
We have a setup with ECS and AWS's Service Discovery being load balanced by 
HAProxy in order to support sticky sessions for WebSocket handshakes, and we're 
working on making it more efficient by upgrading to 1.8.9 and taking advantage 
of seamless reloads and DNS service discovery. We have a solution almost 
working, however, we're seeing an issue during scaling when the DNS response 
crosses a certain size.


We're using the following config (anonymized): 
https://gist.github.com/jredville/523de951d5ab6b60a0d345516bcf46d4

What we're seeing is:
  * if we bring up 3 target servers, they come up as healthy, and traffic is 
routed appropriately. If we restart haproxy, it comes up healthy
  * if we then scale to 4 or more servers, the 4th and additional are never 
recognized, however, the first 3 stay healthy
  * if we restart haproxy with 4 or more servers, no servers come up healthy

We've attempted to modify the init-addr setting, accepted_payload_size, check 
options, and we've tried with and without a server-template and this is the 
behavior we consistently see. If we run strace over haproxy, we see it making 
the DNS requests but never updating the state of the servers. At this point 
we're not sure if we have something wrong in config or if there is a bug in how 
haproxy parses responses from AWS. Johnathan (cc'd) has pcap's if that would be 
helpful as well.

Thanks,
Jim


stable-bot: WARNING: 29 bug fixes in queue for next release

2018-06-19 Thread stable-bot
Hi,

This is a friendly bot that watches fixes pending for the next haproxy-stable 
release!  One such e-mail is sent every week once patches are waiting in the 
last maintenance branch, and an ideal release date is computed based on the 
severity of these fixes and their merge date.  Responses to this mail must be 
sent to the mailing list.

Last release 1.8.9 was issued on 2018/05/18.  There are currently 29 patches in 
the queue cut down this way:
- 2 BUILD, first one merged on 2018/05/23
- 2 MAJOR, first one merged on 2018/06/06
- 15 MEDIUM, first one merged on 2018/05/23
- 10 MINOR, first one merged on 2018/05/23

Thus the computed ideal release date for 1.8.10 would be 2018/06/20, which was 
within the last week.

The current list of patches in the queue is:
- BUILD   : threads: unbreak build without threads
- BUILD   : fd: fix typo causing a warning when threads are disabled
- MAJOR   : map: fix a segfault when using http-request set-map
- MAJOR   : lua: Dead lock with sockets
- MEDIUM  : spoe: Return an error when the wrong ACK is received in sync 
mode
- MEDIUM  : lua/socket: Sheduling error on write: may dead-lock
- MEDIUM  : cache: don't cache when an Authorization header is present
- MEDIUM  : threads: handle signal queue only in thread 0
- MEDIUM  : contrib/modsecurity: Use network order to encode/decode flags
- MEDIUM  : contrib/mod_defender: Use network order to encode/decode flags
- MEDIUM  : fd: Only check update_mask against all_threads_mask.
- MEDIUM  : spoe: Flags are not encoded in network order
- MEDIUM  : stick-tables: Decrement ref_cnt in table_* converters
- MEDIUM  : lua/socket: wrong scheduling for sockets
- MEDIUM  : dns: Delay the attempt to run a DNS resolution on check failure.
- MEDIUM  : lua/socket: Length required read doesn't work
- MEDIUM  : lua/socket: Notification error
- MEDIUM  : lua/socket: Buffer error, may segfault
- MEDIUM  : servers: Add srv_addr default placeholder to the state file
- MINOR   : contrib/mod_defender: update pointer on the end of the frame
- MINOR   : lua: Socket.send threw runtime error: 'close' needs 1 arguments.
- MINOR   : unix: Make sure we can transfer abns sockets on seamless reload.
- MINOR   : contrib/mod_defender: Don't reset the status code during 
disconnect
- MINOR   : don't ignore SIG{BUS,FPE,ILL,SEGV} during signal processing
- MINOR   : contrib/modsecurity: Don't reset the status code during 
disconnect
- MINOR   : signals: ha_sigmask macro for multithreading
- MINOR   : contrib/modsecurity: update pointer on the end of the frame
- MINOR   : ssl/lua: prevent lua from affecting automatic maxconn 
computation
- MINOR   : contrib/spoa_example: Don't reset the status code during 
disconnect

---
The haproxy stable-bot is freely provided by HAProxy Technologies to help 
improve the quality of each HAProxy release.  If you have any issue with these 
emails or if you want to suggest some improvements, please post them on the 
list so that the solutions suiting the most users can be found.



Re: remaining process after (seamless) reload

2018-06-19 Thread William Dauchy
On Tue, Jun 19, 2018 at 4:30 PM William Lallemand
 wrote:
> That's interesting, we can suppose that this bug is not related anymore to the
> signal problem we had previously.
> Looks like it's blocking in the thread sync point.
> Are you able to do a backtrace with gdb? that could help a lot.

yes, sorry, forgot to paste it.

#0  0x56269318f0fb in thread_sync_barrier (barrier=0x5626933fa528
) at src/hathreads.c:112
#1  thread_enter_sync () at src/hathreads.c:125
#2  0x5626931377a2 in sync_poll_loop () at src/haproxy.c:2376
#3  run_poll_loop () at src/haproxy.c:2433
#4  run_thread_poll_loop (data=data@entry=0x5626a7aa1000) at src/haproxy.c:2463
#5  0x5626930b3856 in main (argc=, argv=) at src/haproxy.c:3065

-- 
William



Re: remaining process after (seamless) reload

2018-06-19 Thread William Lallemand
On Tue, Jun 19, 2018 at 04:09:51PM +0200, William Dauchy wrote:
> Hello William,
> 
> Not much progress on my side, apart from the fact I forgot to mention
> where the process are now stuck using all the cpu, in
> src/hathreads.c:112
> while (*barrier != all_threads_mask)
>   pl_cpu_relax();
> 

That's interesting, we can suppose that this bug is not related anymore to the
signal problem we had previously.

Looks like it's blocking in the thread sync point.

Are you able to do a backtrace with gdb? that could help a lot.

Thanks,

-- 
William Lallemand



Re: [PATCH] Re: Random crash (segfault, double free, ...) with a mix of SSL + cipherlist hash

2018-06-19 Thread Emmanuel Hocdet


> Le 18 juin 2018 à 15:54, Thierry Fournier  a 
> écrit :
> 
> I don’t known. In fact it works, so it is not a bug. But, when I use the
> reservation for an ex_data slot, it returns the slot 0, and this slot is
> used for the compatibility layer and can be crush some data. I conclude
> that is a bug in Openssl. The reservation function must give a slot
> starting to 1.
> 

Indeed, it’s unexpected.

> Maybe, the recommendation is to not mix the compatibility functions
> like set_app_data() with *ex_data*() functions. But, I don’t see 
> anything about this.
> 

Neither do I...
In any case it's cleaner with  *_get_ex_new_index.

++
Manu





Re: remaining process after (seamless) reload

2018-06-19 Thread William Dauchy
Hello William,

Not much progress on my side, apart from the fact I forgot to mention
where the process are now stuck using all the cpu, in
src/hathreads.c:112
while (*barrier != all_threads_mask)
  pl_cpu_relax();

-- 
William



Re: [PATCH] MINOR: reg-tests: Add reg-tests/README file.

2018-06-19 Thread Willy Tarreau
On Tue, Jun 19, 2018 at 02:24:23PM +0200, Frederic Lecaille wrote:
> Here is a new patch to add reg-tests/README file and update CONTRIBUTING
> file.

Applied, thanks.

Willy



[PATCH] MINOR: reg-tests: Add reg-tests/README file.

2018-06-19 Thread Frederic Lecaille

On 06/19/2018 10:18 AM, Willy Tarreau wrote:

On Mon, Jun 18, 2018 at 07:50:38PM +0200, Frederic Lecaille wrote:

Hello,

Here is a simple patch to add a Makefile target to run all "*.vtc"
regression testing files found in 'reg-tests' directory.

(...)

Thank you very much for this, Fred! I hope this will ignite a long
series of such tests. It may be useful to add a README in the reg-tests
directory indicating how to install varnishtest (especially the version
compatible with haproxy), and probably suggest contributing new tests
in the CONTRIBUTING file to encourage the practice.


Here is a new patch to add reg-tests/README file and update CONTRIBUTING 
file.


Fred.
>From cc4ea839d4783de2e38389801d469cd908f3a469 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= 
Date: Tue, 19 Jun 2018 14:06:07 +0200
Subject: [PATCH] MINOR: reg-tests: Add reg-tests/README file.

Add reg-tests/README file about how to compile and use varnishtest, and
how to produce patches to add regression testing files to HAProxy sources.

Also update CONTRIBUTING file to encourage the contributors to write
regression testing files.
---
 CONTRIBUTING |  9 
 reg-tests/README | 63 
 2 files changed, 72 insertions(+)
 create mode 100644 reg-tests/README

diff --git a/CONTRIBUTING b/CONTRIBUTING
index b2c2b49..9e0d625 100644
--- a/CONTRIBUTING
+++ b/CONTRIBUTING
@@ -467,6 +467,12 @@ do not think about them anymore after a few patches.
harm when nobody is able to tell whether or not the patch needs to be
backported or can be reverted in case of regression.
 
+   When fixing a bug which is reproducible, if possible, the contributors are
+   strongly encouraged to write a regression testing VTC file for varnishtest
+   to add to reg-tests directory. More information about varnishtest may be
+   found in README file of reg-tests directory and in doc/regression-testing.txt
+   file.
+
 12) Discuss on the mailing list
 
When submitting changes, please always CC the mailing list address so that
@@ -615,6 +621,9 @@ part is being touched. The following tags are suggested but not limitative :
 
   - tests regression test files. No code is affected, no need to upgrade.
 
+  - reg-tests regression test files for varnishtest. No code is affected, no
+  need to upgrade.
+
   - init  initialization code, arguments parsing, etc...
 
   - configconfiguration parser, mostly used when adding new config keywords
diff --git a/reg-tests/README b/reg-tests/README
new file mode 100644
index 000..5427ad6
--- /dev/null
+++ b/reg-tests/README
@@ -0,0 +1,63 @@
+ * Regression testing for HAProxy with varnishtest *
+
+
+This little README file is about how to compile and run varnishtest test case files (VTC files) to test HAProxy for any regression.
+
+To do so, you will have to compile varnishtest program sources which comes with
+Varnish cache application. varnishtest is a very useful program which has been
+developed to test Varnish cache application. varnishtest has been modified in
+collaboration with Varnish cache conceptor Poul-Henning Kamp to support HAProxy in
+addition to Varnish cache.
+
+
+* varnishtest compilation *
+
+  - Clone recent Varnish cache sources which may be found in this GitHub repository
+https://github.com/varnishcache/varnish-cache:
+
+$ git clone https://github.com/varnishcache/varnish-cache
+
+  - Compile Varnish cache sources:
+
+$ cd varnish-cache && ./autogen.sh && ./configure && make
+
+  Then varnishtest program may be found in bin/varnishtest directory.
+  VTC files for Varnish cache may be found in bin/varnishtest/tests directory.
+  The Varnish cache manuals are located in 'man' directory. You will have to
+  have a look at varnishtest(7) and vtc(7) manuals.
+
+  Some information may also be found in doc/regression-testing.txt in HAProxy
+  sources.
+
+
+* varnishtest execution *
+
+  You must set HAPROXY_PROGRAM environment variable to give the location
+  of the HAProxy program to test to varnishtest:
+
+$ HAPROXY_PROGRAM= varnishtest ...
+
+  The HAProxy VTC files found in HAProxy sources may be run with the reg-tests
+  Makefile target. You must set the VARNISHTEST_PROGRAM environment variable to
+  give the location of the varnishtest program which has been previously compiled.
+
+$ VARNISHTEST_PROGRAM= HAPROXY_PROGRAM= make reg-tests
+
+  Note that varnishtest is run with -t5 and -l option. -l option is to keep
+  keep varnishtest temporary directory in case of failed test cases. core files
+  may be found in this directory (if enabled by ulimit).
+
+
+* varnishtest patches for HAProxy VTC files *
+
+  When producing a patch to add a VCT regression testing file to reg-tests directory,
+  please follow these simple rules:
+
+- Add the commit number on the first line.
+- Then, copy and paste the commit log.
+- If your VTC file needs others files, 

Re: [PATCH] MINOR: tests: First regression testing file.

2018-06-19 Thread Willy Tarreau
On Mon, Jun 18, 2018 at 07:50:38PM +0200, Frederic Lecaille wrote:
> Hello,
> 
> Here is a simple patch to add a Makefile target to run all "*.vtc"
> regression testing files found in 'reg-tests' directory.
(...)

Thank you very much for this, Fred! I hope this will ignite a long
series of such tests. It may be useful to add a README in the reg-tests
directory indicating how to install varnishtest (especially the version
compatible with haproxy), and probably suggest contributing new tests
in the CONTRIBUTING file to encourage the practice.

Willy