Re: HAProxy won't shut down

[Frederic Lecaille]

On 05/29/2017 06:12 PM, Patrick Hemmer wrote:


On 2017/5/29 08:22, Frederic Lecaille wrote:


Hi Patrick,

First thank you for this nice and helpful report.

Would it be possible to have an output of this command the next time
you reproduce such an issue please?

echo "show sess" | socat stdio 


Unfortunately this would not be possible. When the issue occurs, the
haproxy process has stopped accepting connections on all sockets. If I
were to run this command, it would be sent to the new process, not the
one that won't shut down.



If you send a SIGHUP to haproxy-systemd-wrapper it asks the old process 
to graceful stop.


Please have a look to this documentation:

https://cbonte.github.io/haproxy-dconv/1.7/management.html#4

So you are true, if everything goes well no more connection are 
accept()'ed by the old process (the sockets have been unbound). But in 
your reported case the peers sockets are not closed because still in 
CLOSE_WAIT state, so are still being processed, so stats information are 
still available from the socket stats.


If I have missed something please does not hesitate to yell at me ;) .

I have been told that "show sess *all*" give more information.



I have only one question (see below).

On 05/24/2017 10:40 AM, Willy Tarreau wrote:

Hi Patrick,

On Tue, May 23, 2017 at 01:49:42PM -0400, Patrick Hemmer wrote:
(...)

haproxy 28856 root1u IPv4  420797940  0t0
TCP 10.0.33.145:35754->10.0.33.147:1029 (CLOSE_WAIT)
haproxy 28856 root2u IPv4  420266351  0t0
TCP 10.0.33.145:52898->10.0.33.147:1029 (CLOSE_WAIT)
haproxy 28856 root3r  REG0,30
4026531956 net
haproxy 28856 root4u IPv4  422150834  0t0
TCP 10.0.33.145:38874->10.0.33.147:1029 (CLOSE_WAIT)


These ones are very interesting.


These traces also seem interesting to me.

# strace -p 28856
Process 28856 attached
epoll_wait(0, {}, 200, 319) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 362) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 114) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 203) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 331) = 0
epoll_wait(0, {}, 200, 0)


Were such "epoll_wait(0, 0, 200, 0)" calls infinitively displayed?

Yes




In fact I am wondering if it is normal to have so much epoll_wait(0,
{}, 200, 0) calls for a haproxy process which has shut down.

I suspect they are in relation with peer tasks (obviously which has
expired).

If this is the case, and with configurations with only peer tasks,
haproxy would definitively hang consuming a lot of CPU resources.

HAProxy was not consuming high CPU. Note that in every other call to
`epoll_wait`, the 4th value was >0. If every single timeout value were
0, then yes, it would spin consuming CPU.



agreed... but perhaps your configuration does not use only peer tasks, 
contrary to my configuration... this is your traces which lead me to 
check how the peer task expiration is handled with configurations with 
only peers as backends.


In my case with only two peers I see such following traces, after a peer 
has sent a synchronization request:


epoll_wait(0, {}, 200, 1000}
epoll_wait(0, {}, 200, 1000}
epoll_wait(0, {}, 200, 1000}
epoll_wait(0, {}, 200, 1000}
epoll_wait(0, {}, 200, X}# with X < 1000

followed by a big loop of

epoll_wait(0, {}, 200, 0}# so consuming high CPU resources 
during a fraction of second


then:

shutdown(SHUT_WR)# FIN TCP segment at about 5s after 
the first epoll_wait(0, 0, 200, 1000} above.


then again:

epoll_wait(0, {}, 200, 0}

until the remote peer, which is in CLOSE_WAIT state shuts down its socket.

I have not told you that a synchronization request is the first thing a 
peer launches: the peers of the new process try to synchronize with old 
process peers.


With the fix I provided the process epoll_wait(0, {}, 200, *1000*} after
having shutdown(SHUT_WR} its socket... but it concerns only peers which 
are remote from each others.





So, I had a look at the peer struct task 'expire' member handling
code, and I have just found a situation where pollers in relation with
peer tasks are often called with an expired timeout leading haproxy to
consume a lot of CPU resources. In fact this happens each time the
peer task has expired during a fraction of second.

It is easy to reproduce this issue with a sort of peer simulator ;):

strace -ttf socat TCP4-LISTEN:,reuseaddr,fork SYSTEM:"echo
200;sleep 10"

This peer must be started *before* the other remote haproxy process
with only peers as backends.

strace is here only to have an idea of the moment where the remote
haproxy peer has just connected.

The sleep command is here to have enough time to block (ctrl + s) our
peer simulator process after the haproxy peer has just connected.

So 

Re: HAProxy won't shut down

[Patrick Hemmer]

On 2017/5/29 08:22, Frederic Lecaille wrote:
>
> Hi Patrick,
>
> First thank you for this nice and helpful report.
>
> Would it be possible to have an output of this command the next time
> you reproduce such an issue please?
>
> echo "show sess" | socat stdio 

Unfortunately this would not be possible. When the issue occurs, the
haproxy process has stopped accepting connections on all sockets. If I
were to run this command, it would be sent to the new process, not the
one that won't shut down.

>
> I have only one question (see below).
>
> On 05/24/2017 10:40 AM, Willy Tarreau wrote:
>> Hi Patrick,
>>
>> On Tue, May 23, 2017 at 01:49:42PM -0400, Patrick Hemmer wrote:
>> (...)
>>> haproxy 28856 root1u IPv4  420797940  0t0   
>>> TCP 10.0.33.145:35754->10.0.33.147:1029 (CLOSE_WAIT)
>>> haproxy 28856 root2u IPv4  420266351  0t0   
>>> TCP 10.0.33.145:52898->10.0.33.147:1029 (CLOSE_WAIT)
>>> haproxy 28856 root3r  REG0,30
>>> 4026531956 net
>>> haproxy 28856 root4u IPv4  422150834  0t0   
>>> TCP 10.0.33.145:38874->10.0.33.147:1029 (CLOSE_WAIT)
>>
>> These ones are very interesting.
>
> These traces also seem interesting to me.
>
> # strace -p 28856
> Process 28856 attached
> epoll_wait(0, {}, 200, 319) = 0
> epoll_wait(0, {}, 200, 0)   = 0
> epoll_wait(0, {}, 200, 362) = 0
> epoll_wait(0, {}, 200, 0)   = 0
> epoll_wait(0, {}, 200, 114) = 0
> epoll_wait(0, {}, 200, 0)   = 0
> epoll_wait(0, {}, 200, 203) = 0
> epoll_wait(0, {}, 200, 0)   = 0
> epoll_wait(0, {}, 200, 331) = 0
> epoll_wait(0, {}, 200, 0)
>
>
> Were such "epoll_wait(0, 0, 200, 0)" calls infinitively displayed?
Yes

>
>
> In fact I am wondering if it is normal to have so much epoll_wait(0,
> {}, 200, 0) calls for a haproxy process which has shut down.
>
> I suspect they are in relation with peer tasks (obviously which has
> expired).
>
> If this is the case, and with configurations with only peer tasks,
> haproxy would definitively hang consuming a lot of CPU resources.
HAProxy was not consuming high CPU. Note that in every other call to
`epoll_wait`, the 4th value was >0. If every single timeout value were
0, then yes, it would spin consuming CPU.

>
> So, I had a look at the peer struct task 'expire' member handling
> code, and I have just found a situation where pollers in relation with
> peer tasks are often called with an expired timeout leading haproxy to
> consume a lot of CPU resources. In fact this happens each time the
> peer task has expired during a fraction of second.
>
> It is easy to reproduce this issue with a sort of peer simulator ;):
>
> strace -ttf socat TCP4-LISTEN:,reuseaddr,fork SYSTEM:"echo
> 200;sleep 10"
>
> This peer must be started *before* the other remote haproxy process
> with only peers as backends.
>
> strace is here only to have an idea of the moment where the remote
> haproxy peer has just connected.
>
> The sleep command is here to have enough time to block (ctrl + s) our
> peer simulator process after the haproxy peer has just connected.
>
> So this peer accepts any remote peer sessions sending "200" status
> messages (and that's all).
>
> A haproxy peer which connects to such a peer which does not reply to a
> synchronization request would endlessly consume high CPU ressources
> until you unblock (ctrl + q) the peer simulator process.
>
> *Unhappily, I do not see any relation between this bug and the
> "CLOSE_WAIT peer state issue" which prevents haproxy from correctly
> shutting down.*
>
> I have attached a patch to this mail which fixes this issue.
Again, we're not seeing high CPU usage in this specific case. We have
reported a completely different scenario where haproxy starts consuming
CPU doing `epoll_wait(x,x,x,0)`, but this is not that. Every time this
shutdown issue occurs, the process is not consuming CPU.
However it is possible the 2 issues might have the same root cause. I
will try out the patch and see what happens.

Thanks

-Patrick
>
> Regards,
>
> Fred.
>
>
>
>
>
>

[PATCH 9/9] MAJOR: systemd-wrapper: get rid of the wrapper

[William Lallemand]

The master worker mode obsoletes the systemd-wrapper, to ensure that
nobody uses it anymore, the code has been removed.
---
 Makefile   |  18 +--
 contrib/systemd/haproxy.service.in |   2 +-
 src/haproxy-systemd-wrapper.c  | 319 -
 3 files changed, 3 insertions(+), 336 deletions(-)
 delete mode 100644 src/haproxy-systemd-wrapper.c

diff --git a/Makefile b/Makefile
index 5b5ecca..1c90a3e 100644
--- a/Makefile
+++ b/Makefile
@@ -61,8 +61,7 @@
 #  by "haproxy -vv" in CFLAGS.
 #   SILENT_DEFINE may be used to specify other defines which will not be
 # reported by "haproxy -vv".
-#   EXTRA   is used to force building or not building some extra tools. By
-#   default on Linux 2.6+, it contains "haproxy-systemd-wrapper".
+#   EXTRA   is used to force building or not building some extra tools.
 #   DESTDIR is not set by default and is used for installation only.
 #   It might be useful to set DESTDIR if you want to install haproxy
 #   in a sandbox.
@@ -175,7 +174,7 @@ ADDLIB =
 DEFINE =
 SILENT_DEFINE =
 
- extra programs to build (eg: haproxy-systemd-wrapper)
+ extra programs to build
 # Force this to enable building extra programs or to disable them.
 # It's automatically appended depending on the targets.
 EXTRA =
@@ -266,7 +265,6 @@ ifeq ($(TARGET),linux26)
   USE_TPROXY  = implicit
   USE_LIBCRYPT= implicit
   USE_FUTEX   = implicit
-  EXTRA  += haproxy-systemd-wrapper
   USE_DL  = implicit
 else
 ifeq ($(TARGET),linux2628)
@@ -282,7 +280,6 @@ ifeq ($(TARGET),linux2628)
   USE_FUTEX   = implicit
   USE_CPU_AFFINITY= implicit
   ASSUME_SPLICE_WORKS= implicit
-  EXTRA  += haproxy-systemd-wrapper
   USE_DL  = implicit
 else
 ifeq ($(TARGET),solaris)
@@ -835,7 +832,6 @@ ifneq ($(TRACE),)
 OBJS += src/trace.o
 endif
 
-WRAPPER_OBJS = src/haproxy-systemd-wrapper.o
 
 # Not used right now
 LIB_EBTREE = $(EBTREE_DIR)/libebtree.a
@@ -850,9 +846,6 @@ DEP = $(INCLUDES) .build_opts
 haproxy: $(OPTIONS_OBJS) $(EBTREE_OBJS) $(OBJS)
$(LD) $(LDFLAGS) -o $@ $^ $(LDOPTS)
 
-haproxy-systemd-wrapper: $(WRAPPER_OBJS)
-   $(LD) $(LDFLAGS) -o $@ $^ $(LDOPTS)
-
 $(LIB_EBTREE): $(EBTREE_OBJS)
$(AR) rv $@ $^
 
@@ -875,11 +868,6 @@ src/haproxy.o: src/haproxy.c $(DEP)
  -DBUILD_OPTIONS='"$(strip $(BUILD_OPTIONS))"' \
   -c -o $@ $<
 
-src/haproxy-systemd-wrapper.o: src/haproxy-systemd-wrapper.c $(DEP)
-   $(CC) $(COPTS) \
- -DSBINDIR='"$(strip $(SBINDIR))"' \
-  -c -o $@ $<
-
 src/dlmalloc.o: $(DLMALLOC_SRC) $(DEP)
$(CC) $(COPTS) -DDEFAULT_MMAP_THRESHOLD=$(DLMALLOC_THRES) -c -o $@ $<
 
@@ -915,14 +903,12 @@ uninstall:
done
-rmdir "$(DESTDIR)$(DOCDIR)"
rm -f "$(DESTDIR)$(SBINDIR)"/haproxy
-   rm -f "$(DESTDIR)$(SBINDIR)"/haproxy-systemd-wrapper
 
 clean:
rm -f *.[oas] src/*.[oas] ebtree/*.[oas] haproxy test .build_opts 
.build_opts.new
for dir in . src include/* doc ebtree; do rm -f $$dir/*~ $$dir/*.rej 
$$dir/core; done
rm -f haproxy-$(VERSION).tar.gz haproxy-$(VERSION)$(SUBVERS).tar.gz
rm -f haproxy-$(VERSION) haproxy-$(VERSION)$(SUBVERS) nohup.out gmon.out
-   rm -f haproxy-systemd-wrapper
 
 tags:
find src include \( -name '*.c' -o -name '*.h' \) -print0 | \
diff --git a/contrib/systemd/haproxy.service.in 
b/contrib/systemd/haproxy.service.in
index ca38d70..81b4951 100644
--- a/contrib/systemd/haproxy.service.in
+++ b/contrib/systemd/haproxy.service.in
@@ -7,7 +7,7 @@ After=network.target
 # socket if you want seamless reloads.
 Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
 ExecStartPre=@SBINDIR@/haproxy -f $CONFIG -c -q
-ExecStart=@SBINDIR@/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE
+ExecStart=@SBINDIR@/haproxy -W -f $CONFIG -p $PIDFILE
 ExecReload=@SBINDIR@/haproxy -f $CONFIG -c -q
 ExecReload=/bin/kill -USR2 $MAINPID
 KillMode=mixed
diff --git a/src/haproxy-systemd-wrapper.c b/src/haproxy-systemd-wrapper.c
deleted file mode 100644
index 457f5bd..000
--- a/src/haproxy-systemd-wrapper.c
+++ /dev/null
@@ -1,319 +0,0 @@
-/*
- * Wrapper to make haproxy systemd-compliant.
- *
- * Copyright 2013 Marc-Antoine Perennou 
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; either version
- * 2 of the License, or (at your option) any later version.
- *
- */
-
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-#include 
-
-#define REEXEC_FLAG "HAPROXY_SYSTEMD_REEXEC"
-#define SD_DEBUG "<7>"
-#define SD_NOTICE "<5>"
-
-static volatile sig_atomic_t caught_signal;
-
-static char *pid_file = "/run/haproxy.pid";
-static int wrapper_argc;
-static char **wrapper_argv;
-
-static void setup_signal_handler();
-static void 

[PATCH 7/9] DOC: add documentation for the master-worker mode

[William Lallemand]

---
 doc/configuration.txt | 16 
 doc/management.txt| 15 +--
 2 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index ad7d3a8..5fa49d3 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -803,6 +803,20 @@ lua-load 
   This global directive loads and executes a Lua file. This directive can be
   used multiple times.
 
+master-worker [exit-on-failure]
+  Master-worker mode. It is equivalent to the command line "-W" argument.
+  This mode will launch a "master" which will monitor the "workers". Using
+  this mode, you can reload HAProxy directly by sending a SIGUSR2 signal to
+  the master.  The master-worker mode is compatible either with the foreground
+  or daemon mode. It is recommended to use this mode with multiprocess and
+  systemd.
+  The "exit-on-failure" option allows the master to kill every workers and
+  exit when one of the current workers died. It is convenient to combine this
+  option with Restart=on-failure in a systemd unit file in order to relaunch
+  the whole process.
+
+  See alors "-W" in the management guide.
+
 nbproc 
   Creates  processes when going daemon. This requires the "daemon"
   mode. By default, only one process is created, which is the recommended mode
@@ -10431,6 +10445,8 @@ defer-accept
 expose-fd listeners
   This option is only usable with the stats socket. It gives your stats socket
   the capability to pass listeners FD to another HAProxy process.
+  During a reload with the master-worker mode, the process is automatically
+  reexecuted adding -x and one of the stats socket with this option.
   See alors "-x" in the management guide.
 
 force-sslv3
diff --git a/doc/management.txt b/doc/management.txt
index 64d6a2d..df091bb 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -160,8 +160,6 @@ list of options is :
 configuration. It is recommended to always force it in any init script so
 that a faulty configuration doesn't prevent the system from booting.
 
-  -Ds : work in systemd mode. Only used by the systemd wrapper.
-
   -L  : change the local peer name to , which defaults to the local
 hostname. This is used only with peers replication.
 
@@ -171,6 +169,13 @@ list of options is :
   -V : enable verbose mode (disables quiet mode). Reverts the effect of "-q" or
 "quiet".
 
+  -W : master-worker mode. It is equivalent to the "master-worker" keyword in
+the "global" section of the configuration. This mode will launch a "master"
+which will monitor the "workers". Using this mode, you can reload HAProxy
+directly by sending a SIGUSR2 signal to the master.  The master-worker mode
+is compatible either with the foreground or daemon mode.  It is
+recommended to use this mode with multiprocess and systemd.
+
   -c : only performs a check of the configuration files and exits before trying
 to bind. The exit status is zero if everything is OK, or non-zero if an
 error is encountered.
@@ -419,6 +424,12 @@ reload or restart, so that they are sent at the latest 
possible moment and only
 if absolutely required. This is what is performed by the "-st" (hard) and "-sf"
 (graceful) options respectively.
 
+In master-worker mode, it is not needed to start a new haproxy process in
+order to reload the configuration. The master process reacts to the SIGUSR2
+signal by reexecuting itself with the -sf parameter followed by the PIDs of
+the workers. The master will then parse the configuration file and fork new
+workers.
+
 To understand better how these signals are used, it is important to understand
 the whole restart mechanism.
 
-- 
2.10.2

[PATCH 2/9] MEDIUM: mworker: handle reload and signals

[William Lallemand]

The master-worker will reload itself on SIGUSR2/SIGHUP

It's inherited from the systemd wrapper, when the SIGUSR2 signal is
received, the master process will reexecute itself with the -sf flag
followed by the PIDs of the children.

In the systemd wrapper, the children were using a pipe to notify when
the config has been parsed and when the new process is ready. The goal
was to ensure that the process couldn't reload during the parsing of the
configuration, before signals were send to old process.

With the new mworker model, the master parses the configuration and is
aware of all the children. We don't need a pipe, but we need to block
those signals before the end of a reload, to ensure that the process
won't be killed during a reload.

The SIGUSR1 signal is forwarded to the children to soft-stop HAProxy.

The SIGTERM and SIGINT signals are forwarded to the children in order to
terminate them.
---
 src/haproxy.c | 304 --
 src/signal.c  |   4 +
 2 files changed, 279 insertions(+), 29 deletions(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index 0e8511b..2454dfd 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -158,6 +158,8 @@ struct global global = {
 
 /*/
 
+#define REEXEC_FLAG "HAPROXY_MWORKER_REEXEC"
+
 int stopping;  /* non zero means stopping in progress */
 int killed;/* non zero means a hard-stop is triggered */
 int jobs = 0;   /* number of active jobs (conns, listeners, active tasks, ...) 
*/
@@ -194,6 +196,11 @@ char localpeer[MAX_HOSTNAME_LEN];
  */
 int shut_your_big_mouth_gcc_int = 0;
 
+int *children = NULL; /* store PIDs of children in master workers mode */
+
+static volatile sig_atomic_t caught_signal = 0;
+static char **next_argv = NULL;
+
 /* list of the temporarily limited listeners because of lack of resource */
 struct list global_listener_queue = LIST_HEAD_INIT(global_listener_queue);
 struct task *global_listener_queue_task;
@@ -389,6 +396,232 @@ static void usage(char *name)
 /*   more specific functions   ***/
 /*/
 
+/* sends the signal  to all pids found in . Returns the number of
+ * pids the signal was correctly delivered to.
+ */
+static int tell_old_pids(int sig)
+{
+   int p;
+   int ret = 0;
+   for (p = 0; p < nb_oldpids; p++)
+   if (kill(oldpids[p], sig) == 0)
+   ret++;
+   return ret;
+}
+
+/* return 1 if a pid is a current child otherwise 0 */
+
+int current_child(int pid)
+{
+   int i;
+
+   for (i = 0; i < global.nbproc; i++) {
+   if (children[i] == pid)
+   return 1;
+   }
+   return 0;
+}
+
+static void mworker_signalhandler(int signum)
+{
+   caught_signal = signum;
+}
+
+static void mworker_register_signals()
+{
+   struct sigaction sa;
+   /* Here we are not using the haproxy async way
+   for signals because it does not exists in
+   the master */
+   memset(, 0, sizeof(struct sigaction));
+   sa.sa_handler = _signalhandler;
+   sigaction(SIGHUP, , NULL);
+   sigaction(SIGUSR1, , NULL);
+   sigaction(SIGUSR2, , NULL);
+   sigaction(SIGINT, , NULL);
+   sigaction(SIGTERM, , NULL);
+}
+
+static void mworker_block_signals()
+{
+   sigset_t set;
+
+   sigemptyset();
+   sigaddset(, SIGUSR1);
+   sigaddset(, SIGUSR2);
+   sigaddset(, SIGHUP);
+   sigaddset(, SIGINT);
+   sigaddset(, SIGTERM);
+   sigprocmask(SIG_SETMASK, , NULL);
+}
+
+static void mworker_unblock_signals()
+{
+   sigset_t set;
+
+   sigemptyset();
+   sigaddset(, SIGUSR1);
+   sigaddset(, SIGUSR2);
+   sigaddset(, SIGHUP);
+   sigaddset(, SIGINT);
+   sigaddset(, SIGTERM);
+   sigprocmask(SIG_UNBLOCK, , NULL);
+}
+
+static void mworker_unregister_signals()
+{
+   signal(SIGINT, SIG_DFL);
+   signal(SIGTERM, SIG_DFL);
+   signal(SIGHUP,  SIG_IGN);
+   signal(SIGUSR1, SIG_IGN);
+   signal(SIGUSR2, SIG_IGN);
+}
+
+/*
+ * Send signal to every known children.
+ */
+
+static void mworker_kill(int sig)
+{
+   int i;
+
+   tell_old_pids(sig);
+   if (children) {
+   for (i = 0; i < global.nbproc; i++)
+   kill(children[i], sig);
+   }
+}
+
+/*
+ * remove a pid forom the olpid array and decrease nb_oldpids
+ * return 1 pid was found otherwise return 0
+ */
+
+int delete_oldpid(int pid)
+{
+   int i;
+
+   for (i = 0; i < nb_oldpids; i++) {
+   if (oldpids[i] == pid) {
+   oldpids[i] = oldpids[nb_oldpids - 1];
+   oldpids[nb_oldpids - 1] = 0;
+   nb_oldpids--;
+   return 1;
+   }
+   }
+   return 0;
+}
+
+/*
+ * When called, this function reexec haproxy with -sf followed by current
+ * 

[PATCH 6/9] MEDIUM: mworker: workers exit when the master leaves

[William Lallemand]

This patch ensure that the children will exit when the master quits,
even if the master didn't send any signal.

The master and the workers are connected through a pipe, when the pipe
closes the children leave.
---
 src/haproxy.c | 55 +++
 1 file changed, 55 insertions(+)

diff --git a/src/haproxy.c b/src/haproxy.c
index d23bf3a..01681d4 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -205,6 +205,8 @@ int *children = NULL; /* store PIDs of children in master 
workers mode */
 static volatile sig_atomic_t caught_signal = 0;
 static char **next_argv = NULL;
 
+int mworker_pipe[2];
+
 /* list of the temporarily limited listeners because of lack of resource */
 struct list global_listener_queue = LIST_HEAD_INIT(global_listener_queue);
 struct task *global_listener_queue_task;
@@ -2227,6 +2229,34 @@ static struct task *manage_global_listener_queue(struct 
task *t)
return t;
 }
 
+void mworker_pipe_handler(int fd)
+{
+   char c;
+
+   if (read(fd, , 1) > -1) {
+   fd_delete(fd);
+   deinit();
+   exit(EXIT_FAILURE);
+   } else {
+   /* should never happened */
+   fd_delete(fd);
+   }
+
+   return;
+}
+
+void mworker_pipe_register(int pipefd[2])
+{
+   close(mworker_pipe[1]); /* close the write end of the master 
pipe in the children */
+
+   fcntl(mworker_pipe[0], F_SETFL, O_NONBLOCK);
+   fdtab[mworker_pipe[0]].owner = mworker_pipe;
+   fdtab[mworker_pipe[0]].iocb = mworker_pipe_handler;
+   fd_insert(mworker_pipe[0]);
+   fd_want_recv(mworker_pipe[0]);
+   }
+
+
 int main(int argc, char **argv)
 {
int err, retry;
@@ -2478,6 +2508,28 @@ int main(int argc, char **argv)
exit(0);
}
 
+   if (global.mode & MODE_MWORKER) {
+   if ((getenv(REEXEC_FLAG) == NULL)) {
+   char *msg = NULL;
+   /* master pipe to ensure the master is still 
alive  */
+   ret = pipe(mworker_pipe);
+   if (ret < 0) {
+   Warning("[%s.main()] Cannot create 
master pipe.\n", argv[0]);
+   } else {
+   memprintf(, "%d", mworker_pipe[0]);
+   setenv("MWORKER_PIPE_RD", msg, 1);
+   memprintf(, "%d", mworker_pipe[1]);
+   setenv("MWORKER_PIPE_WR", msg, 1);
+   free(msg);
+   }
+   } else {
+   mworker_pipe[0] = 
atol(getenv("MWORKER_PIPE_RD"));
+   mworker_pipe[1] = 
atol(getenv("MWORKER_PIPE_WR"));
+   if (mworker_pipe[0] <= 0 || mworker_pipe[1] <= 
0) {
+   Warning("[%s.main()] Cannot get master 
pipe FDs.\n", argv[0]);
+   }
+   }
+   }
 
/* the father launches the required number of processes */
for (proc = 0; proc < global.nbproc; proc++) {
@@ -2637,6 +2689,9 @@ int main(int argc, char **argv)
if (!dns_init_resolvers(1))
exit(1);
 
+   if (global.mode & MODE_MWORKER)
+   mworker_pipe_register(mworker_pipe);
+
protocol_enable_all();
/*
 * That's it : the central polling loop. Run until we stop.
-- 
2.10.2

[PATCH 8/9] MEDIUM: systemd: Type=forking in unit file

[William Lallemand]

Adding Type=forking in the unit file ensure better monitoring from
systemd. During a systemctl start the tool is able to return an error if
it didn't work with this option.
---
 contrib/systemd/haproxy.service.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/contrib/systemd/haproxy.service.in 
b/contrib/systemd/haproxy.service.in
index 05bb716..ca38d70 100644
--- a/contrib/systemd/haproxy.service.in
+++ b/contrib/systemd/haproxy.service.in
@@ -12,6 +12,7 @@ ExecReload=@SBINDIR@/haproxy -f $CONFIG -c -q
 ExecReload=/bin/kill -USR2 $MAINPID
 KillMode=mixed
 Restart=always
+Type=forking
 
 [Install]
 WantedBy=multi-user.target
-- 
2.10.2

[PATCH 3/9] MEDIUM: mworker: wait mode on reload failure

[William Lallemand]

In Master Worker mode, when the reloading of the configuration fail,
the process is exiting leaving the children without their father.

To handle this, we register an exit function with atexit(3), which is
reexecuting the binary in a special mode. This particular mode of
HAProxy don't reload the configuration, it only loops on wait().
---
 src/haproxy.c | 37 -
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index 2454dfd..61114b6 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -183,6 +183,8 @@ struct chunk trash = { };
  */
 char *swap_buffer = NULL;
 
+int atexit_flag = 0;
+
 int nb_oldpids = 0;
 const int zero = 0;
 const int one = 1;
@@ -594,6 +596,7 @@ static void mworker_wait()
 
if (exitpid == -1 && errno == ECHILD) {
Warning("All workers are left. Leaving... (%d)\n", 
status);
+   atexit_flag = 0;
exit(status); /* parent must leave using the latest 
status code known */
}
 
@@ -621,6 +624,22 @@ static void mworker_wait()
 }
 
 
+/*
+ * Reexec the process in failure mode, instead of exiting
+ */
+void reexec_on_failure()
+{
+   if (!atexit_flag)
+   return;
+
+   setenv("WAIT_ONLY", "1", 1);
+
+   Warning("Reexecuting Master process in waitpid mode\n");
+   mworker_reload();
+
+   Warning("Failed to reexecute the master processs\n");
+}
+
 
 /*
  * upon SIGUSR1, let's have a soft stop. Note that soft_stop() broadcasts
@@ -1272,6 +1291,18 @@ static void init(int argc, char **argv)
(arg_mode & (MODE_DAEMON | MODE_MWORKER | MODE_FOREGROUND | 
MODE_VERBOSE
 | MODE_QUIET | MODE_CHECK | MODE_DEBUG));
 
+   /* Master workers wait mode */
+   if ((global.mode & MODE_MWORKER) && (getenv("WAIT_ONLY") != NULL)) {
+
+   unsetenv("WAIT_ONLY");
+   mworker_wait();
+   }
+
+   if ((global.mode & MODE_MWORKER) && (getenv(REEXEC_FLAG) != NULL)) {
+   atexit_flag = 1;
+   atexit(reexec_on_failure);
+   }
+
if (change_dir && chdir(change_dir) < 0) {
Alert("Could not change to directory %s : %s\n", change_dir, 
strerror(errno));
exit(1);
@@ -2425,9 +2456,13 @@ int main(int argc, char **argv)
 #ifndef OPENSSL_NO_DH
ssl_free_dh();
 #endif
-   exit(0); /* parent must leave */
+   /* should never get there */
+   exit(EXIT_FAILURE);
}
 
+   /* child must never use the atexit function */
+   atexit_flag = 0;
+
/* Must chroot and setgid/setuid in the children */
/* chroot if needed */
if (global.chroot != NULL) {
-- 
2.10.2

[PATCH 5/9] MEDIUM: mworker: exit-on-failure option

[William Lallemand]

This option exits every workers when one of the current workers die.

It allows you to monitor the master process in order to relaunch
everything on a failure.

For example it can be used with systemd and Restart=on-failure in a spec
file.
---
 include/types/global.h |  1 +
 src/cfgparse.c | 11 ++-
 src/haproxy.c  |  5 +
 3 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/include/types/global.h b/include/types/global.h
index ee8e95e..cce3de7 100644
--- a/include/types/global.h
+++ b/include/types/global.h
@@ -63,6 +63,7 @@
 #define GTUNE_RESOLVE_DONTFAIL   (1<<7)
 
 #define GTUNE_SOCKET_TRANSFER   (1<<8)
+#define GTUNE_EXIT_ONFAILURE (1<<9)
 
 /* Access level for a stats socket */
 #define ACCESS_LVL_NONE 0
diff --git a/src/cfgparse.c b/src/cfgparse.c
index bdf55a8..76a4f31 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -625,8 +625,17 @@ int cfg_parse_global(const char *file, int linenum, char 
**args, int kwm)
global.mode |= MODE_DAEMON;
}
else if (!strcmp(args[0], "master-worker")) {
-   if (alertif_too_many_args(0, file, linenum, args, _code))
+   if (alertif_too_many_args(1, file, linenum, args, _code))
goto out;
+   if (*args[1]) {
+   if (!strcmp(args[1], "exit-on-failure")) {
+   global.tune.options |= GTUNE_EXIT_ONFAILURE;
+   } else {
+   Alert("parsing [%s:%d] : '%s' only supports 
'exit-on-failure' option.\n", file, linenum, args[0]);
+   err_code |= ERR_ALERT | ERR_FATAL;
+   goto out;
+   }
+   }
global.mode |= MODE_MWORKER;
}
else if (!strcmp(args[0], "debug")) {
diff --git a/src/haproxy.c b/src/haproxy.c
index 57a5db6..d23bf3a 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -673,6 +673,11 @@ static void mworker_wait()
/* check if exited child was in the current children 
list */
if (current_child(exitpid)) {
Alert("Current worker %d left with exit code 
%d\n", exitpid, status);
+   if (status != 0 && status != 130 && status != 
143
+   && global.tune.options & 
GTUNE_EXIT_ONFAILURE) {
+   Alert("exit-on-failure: killing every 
workers with SIGTERM\n");
+   mworker_kill(SIGTERM);
+   }
} else {
Warning("Former worker %d left with exit code 
%d\n", exitpid, status);
delete_oldpid(exitpid);
-- 
2.10.2

[PATCH 4/9] MEDIUM: mworker: try to guess the next stats socket to use with -x

[William Lallemand]

In master worker mode, you can't specify the stats socket where you get
your listeners FDs on a reload, because the command line of the re-exec
is launched by the master.

To solve the problem, when -x is found on the command line, its
parameter is rewritten on a reexec with the first stats socket with the
capability to send sockets. It tries to reuse the original parameter if
it has this capability.
---
 src/haproxy.c | 74 +++
 1 file changed, 69 insertions(+), 5 deletions(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index 61114b6..57a5db6 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -175,6 +175,8 @@ static int oldpids_sig; /* use USR1 or TERM */
 /* Path to the unix socket we use to retrieve listener sockets from the old 
process */
 static const char *old_unixsocket;
 
+static char *cur_unixsocket = NULL;
+
 /* this is used to drain data, and as a temporary buffer for sprintf()... */
 struct chunk trash = { };
 
@@ -514,6 +516,42 @@ int delete_oldpid(int pid)
return 0;
 }
 
+
+static void get_cur_unixsocket()
+{
+   /* if -x was used, try to update the stat socket if not available 
anymore */
+   if (global.stats_fe) {
+   struct bind_conf *bind_conf;
+
+   /* pass through all stats socket */
+   list_for_each_entry(bind_conf, _fe->conf.bind, 
by_fe) {
+   struct listener *l;
+
+   list_for_each_entry(l, _conf->listeners, by_bind) {
+
+   if (l->addr.ss_family == AF_UNIX &&
+   (bind_conf->level & ACCESS_FD_LISTENERS)) {
+   const struct sockaddr_un *un;
+
+   un = (struct sockaddr_un *)>addr;
+   /* priority to old_unixsocket */
+   if (!cur_unixsocket) {
+   cur_unixsocket = 
strdup(un->sun_path);
+   } else {
+   if (old_unixsocket && 
!strcmp(un->sun_path, old_unixsocket)) {
+   free(cur_unixsocket);
+   cur_unixsocket = 
strdup(old_unixsocket);
+   return;
+   }
+   }
+   }
+   }
+   }
+   }
+   if (!cur_unixsocket && old_unixsocket)
+   cur_unixsocket = strdup(old_unixsocket);
+}
+
 /*
  * When called, this function reexec haproxy with -sf followed by current
  * children PIDs and possibily old children PIDs if they didn't leave yet.
@@ -532,8 +570,8 @@ static void mworker_reload()
while (next_argv[next_argc])
next_argc++;
 
-   /* 1 for haproxy -sf */
-   next_argv = realloc(next_argv, (next_argc + 1 + global.nbproc + 
nb_oldpids + 1) * sizeof(char *));
+   /* 1 for haproxy -sf, 2 for -x /socket */
+   next_argv = realloc(next_argv, (next_argc + 1 + 2 + global.nbproc + 
nb_oldpids + 1) * sizeof(char *));
if (next_argv == NULL)
goto alloc_error;
 
@@ -557,6 +595,26 @@ static void mworker_reload()
msg = NULL;
}
next_argv[next_argc] = NULL;
+
+   /* if -x was used, try to update the stat socket if not available 
anymore */
+   if (cur_unixsocket) {
+
+   if (old_unixsocket) {
+
+   /* look for -x  */
+   for (j = 0; next_argv[j]; j++) {
+   if (!strcmp(next_argv[j], "-x"))
+   next_argv[j + 1] = (char 
*)cur_unixsocket;
+   }
+   } else {
+   /* if -x is not specified but we know the socket, add 
-x with it */
+   next_argv[next_argc++] = "-x";
+   next_argv[next_argc++] = (char *)cur_unixsocket;
+   next_argv[next_argc++] = NULL;
+
+   }
+   }
+
deinit(); /* we don't want to leak FD there */
Warning("Reexecuting Master process\n");
execv(next_argv[0], next_argv);
@@ -2218,11 +2276,16 @@ int main(int argc, char **argv)
}
 
if (old_unixsocket) {
-   if (get_old_sockets(old_unixsocket) != 0) {
-   Alert("Failed to get the sockets from the old 
process!\n");
-   exit(1);
+   if (strcmp("/dev/null", old_unixsocket) != 0) {
+   if (get_old_sockets(old_unixsocket) != 0) {
+   Alert("Failed to get the sockets from the old 
process!\n");
+   if (!(global.mode & MODE_MWORKER))
+ 

[PATCH 1/9] MEDIUM: mworker: replace systemd mode by master worker mode

[William Lallemand]

This commit remove the -Ds systemd mode in HAProxy in order to replace
it by a more generic master worker system. It aims to replace entirely
the systemd wrapper in the near future.

The master worker mode implements a new way of managing HAProxy
processes. The master is in charge of parsing the configuration
file and is responsible for spawning child processes.

The master worker mode can be invoked by using the -W flag.  It can be
used either in background mode (-D) or foreground mode. When used in
background mode, the master will fork to daemonize.

In master worker background mode, chroot, setuid and setgid are done in
each child rather than in the master process, because the master process
will still need access to filesystem to reload the configuration.
---
 include/types/global.h |   2 +-
 src/cfgparse.c |   5 ++
 src/haproxy.c  | 136 +
 src/listener.c |   4 +-
 4 files changed, 99 insertions(+), 48 deletions(-)

diff --git a/include/types/global.h b/include/types/global.h
index aeb82ea..ee8e95e 100644
--- a/include/types/global.h
+++ b/include/types/global.h
@@ -44,7 +44,7 @@
 #defineMODE_VERBOSE0x10
 #defineMODE_STARTING   0x20
 #defineMODE_FOREGROUND 0x40
-#defineMODE_SYSTEMD0x80
+#defineMODE_MWORKER0x80/* Master Worker */
 
 /* list of last checks to perform, depending on config options */
 #define LSTCHK_CAP_BIND0x0001  /* check that we can bind to 
any port */
diff --git a/src/cfgparse.c b/src/cfgparse.c
index 4c0e2d4..bdf55a8 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -624,6 +624,11 @@ int cfg_parse_global(const char *file, int linenum, char 
**args, int kwm)
goto out;
global.mode |= MODE_DAEMON;
}
+   else if (!strcmp(args[0], "master-worker")) {
+   if (alertif_too_many_args(0, file, linenum, args, _code))
+   goto out;
+   global.mode |= MODE_MWORKER;
+   }
else if (!strcmp(args[0], "debug")) {
if (alertif_too_many_args(0, file, linenum, args, _code))
goto out;
diff --git a/src/haproxy.c b/src/haproxy.c
index 6ecae25..0e8511b 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -348,6 +348,7 @@ static void usage(char *name)
"-dM[] poisons memory with  (defaults to 
0x50)\n"
"-V enters verbose mode (disables quiet mode)\n"
"-D goes daemon ; -C changes to  before loading 
files.\n"
+   "-W master-worker mode.\n"
"-q quiet mode : don't display messages\n"
"-c check mode : only check config files and exit\n"
"-n sets the maximum total # of connections (%d)\n"
@@ -921,11 +922,10 @@ static void init(int argc, char **argv)
arg_mode |= MODE_DEBUG;
else if (*flag == 'c')
arg_mode |= MODE_CHECK;
-   else if (*flag == 'D') {
+   else if (*flag == 'D')
arg_mode |= MODE_DAEMON;
-   if (flag[1] == 's')  /* -Ds */
-   arg_mode |= MODE_SYSTEMD;
-   }
+   else if (*flag == 'W')
+   arg_mode |= MODE_MWORKER;
else if (*flag == 'q')
arg_mode |= MODE_QUIET;
else if (*flag == 'x') {
@@ -1001,7 +1001,7 @@ static void init(int argc, char **argv)
}
 
global.mode = MODE_STARTING | /* during startup, we want most of the 
alerts */
-   (arg_mode & (MODE_DAEMON | MODE_SYSTEMD | MODE_FOREGROUND | 
MODE_VERBOSE
+   (arg_mode & (MODE_DAEMON | MODE_MWORKER | MODE_FOREGROUND | 
MODE_VERBOSE
 | MODE_QUIET | MODE_CHECK | MODE_DEBUG));
 
if (change_dir && chdir(change_dir) < 0) {
@@ -1330,24 +1330,24 @@ static void init(int argc, char **argv)
 
if (arg_mode & (MODE_DEBUG | MODE_FOREGROUND)) {
/* command line debug mode inhibits configuration mode */
-   global.mode &= ~(MODE_DAEMON | MODE_SYSTEMD | MODE_QUIET);
+   global.mode &= ~(MODE_DAEMON | MODE_QUIET);
global.mode |= (arg_mode & (MODE_DEBUG | MODE_FOREGROUND));
}
 
-   if (arg_mode & (MODE_DAEMON | MODE_SYSTEMD)) {
+   if (arg_mode & MODE_DAEMON) {
/* command line daemon mode inhibits foreground and debug modes 
mode */
global.mode &= ~(MODE_DEBUG | MODE_FOREGROUND);
-   global.mode |= (arg_mode & (MODE_DAEMON | MODE_SYSTEMD));
+   global.mode |= arg_mode & MODE_DAEMON;
}
 
global.mode |= (arg_mode & (MODE_QUIET | 

Replace the systemd-wrapper by the master worker mode

[William Lallemand]

The master worker mode replaces the systemd wrapper, it does not need a
separated binary anymore, everything is builtin.

This mode will launch a "master" which will monitor the "workers". Using this
mode, you can reload HAProxy directly by sending a SIGUSR2 signal to the
master. The master-worker mode is compatible either with the foreground or
daemon mode. It is recommended to use this mode with multiprocess and/or 
systemd.

The master take advantage of the seamless reload feature (-x on the command
line), it will try to add automatically this option during a reload when a
stats socket exposes its listeners. The seamless reload is transparent.

Re: Mod Defender (a NAXSI clone) integration patch

[Aleksandar Lazic]

Hi Dragan Dosen.

Dragan Dosen  have written on Mon, 29 May 2017
15:56:06 +0200:

> Hi Aleksandar,
> 
> Thank you for your comments and feedback.
> 
> 
> On 29.5.2017. 14:58, Aleksandar Lazic wrote:
> > 
> > Is there a comparison table what's the difference between mod
> > defender and mod security?
> >   
> 
> Not really, as far as I know. But existing comparisons of ModSecurity
> and Naxsi functionality can be found.

I added this links to my README in the repo

https://www.haproxy.com/blog/high-performance-waf-platform-with-naxsi-and-haproxy/
thanks Baptiste ;-)

http://blog.memze.ro/?p=39
https://geekflare.com/open-source-web-application-firewall/

> > 
> > Some feedback.
> > 
> > I have tried to build it on centos latest and got the following
> > error.
> > 
> > ###
> > + make MOD_DEFENDER_SRC=/usr/src/mod_defender
> > APACHE2_INC=/usr/include/httpd APR_INC=/usr/include/apr-1

[snipp]

> That's probably because you're using gcc/g++ version 4.8.2 and >= 4.9
> is required for Mod Defender, since it uses std::regex.

Yep that was it.

[root@188f1bea6edb /]# /usr/local/bin/defender -f naxsi_core.rules
1496068005.222163 [00] Defender active on server 188f1bea6edb: 42
MainRules loaded 1496068005.87 [00] Defender scanner disabled for
loc / 1496068010.228776 [01] 0 clients connected
1496068010.241821 [02] 0 clients connected

How about to add a -v or --version to the defender, I thought this also
for the modsecurity contrib but forgotten to write.

Does it makes sense to combine mod_defender and mod_security?

> Best regards,
> Dragan Dosen

Cheers
Aleks

Re: Mod Defender (a NAXSI clone) integration patch

[Dragan Dosen]

Hi Aleksandar,

Thank you for your comments and feedback.


On 29.5.2017. 14:58, Aleksandar Lazic wrote:
> 
> Is there a comparison table what's the difference between mod defender
> and mod security?
> 

Not really, as far as I know. But existing comparisons of ModSecurity
and Naxsi functionality can be found.


> 
> Some feedback.
> 
> I have tried to build it on centos latest and got the following error.
> 
> ###
> + make MOD_DEFENDER_SRC=/usr/src/mod_defender
> APACHE2_INC=/usr/include/httpd APR_INC=/usr/include/apr-1
> gcc -g -Wall -pthread -I../../include -I../../ebtree
> -I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> spoa.o spoa.c
> gcc -g -Wall -pthread -I../../include -I../../ebtree
> -I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> defender.o defender.c
> gcc -g -Wall -pthread -I../../include -I../../ebtree
> -I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/deps/libinjection/libinjection_sqli.o 
> /usr/src/mod_defender/deps/libinjection/libinjection_sqli.c
> gcc -g -Wall -pthread -I../../include -I../../ebtree
> -I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/deps/libinjection/libinjection_xss.o 
> /usr/src/mod_defender/deps/libinjection/libinjection_xss.c
> gcc -g -Wall -pthread -I../../include -I../../ebtree
> -I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/deps/libinjection/libinjection_html5.o 
> /usr/src/mod_defender/deps/libinjection/libinjection_html5.c
> g++ -g -std=gnu++11 -I/usr/src/mod_defender
> -I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/JsonValidator.o /usr/src/mod_defender/JsonValidator.cpp
> g++ -g -std=gnu++11 -I/usr/src/mod_defender
> -I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/RuntimeScanner.o 
> /usr/src/mod_defender/RuntimeScanner.cpp
> g++ -g -std=gnu++11 -I/usr/src/mod_defender
> -I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/mod_defender.o /usr/src/mod_defender/mod_defender.cpp
> g++ -g -std=gnu++11 -I/usr/src/mod_defender
> -I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/Util.o /usr/src/mod_defender/Util.cpp
> g++ -g -std=gnu++11 -I/usr/src/mod_defender
> -I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
> /usr/src/mod_defender/RuleParser.o /usr/src/mod_defender/RuleParser.cpp
> gcc -o defender standalone.o spoa.o
> defender.o /usr/src/mod_defender/deps/libinjection/libinjection_sqli.o 
> /usr/src/mod_defender/deps/libinjection/libinjection_xss.o 
> /usr/src/mod_defender/deps/libinjection/libinjection_html5.o 
> /usr/src/mod_defender/JsonValidator.o /usr/src/mod_defender/RuntimeScanner.o 
> /usr/src/mod_defender/mod_defender.o /usr/src/mod_defender/Util.o 
> /usr/src/mod_defender/RuleParser.o -lpthread  -levent -levent_pthreads 
> -lapr-1 -laprutil-1 -lstdc++
> /usr/src/mod_defender/RuntimeScanner.o: In function
> `RuntimeScanner::processRuleBuffer(std::string const&, http_rule_t const&, 
> unsigned long&)':
> /usr/src/mod_defender/RuntimeScanner.cpp:146: undefined reference to
> `std::regex_iterator<__gnu_cxx::__normal_iterator, 
> char, std::regex_traits >::regex_iterator()'
> /usr/src/mod_defender/RuntimeScanner.cpp:146: undefined reference to
> `std::regex_iterator<__gnu_cxx::__normal_iterator, 
> char, std::regex_traits 
> >::regex_iterator(__gnu_cxx::__normal_iterator, 
> __gnu_cxx::__normal_iterator, 
> std::basic_regex const&, std::bitset<11ul>)'
> /usr/src/mod_defender/RuntimeScanner.o: In function
> `std::iterator_traits, char, std::regex_traits > >::difference_type 
> std::distance, char, std::regex_traits > 
> >(std::regex_iterator<__gnu_cxx::__normal_iterator, 
> char, std::regex_traits >, 
> std::regex_iterator<__gnu_cxx::__normal_iterator, 
> char, std::regex_traits >)':
> /usr/include/c++/4.8.2/bits/stl_iterator_base_funcs.h:118: undefined
> reference to `std::regex_iterator<__gnu_cxx::__normal_iterator std::string>, char, std::regex_traits 
> >::regex_iterator(std::regex_iterator<__gnu_cxx::__normal_iterator const*, std::string>, char, std::regex_traits > const&)'
> /usr/include/c++/4.8.2/bits/stl_iterator_base_funcs.h:118: undefined
> reference to `std::regex_iterator<__gnu_cxx::__normal_iterator std::string>, char, std::regex_traits 
> >::regex_iterator(std::regex_iterator<__gnu_cxx::__normal_iterator const*, std::string>, char, std::regex_traits > const&)'
> /usr/src/mod_defender/RuntimeScanner.o: In function
> `std::iterator_traits, char, std::regex_traits > >::difference_type 
> 

Re: Mod Defender (a NAXSI clone) integration patch

[Willy TARREAU]

On Mon, May 29, 2017 at 03:02:57PM +0200, Aleksandar Lazic wrote:
Hi Aleks,

> > Since both of them are at the exact same commit ID, do you know if the
> > project simply moved or is forked ? Does this mean we should expect to
> > find updates only at the new URL above and not at the previous one ?
> > Or maybe someone should just contact the project maintainer to know
> > which one is supposed to be the right one.
> 
> +1 and some more English doc would be nice ;-)
> 
> I'm still not that got in French :-(

That's because you don't visit France often enough ;-)

Willy

Re: Mod Defender (a NAXSI clone) integration patch

[Aleksandar Lazic]

Hi Willy TARREAU.

Willy TARREAU  have written on Mon, 29 May 2017
11:40:18 +0200:

> Hi Thierry, Dragan,
> 
> On Mon, May 29, 2017 at 11:25:48AM +0200, Thierry Fournier wrote:
> > Hi dragan, thats a great news.  
> 
> Yep great news and apparently great work (as usual).
> 
> > Just for information, the official project “mod_defenderâ€_ is
> > now here 
> > 
> >https://github.com/VultureProject/mod_defender  
> 
> Since both of them are at the exact same commit ID, do you know if the
> project simply moved or is forked ? Does this mean we should expect to
> find updates only at the new URL above and not at the previous one ?
> Or maybe someone should just contact the project maintainer to know
> which one is supposed to be the right one.

+1 and some more English doc would be nice ;-)

I'm still not that got in French :-(

Cheers
Aleks

> Two other comments while I'm thinking about this :
>   - Dragan, I think it could be useful to mention in the README that
> in its current state, the module is limited by haproxy to the analysis
> of the first buffer and that just like for the mod_sec equivalent,
> one workaround may consist in significantly increasing haproxy's
> buffer size ;
> 
>   - Thierry/Dragan, given that both of your contribs were made from
> Apache modules, do you think it would be useful/feasible to have
> a more generic SPOE<->APR agent to natively support more Apache
> modules ? Some people might want to recompress images or inline
> CSS and JS for example, and while I totally despise these
> practises which modify the delivered contents and corrupt caches,
> I can understand why some people would prefer to run this on the
> edge LB than having to configure it on all hosted servers.
> 
> Let's wait for a bit more feedback (if any) but at least from a
> quality perspective I'm fine with merging it as-is.
> 
> Cheers,
> Willy

Re: Mod Defender (a NAXSI clone) integration patch

[Aleksandar Lazic]

Hi Dragan Dosen.

Dragan Dosen  have written on Mon, 29 May 2017
10:29:55 +0200:

> Hi all,
> 
> I'm sending you a patch for Mod Defender (a NAXSI clone) integration
> -- a service that talks SPOE and uses the Mod Defender
> (https://github.com/Annihil/mod_defender) functionality to detect HTTP
> attacks. It returns a HTTP status code to indicate whether the request
> is suspicious or not, based on NAXSI rules. The value of the returned
> status can be used in HAProxy rules to determine if the HTTP request
> should be blocked/rejected.
> 
> Unlike ModSecurity, Mod Defender is a whitelist based WAF (everything
> is disallowed, unless there are rules saying otherwise). It's a
> partial replication of NAXSI and it uses NAXSI compatible rules
> configuration format.

Is there a comparison table what's the difference between mod defender
and mod security?

> Any comments are welcome.

Some feedback.

I have tried to build it on centos latest and got the following error.

###
+ make MOD_DEFENDER_SRC=/usr/src/mod_defender
APACHE2_INC=/usr/include/httpd APR_INC=/usr/include/apr-1
gcc -g -Wall -pthread -I../../include -I../../ebtree
-I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o spoa.o 
spoa.c
gcc -g -Wall -pthread -I../../include -I../../ebtree
-I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
defender.o defender.c
gcc -g -Wall -pthread -I../../include -I../../ebtree
-I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/deps/libinjection/libinjection_sqli.o 
/usr/src/mod_defender/deps/libinjection/libinjection_sqli.c
gcc -g -Wall -pthread -I../../include -I../../ebtree
-I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/deps/libinjection/libinjection_xss.o 
/usr/src/mod_defender/deps/libinjection/libinjection_xss.c
gcc -g -Wall -pthread -I../../include -I../../ebtree
-I/usr/src/mod_defender -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/deps/libinjection/libinjection_html5.o 
/usr/src/mod_defender/deps/libinjection/libinjection_html5.c
g++ -g -std=gnu++11 -I/usr/src/mod_defender
-I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/JsonValidator.o /usr/src/mod_defender/JsonValidator.cpp
g++ -g -std=gnu++11 -I/usr/src/mod_defender
-I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/RuntimeScanner.o /usr/src/mod_defender/RuntimeScanner.cpp
g++ -g -std=gnu++11 -I/usr/src/mod_defender
-I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/mod_defender.o /usr/src/mod_defender/mod_defender.cpp
g++ -g -std=gnu++11 -I/usr/src/mod_defender
-I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/Util.o /usr/src/mod_defender/Util.cpp
g++ -g -std=gnu++11 -I/usr/src/mod_defender
-I/usr/src/mod_defender/deps -I/usr/include/httpd -I/usr/include/apr-1 -c -o 
/usr/src/mod_defender/RuleParser.o /usr/src/mod_defender/RuleParser.cpp
gcc -o defender standalone.o spoa.o
defender.o /usr/src/mod_defender/deps/libinjection/libinjection_sqli.o 
/usr/src/mod_defender/deps/libinjection/libinjection_xss.o 
/usr/src/mod_defender/deps/libinjection/libinjection_html5.o 
/usr/src/mod_defender/JsonValidator.o /usr/src/mod_defender/RuntimeScanner.o 
/usr/src/mod_defender/mod_defender.o /usr/src/mod_defender/Util.o 
/usr/src/mod_defender/RuleParser.o -lpthread  -levent -levent_pthreads -lapr-1 
-laprutil-1 -lstdc++
/usr/src/mod_defender/RuntimeScanner.o: In function
`RuntimeScanner::processRuleBuffer(std::string const&, http_rule_t const&, 
unsigned long&)':
/usr/src/mod_defender/RuntimeScanner.cpp:146: undefined reference to
`std::regex_iterator<__gnu_cxx::__normal_iterator, 
char, std::regex_traits >::regex_iterator()'
/usr/src/mod_defender/RuntimeScanner.cpp:146: undefined reference to
`std::regex_iterator<__gnu_cxx::__normal_iterator, 
char, std::regex_traits 
>::regex_iterator(__gnu_cxx::__normal_iterator, 
__gnu_cxx::__normal_iterator, std::basic_regex const&, std::bitset<11ul>)'
/usr/src/mod_defender/RuntimeScanner.o: In function
`std::iterator_traits >::difference_type 
std::distance 
>(std::regex_iterator<__gnu_cxx::__normal_iterator, 
char, std::regex_traits >, 
std::regex_iterator<__gnu_cxx::__normal_iterator, 
char, std::regex_traits >)':
/usr/include/c++/4.8.2/bits/stl_iterator_base_funcs.h:118: undefined
reference to `std::regex_iterator<__gnu_cxx::__normal_iterator, char, std::regex_traits 
>::regex_iterator(std::regex_iterator<__gnu_cxx::__normal_iterator, char, std::regex_traits > const&)'
/usr/include/c++/4.8.2/bits/stl_iterator_base_funcs.h:118: undefined
reference to 

New feature request

[John Dison]

Hello,

in ROADMAP I see:
- spare servers : servers which are used in LB only when a minimum farm
weight threshold is not satisfied anymore. Useful for inter-site LB with
local pref by default.


Is it possible to push this item priority to get it done for 1.8 please?  It 
looks like it should not require major code refactoring, just another LB scheme.

What I want to achieve is an ability to route request to "local" pool until is 
get some
pre-defined maximum load, and route extra request to "remote" pool of servers.

Thanks in advance.

[PATCH] BUILD: ssl: fix build with OPENSSL_NO_ENGINE

[Emmanuel Hocdet]

Hi,

Last patches with openssl engine break build with boringssl.
Fix include in the mail.

Manu



0001-BUILD-ssl-fix-build-with-OPENSSL_NO_ENGINE.patch
Description: Binary data

Re: New feature proposal: Add support for decompressing proxyed gziped requests

[Vasileios Kyrillidis]

Hi Willy,

Thank you for your reply.

The time schedule is not yet set, but I believe we will require one/two 
months as this is not high priority on our side at the moment.
Since we have confirmed that merging is indeed promising (depending on 
the quality of the patches), I can give the green light to start working 
on this.


Thank you also for your points, we will have a more thorough look into 
the code and we will get back to you in the near future with questions 
etc :)


Cheers,
Vasilis

On 27/05/17 08:09, Willy Tarreau wrote:

Hi,

On Fri, May 26, 2017 at 06:57:14PM +0200, Aleksandar Lazic wrote:

Hi Vasileios Kyrillidis.

Vasileios Kyrillidis  have
written on Fri, 26 May 2017 16:17:48 +0200:


Hi Aleksandar,

No patches yet. We wanted confirmation that it has a chance of
getting merged before spending the time implementing. We have other
solutions that we could pursue instead.

If merging looks promising we will start working on patches.

Well I'm not a decision guy but when the patche's look good and does
match the event based system of haproxy I think there a good chances to
be part of haproxy.

It could be interesting, and theorically possible using the filters (ie
do the opposite of the compression). However you'll face some painful
difficulties : while with compression it's trivial to ensure that an
output chunk will never be larger than the input one, you can't do that
with decompression as just a few bytes can decompress into thousands.

This will require to keep some large memory contexts allocated, except
that contrary to the compression it will not even be possible to limit
this and decide not to decompress.

Another point is that usually for compression we build with USE_SLZ, which
doesn't provide decompression so you'll either need to make this feature
dependant on USE_ZLIB only or ensure it's disabled when using USE_SLZ.

Finally, please be aware that we're not merging new features for 1.8 now
so if you do this, it will be for 1.9. But if you have some good quality
patches ready in time, I'm fine with opening a "next" branch to accumulate
all the pending work for when 1.9-dev opens.

Cheers,
Willy


--
*Vasileios Kyrillidis*  | Software Developer  | Sociomantic Labs 
www.sociomantic.com 
  | 

Twitter  + Facebook 
 + Resources 
  | 
*Subscribe* to our Newsletter 
. 


Sociomantic Labs Logo
Sociomantic Labs GmbH, Location: Berlin, Commercial Register - AG 
Charlottenburg: HRB 121302 B, VAT No. - USt-ID: DE 266262100, Managing 
Directors: Thomas Nicolai, Sarah McCarthy, Mark Anthony Hinds. This 
message and any attachments are confidential and intended solely for the 
use of the individual to whom it is addressed.

Re: HAProxy won't shut down

[Frederic Lecaille]

Hi Patrick,

First thank you for this nice and helpful report.

Would it be possible to have an output of this command the next time you 
reproduce such an issue please?


echo "show sess" | socat stdio 

I have only one question (see below).

On 05/24/2017 10:40 AM, Willy Tarreau wrote:

Hi Patrick,

On Tue, May 23, 2017 at 01:49:42PM -0400, Patrick Hemmer wrote:
(...)

haproxy 28856 root1u IPv4  420797940  0t0TCP 
10.0.33.145:35754->10.0.33.147:1029 (CLOSE_WAIT)
haproxy 28856 root2u IPv4  420266351  0t0TCP 
10.0.33.145:52898->10.0.33.147:1029 (CLOSE_WAIT)
haproxy 28856 root3r  REG0,30 4026531956 net
haproxy 28856 root4u IPv4  422150834  0t0TCP 
10.0.33.145:38874->10.0.33.147:1029 (CLOSE_WAIT)


These ones are very interesting.


These traces also seem interesting to me.

# strace -p 28856
Process 28856 attached
epoll_wait(0, {}, 200, 319) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 362) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 114) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 203) = 0
epoll_wait(0, {}, 200, 0)   = 0
epoll_wait(0, {}, 200, 331) = 0
epoll_wait(0, {}, 200, 0)


Were such "epoll_wait(0, 0, 200, 0)" calls infinitively displayed?


In fact I am wondering if it is normal to have so much epoll_wait(0, {}, 
200, 0) calls for a haproxy process which has shut down.


I suspect they are in relation with peer tasks (obviously which has 
expired).


If this is the case, and with configurations with only peer tasks, 
haproxy would definitively hang consuming a lot of CPU resources.


So, I had a look at the peer struct task 'expire' member handling code, 
and I have just found a situation where pollers in relation with peer 
tasks are often called with an expired timeout leading haproxy to 
consume a lot of CPU resources. In fact this happens each time the peer 
task has expired during a fraction of second.


It is easy to reproduce this issue with a sort of peer simulator ;):

strace -ttf socat TCP4-LISTEN:,reuseaddr,fork SYSTEM:"echo 
200;sleep 10"


This peer must be started *before* the other remote haproxy process with 
only peers as backends.


strace is here only to have an idea of the moment where the remote 
haproxy peer has just connected.


The sleep command is here to have enough time to block (ctrl + s) our 
peer simulator process after the haproxy peer has just connected.


So this peer accepts any remote peer sessions sending "200" status 
messages (and that's all).


A haproxy peer which connects to such a peer which does not reply to a 
synchronization request would endlessly consume high CPU ressources 
until you unblock (ctrl + q) the peer simulator process.


*Unhappily, I do not see any relation between this bug and the 
"CLOSE_WAIT peer state issue" which prevents haproxy from correctly 
shutting down.*


I have attached a patch to this mail which fixes this issue.

Regards,

Fred.






>From 855635c42b89e25a52255dd5edc8c872e888656e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= 
Date: Mon, 29 May 2017 13:47:16 +0200
Subject: [PATCH] BUG/MINOR: Wrong peer task expiration handling during
 synchronization processing.

When a peer task has sent a synchronization request to remote peers
its next expiration date was updated based on a resynchronization timeout
value which itself may have already expired leading the underlying
poller to wait for 0ms during a fraction of second (consuming high CPU
resources).

With this patch we update such peer task expiration dates only if
the resynchronization timeout is not already expired.

Thanks to Patrick Hemmer who reported an issue with nice traces
which helped in finding this one.

This patch may be backported to 1.7 and 1.6.
---
 src/peers.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/peers.c b/src/peers.c
index 0c8861f..0ebc040 100644
--- a/src/peers.c
+++ b/src/peers.c
@@ -1957,8 +1957,9 @@ static struct task *process_peer_sync(struct task * task)
 
 		if ((peers->flags & PEERS_RESYNC_STATEMASK) != PEERS_RESYNC_FINISHED) {
 			/* Resync not finished*/
-			/* reschedule task to resync timeout, to ended resync if needed */
-			task->expire = tick_first(task->expire, peers->resync_timeout);
+			/* reschedule task to resync timeout if not expired, to ended resync if needed */
+			if (!tick_is_expired(peers->resync_timeout, now_ms))
+task->expire = tick_first(task->expire, peers->resync_timeout);
 		}
 	} /* !stopping */
 	else {
-- 
2.1.4

Re: [PATCHES] Major DNS changes

[Aleksandar Lazic]

Hi Baptiste.

Baptiste  have written on Mon, 29 May 2017 11:14:14
+0200:

> Hi Aleksandar,
> 
> I have take a look into the code and have just some questions about
> > calloc in [PATCH 03/11] & [PATCH 07/11]
> >
> > In the function dns_alloc_resolution is calloc used, would the use
> > of haproxy pools bring any benefit?
> >
> >  
> it may help a bit from memory usage point of view.
> I planned to use them, but since I was blocking other devs, I used the
> quickest and safest way for now.
> Let say I'll improve this for HAProxy 1.9, unless I find some time
> before 1.8 is ready.

Thanks for info.

> > Do I have understand the code right that you have per resolver a
> > cache and this resolver belongs to a backend/frontend.
> >
> >  
> A cache belong to a "resolvers" section. Many servers (for now) or any
> requester (later) can use the same "resolvers", hence benefit from the
> cached responses.
>
> > So in case several backends have the same server names all resolver
> > requests this server individual.
> > There is no 'global dns cache' for all or I missed something.
> >
> >  
> No.
> Take a look at this configuration:
> 
>resolvers mydns
> ...
>frontend ft
> ...
> 
>backend b1
> server s1 myappsrv1.domain.com:80 resolvers mydns   # no need
> "check" anymore :)
> 
>backend b2
> server s1 myappsrv1.domain.com:80 resolvers mydns   # no need
> "check" anymore :)
> 
> Both b1/s1 and b2/s1 points to the same resolvers and one will
> benefit from the cached response of the other one.
> 
> There is still one point though, the cache is per resolvers, but (for
> now) based on family preference. Soon, Olivier or I will improve this
> by enforcing the resolvers to perform both A and  queries and
> cache both response and let the requester pick-up the one he wants.
> For now, the cache only stores the response of the latest query...

thanks. got the point.

> Baptiste
> 
> 
> >  
> > > Please give it a try and report any issues you may spot :)
> > >
> > > Baptiste  
> >
> > Regards
> > Aleks
> >  
> 

Re: Mod Defender (a NAXSI clone) integration patch

[Willy TARREAU]

Hi Thierry, Dragan,

On Mon, May 29, 2017 at 11:25:48AM +0200, Thierry Fournier wrote:
> Hi dragan, thats a great news.

Yep great news and apparently great work (as usual).

> Just for information, the official project “mod_defender” is now here 
> 
>https://github.com/VultureProject/mod_defender

Since both of them are at the exact same commit ID, do you know if the
project simply moved or is forked ? Does this mean we should expect to
find updates only at the new URL above and not at the previous one ? Or
maybe someone should just contact the project maintainer to know which
one is supposed to be the right one.

Two other comments while I'm thinking about this :
  - Dragan, I think it could be useful to mention in the README that in
its current state, the module is limited by haproxy to the analysis
of the first buffer and that just like for the mod_sec equivalent,
one workaround may consist in significantly increasing haproxy's
buffer size ;

  - Thierry/Dragan, given that both of your contribs were made from
Apache modules, do you think it would be useful/feasible to have
a more generic SPOE<->APR agent to natively support more Apache
modules ? Some people might want to recompress images or inline
CSS and JS for example, and while I totally despise these
practises which modify the delivered contents and corrupt caches,
I can understand why some people would prefer to run this on the
edge LB than having to configure it on all hosted servers.

Let's wait for a bit more feedback (if any) but at least from a quality
perspective I'm fine with merging it as-is.

Cheers,
Willy

Re: Mod Defender (a NAXSI clone) integration patch

[Thierry Fournier]

Hi dragan, thats a great news.
Just for information, the official project “mod_defender” is now here 

   https://github.com/VultureProject/mod_defender 


Thierry

> On 29 May 2017, at 10:29, Dragan Dosen  wrote:
> 
> Hi all,
> 
> I'm sending you a patch for Mod Defender (a NAXSI clone) integration --
> a service that talks SPOE and uses the Mod Defender
> (https://github.com/Annihil/mod_defender) functionality to detect HTTP
> attacks. It returns a HTTP status code to indicate whether the request
> is suspicious or not, based on NAXSI rules. The value of the returned
> status can be used in HAProxy rules to determine if the HTTP request
> should be blocked/rejected.
> 
> Unlike ModSecurity, Mod Defender is a whitelist based WAF (everything is
> disallowed, unless there are rules saying otherwise). It's a partial
> replication of NAXSI and it uses NAXSI compatible rules configuration
> format.
> 
> Any comments are welcome.
> 
> 
> Best regards,
> Dragan Dosen
> <0001-MINOR-Add-Mod-Defender-integration-as-contrib.patch>

Re: [PATCHES] Major DNS changes

[Baptiste]

Hi Aleksandar,

I have take a look into the code and have just some questions about
> calloc in [PATCH 03/11] & [PATCH 07/11]
>
> In the function dns_alloc_resolution is calloc used, would the use of
> haproxy pools bring any benefit?
>
>
it may help a bit from memory usage point of view.
I planned to use them, but since I was blocking other devs, I used the
quickest and safest way for now.
Let say I'll improve this for HAProxy 1.9, unless I find some time before
1.8 is ready.



> [PATCH 08/11]
> +static int dns_cache_size = 1024;   /* arbitrary DNS cache size */
>
> For the future maybe there could be a glolbal.tune.dns-cache-size or
> something similar.
>

Ah yes!
I take the point :)


>
> Do I have understand the code right that you have per resolver a cache
> and this resolver belongs to a backend/frontend.
>
>
A cache belong to a "resolvers" section. Many servers (for now) or any
requester (later) can use the same "resolvers", hence benefit from the
cached responses.



> So in case several backends have the same server names all resolver
> requests this server individual.
> There is no 'global dns cache' for all or I missed something.
>
>
No.
Take a look at this configuration:

   resolvers mydns
...
   frontend ft
...

   backend b1
server s1 myappsrv1.domain.com:80 resolvers mydns   # no need "check"
anymore :)

   backend b2
server s1 myappsrv1.domain.com:80 resolvers mydns   # no need "check"
anymore :)

Both b1/s1 and b2/s1 points to the same resolvers and one will benefit from
the cached response of the other one.

There is still one point though, the cache is per resolvers, but (for now)
based on family preference. Soon, Olivier or I will improve this by
enforcing the resolvers to perform both A and  queries and cache both
response and let the requester pick-up the one he wants.
For now, the cache only stores the response of the latest query...

Baptiste


>
> > Please give it a try and report any issues you may spot :)
> >
> > Baptiste
>
> Regards
> Aleks
>

Re: Haproxy first core 100%

[Haim Ari]

Hello Lukas,

Since this is a production environment i had to find a window to run the 
tasks...


  1.  Upgraded to 1.7.5
  2.  below is the output of haproxy -vv (before and after upgrade)

HA-Proxy version 1.7.3 2017/02/28
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe


--


HA-Proxy version 1.7.5-2ppa1~xenial 2017/05/27
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 
USE_PCRE=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Built with network namespace support

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe


So far i don't see the haproxy process hangs on 100% utilization.
I returned the configuration to run on a single core (#1) and will update in 
case the issue returns

Thanks




From: Lukas Tribus 
Sent: Thursday, May 25, 2017 7:33 PM
To: Haim Ari; haproxy@formilux.org
Subject: Re: Haproxy first core 100%

Hello Haim,


Am 25.05.2017 um 09:23 schrieb Haim Ari:
>
> Hello,
>
>
> I'll try do describe the issue as clear as possible:
>
>
> We set up an haproxy Cluster on Ubuntu16.04 + pacemaker + corosync
>
> We faced an issue where after working for a few hours with single core
> (the haproxy process was bind to core #1 not #0 - which is handling
> the IRQ interrupts )
>
> it is 100% utilized.
>
>
> [...]
> Haproxy: HA-Proxy version 1.7.3 2017/02/28
> OS: Ubuntu 16.04.2 LTS
> Kernel: 4.4.0-66-generic
> Pacemaker: 1.1.14-2
> Corosync 2.3.5-3
>
> Any advise on this ?


First of all, please upgrade to latest stable, by adding Vincent PPA:
https://haproxy.debian.net/#?distribution=Ubuntu=xenial=1.7


If it still happens when running haproxy 1.7.5, please provide the
output of haproxy -vv and some details about the loop as it appears in
strace (let "strace -ttp" running for a few seconds - be careful
about your privacy when sharing the strace output).



Thanks,

Lukas

Re: haproxy 1.7.5 segfault on cookie/header parsing.

[Willy Tarreau]

Hi Jean,

On Sun, May 28, 2017 at 10:15:28AM +, Jean LUBATTI wrote:
> There was a tcp-request inspect-delay of 2s in the configuration when running
> the repro, so it should be fine.

OK. However, I totally fail to reproduce the problem here using your config,
the build options I found in your executable, and the captured requests, they
are properly handled and passed. And since the crash happens inside memmove(),
the memory is displaced everywhere and the internal structures are not much
analyzable.

Could you please try again with the attached patch ? It's supposed to provoke
a segfault before the bogus call to memmove(), resulting in a "clean" core.

Thanks,
Willy
diff --git a/src/buffer.c b/src/buffer.c
index 4f8f647..70a01dd 100644
--- a/src/buffer.c
+++ b/src/buffer.c
@@ -87,6 +87,9 @@ int buffer_replace2(struct buffer *b, char *pos, char *end, 
const char *str, int
return 0;  /* no space left before wrapping data */
 
/* first, protect the end of the buffer */
+   if (bi_end(b) < end)
+   *(volatile int *)0 = 0;
+
memmove(end + delta, end, bi_end(b) - end);
 
/* now, copy str over pos */