Hi Willy,
I believe your workaround fits socket data pollution, but it does not
really help with epoll connection drop. However, instead of forcibly
closing the sockets on each fork(), perhaps, setting FD_CLOEXEC right
after accept()/socket() is much safer and potentially better
performing with large number of descriptors and external-checks.
Alternative way, is to try to use posix_spawn().
Also,I have noticed around 0.6% of dropped connections which can be a
consequence of race condition on close() in child. I'll try to look
more.
Regarding, the epoll connection drop, it seems the issue comes from
epoll FD being shared with child process and therefore affected by
changes in it.
Below is patch which fixes the issues remaining after close()
workaround (at least in my tests). Note: maxsock is unused parameter -
it is used to be utilized only as a hint in pre 2.6.8 kernels.
Just in case, I contribute the patch according to LICENSE and yes, I
am a bit violating CONTRIBUTING rules. Sorry :(
NOTE: it also requires a separate socket data pollution fix as well!
--- haproxy-1.6.5.orig/src/ev_epoll.c 2016-05-10 16:42:00.000000000 +0300
+++ haproxy-1.6.5/src/ev_epoll.c 2016-06-14 01:21:45.957098610 +0300
@@ -176,7 +176,7 @@ REGPRM1 static int _do_init(struct polle
{
p->private = NULL;
- epoll_fd = epoll_create(global.maxsock + 1);
+ epoll_fd = epoll_create1(EPOLL_CLOEXEC);
if (epoll_fd < 0)
goto fail_fd;
@@ -222,7 +222,7 @@ REGPRM1 static int _do_test(struct polle
{
int fd;
- fd = epoll_create(global.maxsock + 1);
+ fd = epoll_create1(EPOLL_CLOEXEC);
if (fd < 0)
return 0;
close(fd);
@@ -239,7 +239,7 @@ REGPRM1 static int _do_fork(struct polle
{
if (epoll_fd >= 0)
close(epoll_fd);
- epoll_fd = epoll_create(global.maxsock + 1);
+ epoll_fd = epoll_create1(EPOLL_CLOEXEC);
if (epoll_fd < 0)
return 0;
return 1;
Thanks,
Andrey Galkin
Андрей
On Mon, Jun 13, 2016 at 10:35 PM, Willy Tarreau <[email protected]> wrote:
> Hi Andrey,
>
> On Sun, Jun 12, 2016 at 04:15:49PM +0300, Andrey G. wrote:
>> Hi,
>>
>> I am seeing quite strange and not simple to reproduce behavior found
>> during stress testing of https://github.com/codingfuture/puppet-cfdb .
>> After a long testing with different options, I am quite sure the issue
>> comes from epoll support as "noepoll" fixes the problem.
>>
>> Here's the scenario:
>>
>> 1. There are several DB clusters accessible through TCP
>> 2. HAProxy 1.6.5 on Debian Jessie accepts clients on UNIX sockets
>> under /run/{service}/ (easier auto-configuration, security and per
>> role maxconn control)
>> 3. external-check python-based (yep..) scripts are used to properly
>> identify ready nodes in clusters
>> 4. As cluster connections may be wrapped in TLS, external-check
>> scripts use dedicated no-check "listen" blocks in HAProxy, but do not
>> connect directly.
>> 5. As there is a known recently discovered issue with stdout/stderr
>> data clobbering, I made sure external-check scripts to close mentioned
>> descriptors (the example below uses /dev/null instead).
>
> I think the issue is very similar. In fact, closing stdin/stdout is not
> enough, we need to close *all* FDs there. epoll uses a specific FD and
> I suspect that it is what is causing the issue. An alternate reason could
> be that its presence simply shifts another FD by 1 and makes it conflict
> with one FD used by the external check. Normal scripts should not assume
> that fds above 2 may be used without initialization. But note that it is
> also possible that epoll is assigned fd #2 and causes all the trouble.
>
> Thus now I'd do this after the fork for external checks instead in
> src/checks.c to guarantee we close all open FDs :
>
> if (pid == 0) {
> /* Child */
> extern char **environ;
> + int fd;
> +
> + for (fd = 0; fd <= maxfd; fd++)
> + close(fd);
>
> Is it something you could check ? By the way, thanks a lot for your bug
> report, it's very well detailed!
>
> Willy
