Hi guys,
I'm replying to this thread and adding Andrey and Veiko in Cc to keep
everyone up to date.
On Wed, Jun 08, 2016 at 10:17:19AM +0900, Simon Horman wrote:
> Lukas, could you try this?
>
> diff --git a/src/checks.c b/src/checks.c
> index c4ac947b6051..e65d28f7c3c6 100644
> --- a/src/checks.c
> +++ b/src/checks.c
> @@ -1836,6 +1836,12 @@ static int connect_proc_chk(struct task *t)
> if (pid == 0) {
> /* Child */
> extern char **environ;
> +
> + if ((global.mode & MODE_QUIET) && !(global.mode &
> MODE_VERBOSE)) {
> + close(0);
> + close(1);
> + }
> +
> environ = check->envp;
> extchk_setenv(check, EXTCHK_HAPROXY_SERVER_CURCONN,
> ultoa_r(s->cur_sess, buf, sizeof(buf)));
> execvp(px->check_command, check->argv);
So since this patch we got Andrey's report and figured other issues
caused by sharing FDs between the processes (namely the fact that
automatic FD deletion doesn't happen in epoll until the child closes
it as well, possibly causing polling loops or fake event reports).
Thus I went down the safe route for now, by closing *all* FDs. This is
fast enough, on my machine I can close around 17 million FDs per second.
However I liked Simon's approach above consisting in keeping stdin/stdout/
stderr open when in verbose mode. That's very handy to debug.
While testing this change, I discovered a very interesting issue. My
process would randomly segfault after a delay ranging from 1 to 10
seconds. The core files didn't make any sense, proving that the memory
was corrupted. In the end I found the culprit. The problem is random
signal delivery. In the signal handler, a call to task_wakeup() is
made, and this sometimes destroys the scheduler when it's already in
the signal handler, as the rq_next pointer is sometimes nulled then
dereferenced, and sometimes the ebtree structure is corrupted. So I
changed this to make use of the asynchronous signal delivery and now
it's OK since the functions are called after the poll loop.
This made me remember that Veiko who used to experience random segfaults
with zlib also has external checks configured and also had a non-sensical
core file, and zlib has a few orders of magniture higher latency than the
rest of the process, that makes it much more likely that a signal is
delivered here than anywhere else.
Thus I backported all of this to 1.6 planning for 1.6.6. It would be nice
if the people affected with issues could give it a try this week (either
from git or just wait for tomorrow morning to get the latest snapshot).
I also developped a patch to block all signals out of the poll loop. This
makes the code cleaner but I realized that it only improves the performance
for fork-limited workloads and will decrease it for regular workloads. So
for now it remains unmerged before I run more tests. On Linux we can use
epoll_pwait() to remove a few syscalls so the impact might be less important.
Anyway I suspect that over the longterm we'll go this way.
Thanks to all those who helped on this, and do not hesitate to report
any remaining issue!
Willy
