Hello Thomas,
On Fri, Jul 05, 2013 at 10:58:43AM +0200, Tomas Pospisek wrote:
> >> When I strace that instance, it's showing lots and lots of:
> >>
> >> epoll_wait(0, {}, 200, 0) = 0
> >
> > Is it the only output for epoll_wait() or do you also see some values
> > where the timeout is non-zero (the last field) ? If it's always this,
> > then it probably means that a task supposed to run does not correctly
> > run or does not sleep.
>
> It's allways like this.
OK.
> > Could you please try to run it with "nosepoll" in the global section ?
> > It will disable the speculative epoll and use only epoll. Can you
> > confirm that you don't have any health checks in your config ?
> > Health-checks are a good examle of a background task. I'm trying
> > to understand what could happen...
> >
> > Other than that, I'm not thinking about anything that could induce
> > this. If you can, please try to disable the transparent proxy. We
> > could imagine that it reports an error very early that is not
> > properly caught and that causes the process to loop until a connect
> > timeout for example.
>
> We had another, aparently unrelated problem. Once we fixed it, haproxy
> returned to normal operation mode.
Interesting case to investigate.
> The problem we had is that we have most of our virtual servers on an
> internal, private network (this can be seen in the config posted in my
> original mail). This was working fine before we switched to TPROXY.
>
> Once we switched certain ports to TPROXY mode, one could no more connect
> (TCP) from one internal server to a TPROXied service.
>
> The problem was that communication from an internal server to the
> service was going over the external IP and thus over haproxy.
>
> Haproxy would then TPROXY the communication and thus preserve the
> original sender IP.
>
> Once the service got the IP packet it would reply to the *internal* IP
> wich would not get routed via haproxy, but instead directly.
>
> So there would be a socket communicating from the internal client
> machine to haproxy and from haproxy to the internal server. And replies
> from the server would go directly to the client. Which evidently the IP
> stack is unable to grok.
Obviously. Don't worry, this mess is much more common than you can
think, and personally, I'm really not a fan of transparent proxying.
I was proud of it when I managed to get it and quickly got fed up
with all the issues it causes due to the various possible configuration
errors in field. Not to mention the fact that you have to carefully set
your timewait timeouts on your machines or ensure that your LB doesn't
forward IP, otherwise you risk some ACK storms between the clients and
the server when source ports get reused too fast.
> Was that where the 100% CPU/epoll would come from? Haproxy tirelessly
> but without success trying to establish a connection to forward the TCP
> stream?
I'll dig into this direction. For haproxy this means that it never
received the SYN-ACK so it was exactly like a server not responding
(eg: suddenly unplugged), except that this time health checks probably
said it was OK so the problem remained permanent. It is very possible
that this happens for other connections in some other cases but the
short duration of the issue hides it.
> Our solution was to split the configuration into connections from the
> internal private networks and into connections from the external
> networks aka:
>
> frontend smtp
> bind 4.3.2.1:25,4.3.2.1:465
> mode tcp
> maxconn 100
> acl client_from_lan src 192.168.138.0/24
> use_backend smtp_b0 if client_from_lan
> default_backend smtp_b1
>
> backend smtp_b0
> mode tcp
> server vilan-mail 192.168.1.1
>
> backend smtp_b1
> mode tcp
> source 0.0.0.0 usesrc clientip
> server vilan-mail 192.168.1.1
>
> Now "everthing's OK".
Thanks for the details. If you want, you can as well set the source
parameter on each server line, it case you would need to mix them,
but given your config I guess this is not the case and what you did
is the right thing.
> Hope this will help someone who runs into the same problem.
Could be useful, yes, and the solution is clean and easy to understand.
> And maybe make haproxy more resilient in such a situation or
> respectively emit some helpful diagnostic messages.
It should just time-out when establishing the connections, the spinning
loop certainly is not expected and is a bug. Did you by chance reproduce
this with 1.5 or didn't you try ? Since the connection management is
very different between the two, I would doubt about this.
> Credits: the solution to the problem connection problem was found by
> Michal Fiala.
Kudos to Michal then :-)
> Many thanks Willy for your helpfulnes!!!
Thanks to you for this detailed explanation, it will certainly be useful.
Best regards,
Willy
