When you closing rabbit gracefully it sends 'DOWN' message to connected clients and you notice your clients reconnecting. I've tried in the past various ways to detect disconnects on HAProxy side, including playing with various TCPKA settings but nothing helped. (I was playing with 1.4)
2014/1/17 Andrei Chevenkov <[email protected]>: > Dmitry, thank you for the reply, but I would imagine that haproxy would > close all sessions on the DOWN nodes, regardless of the type of client > connecting and the protocol? Can this be enforced? > > I did implement producer ack, but that slows down publishes big time. Have > also tried heartbeat and that did help, but this is initiated by the client > and I would like to see if haproxy can manage this. I.e. close all sessions > on a node that is marked as DOWN. > > On 17/01/2014 8:09 pm, "Dmitriy Samsonov" <[email protected]> > wrote: >> >> Hi! >> >> That's related to rabbiit's default heartbeat timeout. You can set it to >> lower value when connecting. Also there is a way to check if message was >> actually delivered to the broker, take a look at rabbiit's docs. >> >> >> пятница, 17 января 2014 г. пользователь Andrei Chevenkov написал: >>> >>> Hi, >>> >>> I am pretty new to haproxy and trying to do ha setup for rabbitmq. >>> >>> Setup is pretyt simple: >>> >>> listen rabbitmq 192.168.69.106:5672 >>> mode tcp >>> balance leastconn >>> option tcplog >>> option tcpka >>> server rabbit01 192.168.69.107:5672 check inter 1000 downinter 5000 >>> fall 1 on-error mark-down >>> server rabbit02 192.168.69.108:5672 check inter 1000 downinter 5000 >>> fall 1 on-error mark-down backup >>> >>> Everything seems to work well, but I noticed one strange behaviour. If I >>> gracefully shut down rabbit01, message producer (client) connected to >>> 192.168.69.106:5672 notices broken connection pretty much straight away, >>> attempts a reconnect, hits rabbit02 and all is well. >>> >>> However if I simply "turn off" rabbit01, message producer (client) >>> notices broken connection only in around 30 seconds or so. And while the >>> break is undetected it keeps pushing messages thinking there is still >>> someone on another side. >>> >>> Now, from the statistic report I can see that as soon as I turn off the >>> box, rabbit01 is marked as "DOWN", but its "Sessions Current" is still set >>> to 1? In the logs this comes up pretty much immediately after the hard turn >>> off: >>> >>> Jan 17 17:13:51 prodlb01 haproxy[38459]: Server rabbitmq/rabbit01 is >>> DOWN, reason: Layer4 timeout, check duration: 1008ms. 0 active and 1 backup >>> servers left. Running on backup. 1 sessions active, 0 requeued, 0 remaining >>> in queue. >>> >>> After around 30 seconds, the connection is detected as broken and >>> failover happens successfully. At that time "Sessions Current" is set to 0 >>> fo rabbit01. >>> >>> Tried option nolinger and option abortonclose, but no luck. >>> >>> Any help would be much appreciated... >>> >>> Cheers, >>> Andrei
