Hello, On 2013-02-07 01:13, Julian Anastasov wrote: > > Hello, > > On Tue, 5 Feb 2013, Dmitry Akindinov wrote: > >> Hello, >> >> We have met a quite troublesome situation which causes an internal SYN >> storm. >> >> The simplified version of the configuration consists of 2 servers - A >> and B, both running Linux kernel 3.7.4-20. >> >> Both have the IPVS software enabled, A is acting as the active load >> balancer, B as a backup. >> Both servers act as real servers also. >> >> At some point, there is an incoming TCP connection from IPpair >> (address:port) I. >> The load balancer A decides to process it locally. Connection is >> established, and the balancer status is distributed to server B via >> syncing broadcast. >> >> The client closes connection, and again the status is updated on B via >> the broadcast - the connection is now in the "TCP_WAIT" state. >> >> Pretty soon (within 10 seconds) the client opens the new TCP connection >> using the same IP pair I. >> It is not a good TCP practice, but nevertheless, some clients work this way. >> >> This time the load balancer A decides that the connection is to be >> handled on the server B (persistence is switched off). > > If connection still exists in balancer A it is > not going to select new real server. May be only if > expire_nodest_conn is set and when current real server > becomes unavailable a new real server can be selected > for next packets (2nd SYN). > >> The SYN packet is relayed to the server B, which finds an existing >> routing record for that pair I. >> And that record (in the CLOSE state) - points to the server A, and the >> SYN packet is relayed there. >> >> The server A processes it again, directs it to the server B again, and >> the loop spirals, since the server B does not have the new connection >> table element I synced. > > More likely the SYN comes short after the conn > in server A is expired but the synced conn in server B > is not expired yet. This can happen often because > the sync protocol is not perfect, conns in backup > tend to expire later.
We tried to run with persistence but this made the situation even worse: after successfully accepting a connection from the client IP, a new (now from a different source port) should be handled by the same server (the master) but the SYN packet is redirected... >> We can send packet dumps illustrating the problem. >> >> If our analysis is correct, what are the available workarounds? > > I see that we discussed this problem August 2012. > I assume this is DR method and all IPVS rules are present > in backup? Are you using the old "sync_threshold" algorithm > or the new one with sync_refresh_period=10? Yes, we use DR method and rules are set up on all members. We use as few configuration options as possible, e.g.: [root@fm1 ~]# ipvsadm-save -A -f 1000 -s rr -O -a -f 1000 -r fm1.service.com:0 -g -w 75 -a -f 1000 -r fm2.service.com:0 -g -w 100 -A -f 1061 -s wlc -a -f 1061 -r fm1.service.com:0 -g -w 1 -A -f 1062 -s wlc -a -f 1062 -r fm2.service.com:0 -g -w 1 and the daemons are started without any additional parameters: ipvsadm --start-daemon [backup|master] --mcast-interface $IPINTF --syncid 0 >> a) we can always use "persistent" option with time larger than CLOSE >> (TIME_WAIT?) state time. > > May be the problem will move from the normal conns > to the persistent conn templates. Just starting the daemons with the -p flag did worse: now all new connections from the same client IP for which a live connection exists would start a SYN storm. > The simplest solution is > to use: > > if (ipvs->sync_state & IP_VS_STATE_BACKUP) > return NF_ACCEPT; > > in all hooks. This will stop all traffic. Currently we use no hooks. > The problem is that we do not know if the backup > function is used for part of the virtual services, > other virtual services can be in normal mode, > possibly using IP_VS_STATE_MASTER. I assume that > was the reason the master and backup functions to > be able to run together, with different sync_id. > > I'm not sure what is more appropriate > to apply here, some sysctl var that will stop > the forwarding mode while backup function is > enabled. This solution will work better if we > don't want to change the tools that manage IPVS > rules and it is easier to implement, eg. > "backup_only=1" to activate such mode. The > result would be: > > if (ipvs->sync_state & IP_VS_STATE_BACKUP && > ipvs->backup_only) > return NF_ACCEPT; /* Try local server */ > > Another solution would be to add > optional syncid attribute to the virtual > server. By this way we will know that received > packet matches the backup_syncid (we are > used as real server from director) or else it is > directed from client to our virtual server. In the > second case if the packet matches master_syncid (as > additional check to the IP_VS_STATE_MASTER flag) a sync > message would be sent. Hmm. Do you mean the workaround is to customize the kernel? We would rather avoid that... >> b) on the server B we can remove the iptables records marking incoming >> packets with a flag used with the IPVS uses. >> We can insert those iptable rule(s) only when the server B becomes the >> main load balancer. But will it stop IPVS from running all incoming >> packets via its (synced) connections table? > > What is the case now, do you have IPVS rules > on server B while the backup function is enabled? Yes. -- Best regards, Dmitry Akindinov _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users
