On 30.04.2013, Konstantin Boyanov wrote: > 1. I open some connections from a client browser (IE 8) to the sites that > are hosted on the real servers > 2. I cange the weight of the real server which server the above connections > to 0 and leave only the other real server alive > 3. I reload the pages to regenerate the connections > > What I am seeing with ipvsadm -Ln is that the connections are still on the > "dead" server. I have to wait up to one minute (I suppose some TCP timeout > from the browser-side) for them to transfer to the "living" server. And If > in this one minute I continue pressing the reload button the connections > stay at the "dead" server and their TCP timeout counter gets restarted. > > So my question is: Is there a way to tell the load balancer in NAT mode to > terminate / redirect existing connections to a dead server *immediately* > (or close to immediately)?
The man page for ipvsadm (the --weight parameter) helps understanding this :) To remove a dead server "immediately", you need to remove it from the ipvs table (ipvsadm --delete-server). This also breaks any open tcp connections, probably resulting in "connection refused"-error messages or "broken images" on the client. A weight of zero essentially means "don't put new connections on this realserver, but continue serving existing connections". This is something you're usually using for maintenance: let the server fulfill any pending requests, and after a few minutes, all new connections usually have shifted to other realservers. Of course, as long as the existing connection is being used, IPVS does recognize your connection to be "in use". If you don't like this, you'll actively have to tell your webserver to shut down any connections (e.g. by shutting down IIS), forcing the client to reconnect or remove your realserver from the IPVS table (which will result in timeouts for any open connections, but ultimately the client will be forced to reconnect as well). > It seems to me a blunder that a reload on the client-side can make a > connection become a "zombie", e.g. be bound to a dead real server although > persistance is not used and the other server is ready and available. The maintenance usecase is usually quite obvious to understand, and that's the "zero weight" usecase. It's for "draining" connections: new connections won't sent to this box, but any current connections will be served (until they're being closed). > The only thing that I found affecting this timeout is changing the > keepAliveTimeout in the Windows machine running the IE8 which I use for the > tests. When I cahnged it from the dafault value of 60 seconds to 30 seconds > the connections could be transferred after 30 seconds. It seems to me very > odd that a client setting can affect the operation of a network component > as the load balancer. The server does also have similiar options to configure: -you can turn off HTTP Keepalive completely -IIS 7 per default uses 2 minutes as a keepalive timeout; after this idle time, IIS will close the connection. Of course, these options may have some (small) performance impact. If you're turning off HTTP Keepalive completely, the browser is forced to setup a new tcp connection, including the full tcp handshake for every object to retrieve and the two packets for tearing down the connection. If your website e.g. contains 40 images, you'll open and close at least 41 connetions: the first for retrieving the html-part, another 40 connections for every single image (that's over-simplified, about every browser does open a few connections in parallel - yet much fewer than those 40). If the network latency between client and server is high, opening that many connections one after another may result in a noticable delay. By using HTTP keepalive, your browser will open one connection and ask one image after the other via this connection, without setting up new connections and tearing down connections, the website is somehow faster. Additionally, long-living tcp connections do automatically increase their tcp receive window size, which may give you a little bit of extra performance as well. If you decrease your IIS keepalive timeout lower than the limit of your clients, your server is likely the one who'll be initiating closing of idle tcp connections. If your server does this frequently, you'll end up collecting lots of connections in the TIME_WAIT state (well, usually less than without keepalive at all, but depending on the exact setup, this may result in some trouble opening outgoing connections). > And another thing - what is the colum named "Inactive Conenctions" in the > output from ipvsadm used for? Which connections are considered inactive? > > And also in the output of ipvsadm i see a couple of connections with the > state TIME_WAIT. What are these for? TCP connections do have some kind of lifecycle and many states. -setting up a connection works by sending a few packets back and forth, before the connection is known to be established and ready for data transfer. Search "tcp 3way handshake", if you'd like to know more about this. -tearing down a connection also involves sending a few packets back and forth as well. The host initiating the closing tracks this connection as "TIME_WAIT", the other host tracks this connection as "CLOSE_WAIT". In case some late packets arrive for this connection, the host can handle them more appropriately (discard them, don't mistake them for being part of a subsequent connection). And in case the final FIN packet didn't arrive, the WAIT-state still gives that host the idea of temporarily not re-using this connection. Read: if you do have sockets in a TIME_WAIT-state, your host tries not to reuse them immediately for something else, but waits a few minutes before reusing it. If you're closing too many connections and try to open new outgoing connections, you may be out of usable sockets. That problem does rarely occur on systems with either poor software or some poor connection access pattern. In IPVS, only established connections are counted as "active", while "inactive" refers to connections in any other state. A few minutes after TIME_WAIT, the connection will become "closed", its state will be forgotten and won't show up both in ipvsadm nor netstat anymore. Anders -- 1&1 Internet AG Expert Systems Architect (IT Operations) Brauerstrasse 50 v://49.721.91374.0 D-76135 Karlsruhe f://49.721.91374.225 Amtsgericht Montabaur HRB 6484 Vorstände: Henning Ahlert, Ralph Dommermuth, Matthias Ehrlich, Robert Hoffmann, Andreas Hofmann, Markus Huhn, Hans-Henning Kettler, Dr. Oliver Mauss, Jan Oetjen, Martin Witt, Christian Würst Aufsichtsratsvorsitzender: Michael Scheeren _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-requ...@linuxvirtualserver.org or go to http://lists.graemef.net/mailman/listinfo/lvs-users
