On Sat, Oct 27, 2012 at 9:02 AM, Willy Tarreau <[email protected]> wrote: > On Wed, Oct 24, 2012 at 04:54:57PM +0200, Finn Arne Gangstad wrote: >> On Tue, Oct 23, 2012 at 4:02 PM, Mariusz Gronczewski <[email protected]> >> wrote: >> > 2012/10/23 Thomas Heil <[email protected]>: >> >> Hi, >> >> >> >> On 23.10.2012 13:55, Finn Arne Gangstad wrote: >> >>> >> >>> Each request is a reasonably simple GET request that typically takes >> >>> 10-20ms to process. This works great until a server needs to GC, then >> >>> the query will hang for a few seconds. >> >> Iam not quit sure, but I think you can play with timeout server and >> >> option redispatch and retries, so that when GC occours the request would >> >> be >> >> redispatched to the next server in the backend. >> >> >> > Try using "balance leastconn", if server will slow down/halt because >> > of GC his queue will quickly be higher than rest of servers and new >> > request will hit non-GCing ones, only disadvantage is that servers >> > which respond faster will on average get more requests but that can be >> > a good thing, if for any reason (backup, system update etc.) one of >> > servers will start answering slower it will automatically get less >> > requests. >> >> leastconn helps slighty in this particular situation, we'd lose maybe >> 6-7 queries >> instead of 10 (depending a bit on the load), but we still lose queries and we >> don't want to lose any queries at all. Any query that takes more than a >> second >> or two is effectively lost. > > Well, here you have a tuning problem then. If your GC lasts longer than the > maximum critical response time, then you need to correctly tune your JVM to > ensure GCs are more common and take less time.
Tuning the GC to give very small pauses is a possibility, but not the only valid solution. We have found that using a parallel stop-the-world GC will increase the overall throughput with about 30% (e.g. -XX:+UseNUMA -XX:+UseParallelGC), but this gives unpredictable multi-second GC pauses. As long as we distribute the load over 5+ servers, there are always enough servers that are responsive, and we can reduce the number of servers by roughly 25%. > >> haproxy doesn't currently support resubmitting a query, but it would be very >> nice if it could do something along the nginx feature "proxy_next_upstream". >> nginx lets you resubmit a query until you have started sending data back >> to the client, haproxy only lets you resubmit until a connection to the >> backend server has been established. > > No, believe me, this must *absolutely not* be done. HTTP provides no way to > abort a request that was started, nor to know whether a request has been > completed. Doing so is explicitly forbidden in the HTTP spec for a good > reason. What you describe caused a coworker to receive two books he ordered > online (and obviously he paid twice). Only the client is allowed to decide > whether or not to replay a non-idempotent request. As a general rule you are correct of course, but we have our services split into many different categories. For some of them request duplication is not so good, but most of them are idempotent. We also use haproxy a lot between internal applications for providing robust load distribution and failover. In that case we are in control of both clients and servers, and we don't have to worry too much about HTTP guarantees. It would be good however to get the failover-and-retry functionality done right in one place, and haproxy could be such a place. > > Also, I know about a very large web site which was using the weblogic apache > plugin for load balancing. It was doing exactly what you describe here. The > end result is that when things start to go wrong, you have a domino effect > because the killer request is sent to all servers in turn. Very commonly, > the bottleneck aren't the frontend web servers, but the backend database. > By replaying a request, you really multiply the load on this database by > as much, and you redistribute that extra response time to all the servers, > which at one point might all experience too long a response time. A complicated/costly request is a weakness if you do invividual request timeouts, but less so if you redispatch on total silence from a server (if all pending requests and the monitoring query fails to generate activity for a while). > > So really, you need to tune the GC. Pausing several seconds is not acceptable > in my opinion. I work with people who use a lot of Java applications, and > I've seen them spend as much time on tuning the JVM as they spend writing > the code, and the result is really worth it. In your case, maybe a 50ms > pause every 10s will remain unnoticed for example. We have tried different GC strategies. I realize we can make the system work for some definition of work by reducing throughput and going with a suboptimal GC strategy, at the cost of additional servers. Would you strongly object to a patch that added a feature along these lines: If a server goes silent for X ms, immediately send a monitoring query. If the monitoring query is not handled within Y ms, flag the server as "stopped", redispatch all pending requests on the server. "stopped" flag is removed as soon as the server responds to any pending request or a monitoring query. - Finn Arne
