Hi Michael, On Thu, Mar 05, 2009 at 01:04:06PM -0800, Michael Fortson wrote: > I'm trying to understand why our proxied requests have a much greater > chance of significant delay than non-proxied requests. > > The server is an 8-core (dual quad) Intel machine. Making requests > directly to the nginx backend is just far more reliable. Here's a > shell script output that just continuously requests a blank 0k image > file from nginx directly on its own port, and spits out a timestamp if > the delay isn't 0 or 1 seconds: > > Thu Mar 5 12:36:17 PST 2009 > beginning continuous test of nginx port 8080 > Thu Mar 5 12:38:06 PST 2009 > Nginx Time is 2 seconds > > > > Here's the same test running through haproxy, simultaneously: > > Thu Mar 5 12:36:27 PST 2009 > beginning continuous test of haproxy port 80 > Thu Mar 5 12:39:39 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:39:48 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:39:55 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:40:03 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:40:45 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:40:48 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:40:55 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:40:58 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:41:55 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:42:01 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:42:08 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:42:29 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:42:38 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:43:05 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:43:15 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:08 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:25 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:30 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:33 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:39 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:46 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:44:54 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:45:07 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:45:16 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:45:45 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:45:54 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:45:58 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:05 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:08 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:32 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:48 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:53 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:46:58 PST 2009 > Nginx Time is 3 seconds > Thu Mar 5 12:47:40 PST 2009 > Nginx Time is 3 seconds
3 seconds is typically a TCP retransmit. You have network losses somewhere from/to your haproxy. Would you happen to be running on a gigabit port connected to a 100 Mbps switch ? What type of NIC is this ? I've seen many problems with broadcom netxtreme 2 (bnx2) caused by buggy firmwares, but it seems to work fine for other people after a firmware upgrade. > My sanitized haproxy config is here (mongrel backend was omitted for brevity) > : > http://pastie.org/408729 > > Are the ACLs just too expensive? Not at all. Especially in your case. To reach 3 seconds of latency, you would need hundreds of thousands of ACLs, so this is clearly unrelated to your config. > Nginx is running with 4 processes, and the box shows mostly idle. ... which indicates that you aren't burning CPU cycles processing ACLs ;-) It is also possible that some TCP settings are too low for your load, but I don't know what your load is. Above a few hundreds-thousands of sessions per second, you will need to do some tuning, otherwise you can end up with similar situations. Regards, Willy
