On Wed, Jan 29, 2014 at 04:28:26PM +0100, Jens-Christian Fischer wrote: > My colleague Simon found the "option httpclose" in the configuration. > Removing that changes things for html but not for php: > > ab -c 8 -n 5000 "https://example.com/foo.html"; > Requests per second: 238.83 [#/sec] (mean) > Time per request: 33.497 [ms] (mean) > > ab -kc 8 -n 5000 "https://example.com/foo.html"; > Requests per second: 2035.50 [#/sec] (mean) > Time per request: 3.930 [ms] (mean) > > Progress! > > ab -c 8 -n 5000 "https://example.com/foo.php"; > Requests per second: 223.16 [#/sec] (mean) > Time per request: 35.849 [ms] (mean) > > ab -kc 8 -n 5000 "https://example.com/foo.php"; > Requests per second: 223.52 [#/sec] (mean) > Time per request: 35.791 [ms] (mean) > > No progress :(
Ah indeed there are even two (one per frontend). I didn't notice them. Amusingly I'm currently working on making this more obvious for the user. You should check the exact headers returned by the PHP server, it might be possible that it responds in 1.0 without a content length nor transfer-encoding and that prevents client-side keep-alive from being maintained. Sometimes, using "option http-pretend-keepalive" works around the issue for some bogus servers (those which confuse keep-alive with HTTP version). > >> this is infrastructure we are building and providing ourselves: The load > >> balancing VM is on a host that has 9 VMs running (33 virtual CPUs) on a > >> system with 24 real cores. > > > > OK if you're building it yourself, at least you won't be victim of your > > provider's choices nor of noisy neighbours. That said, as you can see, > > making 33 virtual CPUs out of 24 real ones necessarily means you add a > > lot of latency because you need to schedule between them and you can't > > offer less than a timeslice (generally 1ms, but could even be 10). > > of course - we are aware of that. However, most (all) of the VMs have very > very light loads OK. > >> The VM of the load balancer currently is configured with 8 vCPUs and 16GB > >> of RAM. > > > > You can safely remove 6 CPUs that you will probably never use. If you could > > dedicate 2 real cores instead (one for the interrupts, one for haproxy), it > > would be much more efficient because there would be only full time slices. > > changed the VM type. Will see if I can pin it to 2 real CPUs If you can do, then next step will be to pin the network driver's interrupts to one CPU only (echo 1 > /proc/irq/<irq#>/smp_affinity) and haproxy to the other one (taskset -c 1 haproxy...). You can sometimes almost double the performance this way when you're network-bound. > > You can tune whatever you want if you have control over everything, which > > is good. Most cloud users do not have this chance and only have the option > > to leave and try another one which is less overloaded. But with todays > > dedicated hosting prices, it makes no sense to be hosted outside on VMs, > > it's only seeking trouble. > > Well - we are building a cloud service, and the application we are putting on > this Cloud now is a test case to see how things work and scale. (That's one > of the reasons I came here asking for advice - I still need to establish base > line performance for a lot of components and applications in order to tune > the system). Right now I feel like I'm playing "Whack a Mole" but > blindfolded. > > Of course it doesn't help that the application we putting into production in > a few months time seems to be written by people that thought that 3 users on > their system already was a lot. Ah so you had the same developers as almost any haproxy user then :-) > We are looking to scale it to 50'000 users? > The WTF rate / minute is quite high as I try to scale it (the 60ms for the > status.php is just the beginning? I have seen requests that took 390'000 ms - > because the app goes out and updates several thousand files and corresponding > database entries on the main thread?. sigh) My experience in this area always showed me that when you need to scale to that large numbers from such low ones, you waste *much* less time trashing everything and restarting from scratch with skilled developers than trying to optimise to asymptotically double the performance by winning 10% per month of development. > Back to whacking, it is - and thanks for your inputs. Highly appreciated! You're welcome! Willy
