Hi Konrad,
On Mon, Jul 12, 2010 at 07:49:25PM +0200, Konrad wrote:
> The networking is slow - you're correct. I'll check it why.
>
> But generally I don't know why the stats using "ab" tool can be so low.
> If I add additional server with apache there is hardly any performance
> increase. This makes me worry.
> I got 3 apache servers on every - remotely tested - about 3-4k req/s.
> And with HAproxy about 2k req/s.
but as I understand it, in the first case (apache only), your test is :
ab --[network]--> apache
In the second case, your test is :
ab --[network]--> haproxy --[network]--> apache
So if you pass twice over the network and it's the bottleneck, it's
perfectly expected that the performance cuts in half.
> I made also a configuration with virtualbox just for test (1 HA, 3 vm's
> with apache). Situation is similar.
> To every vm there is about 600 req/s. And using HAproxy about 400 req/s.
Huh, the numbers are even 5 times lower ! Was that on the same physical
machine ? If so, that should ring some bell about the host's tuning. I
think the last time I've seen that low numbers, it was on a Pentium 90.
And even in a VM I don't recall that low numbers.
> Right now I'm not sure but I'll also check the load on cpu on vm's.
The CPU on the host is more important. The one in the VM can report
anything between 0 and 100% depending on what the host lets it really
do.
> Generally I tried different configurations of haproxy.conf. But as you
> can confirm the effect is not good.
> Could you please suggest me what - more or less - effect should I expect
> (in the upper connection schema - 3-4k req/s to 2k req/s)?
>From memories, haproxy under lguest on my Atom 1.6 GHz does slightly more
than 4000 req/s. So any decent machine with hardware virtualization should
do quite more.
There are some important tests you should do :
- move ab to the VMs running apache and launch it locally to check the
difference.
- repeat test above on all apache VMs simultaneously to see how the
numbers go. Do they add or do they cumulate to the same level ? That
will indicate the impact on the CPU of running apache in those VMs.
- move ab to the VM running haproxy, and start ab directly from there
to one apache. Note the numbers and check CPU usage on the haproxy VM.
- run ab from the haproxy VM to the local haproxy, configured to address
the same apache as above. Note the numbers. Also check how CPU spreads
between ab and haproxy on the same machine. If either uses a full CPU
or if both add to 100 or more percent, you'll have to do some math to
see if what you observe is correct versus previous test.
- move ab to another VM and address haproxy (1 server still). Check all
numbers (including host's CPU usage).
- do the same again, but repeat test #1 on the two unused apache VMs to
see if the CPU usage of unused VMs has any impact on the global load.
- do the same again, but use ab from VM1 to address apache in VM2 and
conversely, to see if the network usage of unused VMs has any impact
on the global load.
I really suspect that the issue is making the packets pass between VMs
multiple times, which adds up network traffic that the host cannot handle
between VMs for whatever reasons.
Hoping this helps,
Willy
