On 29/04/2015 12:56 μμ, Krishna Kumar (Engineering) wrote: > Dear all, > > Sorry, my lab systems were down for many days and I could not get back > on this earlier. After > new systems were allocated, I managed to get all the requested > information with a fresh ru > (Sorry, this is a long mail too!). There are now 4 physical servers, > running Debian 3.2.0-4-amd64, > connected directly to a common switch: > > server1: Run 'ab' in a container, no cpu/memory restriction. > server2: Run haproxy in a container, configured with 4 nginx's, > cpu/memory configured as > shown below. > server3: Run 2 different nginx containers, no cpu/mem restriction. > server4: Run 2 different nginx containers, for a total of 4 nginx, > no cpu/mem restriction. > > The servers have 2 sockets, each with 24 cores. Socket 0 has cores > 0,2,4,..,46 and Socket 1 has > cores 1,3,5,..,47. The NIC (ixgbe) is bound to CPU 0.
It is considered bad thing to bind all queues of NIC to 1 CPU as it
creates a major bottleneck. HAProxy will have to wait for the interrupt
to be processed by a single CPU which is saturated.
> Haproxy is started
> on cpu's:
> 2,4,6,8,10,12,14,16, so that is in the same cache line as the nic (nginx
> is run on different servers
> as explained above). No tuning on nginx servers as the comparison is between
how many workers to run on Nginx?
> 'ab' -> 'nginx' and 'ab' and 'haproxy' -> nginx(s). The cpus are
> "Intel(R) Xeon(R) CPU E5-2670 v3
> @ 2.30GHz". The containers are all configured with 8GB, server having
> 128GB memory.
>
> mpstat and iostat were captured during the test, where the capture
> started after 'ab' started and
> capture ended just before 'ab' finished so as to get "warm" numbers.
>
> ------------------------------------------------------------------------------------------------------------------------
> Request directly to 1 nginx backend server, size=256 bytes:
>
> Command: ab -k -n 100000 -c 1000 <nginx>:80/256
> Requests per second: 69749.02 [#/sec] (mean)
> Transfer rate: 34600.18 [Kbytes/sec] received
> ------------------------------------------------------------------------------------------------------------------------
> Request to haproxy configured with 4 nginx backends (nbproc=4), size=256
> bytes:
>
> Command: ab -k -n 100000 -c 1000 <haproxy>:80/256
> Requests per second: 19071.55 [#/sec] (mean)
> Transfer rate: 9461.28 [Kbytes/sec] received
>
> mpstat (first 4 processors only, rest are almost zero):
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %gnice %idle
> Average: all 0.44 0.00 1.59 0.00 0.00 2.96
> 0.00 0.00 0.00 95.01
> Average: 0 0.25 0.00 0.75 0.00 0.00 98.01
> 0.00 0.00 0.00 1.00
All network interrupts are processed by CPU 0 which is saturated.
You need to spread the queues of NIC to different CPUs. Either use
irqbalancer or the following 'ugly' script which you need to modify a
bit as I have 2 NICs and you have only 1. You also need to adjust the
number of queues, grep eth /proc/interrupts and you will find out how
many you have.
#!/bin/sh
awk '
function get_affinity(cpus) {
split(cpus,list,/,/)
mask=0
for (val in list) {
mask+=lshift(1,list[val])
}
return mask
}
BEGIN {
# Interrupt -> CPU core(s) mapping
map["eth0-q0"]="0"
map["eth0-q1"]="1"
map["eth0-q2"]="2"
map["eth0-q3"]="3"
map["eth0-q4"]="4"
map["eth0-q5"]="5"
map["eth0-q6"]="6"
map["eth0-q7"]="7"
map["eth1-q0"]="12"
map["eth1-q1"]="13"
map["eth1-q2"]="14"
map["eth1-q3"]="15"
map["eth1-q4"]="16"
map["eth1-q5"]="17"
map["eth1-q6"]="18"
map["eth1-q7"]="19"
}
/eth/ {
irq=substr($1,0,length($1)-1)
queue=$NF
printf "%s (%s) -> %s
(%08X)\n",queue,irq,map[queue],get_affinity(map[queue])
system(sprintf("echo %08X >
/proc/irq/%s/smp_affinity\n",get_affinity(map[queue]),irq))
}
' /proc/interrupts
> Average: 1 1.26 0.00 5.28 0.00 0.00 2.51
> 0.00 0.00 0.00 90.95
> Average: 2 2.76 0.00 8.79 0.00 0.00 5.78
> 0.00 0.00 0.00 82.66
> Average: 3 1.51 0.00 6.78 0.00 0.00 3.02
> 0.00 0.00 0.00 88.69
>
> pidstat:
> Average: 105 471 5.00 33.50 0.00 38.50 - haproxy
> Average: 105 472 6.50 44.00 0.00 50.50 - haproxy
> Average: 105 473 8.50 40.00 0.00 48.50 - haproxy
> Average: 105 475 2.50 14.00 0.00 16.50 - haproxy
> ------------------------------------------------------------------------------------------------------------------------
> Request directly to 1 nginx backend server, size=64K
>
I would like to see pidstat and mpstat while you test nginx.
Cheers,
Pavlos
signature.asc
Description: OpenPGP digital signature
