RE: [EXTERNAL] Re: Tuning suggestions for high-core-count Linux servers

2017-06-04 Thread Browne, Stuart
Ugh, let me try that again (apologies if you got the half-composed version).



> The lab uses Dell R430s running Fedora Core 23 with Intel X710 10GB NICs
> and each populated with a single Xeon E5-2680 v3 2.5 GHz 12-core CPU.

R630 chassis I believe, same NIC's, smaller processor (E5-2650v4@2.2Ghz).



> The only major setting I've found which both helps performance and
> improves consistency is to ensure that each NIC rx/tx queue IRQ is
> assigned to a specific CPU core, with irqbalance disabled.

I've been stopping irqbalance, and have confirmed that the rx/tx queue IRQ's 
aren't jumping around.

> This is with a _single_ dnsperf client, too.  The settings I use are
> -c24 -q82 -T6 -x2048.   However I do use a tweaked version of dnsperf
> which assigns each thread pair (it uses separate threads for rx and tx)
> to its own core.

I didn't think of using -T. *tries that* ..

> You may find the presentation I made at the recent DNS-OARC workshop of
> interest:
> 
> https://indico.dns-oarc.net/event/26/session/3/contribution/18

Reading it now. Many thanks.
 
> You didn't mention precisely which 9.10 series version you're running.
> Note that versions prior to 9.10.4 defaulted to a -U value of ncores/2,
> but investigation showed that on modern systems this was sub-optimal so
> it was changed to ncores-1.  This makes a *very* big difference.

BIND 9.10.4-P8.

> kind regards,
> 
> Ray Bellis
> ISC Research Fellow

Stuart
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users


RE: [EXTERNAL] Re: Tuning suggestions for high-core-count Linux servers

2017-06-01 Thread Browne, Stuart


> -Original Message-
> From: Plhu [mailto:p...@seznam.cz]


> a few simple ideas to your tests:
>  - have you inspected the per-thread CPU? Aren't some of the threads
> overloaded?

I've tested both the auto-calculated values (one thread per available core) and 
explicitly overridden this. NUMA boundaries seem to be where things get wonky.

>  - have you tried to get the statistics from the Bind server using the
>  XML or JSON interface? It may bring you another insight to the errors.


>  - I may have missed the connection count you use for testing - can you
>  post it? More, how may entries do you have in your database? Can you
>  share your named.conf (without any compromising entries)?

I'm testing to flood, so approximately 5 x 400 client count (dnsperf) with a 
500 query backlog per test instance.

Theoretically this should mean up to 4k5 active or back-logged connections (or 
just 2k5 if I read that documentation wrong).

>  - what is your network environment? How many switches/routers are there
>  between your simulator and the Bind server host?

This is a very closed environment. Server-Switch-Server, all 10Git or 25Gbit. 
Verified the switch stats today, capable of 10x what I'm pushing through it 
currently.

>  - is Bind the only running process on the tested server?

As always, there's the rest of the OS helper stuff, but BIND is the only thing 
actively doing anything (beyond the monitoring I'm doing). So no, nothing else 
is drawing massive amounts of either CPU or network resources.

>  - what CPUs is the Bind server being run on?

>From procinfo:
Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz

2 of them.


>  - is there numad running and while trying the taskset, have you
>  selected the CPUs on the same processor? What does numastat show during
>  the test?

I was manually issuing taskset after confirming the CPU allocations:

taskset 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,46,47 
/usr/sbin/named -u named -n 24 -f

This is all of the cores (including HT) on the 2nd socket. There wwas almost no 
performance difference between 12 (just the actual cores, no HT's) and 24 (with 
the HT's).

>  - how many UDP sockets are in use during your test?

See above.

> 
> Curious for the responses.
> 
>   Lukas

Stuart
___
Please visit https://lists.isc.org/mailman/listinfo/bind-users to unsubscribe 
from this list

bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users