On 03/28/2015 02:16 AM, Michał Purzyński wrote:
So far so good. Thanks for the idea, both vm.dirty_ratio and
vm.dirty_background_ratio are lowered and I can see much more frequent
disk writes of much smaller chunks, quite close to have it running all
the time.
I also made some tuning and experiments with the system and
netsniff-ng and would like to share results
1. booted system with
audit=1 processor.max_cstate=0 intel_idle.max_cstate=0 selinux=0
nohz_full=1-99 isolcpus=1-99
(note - max cstate 0 is in fact 1, look in the kernel source)
2. ixgbe from intel
modprobe ixgbe MQ=0 DCA=1 RSS=0 InterruptThrottleRate=956 LRO=0
3. move interrupts from netsniff cores
for D in $(ls /proc/irq)
do
if [[ -x "/proc/irq/$D" && $D != "0" ]]
then
echo $D
echo 1 > /proc/irq/$D/smp_affinity
fi
done
4. pin the capture card irq to core 2
echo 2 > /proc/irq/137/smp_affinity
echo 2 > /proc/irq/138/smp_affinity
5. move other processes including some kernel threads off the cores 1-24
for i in `pgrep rcu` ; do taskset -pc 0 $i ; done
echo 1 > /sys/bus/workqueue/devices/writeback/cpumask
echo 0 > /sys/bus/workqueue/devices/writeback/numa
6. started netsniff-ng from supervisord
netsniff-ng -i <card> -o <path to your pcaps> --user <id> --group <id>
-s --prefix <your prefix> --ring-size 4000MiB --interval 60GiB -Q -b 2
--mmap --filter <path to your filter>
-Q means do not touch the carefuly hand crafted IRQ affinity
-b 2 is obviously bind to core 2
So far so good, traffic up to 4.5Gbit/sec, lots of it filtered out.
Thanks for the report and details!
Can you elaborate on how much you can actually capture?
Have you tuned socket rmem/wmem so you have enough capacities?
Have you tried a smaller ring size e.g. very few MB to reduce
pressure from MMU? Did you see an improvement for SG I/O on the
writeback?
There's also a tool called *ifpps* in the netsniff-ng repository,
might be helpful as well.
How many instances of netsniff-ng are you running? Is this a
middle box or rather server? If the latter, you could let the
NIC do flow steering (e.g. aRFS) and have e.g. 2 or 4 instances
of netsniff-ng filtering in BPF by queue mapping.
Alternatively, if all else doesn't help, you could also sample
flows in BPF, so you get a portion of the traffic for analysis.
Early results with 'perf' and performance counters indicate that
packets make it to the card, get written into cache (I think DCA
writes directly into L1, so on Nehalem and later that's L3, too) of
core 1 and then core 2 picks that up from L3 and writes to disk.
So far 0 packet loss.
Comments, questions? I am doing everything wrong?
On Fri, Mar 27, 2015 at 3:35 PM, Mike Reeves <[email protected]> wrote:
One thing you can do is make sure you tune your disk writes. I have noticed
that I get the best results when pdflush is almost constantly running. There
is a ton of info out there about what you can tweak but take a look at
vm.dirty_ratio and vm.dirty_background_ratio.
You have to play with these settings until you find what works best for your
system.
Thanks
Mike
On Mar 27, 2015, at 7:47 AM, Michał Purzyński <[email protected]>
wrote:
Hello.
I went through a massive number of documents regarding the netsniff itself,
including reading code of both the application, libpcap and kernel parts,
intel manuals for X520 and CPUs. Enough of reading, let's do something.
How to tune netsniff-ng for the maximum performance? Question seems trivial,
answer is not so. Feel free to correct me. X520 and a few gbit/sec and a
NUMA (Sandy Bridge) system is what I have here.
For the best cache utilisation, netsniff-ng should be on the same core that
card delivers packets to. With IO/AT and DCA we could have something like
this:
-- packets --> card --> cpu cache --> netsniff-ng
It makes sense with DCA, actually.
Now, does it mean we want RSS disabled and a single queue pinned to a single
core?
Won't the number of interrupts (even using interrupt mitigation that's in
the ixgbe driver) kill the single core?
Alternatively (I have only 2.0Ghz CPU and that's not enough to keep up with
the load from interrupts and netsniff-ng together on the same core) -
interrupts on core1, netsniff-ng on core2 (we're talking physical cores from
the same socket).
This will generate L1 and L2 miss most of the time but we still have L3
that's inclusive (and huge). Win?
With the kernel parameters "nohz_full=1-23 isolcpus=1-23" scheduler will
keep other tasks off the cores (1-23) by setting affinity on the init
process and disable ticks on them if there is a single task running (other
kernel threads can be set to run on core 0).
What gives the best write performance - sg or mmap?
--
You received this message because you are subscribed to the Google Groups
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
