Overall looks like this configuration has no problem writing over 6
Gbit/sec and can jump to 8-10 Gbit/sec with a minor packet loss. Now
remember it's just a 2.0Ghz.
Enabled RPS by doing
cd /sys/class/net/eth2/queues/rx-0
echo 8 > rps_cpus
So that packets are ACK'ed on core 1, processed on core 3 and written
on core 2. I'm not afraid of the caching to ruin my dream, L3 on sandy
bridge is inclusive.
Instant effect of 10x packet loss in ifpps/ifconfig statistics with
traffic constantly above 4.5Gbit/sec with frequent and long spikes to
around 8-9Gbit/sec
Now only to tune the disk subsystem, because it started choking.
Core 0 - doing system level work, it has every process and every
kernel thread pinned to it (except for what is below)
nsm1 :: ~ » perf stat -e
L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,iTLB-loads,iTLB-load-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,stalled-cycles-frontend,stalled-cycles-backend,cpu-cycles,instructions
-C 0
^C
Performance counter stats for 'CPU(s) 0':
55,025,567,915 L1-dcache-loads
[27.27%]
2,196,598,080 L1-dcache-load-misses # 3.99% of all
L1-dcache hits [27.27%]
34,703,865,975 L1-dcache-stores
[27.27%]
686,894,785 L1-dcache-store-misses
[27.27%]
588,116,260 L1-dcache-prefetch-misses
[18.18%]
891,250,368 L1-icache-load-misses
[18.18%]
55,318,537,337 dTLB-loads
[18.18%]
364,857,560 dTLB-load-misses # 0.66% of all dTLB
cache hits [18.18%]
34,731,794,924 dTLB-stores
[18.18%]
48,623,748 dTLB-store-misses
[18.18%]
9,414,057 iTLB-loads
[18.18%]
9,621,508 iTLB-load-misses # 102.20% of all iTLB
cache hits [18.18%]
843,647,327 LLC-loads
[18.18%]
566,128,176 LLC-load-misses # 67.10% of all
LL-cache hits [18.18%]
202,846,495 LLC-stores
[ 9.09%]
57,555,948 LLC-store-misses
[ 9.09%]
697,626,442 LLC-prefetches
[ 9.09%]
436,674,721 LLC-prefetch-misses
[ 9.09%]
137,707,645,647 stalled-cycles-frontend # 61.45% frontend
cycles idle [13.63%]
101,026,169,008 stalled-cycles-backend # 45.08% backend
cycles idle [18.18%]
224,084,448,825 cpu-cycles [22.72%]
218,496,348,702 instructions # 0.98 insns per cycle
# 0.63 stalled cycles
per insn [27.27%]
188.058735333 seconds time elapsed
Core 1 - doing IRQ ACK (without the software IRQ)
nsm1 :: ~ » perf stat -e
L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,iTLB-loads,iTLB-load-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,stalled-cycles-frontend,stalled-cycles-backend,cpu-cycles,instructions
-C 1
^C
Performance counter stats for 'CPU(s) 1':
78,057,035,645 L1-dcache-loads
[27.26%]
4,454,918,769 L1-dcache-load-misses # 5.71% of all
L1-dcache hits [27.27%]
53,811,910,715 L1-dcache-stores
[27.27%]
2,125,376,211 L1-dcache-store-misses
[27.27%]
1,419,443,049 L1-dcache-prefetch-misses
[18.18%]
1,717,614,461 L1-icache-load-misses
[18.18%]
78,183,469,353 dTLB-loads
[18.18%]
138,910,074 dTLB-load-misses # 0.18% of all dTLB
cache hits [18.18%]
53,746,818,686 dTLB-stores
[18.18%]
20,372,871 dTLB-store-misses
[18.18%]
58,735,482 iTLB-loads
[18.18%]
51,692,750 iTLB-load-misses # 88.01% of all iTLB
cache hits [18.18%]
1,950,101,712 LLC-loads
[18.18%]
797,955,820 LLC-load-misses # 40.92% of all
LL-cache hits [18.19%]
938,099,866 LLC-stores
[ 9.09%]
65,799,139 LLC-store-misses
[ 9.09%]
1,711,222,748 LLC-prefetches
[ 9.09%]
780,061,723 LLC-prefetch-misses
[ 9.09%]
202,194,514,806 stalled-cycles-frontend # 64.13% frontend
cycles idle [13.63%]
158,965,673,867 stalled-cycles-backend # 50.42% backend
cycles idle [18.17%]
315,271,440,989 cpu-cycles [22.72%]
259,705,745,673 instructions # 0.82 insns per cycle
# 0.78 stalled cycles
per insn [27.26%]
282.624443984 seconds time elapsed
Core 2 - netsniff-ng
nsm1 :: ~ » perf stat -e
L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,iTLB-loads,iTLB-load-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,stalled-cycles-frontend,stalled-cycles-backend,cpu-cycles,instructions
-C 2
^C
Performance counter stats for 'CPU(s) 2':
52,794,266,975 L1-dcache-loads
[27.26%]
6,130,111,113 L1-dcache-load-misses # 11.61% of all
L1-dcache hits [27.27%]
37,736,708,312 L1-dcache-stores
[27.27%]
2,677,627,657 L1-dcache-store-misses
[27.27%]
2,633,332,874 L1-dcache-prefetch-misses
[18.19%]
1,054,638,467 L1-icache-load-misses
[18.19%]
52,900,667,535 dTLB-loads
[18.19%]
240,165,561 dTLB-load-misses # 0.45% of all dTLB
cache hits [18.19%]
37,791,942,529 dTLB-stores
[18.19%]
144,209,002 dTLB-store-misses
[18.19%]
3,622,626 iTLB-loads
[18.19%]
13,438,235 iTLB-load-misses # 370.95% of all iTLB
cache hits [18.18%]
3,034,756,954 LLC-loads
[18.18%]
585,953,807 LLC-load-misses # 19.31% of all
LL-cache hits [18.18%]
2,694,340,367 LLC-stores
[ 9.09%]
1,154,031,604 LLC-store-misses
[ 9.09%]
2,594,543,444 LLC-prefetches
[ 9.09%]
519,435,539 LLC-prefetch-misses
[ 9.09%]
139,893,734,126 stalled-cycles-frontend # 64.94% frontend
cycles idle [13.63%]
100,973,469,345 stalled-cycles-backend # 46.88% backend
cycles idle [18.17%]
215,407,185,242 cpu-cycles [22.72%]
166,023,453,084 instructions # 0.77 insns per cycle
# 0.84 stalled cycles
per insn [27.26%]
121.128926406 seconds time elapsed
Core 3 - used with RPS
nsm1 :: ~ » perf stat -e
L1-dcache-loads,L1-dcache-load-misses,L1-dcache-stores,L1-dcache-store-misses,L1-dcache-prefetch-misses,L1-icache-load-misses,dTLB-loads,dTLB-load-misses,dTLB-stores,dTLB-store-misses,iTLB-loads,iTLB-load-misses,LLC-loads,LLC-load-misses,LLC-stores,LLC-store-misses,LLC-prefetches,LLC-prefetch-misses,stalled-cycles-frontend,stalled-cycles-backend,cpu-cycles,instructions
-C 3
^C
Performance counter stats for 'CPU(s) 3':
36,022,430,006 L1-dcache-loads
[27.27%]
3,187,109,569 L1-dcache-load-misses # 8.85% of all
L1-dcache hits [27.27%]
23,578,159,321 L1-dcache-stores
[27.27%]
1,367,615,110 L1-dcache-store-misses
[27.27%]
1,139,870,926 L1-dcache-prefetch-misses
[18.18%]
210,316,835 L1-icache-load-misses
[18.18%]
35,994,523,087 dTLB-loads
[18.18%]
95,732,304 dTLB-load-misses # 0.27% of all dTLB
cache hits [18.18%]
23,563,854,293 dTLB-stores
[18.18%]
10,175,312 dTLB-store-misses
[18.18%]
4,639,403 iTLB-loads
[18.18%]
9,947,803 iTLB-load-misses # 214.42% of all iTLB
cache hits [18.18%]
1,570,510,882 LLC-loads
[18.18%]
635,977,799 LLC-load-misses # 40.49% of all
LL-cache hits [18.18%]
1,139,476,381 LLC-stores
[ 9.09%]
709,371,438 LLC-store-misses
[ 9.09%]
1,462,444,758 LLC-prefetches
[ 9.09%]
741,338,556 LLC-prefetch-misses
[ 9.09%]
115,600,249,660 stalled-cycles-frontend # 70.17% frontend
cycles idle [13.64%]
91,550,716,430 stalled-cycles-backend # 55.57% backend
cycles idle [18.18%]
164,754,821,037 cpu-cycles [22.72%]
99,910,386,178 instructions # 0.61 insns per cycle
# 1.16 stalled cycles
per insn [27.27%]
117.477821356 seconds time elapsed
Overall the results are interesting.
Average over 5Gbit/sec over the last half hour with frequent spikes 8-10.
On Tue, Mar 31, 2015 at 3:24 PM, Michał Purzyński
<[email protected]> wrote:
> Sorry for the delay, I wanted to get the week started so I can return
> to a production level of traffic :-)
>
> On average, I can see between 4 - 5 Gbit/sec and netsniff does not
> report any loss in the log now. There is around 3% loss in the NIC
> statistics, visible in both ifpps and ifconfig.
>
>>
>> Thanks for the report and details!
>>
>> Can you elaborate on how much you can actually capture?
>
> Between 4-5 Gbit/sec
>
>>
>> Have you tuned socket rmem/wmem so you have enough capacities?
>
> Not at all - how do I calculate the optimal ratio? Does not netsniff
> do it automatically?
>
>>
>> Have you tried a smaller ring size e.g. very few MB to reduce
>> pressure from MMU? Did you see an improvement for SG I/O on the
>> writeback?
>
> I will test smaller ring size in a day - so far a quick experiment
> with 4MB ring gave me packet loss.
>
> SG I/O is indeed better - no packet loss when writing to disk and
> lower CPU usage, I would say at least 15% of not better.
>
>>
>> There's also a tool called *ifpps* in the netsniff-ng repository,
>> might be helpful as well.
>
> Excellent tool, scales well ,does not lie (hello, iptraf ;-) and shows
> a lot of useful statistics. Thanks.
>
>>
>> How many instances of netsniff-ng are you running? Is this a
>> middle box or rather server? If the latter, you could let the
>> NIC do flow steering (e.g. aRFS) and have e.g. 2 or 4 instances
>> of netsniff-ng filtering in BPF by queue mapping.
>
> I have a single netsniff instance so far. This is a real server with
>
> 2x Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
> 64GB RAM (each channel got 8GB DIMM)
> 12x 3TB SATA 7.2k rpm in RAID5 (hardware)
>
> The storage subsystem is far from being saturated, I could write
> easily 1.2GB/sec.
>
> Anything special I need to do to enable RFS? I've just started reading
> about it and understand the idea behind RPS, just need a hint how to
> use it with netsniff.
>
>>
>> Alternatively, if all else doesn't help, you could also sample
>> flows in BPF, so you get a portion of the traffic for analysis.
>>
>>
>>> Early results with 'perf' and performance counters indicate that
>>> packets make it to the card, get written into cache (I think DCA
>>> writes directly into L1, so on Nehalem and later that's L3, too) of
>>> core 1 and then core 2 picks that up from L3 and writes to disk.
>>>
>>> So far 0 packet loss.
>>>
>>> Comments, questions? I am doing everything wrong?
>>>
>>>
>>>
>>> On Fri, Mar 27, 2015 at 3:35 PM, Mike Reeves <[email protected]> wrote:
>>>>
>>>> One thing you can do is make sure you tune your disk writes. I have
>>>> noticed
>>>> that I get the best results when pdflush is almost constantly running.
>>>> There
>>>> is a ton of info out there about what you can tweak but take a look at
>>>> vm.dirty_ratio and vm.dirty_background_ratio.
>>>>
>>>> You have to play with these settings until you find what works best for
>>>> your
>>>> system.
>>>>
>>>> Thanks
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>> On Mar 27, 2015, at 7:47 AM, Michał Purzyński
>>>> <[email protected]>
>>>> wrote:
>>>>
>>>> Hello.
>>>>
>>>> I went through a massive number of documents regarding the netsniff
>>>> itself,
>>>> including reading code of both the application, libpcap and kernel parts,
>>>> intel manuals for X520 and CPUs. Enough of reading, let's do something.
>>>>
>>>> How to tune netsniff-ng for the maximum performance? Question seems
>>>> trivial,
>>>> answer is not so. Feel free to correct me. X520 and a few gbit/sec and a
>>>> NUMA (Sandy Bridge) system is what I have here.
>>>>
>>>> For the best cache utilisation, netsniff-ng should be on the same core
>>>> that
>>>> card delivers packets to. With IO/AT and DCA we could have something like
>>>> this:
>>>>
>>>> -- packets --> card --> cpu cache --> netsniff-ng
>>>>
>>>> It makes sense with DCA, actually.
>>>>
>>>> Now, does it mean we want RSS disabled and a single queue pinned to a
>>>> single
>>>> core?
>>>>
>>>> Won't the number of interrupts (even using interrupt mitigation that's in
>>>> the ixgbe driver) kill the single core?
>>>>
>>>> Alternatively (I have only 2.0Ghz CPU and that's not enough to keep up
>>>> with
>>>> the load from interrupts and netsniff-ng together on the same core) -
>>>> interrupts on core1, netsniff-ng on core2 (we're talking physical cores
>>>> from
>>>> the same socket).
>>>> This will generate L1 and L2 miss most of the time but we still have L3
>>>> that's inclusive (and huge). Win?
>>>>
>>>> With the kernel parameters "nohz_full=1-23 isolcpus=1-23" scheduler will
>>>> keep other tasks off the cores (1-23) by setting affinity on the init
>>>> process and disable ticks on them if there is a single task running
>>>> (other
>>>> kernel threads can be set to run on core 0).
>>>>
>>>> What gives the best write performance - sg or mmap?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "netsniff-ng" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "netsniff-ng" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"netsniff-ng" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
