> The thing that confuses me though is the following. I do iperf from A to B > using ipsec in the middle (two encryption boxes I'm testing). Now I trace on > the receiving box doing decryption and the softirq is really high for TX on > the > interface handling the TCP reply! So the high interrupt rate is not from the > inbound (bulk) TCP stream, but purely from the ACKs which are going the > way back. Is that expected? I'm not sure that it's only the ACK's. With the overruns happening there are probably a lot of retransmissions happening as well. So I bet if you sniff the traffic you'll see lots of other packets and not just the ACK's.
Cheers, John > -----Original Message----- > From: Anna Fischer [mailto:a.fisc...@sirrix.com] > Sent: Monday, February 29, 2016 7:31 AM > To: e1000-devel@lists.sourceforge.net > Subject: Re: [E1000-devel] Poor IPsec performance and high ksoftirqd load > > > > Am 29.02.2016 um 16:15 schrieb Anna Fischer: > > > > > > Am 26.02.2016 um 17:31 schrieb Ronciak, John: > >> Hi Anna, > >> > >>> I have tried with multiple TCP streams as well but the performance > >>> drops down even more already with two parallel streams. I use iperf3 > >>> for testing on the client side, and then I have some scripts on my > >>> IPSec box to monitor packet rates and bandwidth as well. > >> So are you running out of CPU cycles or are packets being dropped > causing retransmitted packets? Use netstat and ethtool to look for dropped > packets. > > > > I do not see any packet drops on the NIC but I see "overruns" where > > the kernel cannot handle the packet load and the buffers on the NIC > overrun. > > > > I can see that ksoftirqd is running at 100% on the CPU, so I guess I > > am running out of CPU cycles. > > The thing that confuses me though is the following. I do iperf from A to B > using ipsec in the middle (two encryption boxes I'm testing). Now I trace on > the receiving box doing decryption and the softirq is really high for TX on > the > interface handling the TCP reply! So the high interrupt rate is not from the > inbound (bulk) TCP stream, but purely from the ACKs which are going the > way back. Is that expected? > > Cheers, > Anna > > > > > Anna > > > >> > >> Cheers, > >> John > >> > >> > >>> -----Original Message----- > >>> From: Anna Fischer [mailto:a.fisc...@sirrix.com] > >>> Sent: Friday, February 26, 2016 12:10 AM > >>> To: Ronciak, John <john.ronc...@intel.com>; e1000- > >>> de...@lists.sourceforge.net > >>> Subject: Re: Poor IPsec performance and high ksoftirqd load > >>> > >>> > >>> > >>> Am 25.02.2016 um 20:40 schrieb Ronciak, John: > >>>>> -----Original Message----- > >>>>> From: Anna Fischer [mailto:a.fisc...@sirrix.com] > >>>>> Sent: Thursday, February 25, 2016 10:53 AM > >>>>> To: e1000-devel@lists.sourceforge.net > >>>>> Subject: [E1000-devel] Poor IPsec performance and high ksoftirqd > >>>>> load > >>>>> > >>>>> Hi, > >>>>> > >>>>> I'm running Linux 4.3.3 (Gentoo) and standard igb drivers. I have > >>>>> trouble with the performance of IPSec on my platform. The platform > >>>>> has Intel Ethernet Controller I211 (1GbE) and Intel Atom CPUs > >>>>> (4-core) from the C2000 product family. > >>>>> > >>>>> My platform only seems to be able to do ~300Mbit/s when receiving > >>>>> IPSec packets. When sending IPSec packets it can do ~600Mbit/s. > >>>>> The weird thing is, the ksoftirqd handler seems to run at very > >>>>> high CPU load. I don't understand where that comes from. I have > >>>>> traced using kernel function tracing and I can see that it is > >>>>> taking a lot of time do handle do_softirq and functions like > >>>>> igb_poll(). I was wondering if it was normal that this takes up so > >>>>> much CPU? Also it seems that this is > >>> only the case in the following scenario: > >>>>> > >>>>> The platform receives IPSec packets on eth0, then decrypts them, > >>>>> and routes them out via eth1 (plaintext). The packet stream is > >>>>> TCP. So for each amount of packets going this way, there is an TCP > >>>>> ACK packet going back the other way. E.g. plaintext incoming via > >>>>> eth1, IPSec > >>> encryption and TX out via eth0. > >>>>> > >>>>> For me this sounds like a very normal use case of IPSec, but still > >>>>> the platform does not seem to be able to handle it. As my CPU load > >>>>> is high, but not maxed out, the CPU does not seem to be the > bottleneck. > >>>>> So my guess is it might also be the NIC. I do not have any > >>>>> configuration options in my igb driver, and I already tried a few > >>>>> option > >>> with ethtool, but I don't get any improvements. > >>>>> Does anyone have a good idea on how to track what is slowing down > >>>>> the system? > >>>>> > >>>>> Many thanks, > >>>>> Anna > >>>>> > >>>>> > >>>> Hi Anna, > >>>> > >>>> So the i211 is a very low end client Ethernet device (PCIe x1 2with > >>>> limited > >>> buffer space). So it's nowhere near as performant as the server devices. > >>> That said, I'm not sure what you are using to test this with. > >>> Testing with a single TCP stream can always be problematic when > >>> looking at wire bandwidth. Do things change if you start multiple > >>> TCP strings? It will use more CPU as the packets need to be > >>> decrypted but you should see an increase in throughput, at least > >>> until you run out of CPU cycles. The single stream TCP throughput > >>> is well known to be an issue as any single TCP connection can only > >>> have so much data outstanding before being ACK'd. So using multiple > >>> TPC connections changes that and should increase your overall > throughput. Please give that a try. > >>>> > >>>> Thanks. > >>>> > >>>> Cheers, > >>>> John > >>>> > >>> > >>> I have tried with multiple TCP streams as well but the performance > >>> drops down even more already with two parallel streams. I use iperf3 > >>> for testing on the client side, and then I have some scripts on my > >>> IPSec box to monitor packet rates and bandwidth as well. > >>> > >>> Anna > > > > -- > Sirrix AG security technologies - http://www.sirrix.com > Dr. Anna Fischer eMail: a.fisc...@sirrix.com > Tel +49(681) 959 86-139 Mobile +49(176) 47928927 > PGP Key ID 0x96E0708B > > Vorstand: Ammar Alkassar (Vors.), Christian Stüble Vorsitzender des > Aufsichtsrates: Peter Riedel Sitz der Gesellschaft: Homburg/Saar, HRB 3857 > Amtsgericht Saarbrücken > > This message may contain confidential and/or privileged information. If you > are not the addressee, you must not use, copy, disclose or take any action > based on this message or any information herein. If you have received this > message in error, please advise the sender immediately by reply e-mail and > delete this message. > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance APM > + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor > end-to-end web transactions and take corrective actions now Troubleshoot > faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > E1000-devel mailing list > E1000-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
