Am 29.02.2016 um 16:15 schrieb Anna Fischer: > > > Am 26.02.2016 um 17:31 schrieb Ronciak, John: >> Hi Anna, >> >>> I have tried with multiple TCP streams as well but the performance drops >>> down even more already with two parallel streams. I use iperf3 for testing >>> on >>> the client side, and then I have some scripts on my IPSec box to monitor >>> packet rates and bandwidth as well. >> So are you running out of CPU cycles or are packets being dropped causing >> retransmitted packets? Use netstat and ethtool to look for dropped packets. > > I do not see any packet drops on the NIC but I see "overruns" where the > kernel cannot handle the packet load and the buffers on the NIC overrun. > > I can see that ksoftirqd is running at 100% on the CPU, so I guess I am > running out of CPU cycles.
The thing that confuses me though is the following. I do iperf from A to B using ipsec in the middle (two encryption boxes I'm testing). Now I trace on the receiving box doing decryption and the softirq is really high for TX on the interface handling the TCP reply! So the high interrupt rate is not from the inbound (bulk) TCP stream, but purely from the ACKs which are going the way back. Is that expected? Cheers, Anna > > Anna > >> >> Cheers, >> John >> >> >>> -----Original Message----- >>> From: Anna Fischer [mailto:a.fisc...@sirrix.com] >>> Sent: Friday, February 26, 2016 12:10 AM >>> To: Ronciak, John <john.ronc...@intel.com>; e1000- >>> de...@lists.sourceforge.net >>> Subject: Re: Poor IPsec performance and high ksoftirqd load >>> >>> >>> >>> Am 25.02.2016 um 20:40 schrieb Ronciak, John: >>>>> -----Original Message----- >>>>> From: Anna Fischer [mailto:a.fisc...@sirrix.com] >>>>> Sent: Thursday, February 25, 2016 10:53 AM >>>>> To: e1000-devel@lists.sourceforge.net >>>>> Subject: [E1000-devel] Poor IPsec performance and high ksoftirqd load >>>>> >>>>> Hi, >>>>> >>>>> I'm running Linux 4.3.3 (Gentoo) and standard igb drivers. I have >>>>> trouble with the performance of IPSec on my platform. The platform >>>>> has Intel Ethernet Controller I211 (1GbE) and Intel Atom CPUs >>>>> (4-core) from the C2000 product family. >>>>> >>>>> My platform only seems to be able to do ~300Mbit/s when receiving >>>>> IPSec packets. When sending IPSec packets it can do ~600Mbit/s. The >>>>> weird thing is, the ksoftirqd handler seems to run at very high CPU >>>>> load. I don't understand where that comes from. I have traced using >>>>> kernel function tracing and I can see that it is taking a lot of time >>>>> do handle do_softirq and functions like igb_poll(). I was wondering >>>>> if it was normal that this takes up so much CPU? Also it seems that this >>>>> is >>> only the case in the following scenario: >>>>> >>>>> The platform receives IPSec packets on eth0, then decrypts them, and >>>>> routes them out via eth1 (plaintext). The packet stream is TCP. So >>>>> for each amount of packets going this way, there is an TCP ACK packet >>>>> going back the other way. E.g. plaintext incoming via eth1, IPSec >>> encryption and TX out via eth0. >>>>> >>>>> For me this sounds like a very normal use case of IPSec, but still >>>>> the platform does not seem to be able to handle it. As my CPU load is >>>>> high, but not maxed out, the CPU does not seem to be the bottleneck. >>>>> So my guess is it might also be the NIC. I do not have any >>>>> configuration options in my igb driver, and I already tried a few option >>> with ethtool, but I don't get any improvements. >>>>> Does anyone have a good idea on how to track what is slowing down the >>>>> system? >>>>> >>>>> Many thanks, >>>>> Anna >>>>> >>>>> >>>> Hi Anna, >>>> >>>> So the i211 is a very low end client Ethernet device (PCIe x1 2with limited >>> buffer space). So it's nowhere near as performant as the server devices. >>> That said, I'm not sure what you are using to test this with. Testing with >>> a >>> single TCP stream can always be problematic when looking at wire >>> bandwidth. Do things change if you start multiple TCP strings? It will use >>> more CPU as the packets need to be decrypted but you should see an >>> increase in throughput, at least until you run out of CPU cycles. The >>> single >>> stream TCP throughput is well known to be an issue as any single TCP >>> connection can only have so much data outstanding before being ACK'd. So >>> using multiple TPC connections changes that and should increase your overall >>> throughput. Please give that a try. >>>> >>>> Thanks. >>>> >>>> Cheers, >>>> John >>>> >>> >>> I have tried with multiple TCP streams as well but the performance drops >>> down even more already with two parallel streams. I use iperf3 for testing >>> on >>> the client side, and then I have some scripts on my IPSec box to monitor >>> packet rates and bandwidth as well. >>> >>> Anna > -- Sirrix AG security technologies - http://www.sirrix.com Dr. Anna Fischer eMail: a.fisc...@sirrix.com Tel +49(681) 959 86-139 Mobile +49(176) 47928927 PGP Key ID 0x96E0708B Vorstand: Ammar Alkassar (Vors.), Christian Stüble Vorsitzender des Aufsichtsrates: Peter Riedel Sitz der Gesellschaft: Homburg/Saar, HRB 3857 Amtsgericht Saarbrücken This message may contain confidential and/or privileged information. If you are not the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
