Hi Gabi,
It looks like aesni_mb and aesni_gcm are disabled in VPP's DPDK build
configuration. see build/external/packages/dpdk.mk. You would probably need
to remove those from DPDK_DRIVERS_DISABLED and rebuild if you want to use
them. That said, I doubt you would see much improvement as a result of
using them. VPP's ipsecmb crypto plugin uses the same optimized crypto
library that those vdev's use. I think VPP's native crypto plugin is
assigned the highest priority, so that plugin is likely handling crypto
operations for your tunnels by default. If you want to use the ipsecmb
crypto plugin instead you can use a command like "vppctl set crypto
handler ipsecmb" for the ciphers used by your tunnels. I don't
know if you'll see any difference in performance by using ipsecmb instead
of native, but it doesn't hurt to try it.
Here are some thoughts and questions on tuning to improve IPsec throughput:
- If you haven't already, you should configure at least one worker
thread so your crypto operations are not being executed on the same CPU as
the main thread.
- Are you using one tunnel or multiple tunnels? An SA will be bound to a
particular thread in order to keep packets in order. With synchronous
crypto, all of the operations for the SA will be handled by that one thread
and throughput will be limited to how much crypto the CPU that thread is
bound to can handle. So you might get higher throughput by distributing
traffic across multiple tunnels if possible. Or if you enable asynchronous
crypto, the sw_scheduler plugin tries to distribute crypto operations to
other threads, which might help.
- With multiple workers, you could get encrypt & decrypt operations
handled by different threads/cores. If you have a LAN interface and a WAN
interface and your tunnel is terminated on the WAN interface to allow VMs
on your LAN subnet to communicate with some remote systems on the other
side of the tunnel, you could bind the RX queues for the interfaces to
different threads. Outbound packets would be encrypted by the threads which
handle the queues for the LAN interface. Inbound packets will be decrypted
by the threads which handle the queues for the WAN interface.
- You mentioned that you can't get better throughput from VPP than you
can with kernel IPsec. Is the kernel getting the same throughput as VPP or
higher? If it's close to the same, you may be hitting some external
resource limit. E.g. the other end of the tunnel could be the bottleneck.
Or AWS's traffic shaping might be preventing you from sending any faster.
- Are you using policy-based IPsec or routed IPsec (creating a tunnel
interface)? There have been patches merged recently which are intended to
improve performance for policy-based IPsec, but if you are using
policy-based IPsec you might try using a tunnel interface instead and see
if your measurements improve.
- Fragmentation and reassembly can impact IPsec throughput. If your
packets are close to the size of the hardware interface that packets will
be sent out, the encapsulation & crypto padding may push the packet size
over the MTU and the encrypted packet may need to be fragmented before
being sent. That means the other end of the tunnel will need to wait for
all the fragments to arrive and reassemble them before it can decrypt the
packet. If you are using a tunnel interface, you can set the MTU on the
tunnel interface lower than the MTU on the hardware interface. Then packets
would be fragmented by the tunnel interface before being encrypted and the
other end would not need to reassemble them.
-Matt
On Fri, Jun 3, 2022 at 7:52 AM wrote:
> Hi,
> I am a beginner in VPP and DPDK stuff, I am trying to create a high
> performance AWS VM which should do IPSec tunneling.
>
> The IPSEc traffic is running well, but I can not exceed 8Gb traffic
> throughput and I can not convince VPP to beat the "ip xfrm" in terms of
> IPSec traffic throughput.
>
> When the VPP starts, I get this warning all the times:
>
> dpdk/cryptodev [warn ]: dpdk_cryptodev_init: Not enough cryptodev
> resources
>
> whatever CPU I have enabled.
>
> If I specify
> vdev crypto_aesni_mb
> or
> vdev crypto_aesni_gcm
> on the dpdk section of startup.conf file, I always hit this error:
> 0: dpdk_config: rte_eal_init returned -1
>
> I am using Ubuntu 20.04 LTS and the CPU flags are:
>
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
> pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm
> constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid
> aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1
> sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
> hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust
> bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx
> smap clflushopt clwb
