Hi, So there's a problem in the dpdk crypto code I think. In dpdk's ipsec.c in crypto_scan_devs() each crypto device is assigned a numa domain number
dev->numa = rte_cryptodev_socket_id (i);
Which I figured out meant I need to allocate a vdev cryptodev per numa (for
numa domains I will use) specifying the numa value as the socket id.
When dpdk does the crypto_auto_placement assigning queue pairs (resources) to
each worker thread, it doesn't pay any attention to this numa value.
I think it might should only allocate resources (queue pairs) from crypto
devices that co-locate with that workers nuam domain. A per numa rte_mempool is
allocated and shared by all crypto devices associated with that numa domain.
I hit a problem b/c I wasn't creating a cryptodev for numa node 2
(socket_id=2), but the worker was running on that numa node (2), so I SIGSEGV'd
in crypto_alloc_ops() b/c it's mempool was NULL; however, even if the user does
allocated enough devices (with enough queue pairs to cover all workers for each
crypto device) the wrong cryptodev will be used by a given worker if it doesn't
share it's numa domain (and so the memory will not be local)
I have a simple fix for this I could submit.
However, getting this right might be even more important when the crypto device
is an actual physical device (e.g., QAT) and it's PCI root is located in a
given numa domain. Here you would want to *prefer* to use the device associate
with the numa node, however, I think you might want worker threads running in
other numa domains to fallback to using a non-local crypto offload device if
there was no numa local one.
It seems important for high performance setups where one has a NIC and QAT card
per socket/pci bus to be choosing the correct (local) crypto offload device,
but for less perfect multi-socket configurations one should probably still be
able to use a singular QAT card. :)
Thoughts?
Thanks,
Chris.
signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#17280): https://lists.fd.io/g/vpp-dev/message/17280 Mute This Topic: https://lists.fd.io/mt/76343919/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
