Frag-list GRO exposes later encrypted records before the receive path
decrypts the current one. Use this lookahead to request write ownership
of linear ciphertext cache lines two records in advance, and maintain
the same distance throughout the batch, overlapping their memory access
latency with the current AEAD operation. An ordinary single-record UDP
receive has no later record and therefore skips the prefetch path.
Only prefetch the linear part of each skb. Walking non-linear fragments
here would duplicate the scatterlist walk performed by crypto and could
cost more than the cache hint saves.
This optimization primarily targets synchronous software AEAD, where the
CPU which issues the hint subsequently overwrites the prefetched lines.
An asynchronous accelerator may instead access them through DMA and is
not expected to benefit in the same way; the cache hint remains
correctness-neutral.
With the ovpn UDP GSO transmit series also applied, five interleaved
60-second iperf3 -Z single-flow AES-128-GCM runs in each direction over
two directly connected 100-Gbit/s mlx5 ports, after a 10-second warm-up,
produced the following mean throughput:
Forward Reverse
Without RX prefetch 21.542 Gbit/s 21.366 Gbit/s
With RX prefetch 25.741 Gbit/s 25.150 Gbit/s
RX prefetch improved throughput by 19.5% forward and 17.7% reverse. Mean
cycles per byte across the monitored data-path CPUs fell from 4.207 to
3.896, a 7.4% reduction.
A same-binary comparison using the aforementioned setup found distances
one and two effectively tied forward, while distance two was 3.3% faster
reverse and less variable.
Signed-off-by: Ralf Lici <[email protected]>
---
No changes since v1
https://lore.kernel.org/openvpn-devel/e18ef3363a3d5167cddb8b0528d447ba1d5aa2ed.1794314969.git.r...@mandelbit.com/
drivers/net/ovpn/udp.c | 34 +++++++++++++++++++++++++++++++++-
1 file changed, 33 insertions(+), 1 deletion(-)
diff --git a/drivers/net/ovpn/udp.c b/drivers/net/ovpn/udp.c
index f2deec4b44fb..5d3d4cd11988 100644
--- a/drivers/net/ovpn/udp.c
+++ b/drivers/net/ovpn/udp.c
@@ -6,9 +6,11 @@
* Author: Antonio Quartulli <[email protected]>
*/
+#include <linux/cache.h>
#include <linux/netdevice.h>
#include <linux/inetdevice.h>
#include <linux/if_vlan.h>
+#include <linux/prefetch.h>
#include <linux/skbuff.h>
#include <linux/socket.h>
#include <linux/udp.h>
@@ -33,6 +35,21 @@
/* like UDP and TCP frag-list GRO */
#define OVPN_UDP_GRO_CNT_MAX 64
+/* leave enough work between a cache hint and the record which consumes it */
+#define OVPN_UDP_GRO_PREFETCH_DISTANCE 2
+
+static void ovpn_udp_prefetchw_linear(const struct sk_buff *skb)
+{
+ unsigned long addr = (unsigned long)skb->data;
+ unsigned long end = addr + skb_headlen(skb);
+
+ /* crypto overwrites data in place, so request write ownership of each
+ * linear cache line before the AEAD implementation reaches it
+ */
+ for (; addr < end; addr = ALIGN(addr + 1, L1_CACHE_BYTES))
+ prefetchw((void *)addr);
+}
+
static bool ovpn_udp_gro_header(struct sk_buff *skb, u32 *header)
{
const unsigned int offset = skb_gro_offset(skb);
@@ -222,6 +239,7 @@ static struct sk_buff *ovpn_udp_gso_prepare(struct sk_buff
*skb, u32 header)
static void ovpn_udp_recv(struct ovpn_peer *peer, struct sk_buff *list)
{
+ struct sk_buff *skb, *next, *prefetch;
const __be16 proto = list->protocol;
const u32 priority = list->priority;
#ifdef CONFIG_NET_SCHED
@@ -231,9 +249,23 @@ static void ovpn_udp_recv(struct ovpn_peer *peer, struct
sk_buff *list)
const u32 secmark = list->secmark;
#endif
const int skb_iif = list->skb_iif;
- struct sk_buff *skb, *next;
+ int i;
+
+ /* a fraglist GRO aggregate makes later ciphertext visible before the
+ * current record is decrypted, so we prime the first two records, then
+ * keep the cache hints the same distance ahead while draining the list
+ */
+ prefetch = list->next ? list : NULL;
+ for (i = 0; i < OVPN_UDP_GRO_PREFETCH_DISTANCE && prefetch; i++) {
+ ovpn_udp_prefetchw_linear(prefetch);
+ prefetch = prefetch->next;
+ }
skb_list_walk_safe(list, skb, next) {
+ if (prefetch) {
+ ovpn_udp_prefetchw_linear(prefetch);
+ prefetch = prefetch->next;
+ }
/* propagate post-GRO metadata from the list head, and clear
* the state which the receive stack consumed on the head only
*/
--
2.56.0
_______________________________________________
Openvpn-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/openvpn-devel
