Software segmentation exposes the complete skb list before encryption
begins. Use encryption of the current segment to overlap fetching data
needed by the next one.
For segments encrypted in place, request write ownership of every cache
line in the next segment's linear area. When converting ordinary GSO
input into an out-of-place UDP GSO aggregate, the linear head has just
been touched by segmentation while most payload remains in shared page
fragments. Read-prefetch the first direct fragment without repeating the
complete scatter-gather walk performed by crypto.
Fully linear segments and fragments without a directly accessible kernel
virtual address skip the fragment hint. A single skb naturally skips
both forms of lookahead.
On two directly connected 100-Gbit/s mlx5 ports, five interleaved
iperf3 -t 60 -O 10 single-flow AES-128-GCM runs in each direction
produced the following mean throughput:
Forward Reverse
Without TX prefetch 24.028 Gbit/s 23.405 Gbit/s
With TX prefetch 24.166 Gbit/s 23.558 Gbit/s
This is a modest 0.6% increase in both directions.
Signed-off-by: Ralf Lici <[email protected]>
---
Changes since RFC v4
https://lore.kernel.org/openvpn-devel/d085dda71a91670680cbaee1029a37d952338c35.1789558856.git.r...@mandelbit.com/
- Retain write-prefetch for segments encrypted in place and add
read-prefetch of the first payload fragment for segments encrypted
into a UDP GSO aggregate.
- Account for fragments without a directly accessible kernel virtual
address.
- Refresh the commit message and add performance measurements.
drivers/net/ovpn/io.c | 34 ++++++++++++++++++++++++++++++++++
1 file changed, 34 insertions(+)
diff --git a/drivers/net/ovpn/io.c b/drivers/net/ovpn/io.c
index c6f24b5d85dd..6e28800286e1 100644
--- a/drivers/net/ovpn/io.c
+++ b/drivers/net/ovpn/io.c
@@ -388,6 +388,28 @@ static bool ovpn_encrypt_one(struct ovpn_peer *peer,
struct sk_buff *skb)
return true;
}
+static void ovpn_skb_prefetch_frag(const struct sk_buff *skb)
+{
+ unsigned long addr, end;
+ const skb_frag_t *frag;
+
+ if (!skb_shinfo(skb)->nr_frags)
+ return;
+
+ frag = &skb_shinfo(skb)->frags[0];
+ /* prefetch requires a directly accessible kernel virtual address */
+ addr = (unsigned long)skb_frag_address_safe(frag);
+ if (!addr)
+ return;
+
+ end = addr + skb_frag_size(frag);
+ /* the linear head was just touched by segmentation, so prefetch the
+ * first shared payload frag without repeating crypto's full sg walk
+ */
+ for (; addr < end; addr = ALIGN(addr + 1, L1_CACHE_BYTES))
+ prefetch((void *)addr);
+}
+
static bool ovpn_encrypt_gso_queue(struct sk_buff_head *skbs,
struct ovpn_peer *peer,
struct sk_buff *batch,
@@ -429,6 +451,12 @@ static bool ovpn_encrypt_gso_queue(struct sk_buff_head
*skbs,
skb = __skb_dequeue(skbs);
len = skb->len + OVPN_DATA_V2_OVERHEAD;
+ /* overlap fetching the next segment's shared payload with
+ * encryption of the current segment
+ */
+ if (i + 1 < segments)
+ ovpn_skb_prefetch_frag(skb_peek(skbs));
+
memset(ovpn_skb_cb(skb), 0, sizeof(struct ovpn_cb));
ovpn_skb_cb(skb)->batch = batch;
ovpn_encrypt_post(skb, ovpn_aead_encrypt_gso(peer, ks, skb,
@@ -490,6 +518,12 @@ static void ovpn_send(struct ovpn_priv *ovpn, struct
sk_buff *skb,
* independently
*/
skb_list_walk_safe(skb, curr, next) {
+ /* encrypting this segment can hide the cost of fetching the
+ * next segment's data into the cache
+ */
+ if (next)
+ ovpn_skb_prefetchw_linear(next);
+
if (unlikely(!ovpn_encrypt_one(peer, curr))) {
ovpn_dev_dstats_tx_dropped(ovpn->dev, 1);
kfree_skb(curr);
--
2.55.0
_______________________________________________
Openvpn-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/openvpn-devel
