__csum_partial() walks the buffer linearly and reads each cache line
exactly once. On cores without a hardware prefetcher, such as the 464FP
in the APM82181, every new line is therefore a demand miss that stalls
the loop. Packets received by non-coherent DMA are never in the cache,
so this is the common case for GRO checksum validation: when routing
TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
of all cycles and is the top entry in the profile.
Issue a dcbt MAX_COPY_PREFETCH cache lines ahead in the main loop,
the same distance csum_partial_copy_generic() already uses. That is
128 bytes on the 44x. dcbt is only a hint and never faults, so
prefetching past the end of the buffer is harmless. Lines that are
already cached, for example when checksumming data the CPU just wrote,
are left untouched.
Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
was toggled at runtime, in the same boot:
LAN->WAN WAN->LAN
Mbit/s csum cycles Mbit/s csum cycles
before 495 30.5% 439 27.2%
after 599 15.3% 520 13.7%
That is 58% less time spent in __csum_partial() per byte.
The prefetch distance was chosen with an earlier, slower driver
configuration (median of 3, Mbit/s):
distance LAN->WAN WAN->LAN
none 370 341
64 bytes 430 381
96 bytes 438 397
128 bytes 438 385
160 bytes 434 386
256 bytes 423 383
Prefetching the first lines before entering the loop made no
measurable difference and is not done.
Assisted-by: LLM
Signed-off-by: Rosen Penev <[email protected]>
---
v3: use MAX_COPY_PREFETCH
v2: reword commit message and retest numbers.
arch/powerpc/lib/checksum_32.S | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
index cd00b9bdd772..5bb872e9e68f 100644
--- a/arch/powerpc/lib/checksum_32.S
+++ b/arch/powerpc/lib/checksum_32.S
@@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
bdnz 2b
21: srwi. r6,r4,4 /* # blocks of 4 words to do */
beq 3f
+ li r9,MAX_COPY_PREFETCH*L1_CACHE_BYTES /* prefetch distance */
lwz r0,4(r3)
mtctr r6
lwz r6,8(r3)
@@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
lwzu r8,16(r3)
adde r5,r5,r7
bdz 23f
-22: lwz r0,4(r3)
+22: dcbt r3,r9
+ lwz r0,4(r3)
adde r5,r5,r8
lwz r6,8(r3)
adde r5,r5,r0
--
2.56.0
