do_csum was identified via perf recently as a hot spot when doing
receive on ip over infiniband workloads. After alot of testing and
ideas, we found the best optimization available to us currently is to
prefetch the entire data buffer prior to doing the checksum
Signed-off-by: Neil Horman <nhor...@tuxdriver.com>
CC: sebastien.du...@bull.net
CC: Thomas Gleixner <t...@linutronix.de>
CC: Ingo Molnar <mi...@redhat.com>
CC: "H. Peter Anvin" <h...@zytor.com>
CC: x...@kernel.org
---
arch/x86/lib/csum-partial_64.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/arch/x86/lib/csum-partial_64.c b/arch/x86/lib/csum-partial_64.c
index 9845371..9f2d3ee 100644
--- a/arch/x86/lib/csum-partial_64.c
+++ b/arch/x86/lib/csum-partial_64.c
@@ -29,8 +29,15 @@ static inline unsigned short from32to16(unsigned a)
* Things tried and found to not make it faster:
* Manual Prefetching
* Unrolling to an 128 bytes inner loop.
- * Using interleaving with more registers to break the carry chains.
*/
+
+static inline void prefetch_lines(void *addr, size_t len)
+{
+ void *end = addr + len;
+ for (; addr < end; addr += cache_line_size())
+ asm("prefetch 0(%[buf])\n\t" : : [buf] "r" (addr));
+}
+
static unsigned do_csum(const unsigned char *buff, unsigned len)
{
unsigned odd, count;
@@ -67,7 +74,9 @@ static unsigned do_csum(const unsigned char *buff, unsigned
len)
/* main loop using 64byte blocks */
zero = 0;
count64 = count >> 3;
- while (count64) {
+
+ prefetch_lines((void *)buff, len);
+ while (count64) {
asm("addq 0*8(%[src]),%[res]\n\t"
"adcq 1*8(%[src]),%[res]\n\t"
"adcq 2*8(%[src]),%[res]\n\t"
--
1.8.3.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
