On 31.10.2018 05:08, Andre Tomt wrote:
On 30.10.2018 12:04, Andre Tomt wrote:
On 30.10.2018 11:58, Andre Tomt wrote:
On 27.10.2018 23:41, Andre Tomt wrote:
On 26.10.2018 13:45, Andre Tomt wrote:
On 25.10.2018 19:38, Eric Dumazet wrote:
On 10/24/2018 12:41 PM, Andre Tomt wrote:
It eventually showed up again with mlx4, on 4.18.16 + fix and
also on 4.19. I still do not have a useful packet capture.
It is running a torrent client serving up various linux
distributions.
Have you also applied this fix ?
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=db4f1be3ca9b0ef7330763d07bf4ace83ad6f913
No. I've applied it now to 4.19 and will report back if anything
shows up.
Just hit it on the simpler server; no VRF, no tunnels, no
nat/conntrack. Only a basic stateless nftables ruleset and a vlan
netdev (unlikely to be the one triggering this I guess; it has only
v4 traffic).
I'm currently testing 4.19 with the recomended commit added, plus
these to sort out some GRO issues (on a hunch, unsure if related):
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=a8305bff685252e80b7c60f4f5e7dd2e63e38218
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=992cba7e276d438ac8b0a8c17b147b37c8c286f7
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=ece23711dd956cd5053c9cb03e9fe0668f9c8894
and I *think* it is behaving better now? it's not conclusive as it
could take a while to trip in this environment but some of the test
servers have not shown anything bad in almost 24h.
Sorry, s/some of the/none of the
I think it is fairly safe to say 4.19 + mlx4 + these 4 commits is OK. At
least for my workload. Servers are now 51-61 hours in, no splats. I also
added ntp pool traffic to one of them to make things a little more
exciting.
Not sure what is needed for 4.18, I dont have the mental bandwidth to
test that right now. Also no idea about the similar looking mlx5 splats
reported elsewhere.
As expected conntrack/nat + vlan + forwarding still splats.
sch_cake, IFB and VRF was removed from this setup.
Here is a conntrack splat without IFB/VRF/Cake inteference:
[34458.506346] wanib: hw csum failure
[34458.506371] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.19.0-1 #1
[34458.506374] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 2.0
06/13/2018
[34458.506377] Call Trace:
[34458.506381] <IRQ>
[34458.506388] dump_stack+0x5c/0x80
[34458.506392] __skb_checksum_complete+0xac/0xc0
[34458.506402] icmp_error+0x1c8/0x1f0 [nf_conntrack]
[34458.506406] ? skb_copy_bits+0x13d/0x220
[34458.506411] nf_conntrack_in+0xd8/0x390 [nf_conntrack]
[34458.506416] ? ___pskb_trim+0x192/0x330
[34458.506421] nf_hook_slow+0x43/0xc0
[34458.506426] ip_rcv+0x90/0xb0
[34458.506430] ? ip_rcv_finish_core.isra.0+0x310/0x310
[34458.506435] __netif_receive_skb_one_core+0x42/0x50
[34458.506438] netif_receive_skb_internal+0x24/0xb0
[34458.506441] napi_gro_frags+0x177/0x210
[34458.506446] mlx4_en_process_rx_cq+0x8df/0xb50 [mlx4_en]
[34458.506459] ? mlx4_eq_int+0x38f/0xcb0 [mlx4_core]
[34458.506463] mlx4_en_poll_rx_cq+0x55/0xf0 [mlx4_en]
[34458.506466] net_rx_action+0xe1/0x2c0
[34458.506469] __do_softirq+0xe7/0x2d3
[34458.506475] irq_exit+0x96/0xd0
[34458.506478] do_IRQ+0x85/0xd0
[34458.506483] common_interrupt+0xf/0xf
[34458.506486] </IRQ>
[34458.506491] RIP: 0010:cpuidle_enter_state+0xb9/0x320
[34458.506495] Code: e8 3c 16 bc ff 80 7c 24 0b 00 74 17 9c 58 0f 1f 44 00 00 f6 c4
02 0f 85 3b 02 00 00 31 ff e8 5e fb c0 ff fb 66 0f 1f 44 00 00 <48> b8 ff ff ff
ff f3 01 00 00 48 2b 1c 24 ba ff ff ff 7f 48 39 c3
[34458.506497] RSP: 0018:ffff978d41943ea8 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffffdb
[34458.506500] RAX: ffff8d8f6fa60fc0 RBX: 00001f56ff07af28 RCX: 000000000000001f
[34458.506501] RDX: 00001f56ff07af28 RSI: 000000003a2e90d6 RDI: 0000000000000000
[34458.506503] RBP: ffff8d8f6fa698c0 R08: 0000000000000002 R09: 0000000000020840
[34458.506504] R10: 0004ea58f2899595 R11: ffff8d8f6fa601e8 R12: 0000000000000001
[34458.506505] R13: ffffffff8a0ac638 R14: 0000000000000001 R15: 0000000000000000
[34458.506509] ? cpuidle_enter_state+0x94/0x320
[34458.506512] do_idle+0x1e4/0x220
[34458.506515] cpu_startup_entry+0x5f/0x70
[34458.506519] start_secondary+0x185/0x1a0
[34458.506521] secondary_startup_64+0xa4/0xb0
Stateless filtered non-forwarding host still looks like it has been
fixed (the udp6_gro_* splats are still all gone). Also seems fine when
moving the traffic over a vlan device. These fixes went into 4.19.1-rc1
(checksum_complete + unlink gro packets on overflow fixes)
