Re: [ovs-discuss] crash when restart openvswitch with huge vxlan traffic running

Fri, 28 Dec 2018 10:03:27 -0800 


On 12/27/2018 1:38 PM, Gregory Rose wrote:


On 12/27/2018 11:40 AM, Ben Pfaff wrote:

Greg, this is a kernel issue.  If you have the time, will you take a
look at it sometime?


Yep, will do.

- Greg



I looked into this and there is not much for us to do wrt our kernel 
datapath.  It is a kernel GRO fix and if it
is not in the kernel you're using then you'll have to talk to your 
vendor about getting the fix into their next
distribution kernel package upgrade or else roll (i.e. custom build) 
your own kernel.



I checked all the recent 3.10.x kernel sources using the lexer at 
access.redhat.com and did not find any
of them that carried Lorenzo's fix (commit 8e1da73acded on Dave Miller's 
current net tree) all the way
up to 3.10.0-957.el7.  That is RHEL 7.6 and is the latest so it looks 
like RHEL 7 is stuck with the bug.


Thanks,

- Greg





On Thu, Dec 20, 2018 at 12:42:43PM +0000, 王志克 wrote:

Hi All,


I did below test, and found system crash, does anyone knows whether 
there are already some fix for it?


Setup:
CentOS7.4 3.10.0-693.el7.x86_64,
OVS: 2.10.1

Step:

1.  Build OVS only for userspace, and reuse kernel-builtin 
openvswitch module.

2.  On Host1, create 1 vxlan interface and add 1 VF_rep to OVS.

3.  Attach the VF to one VM, and the VM will do 5 tuples swap using 
DPDK app.
4.  using traffic generator to send huge traffic （7Mpps with 
serveral k connetions）to Host1 PF.

5.  The OVS rue are configured as below.

VM1_PORTNAME=$1
VXLAN_PORTNAME=$2

VM1_PORT=$(ovs-vsctl list interface | grep $VM1_PORTNAME -A1 | grep 
ofport | sed 's/ofport *: \([0-9]*\)/\1/g')
VXLAN_PORT=$(ovs-vsctl list interface | grep $VXLAN_PORTNAME -A1 | 
grep ofport | sed 's/ofport *: \([0-9]*\)/\1/g')

ZONE=8
ovs-ofctl del-flows ovs-sriov

ovs-ofctl add-flow ovs-sriov -O openflow13 "priority=1000 
table=0,arp, actions=NORMAL"
ovs-ofctl add-flow ovs-sriov -O openflow13 "priority=100 
table=0,ip,in_port=$VM1_PORT,action=set_field:$VM1_PORT->reg6,goto_table:5"
ovs-ofctl add-flow ovs-sriov -O openflow13 "priority=100 
table=0,ip,in_port=$VXLAN_PORT, tun_id=0x242, 
action=set_field:$VXLAN_PORT->reg6,set_field:$VM1_PORT->reg7,goto_table:5"



ovs-ofctl add-flow ovs-sriov -O openflow13 "table=5, priority=100, 
ip,actions=ct(table=10,zone=$ZONE)"



ovs-ofctl add-flow ovs-sriov -O openflow13 "table=10, 
priority=100,ip,ct_state=-new+est-rel-inv+trk actions= goto_table:15"
ovs-ofctl add-flow ovs-sriov -O openflow13 "table=10, 
priority=100,ip,ct_state=-new-est-rel+inv+trk actions=drop"
ovs-ofctl add-flow ovs-sriov -O openflow13 "table=10, 
priority=100,ip,ct_state=-new-est-rel-inv-trk actions=drop"



ovs-ofctl add-flow ovs-sriov -O openflow13 "table=10, 
priority=100,ip,ct_state=+new-rel-inv+trk actions= 
ct(commit,table=15,zone=$ZONE)"



ovs-ofctl add-flow ovs-sriov -O openflow13 "priority=100 
table=15,ip, in_port=$VM1_PORT, 
action=set_field:0x242->tun_id,set_field:$VXLAN_PORT->reg7,goto_table:20"
ovs-ofctl add-flow ovs-sriov -O openflow13 "priority=100 
table=15,ip, in_port=$VXLAN_PORT, actions=goto_table:20"



ovs-ofctl add-flow ovs-sriov -O openflow13 "table=20, priority=100, 
ip,action=output:NXM_NX_REG7[0..15]"
ovs-ofctl add-flow ovs-sriov -O openflow13 "table=200, 
priority=100,action=drop"

6. execute serveral times “systemctl restart openvswitch”, then crash.

Crash stack (2 kinds):
One
[  575.459905] device vxlan_sys_4789 left promiscuous mode

[  575.460103] BUG: unable to handle kernel NULL pointer dereference 
at 0000000000000008

[  575.460133] IP: [<ffffffffc09b330b>] gro_cell_poll+0x4b/0x80 [vxlan]
[  575.460210] PGD 0
[  575.460226] Oops: 0002 [#1] SMP

[  575.460254] Modules linked in: vhost_net vhost macvtap macvlan 
vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 
nf_nat_ipv6 nf_defrag_ipv6 vfio_pci vfio_iommu_type1 vfio 
xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 
iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 
xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp 
llc ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) 
ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) 
ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) 
ib_core(OE) mlx4_core(OE) mlx_compat(OE) devlink iTCO_wdt 
iTCO_vendor_support dcdbas sb_edac edac_core intel_powerclamp 
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
[  575.460619]  ghash_clmulni_intel aesni_intel lrw gf128mul 
glue_helper ablk_helper cryptd ipmi_ssif joydev pcspkr sg mei_me mei 
lpc_ich ipmi_si shpchp ipmi_devintf ipmi_msghandler wmi 
acpi_power_meter knem(OE) nfsd auth_rpcgss nfs_acl lockd grace 
sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic 
mgag200 drm_kms_helper syscopyarea ixgbe sysfillrect igb sysimgblt 
fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm 
ahci libahci megaraid_sas libata i2c_algo_bit i2c_core mdio ptp dca 
pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: 
devlink]
[  575.460885] CPU: 2 PID: 20 Comm: ksoftirqd/2 Tainted: G           
OE  ------------   3.10.0-693.el7.x86_64 #1
[  575.460912] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 
1.3.6 06/03/2015
[  575.460933] task: ffff880152ef1fa0 ti: ffff880152efc000 task.ti: 
ffff880152efc000
[  575.460954] RIP: 0010:[<ffffffffc09b330b>] [<ffffffffc09b330b>] 
gro_cell_poll+0x4b/0x80 [vxlan]

[  575.460990] RSP: 0018:ffff880152effd68  EFLAGS: 00010202

[  575.461004] RAX: 0000000000000000 RBX: ffffe8dfff448818 RCX: 
0000000000000000
[  575.461024] RDX: 0000000000000001 RSI: ffff881fa42ebf00 RDI: 
ffffe8dfff448818
[  575.461042] RBP: ffff880152effd88 R08: 0000000000019c40 R09: 
ffffffff815710d7
[  575.461061] R10: ffff881ffec59c40 R11: ffffea007e90ba00 R12: 
0000000000000002
[  575.461079] R13: 0000000000000040 R14: ffffe8dfff448800 R15: 
0000000000000001
[  575.461098] FS:  0000000000000000(0000) GS:ffff881ffec40000(0000) 
knlGS:0000000000000000

[  575.461119] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[  575.461134] CR2: 0000000000000008 CR3: 00000000019f2000 CR4: 
00000000001427e0
[  575.461153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[  575.461172] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400

[  575.461190] Stack:

[  575.461198]  ffffe8dfff448818 0000000000000000 0000000000000040 
0000000000000000
[  575.461221]  ffff880152effe08 ffffffff8158799d ffff881ffec57950 
ffff881ffec57940
[  575.461254]  00000001000432b7 0000012c52f09428 ffff881ffd57eb40 
ffff881ffd57eb40

[  575.461277] Call Trace:
[  575.461290]  [<ffffffff8158799d>] net_rx_action+0x16d/0x380
[  575.461308]  [<ffffffff81090b3f>] __do_softirq+0xef/0x280
[  575.461324]  [<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[  575.462074]  [<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[  575.462780]  [<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[  575.463464]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[  575.464169]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[  575.464862]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[  575.465497]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40

[  575.466192] Code: 49 39 f6 74 40 48 85 f6 74 3b 83 6b f8 01 48 89 
df 41 83 c4 01 48 8b 0e 48 8b 46 08 48 c7 06 00 00 00 00 48 c7 46 08 
00 00 00 00 <48> 89 41 08 48 89 08 e8 29 4f bd c0 45 39 ec 74 14 48 
8b 73 e8
[  575.467663] RIP  [<ffffffffc09b330b>] gro_cell_poll+0x4b/0x80 
[vxlan]

[  575.468412]  RSP <ffff880152effd68>
[  575.469197] CR2: 0000000000000008

TWO:
[  390.626080] device vxlan_sys_4789 left promiscuous mode

[  390.626345] BUG: unable to handle kernel NULL pointer dereference 
at 0000000000000008

[  390.626411] IP: [<ffffffffc09c8b4a>] vxlan_dellink+0x9a/0xf0 [vxlan]
[  390.626462] PGD 0
[  390.626499] Oops: 0002 [#1] SMP

[  390.626529] Modules linked in: vhost_net vhost macvtap macvlan 
vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 
nf_nat_ipv6 nf_defrag_ipv6 vfio_pci vfio_iommu_type1 vfio 
xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 
iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 
xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp 
llc ebtable_filter ebtables ip6table_filter ip6_tables 
iptable_filter rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) 
ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) 
ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) 
ib_core(OE) mlx4_core(OE) mlx_compat(OE) devlink iTCO_wdt 
iTCO_vendor_support dcdbas sb_edac edac_core intel_powerclamp 
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
[  390.627152]  ghash_clmulni_intel ipmi_ssif aesni_intel lrw 
gf128mul glue_helper ablk_helper cryptd ipmi_si pcspkr joydev 
ipmi_devintf ipmi_msghandler mei_me mei sg lpc_ich shpchp 
acpi_power_meter wmi nfsd auth_rpcgss nfs_acl lockd knem(OE) grace 
sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic 
mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops 
ttm drm crct10dif_pclmul crct10dif_common ixgbe crc32c_intel ahci 
igb libahci libata megaraid_sas mdio i2c_algo_bit ptp i2c_core 
pps_core dca dm_mirror dm_region_hash dm_log dm_mod [last unloaded: 
devlink]
[  390.627626] CPU: 11 PID: 6303 Comm: ovs-vswitchd Tainted: 
G           OE  ------------   3.10.0-693.el7.x86_64 #1
[  390.627690] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 
1.3.6 06/03/2015
[  390.627738] task: ffff881fe0e89fa0 ti: ffff881fa3590000 task.ti: 
ffff881fa3590000
[  390.627786] RIP: 0010:[<ffffffffc09c8b4a>] [<ffffffffc09c8b4a>] 
vxlan_dellink+0x9a/0xf0 [vxlan]

[  390.627848] RSP: 0018:ffff881fa3593888  EFLAGS: 00010206

[  390.627883] RAX: 0000000000000000 RBX: 0000000000000010 RCX: 
0000000000000000
[  390.627929] RDX: 0000000000000000 RSI: ffffea007fd7f600 RDI: 
ffff881ff5fd8c00
[  390.627975] RBP: ffff881fa35938b0 R08: ffff881ff5fd8b00 R09: 
000000018040000d
[  390.628020] R10: 00000000f5fd8a01 R11: ffffea007fd7f600 R12: 
ffff88015270e000
[  390.628066] R13: ffffffff81b1caa0 R14: ffff881fa35938c0 R15: 
ffffe8dfff60a1d8
[  390.628112] FS:  00007f4ea1168ac0(0000) GS:ffff883ffe540000(0000) 
knlGS:0000000000000000

[  390.628163] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[  390.628201] CR2: 0000000000000008 CR3: 0000001ff9055000 CR4: 
00000000001427e0
[  390.628246] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[  390.628292] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
0000000000000400

[  390.628337] Stack:

[  390.628354]  ffff881fa35938c0 ffffffff81ad9d40 0000000000000001 
0000000000000000
[  390.628411]  ffffffff81ad9d40 ffff881fa35938e0 ffffffff81599023 
ffff881fa35938c0
[  390.628468]  ffff881fa35938c0 000000001d8239fc ffff883ffd864a00 
ffff881fa3593a70

[  390.628535] Call Trace:
[  390.628561]  [<ffffffff81599023>] rtnl_delete_link+0x43/0x80
[  390.628610]  [<ffffffff8159b761>] rtnl_dellink+0x91/0xf0
[  390.628649]  [<ffffffff81599bd4>] rtnetlink_rcv_msg+0xa4/0x270
[  390.630373]  [<ffffffff815bacd0>] ? __netlink_lookup+0xc0/0x110
[  390.632066]  [<ffffffff81599b30>] ? rtnetlink_rcv+0x30/0x30
[  390.633751]  [<ffffffff815bd929>] netlink_rcv_skb+0xa9/0xc0
[  390.635426]  [<ffffffff81599b28>] rtnetlink_rcv+0x28/0x30
[  390.637081]  [<ffffffff815bd012>] netlink_unicast+0xf2/0x1b0
[  390.638721]  [<ffffffff815bd3ef>] netlink_sendmsg+0x31f/0x6a0
[  390.640371]  [<ffffffff812b4d65>] ? sock_has_perm+0x75/0x90
[  390.642037]  [<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[  390.643722]  [<ffffffff8156a88f>] ? sock_recvmsg+0xbf/0x100
[  390.645411]  [<ffffffff8132c312>] ? put_dec+0x72/0x90
[  390.647075]  [<ffffffff8132d303>] ? number.isra.2+0x323/0x360
[  390.648724]  [<ffffffff8156ae29>] ___sys_sendmsg+0x3a9/0x3c0
[  390.650362]  [<ffffffff811de9d2>] ? kmem_cache_free+0x1e2/0x200
[  390.652010]  [<ffffffff81217af5>] ? __d_free+0x35/0x40
[  390.653623]  [<ffffffff812181b0>] ? d_free+0x60/0x70
[  390.655181]  [<ffffffff812186b4>] ? dentry_kill+0x154/0x1b0
[  390.656702]  [<ffffffff81222744>] ? mntput+0x24/0x40
[  390.658173]  [<ffffffff81203053>] ? __fput+0x183/0x260
[  390.659606]  [<ffffffff8156b5f1>] __sys_sendmsg+0x51/0x90
[  390.660988]  [<ffffffff8156b642>] SyS_sendmsg+0x12/0x20
[  390.662325]  [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b

[  390.663624] Code: a0 bb c0 49 8b 3f 49 39 ff 74 be 48 85 ff 74 b9 
41 83 6f 10 01 48 8b 0f 48 8b 57 08 48 c7 07 00 00 00 00 48 c7 47 08 
00 00 00 00 <48> 89 51 08 48 89 0a e8 8a a2 ba c0 49 8b 3f 49 39 ff 
75 cc eb
[  390.666406] RIP  [<ffffffffc09c8b4a>] vxlan_dellink+0x9a/0xf0 
[vxlan]

[  390.667674]  RSP <ffff881fa3593888>
[  390.668892] CR2: 0000000000000008

Br,
Zhike Wang
JDCloud, Product Development, IaaS

------------------------------------------------------------------------------------------------ 


Mobile／+86 13466719566
E- mail／[email protected]<mailto:[email protected]>

Address／5F Building A,North-Star Century Center,8 Beichen West 
Street,Chaoyang District Beijing

Https://JDCloud.com<https://jdcloud.com/>

------------------------------------------------------------------------------------------------ 


[cid:[email protected]]





_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss




_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss






















					Reply via email to