Re: [ovs-discuss] OVS 3.2.0 crashing setting port QOS
On 1/29/24 20:13, Daryl Wang via discuss wrote:
> After upgrading from Open vSwitch 3.1.2-4 to 3.2.0-2 we've been consistently
> seeing new OVS crashes when setting QoS on ports. Both packages were taken
> from the Debian distribution
> (https://packages.debian.org/source/sid/openvswitch
> we're running on. From the core dump we're seeing the following backtrace:
>
> # gdb --batch -ex bt /usr/sbin/ovs-vswitchd /core
> [NewLWP 67669]
> [NewLWP 67682]
> [NewLWP 67681]
> [NewLWP 67671]
> [NewLWP 67679]
> [NewLWP 67680]
> [Threaddebugging usinglibthread_db enabled]
> Usinghost libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Corewas generated by`ovs-vswitchd unix:/tmp/ovs/db.sock -vconsole:emer
> -vsyslog:err -vfile:info --ml'.
> Program terminated with signal SIGABRT, Aborted.
> #0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> [Current thread is 1 (Thread 0x7fcacf4cfa80 (LWP 67669))]
> #0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x7fcacec6d472 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x7fcacec574b2 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x560787952c7e in ovs_abort_valist (err_no=,
> format=, args=args@entry=0x7ffd14c3bce0) at ../lib/util.c:447
> #4 0x560787952d14 in ovs_abort (err_no=,
> format=format@entry=0x5607879f3ec7 "%s: pthread_%s_%s failed") at
> ../lib/util.c:439
> #5 0x56078791ee11 in ovs_mutex_lock_at (l_=l_@entry=0x56078934c6c8,
> where=where@entry=0x5607879f864b "../lib/netdev-linux.c:2575") at
> ../lib/ovs-thread.c:76
> #6 0x56078796d03d in netdev_linux_get_speed (netdev_=0x56078934c640,
> current=0x7ffd14c3be64, max=0x7ffd14c3be1c) at ../lib/netdev-linux.c:2575
> #7 0x5607878c04f3 in netdev_get_speed
> (netdev=netdev@entry=0x56078934c640, current=current@entry=0x7ffd14c3be64,
> max=0x7ffd14c3be1c, max@entry=0x0) at ../lib/netdev.c:1175
> #8 0x560787968d67 in htb_parse_qdisc_details__
> (netdev=netdev@entry=0x56078934c640, details=details@entry=0x56078934a880,
> hc=hc@entry=0x7ffd14c3beb0) at ../lib/netdev-linux.c:4804
> #9 0x5607879755da in htb_tc_install (details=0x56078934a880,
> netdev=0x56078934c640) at ../lib/netdev-linux.c:4883
> #10 htb_tc_install (netdev=0x56078934c640, details=0x56078934a880) at
> ../lib/netdev-linux.c:4875
> #11 0x560787974937 in netdev_linux_set_qos (netdev_=0x56078934c640,
> type=, details=0x56078934a880) at ../lib/netdev-linux.c:3054
> #12 0x560787814ea5 in iface_configure_qos (qos=0x56078934a780,
> iface=0x560789349fd0) at ../vswitchd/bridge.c:4845
> #13 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x5607892dbf90) at
> ../vswitchd/bridge.c:928
> #14 0x560787817f7d in bridge_run () at ../vswitchd/bridge.c:3321
> #15 0x56078780d205 in main (argc=, argv=)
> at ../vswitchd/ovs-vswitchd.c:130
>
> A shell script to reproduce the issue is:
>
> #!/bin/sh
>
> apt-get install openvswitch-{common,switch}{,-dbgsym}
>
> # Don't need it running on the system
> systemctl stop openvswitch-switch
>
> set-e
>
> cleanup(){
> ip link delveth0
> rm /tmp/ovs/conf.db
> }
>
> trap cleanup EXIT
>
> # Setup our environment
>
> ip link add veth0 type veth peer veth1
>
> mkdir -p /tmp/ovs
>
> exportOVS_RUNDIR=/tmp/ovs
> exportOVS_LOGDIR=/tmp/ovs
> exportOVS_DBDIR=/tmp/ovs
>
> /usr/share/openvswitch/scripts/ovs-ctl start
> ovs-vsctl add-br demo
> ovs-vsctl add-port demo veth1
>
> # Make it crash
>
> ovs-vsctl setPortveth1 qos=@qos\
> -- --id=@qoscreate QoStype=linux-htb queues:1=@highqueues:2=@low\
> -- --id=@highcreate Queueother-config:priority=1other-config:min-rate=0.1\
> -- --id=@low create Queueother-config:priority=6other-config:min-rate=0.05
>
> We built the reproduction script based on speculation that
> https://github.com/openvswitch/ovs/commit/6240c0b4c80ea3d8dd1bf77526b04b55742de2ce
> is related to the crash. Notably we don't seem to run into the problem when we
> pass in a specific maximum bandwidth instead of relying on the interface's
> maximum
> bandwidth.
Hi, Daryl. Thanks for the report!
Looking at the stack trace, the root cause seems to be the following commit:
https://github.com/openvswitch/ovs/commit/b8f8fad8643518551cf742056ae8728c936674c6
It introduced the netdev_get_speed() call in QoS functions.
These functions are running under netdev mutex and the netdev_get_speed()
function is trying to take the same mutex for a second time. That fails
with 'deadlock avoided' or something like that.
From the commit I linked it's clear why it's not happening if the max-rate
is specified. The code just doesn't go that route.
To fix the issue, we need to have a lockless version of netdev_linux_get_speed()
and call it directly from the QoS functions without going through generic
netdev API.
Adrian, since it was your original patch, could you, please, take a look
at the issue?
Or Daryl, maybe you want to fix it yourself?
Best regards, Ilya Maximets.
[ovs-discuss] OVS 3.2.0 crashing setting port QOS
After upgrading from Open vSwitch 3.1.2-4 to 3.2.0-2 we've been
consistently seeing new OVS crashes when setting QoS on ports. Both
packages were taken from the Debian distribution (
https://packages.debian.org/source/sid/openvswitch) we're running on. From
the core dump we're seeing the following backtrace:
# gdb --batch -ex bt /usr/sbin/ovs-vswitchd /core
[New LWP 67669]
[New LWP 67682]
[New LWP 67681]
[New LWP 67671]
[New LWP 67679]
[New LWP 67680]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `ovs-vswitchd unix:/tmp/ovs/db.sock -vconsole:emer
-vsyslog:err -vfile:info --ml'.
Program terminated with signal SIGABRT, Aborted.
#0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7fcacf4cfa80 (LWP 67669))]
#0 0x7fcacecbb0fc in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x7fcacec6d472 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x7fcacec574b2 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x560787952c7e in ovs_abort_valist (err_no=,
format=, args=args@entry=0x7ffd14c3bce0) at ../lib/util.c:447
#4 0x560787952d14 in ovs_abort (err_no=,
format=format@entry=0x5607879f3ec7 "%s: pthread_%s_%s failed") at
../lib/util.c:439
#5 0x56078791ee11 in ovs_mutex_lock_at (l_=l_@entry=0x56078934c6c8,
where=where@entry=0x5607879f864b "../lib/netdev-linux.c:2575") at
../lib/ovs-thread.c:76
#6 0x56078796d03d in netdev_linux_get_speed (netdev_=0x56078934c640,
current=0x7ffd14c3be64, max=0x7ffd14c3be1c) at ../lib/netdev-linux.c:2575
#7 0x5607878c04f3 in netdev_get_speed (netdev=netdev@entry=0x56078934c640,
current=current@entry=0x7ffd14c3be64, max=0x7ffd14c3be1c, max@entry=0x0) at
../lib/netdev.c:1175
#8 0x560787968d67 in htb_parse_qdisc_details__
(netdev=netdev@entry=0x56078934c640,
details=details@entry=0x56078934a880, hc=hc@entry=0x7ffd14c3beb0) at
../lib/netdev-linux.c:4804
#9 0x5607879755da in htb_tc_install (details=0x56078934a880,
netdev=0x56078934c640) at ../lib/netdev-linux.c:4883
#10 htb_tc_install (netdev=0x56078934c640, details=0x56078934a880) at
../lib/netdev-linux.c:4875
#11 0x560787974937 in netdev_linux_set_qos (netdev_=0x56078934c640,
type=, details=0x56078934a880) at ../lib/netdev-linux.c:3054
#12 0x560787814ea5 in iface_configure_qos (qos=0x56078934a780,
iface=0x560789349fd0) at ../vswitchd/bridge.c:4845
#13 bridge_reconfigure (ovs_cfg=ovs_cfg@entry=0x5607892dbf90) at
../vswitchd/bridge.c:928
#14 0x560787817f7d in bridge_run () at ../vswitchd/bridge.c:3321
#15 0x56078780d205 in main (argc=, argv=)
at ../vswitchd/ovs-vswitchd.c:130
A shell script to reproduce the issue is:
#!/bin/sh
apt-get install openvswitch-{common,switch}{,-dbgsym}
# Don't need it running on the system
systemctl stop openvswitch-switch
set -e
cleanup() {
ip link del veth0
rm /tmp/ovs/conf.db
}
trap cleanup EXIT
# Setup our environment
ip link add veth0 type veth peer veth1
mkdir -p /tmp/ovs
export OVS_RUNDIR=/tmp/ovs
export OVS_LOGDIR=/tmp/ovs
export OVS_DBDIR=/tmp/ovs
/usr/share/openvswitch/scripts/ovs-ctl start
ovs-vsctl add-br demo
ovs-vsctl add-port demo veth1
# Make it crash
ovs-vsctl set Port veth1 qos=@qos \
-- --id=@qos create QoS type=linux-htb queues:1=@high queues:2=@low \
-- --id=@high create Queue other-config:priority=1 other-config:min-rate=
0.1 \
-- --id=@low create Queue other-config:priority=6 other-config:min-rate=
0.05
We built the reproduction script based on speculation that
https://github.com/openvswitch/ovs/commit/6240c0b4c80ea3d8dd1bf77526b04b55742de2ce
is related to the crash. Notably we don't seem to run into the problem when
we pass in a specific maximum bandwidth instead of relying on the
interface's maximum bandwidth.
Sincerely,
Daryl Wang
___
discuss mailing list
disc...@openvswitch.org
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
On 1/29/24 10:23, Lim, Derrick wrote:
> Hi Ilya Maximets,
>
> I did some further testing on my end. Just to make sure it's an address
> family issue, I tried to configure all VXLAN interfaces with IPv6, but
> I ran into an issue with the source IP address selection.
>
> I specified the `local_ip` of the tunnel as `2403:400:31da:::18:6`,
> which is also added on the bridge interface, but it picks the link-local
> address of `fe80::dc03:37ff:fee2:1fef` instead. This gets dropped by other
> devices along the way.
>
> ```
> $ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev | grep 192.168.1.33
> [abbreviated]
> ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
> skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
> packets:1, bytes:42, used:0.063s, dp:ovs,
> actions:tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4,eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd),ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)),push_vlan(vid=304,pcp=0),exit_p0,
> dp-extra-info:miniflow_bits(4,0)
> ```
>
> ```
> $ ovs-vsctl show
> Bridge br-int
> datapath_type: netdev
> Port dpdk-vm101
> Interface dpdk-vm101
> type: dpdkvhostuserclient
> options:
> {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
> Port vxlan0
> Interface vxlan0
> type: vxlan
> options: {dst_port="4789", key="1",
> local_ip="2403:400:31da:::18:6", remote_ip="2403:400:31da:::18:3"}
> ```
>
> The bridge interface has the `local_ip` addresses.
>
> ```
> $ ip a
> [abbreviated]
> 27: br-phy: mtu 1500 qdisc fq_codel
> state UP group default qlen 1000
> link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
> inet 100.87.18.6/32 scope global br-phy
> valid_lft forever preferred_lft forever
> inet6 2403:400:31da:::18:6/128 scope global
> valid_lft forever preferred_lft forever
> inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
> valid_lft forever preferred_lft forever
> ```
>
> The kernel routing table show has a `src` of `2403:400:31da:::18:6`
>
> ```
> $ ip -6 route
> [abbreviated]
> ::1 dev lo proto kernel metric 256 pref medium
> 2403:400:31da:::18:3 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy
> proto bgp src 2403:400:31da:::18:6 metric 20 pref medium
> 2403:400:31da:::18:6 dev br-phy proto kernel metric 256 pref medium
> 2403:400:31da::/48 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy proto bgp
> src 2403:400:31da:::18:6 metric 20 pref medium
> ```
> However, in OVS's route table, the link-local address of
> `fe80::dc03:37ff:fee2:1fef` is picked as the the SRC instead.
>
> ```
> $ ovs-appctl ovs/route/show
> Route Table:
> [abbreviated]
> Cached: 2403:400:31da:::18:3/128 dev br-phy GW fe80::920a:84ff:fe9e:9570
> SRC fe80::dc03:37ff:fee2:1fef
> Cached: 2403:400:31da:::18:6/128 dev br-phy SRC 2403:400:31da:::18:6
> ```
>
> I tried to rewrite the source address on bridge br-phy but this not
> seem to have any effect(no packet hits). My idea was to match udp
> packets of port 4789 and rewrite the IPv6 source address. Is my way
> of rewriting it correct?
>
> ```
>
> $ ovs-ofctl add-flow br-phy
> "priority=50,dl_type=0x86dd,ipv6_src=fe80::dc03:37ff:fee2:1fef,nw_proto=17,tp_dst=4789,actions=set_field:2403:400:31da:::18:6->ipv6_src,normal"
>
> $ ovs-ofctl dump-flows br-phy
> cookie=0x0, duration=653.933s, table=0, n_packets=0, n_bytes=0,
> priority=50,udp6,ipv6_src=fe80::dc03:37ff:fee2:1fef,tp_dst=4789
> actions=load:0x180006->NXM_NX_IPV6_SRC[0..63],load:0x2403040031da->NXM_NX_IPV6_SRC[64..127],NORMAL
> cookie=0x0, duration=275973.900s, table=0, n_packets=1638167,
> n_bytes=152347460, priority=0 actions=NORMAL
> ```
>
> What did work was that if I add a static route for destination with> the
> correct source address and add an ARP entry for the source. So if
> the source address was picked up correctly by OVS, the tunnel works
> correctly.
>
> ```
> $ ovs-appctl ovs/route/add 2403:400:31da:::18:3 br-phy
> 2403:400:31da:::18:6
> OK
> $ ovs-appctl tnl/arp/set br-phy 2403:400:31da:::18:6 90:0a:84:9e:95:70
> OK
>
> $ ovs-appctl ovs/route/show
> [abbreviated]
>
> Cached: 2403:400:31da:::18:3/128 dev br-phy GW fe80::920a:84ff:fe9e:9570
> SRC fe80::dc03:37ff:fee2:1fef
> Cached: 2403:400:31da:::18:6/128 dev br-phy SRC 2403:400:31da:::18:6
> User: 2403:400:31da:::18:3/128 dev br-phy GW 2403:400:31da:::18:6 SRC
>
Re: [ovs-discuss] Encapsulate VXLAN and then process other flows
Hi Ilya Maximets,
I did some further testing on my end. Just to make sure it's an address family
issue, I tried to configure all VXLAN interfaces with IPv6, but I ran into an
issue with the source IP address selection.
I specified the `local_ip` of the tunnel as `2403:400:31da:::18:6`, which
is also added on the bridge interface, but it picks the link-local address of
`fe80::dc03:37ff:fee2:1fef` instead. This gets dropped by other devices along
the way.
```
$ ovs-appctl dpctl/dump-flows -m netdev@ovs-netdev | grep 192.168.1.33
[abbreviated]
ufid:7f6b377d-8ee1-4605-91a7-34b1076068f2,
skb_priority(0/0),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),recirc_id(0),dp_hash(0/0),in_port(dpdk-vm101),packet_type(ns=0,id=0),eth(src=52:54:00:3d:cd:0c/00:00:00:00:00:00,dst=ff:ff:ff:ff:ff:ff/00:00:00:00:00:00),eth_type(0x0806),arp(sip=192.168.1.34/0.0.0.0,tip=192.168.1.33/0.0.0.0,op=1/0,sha=52:54:00:3d:cd:0c/00:00:00:00:00:00,tha=00:00:00:00:00:00/00:00:00:00:00:00),
packets:1, bytes:42, used:0.063s, dp:ovs,
actions:tnl_push(tnl_port(vxlan_sys_4789),header(size=70,type=4,eth(dst=90:0a:84:9e:95:70,src=de:03:37:e2:1f:ef,dl_type=0x86dd),ipv6(src=fe80::dc03:37ff:fee2:1fef,dst=2403:400:31da:::18:3,label=0,proto=17,tclass=0x0,hlimit=64),udp(src=0,dst=4789,csum=0x),vxlan(flags=0x800,vni=0x1)),out_port(br-phy)),push_vlan(vid=304,pcp=0),exit_p0,
dp-extra-info:miniflow_bits(4,0)
```
```
$ ovs-vsctl show
Bridge br-int
datapath_type: netdev
Port dpdk-vm101
Interface dpdk-vm101
type: dpdkvhostuserclient
options: {vhost-server-path="/var/run/vhost-sockets/dpdk-vm101"}
Port vxlan0
Interface vxlan0
type: vxlan
options: {dst_port="4789", key="1",
local_ip="2403:400:31da:::18:6", remote_ip="2403:400:31da:::18:3"}
```
The bridge interface has the `local_ip` addresses.
```
$ ip a
[abbreviated]
27: br-phy: mtu 1500 qdisc fq_codel
state UP group default qlen 1000
link/ether de:03:37:e2:1f:ef brd ff:ff:ff:ff:ff:ff
inet 100.87.18.6/32 scope global br-phy
valid_lft forever preferred_lft forever
inet6 2403:400:31da:::18:6/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::dc03:37ff:fee2:1fef/64 scope link
valid_lft forever preferred_lft forever
```
The kernel routing table show has a `src` of `2403:400:31da:::18:6`
```
$ ip -6 route
[abbreviated]
::1 dev lo proto kernel metric 256 pref medium
2403:400:31da:::18:3 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy proto
bgp src 2403:400:31da:::18:6 metric 20 pref medium
2403:400:31da:::18:6 dev br-phy proto kernel metric 256 pref medium
2403:400:31da::/48 nhid 93 via fe80::920a:84ff:fe9e:9570 dev br-phy proto bgp
src 2403:400:31da:::18:6 metric 20 pref medium
```
However, in OVS's route table, the link-local address of
`fe80::dc03:37ff:fee2:1fef` is picked as the the SRC instead.
```
$ ovs-appctl ovs/route/show
Route Table:
[abbreviated]
Cached: 2403:400:31da:::18:3/128 dev br-phy GW fe80::920a:84ff:fe9e:9570
SRC fe80::dc03:37ff:fee2:1fef
Cached: 2403:400:31da:::18:6/128 dev br-phy SRC 2403:400:31da:::18:6
```
I tried to rewrite the source address on bridge br-phy but this not seem to
have any effect (no packet hits). My idea was to match udp packets of port 4789
and rewrite the IPv6 source address. Is my way of rewriting it correct?
```
$ ovs-ofctl add-flow br-phy
"priority=50,dl_type=0x86dd,ipv6_src=fe80::dc03:37ff:fee2:1fef,nw_proto=17,tp_dst=4789,actions=set_field:2403:400:31da:::18:6->ipv6_src,normal"
$ ovs-ofctl dump-flows br-phy
cookie=0x0, duration=653.933s, table=0, n_packets=0, n_bytes=0,
priority=50,udp6,ipv6_src=fe80::dc03:37ff:fee2:1fef,tp_dst=4789
actions=load:0x180006->NXM_NX_IPV6_SRC[0..63],load:0x2403040031da->NXM_NX_IPV6_SRC[64..127],NORMAL
cookie=0x0, duration=275973.900s, table=0, n_packets=1638167,
n_bytes=152347460, priority=0 actions=NORMAL
```
What did work was that if I add a static route for destination with the correct
source address and add an ARP entry for the source. So if the source address
was picked up correctly by OVS, the tunnel works correctly.
```
$ ovs-appctl ovs/route/add 2403:400:31da:::18:3 br-phy
2403:400:31da:::18:6
OK
$ ovs-appctl tnl/arp/set br-phy 2403:400:31da:::18:6 90:0a:84:9e:95:70
OK
$ ovs-appctl ovs/route/show
[abbreviated]
Cached: 2403:400:31da:::18:3/128 dev br-phy GW fe80::920a:84ff:fe9e:9570
SRC fe80::dc03:37ff:fee2:1fef
Cached: 2403:400:31da:::18:6/128 dev br-phy SRC 2403:400:31da:::18:6
User: 2403:400:31da:::18:3/128 dev br-phy GW 2403:400:31da:::18:6 SRC
2403:400:31da:::18:6
$ ovs-appctl tnl/arp/show
IPMAC Bridge
==
fe80::920a:84ff:fe9e:9570
