Hi,
I have functional OVN environment and now trying to configure DPDK
compute node for high performance networking but look like something
is not correct, i can spin up VM and during vm creation it created
socket in /var/run/openvswitch/vhu3209c4f2-ac= and after 30 second or
so it wipe out whole dir to me look like whole openvswitch crashed and
restart which wipe down directory, not sure what is wrong but
following error logs i am seeing in nova-compute
root@ovn-lab-comp-dpdk-1:/var/run/openvswitch# journalctl -u nova-compute -n100
-- Logs begin at Wed 2021-09-22 13:40:57 UTC, end at Wed 2021-09-22
23:13:01 UTC. --
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45497]:
assert len(_active) == 1
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45497]:
AssertionError:
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45500]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45500]:
Traceback (most recent call last):
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45500]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45500]:
assert len(_active) == 1
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45500]:
AssertionError:
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45503]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45503]:
Traceback (most recent call last):
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45503]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45503]:
assert len(_active) == 1
Sep 22 23:09:52 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45503]:
AssertionError:
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[39806]:
2021-09-22 23:09:55.353 39806 INFO nova.virt.libvirt.imagecache
[req-c8bd7c81-2acb-4f0b-8109-e189448d89ec - - - - -] image
023a1566-4320-48c8-b154-a23ead007c38 at (/var>
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45506]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45506]:
Traceback (most recent call last):
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45506]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45506]:
assert len(_active) == 1
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45506]:
AssertionError:
Sep 22 23:09:55 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[39806]:
2021-09-22 23:09:55.412 39806 INFO nova.virt.libvirt.imagecache
[req-c8bd7c81-2acb-4f0b-8109-e189448d89ec - - - - -] Active base
files: /var/lib/nova/instances/_base/07>
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45580]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45580]:
Traceback (most recent call last):
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45580]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45580]:
assert len(_active) == 1
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45580]:
AssertionError:
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45583]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45583]:
Traceback (most recent call last):
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45583]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45583]:
assert len(_active) == 1
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45583]:
AssertionError:
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45586]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45586]:
Traceback (most recent call last):
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45586]:
File "/usr/lib/python3.8/threading.py", line 1454, in _after_fork
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45586]:
assert len(_active) == 1
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45586]:
AssertionError:
Sep 22 23:10:53 ovn-lab-comp-dpdk-1.v1v0x.net nova-compute[45589]:
Exception ignored in: <function _after_fork at 0x7f8194d67670>
###### openvswitch logs
2021-09-22T23:07:01.878Z|00064|netdev_dpdk|INFO|vHost User device
'vhu3209c4f2-ac' created in 'client' mode, using client socket
'/var/run/openvswitch/vhu3209c4f2-ac'
2021-09-22T23:07:01.882Z|00065|dpdk|WARN|VHOST_CONFIG: failed to
connect to /var/run/openvswitch/vhu3209c4f2-ac: No such file or
directory
2021-09-22T23:07:01.882Z|00066|dpdk|INFO|VHOST_CONFIG:
/var/run/openvswitch/vhu3209c4f2-ac: reconnecting...
2021-09-22T23:07:02.045Z|00067|dpif_netdev|INFO|Core 11 on numa node 0
assigned port 'vhu3209c4f2-ac' rx queue 0 (measured processing cycles
0).
2021-09-22T23:07:02.045Z|00068|bridge|INFO|bridge br-int: added
interface vhu3209c4f2-ac on port 1
2021-09-22T23:07:02.045Z|00069|bridge|INFO|bridge br-int: added
interface patch-br-int-to-provnet-d16df70e-fd3b-4755-9777-5c35b3553002
on port 10
2021-09-22T23:07:02.046Z|00070|dpif_netdev|INFO|Core 11 on numa node 0
assigned port 'vhu3209c4f2-ac' rx queue 0 (measured processing cycles
0).
2021-09-22T23:07:02.046Z|00071|bridge|INFO|bridge br-int: added
interface tap32a819cd-80 on port 3
2021-09-22T23:07:02.079Z|00072|dpdk|INFO|Device with port_id=0 already stopped
2021-09-22T23:07:02.217Z|00073|netdev_dpdk|INFO|Port 0: 8c:dc:d4:b3:1d:04
This is what i configured in ml2_conf.ini
[ovn]
ovn_native_dhcp = True
ovn_nb_connection = tcp:10.62.7.252:6641
ovn_sb_connection = tcp:10.62.7.252:6642
ovn_l3_scheduler = leastloaded
ovn_metadata_enabled = True
# Security groups
vhost_sock_dir = /var/run/openvswitch/
vif_type = vhostuser
I am running libvirt with root user to avoid any permission issue
during socket creation.
in ovs-vsctl i can see vhu* path
Port tap32a819cd-80
Interface tap32a819cd-80
Port vhu3209c4f2-ac
Interface vhu3209c4f2-ac
type: dpdkvhostuserclient
options:
{vhost-server-path="/var/run/openvswitch/vhu3209c4f2-ac"}
Question is why does my openvswitch get crashed or reboot after creating vm?
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
