Hi Gerhard/all,
We also saw this on RHEL 7.4 using OVS 2.7.2.
It didn't stop until we restarted openvswitch service as well.
Log messages showed the following:
2017-11-07T20:56:00.688Z|986468|bridge|INFO|bridge br-int: added interface
ha-a1a195a9-c9 on port 12043
2017-11-07T20:56:00.735Z|986469|bridge|INFO|bridge br-int: added interface
ha-8139fe8c-ed on port 12048
2017-11-07T20:56:00.759Z|986470|bridge|INFO|bridge br-int: added interface
qr-63c9d5a3-1f on port 11368
2017-11-07T20:56:00.844Z|986471|bridge|INFO|bridge br-int: added interface
ha-a7d06805-bf on port 11098
2017-11-07T20:56:00.984Z|986472|bridge|INFO|bridge br-int: deleted
interface ha-87309246-d8 on port 12043
2017-11-07T20:56:00.985Z|986473|bridge|INFO|bridge br-int: deleted
interface ha-b5a217b0-48 on port 11098
2017-11-07T20:56:00.986Z|986474|bridge|INFO|bridge br-int: deleted
interface ha-0d126e69-92 on port 12048
2017-11-07T20:56:00.987Z|986475|bridge|INFO|bridge br-int: deleted
interface qr-4b19305a-77 on port 11368
2017-11-10T13:40:09.568Z|6103901|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
send request, method="transact",
params=["Open_vSwitch",{"lock":"ovs_vswitchd","op":"assert"},{"where":[["_uuid","==",["uuid","54208b80-2af8-4393-8a44-d6bd51c4606b"]]],"row":{"mac_in_use":"9a:4f:fe:ad:f5:ea","ifindex":2960482},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","70deb264-70cc-46bd-b9c7-ea8c451d03cf"]]],"row":{"mac_in_use":"ba:65:56:71:58:64","ifindex":2960481},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","d6007dca-f989-4c94-b1b5-9cd9b4ff44de"]]],"row":{"mac_in_use":"ce:d1:92:7b:e7:8d","ifindex":2960479},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","2e60fcbe-6b75-406f-be07-787b345aeba0"]]],"row":{"mac_in_use":"6a:2c:ec:23:5c:42","ifindex":2960480},"op":"update","table":"Interface"}],
id=1317397
2017-11-10T13:40:09.916Z|6103933|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
send request, method="transact",
params=["Open_vSwitch",{"lock":"ovs_vswitchd","op":"assert"},{"where":[["_uuid","==",["uuid","3a680741-02b8-4e63-a018-827c60071d11"]]],"row":{"mac_in_use":"3a:9c:05:55:c2:ec","ifindex":2960486},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","15a84122-a3fb-4fc1-b5bc-cd918f50880d"]]],"row":{"mac_in_use":"f2:7a:37:5d:52:33","ifindex":2960483},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","7dbf8616-cddd-4d09-9363-4d1d68823a52"]]],"row":{"mac_in_use":"62:39:a7:d6:5a:5f","ifindex":2960484},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","20a86e8f-090f-49c5-9bf0-ddcc1032bd3d"]]],"row":{"mac_in_use":"fe:f2:c5:bc:40:2e","ifindex":2960485},"op":"update","table":"Interface"}],
id=1317398
2017-11-10T13:40:09.923Z|6103934|jsonrpc|DBG|unix:/var/run/openvswitch/db.sock:
send request, method="transact",
params=["Open_vSwitch",{"lock":"ovs_vswitchd","op":"assert"},{"where":[["_uuid","==",["uuid","3a680741-02b8-4e63-a018-827c60071d11"]]],"row":{"mac_in_use":"3a:9c:05:55:c2:ec","ifindex":2960486},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","15a84122-a3fb-4fc1-b5bc-cd918f50880d"]]],"row":{"mac_in_use":"f2:7a:37:5d:52:33","ifindex":2960483},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","7dbf8616-cddd-4d09-9363-4d1d68823a52"]]],"row":{"mac_in_use":"62:39:a7:d6:5a:5f","ifindex":2960484},"op":"update","table":"Interface"},{"where":[["_uuid","==",["uuid","20a86e8f-090f-49c5-9bf0-ddcc1032bd3d"]]],"row":{"mac_in_use":"fe:f2:c5:bc:40:2e","ifindex":2960485},"op":"update","table":"Interface"}],
id=1317399
Thanks,
Daniel
On Mon, Nov 13, 2017 at 11:36 AM, Gerhard Muntingh <[email protected]>
wrote:
> Hi everyone,
>
> We experienced an openvswitch issue on our redhat openstack
> infrastructure. Since it's rather exotic, and we haven't been
> able to reproduce it, I'm sharing it here.
>
> After some high CPU, openvswitch got into some sort of
> device-port-contention looking like this:
>
> 2017-11-01T02:35:03.742Z|1036196|bridge|INFO|bridge br-int: added
> interface tap6ec8832c-4a on port 64
> 2017-11-01T02:35:03.828Z|1036197|bridge|INFO|bridge br-int: added
> interface tap87e621ea-f4 on port 92
> 2017-11-01T02:35:03.853Z|1036198|bridge|INFO|bridge br-int: added
> interface tap2f90dc94-d3 on port 64
> 2017-11-01T02:35:03.913Z|1036199|bridge|INFO|bridge br-int: added
> interface tap45874cb0-cd on port 92
> 2017-11-01T02:35:03.936Z|1036200|bridge|INFO|bridge br-int: added
> interface tap6ec8832c-4a on port 64
> 2017-11-01T02:35:03.939Z|1036201|netdev_linux|WARN|Dropped 271 log
> messages in last 12 seconds (most recently, 1 seconds ago) due to excessive
> rate
> 2017-11-01T02:35:03.939Z|1036202|netdev_linux|WARN|tap2f90dc94-d3:
> removing policing failed: No such device
> 2017-11-01T02:35:04.040Z|1036203|bridge|INFO|bridge br-int: added
> interface tap87e621ea-f4 on port 92
> 2017-11-01T02:35:04.070Z|1036204|bridge|INFO|bridge br-int: added
> interface tap2f90dc94-d3 on port 64
> 2017-11-01T02:35:04.131Z|1036205|bridge|INFO|bridge br-int: added
> interface tap45874cb0-cd on port 92
>
> The rate of these messages increased when we drained the
> machine. Maybe the main thread got more spare cpu cycles to
> use for this after draining.
>
> Restarting openvswitch resolved the issue.
>
> Environment:
> - Redhat 7.4
> - Openstack: newton
> - openvswitch: 2.6.1.10.git20161206.el7fdp (redhat rpm)
> - No DPDK or SRV-IO
>
>
> Kind regards,
>
> Gerhard Muntingh.
> _______________________________________________
> discuss mailing list
> [email protected]
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
_______________________________________________
discuss mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
