Thanks Ilya for the quick and useful response! We’ll dig into monitor/db_change_aware logic.
Regards, Vladislav Odintsov > On 24 Jan 2023, at 17:00, Ilya Maximets <i.maxim...@ovn.org> wrote: > > On 1/24/23 14:12, Vladislav Odintsov wrote: >> Hi Ilya, >> >> could you please take a look on this? >> Maybe you can advice any direction how to investigate this issue? >> >> Thanks in advance. >> >> Regards, >> Vladislav Odintsov >> >>> On 24 Nov 2022, at 21:10, Anton Vazhnetsov <dragen15...@gmail.com> wrote: >>> >>> Hi, Terry! >>> >>> In continuation to our yesterday’s conversation [0], we were able to >>> reproduce the issue with KeyError. We found that the problem is not >>> connected with ovsdb-server load but it appears when the ovsdb-server >>> schema is converted online (it even doesn’t matter whether the real ovs >>> schema is changed) while the active connection persists. >>> Please use next commands to reproduce it: >>> >>> # in terminal 1 >>> >>> ovsdb-tool create ./ovs.db /usr/share/ovn/ovn-nb.ovsschema >>> ovsdb-server --remote punix://$(pwd)/ovs.sock <punix://$(pwd)/ovs.sock> >>> $(pwd)/ovs.db -vconsole:dbg >>> >>> >>> # in terminal 2. run python shell >>> python3 >>> # setup connection >>> import ovsdbapp.schema.ovn_northbound.impl_idl as nb_idl >>> from ovsdbapp.backend.ovs_idl import connection >>> >>> remote = "unix:///<path to ovs.sock>" >>> >>> def get_idl(): >>> """Connection getter.""" >>> >>> idl = connection.OvsdbIdl.from_server(remote, "OVN_Northbound", >>> leader_only=False) >>> return nb_idl.OvnNbApiIdlImpl(connection.Connection(idl, 100)) >>> >>> idl = get_idl() >>> >>> >>> # in terminal 1 >>> ovsdb-client convert unix:$(pwd)/ovs.sock /usr/share/ovn/ovn-nb.ovsschema >>> >>> # in terminal 2 python shell: >>> idl.ls_add("test").execute() >>> >>> >>> We get following traceback: >>> >>> Traceback (most recent call last): >>> File >>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", >>> line 131, in run >>> txn.results.put(txn.do_commit()) >>> File >>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", >>> line 143, in do_commit >>> self.post_commit(txn) >>> File >>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", >>> line 73, in post_commit >>> command.post_commit(txn) >>> File >>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", >>> line 94, in post_commit >>> row = self.api.tables[self.table_name].rows[real_uuid] >>> File "/usr/lib64/python3.6/collections/__init__.py", line 991, in >>> __getitem__ >>> raise KeyError(key) >>> KeyError: UUID('0256afa4-6dd0-4c2c-b6a2-686a360ab331') >>> >>> In ovsdb-server debug logs we see that update2 or update3 messages are not >>> sent from server in response to client’s transaction, just reply with >>> result UUID: >>> 2022-11-24T17:42:36Z|00306|poll_loop|DBG|wakeup due to [POLLIN] on fd 18 >>> (///root/ovsdb-problem/ovs.sock<->) at lib/stream-fd.c:157 >>> 2022-11-24T17:42:36Z|00307|jsonrpc|DBG|unix#5: received request, >>> method="transact", >>> params=["OVN_Northbound",{"uuid-name":"row03ef28d6_93f1_43bc_b07a_eae58d4bd1c5","table":"Logical_Switch","op":"insert","row":{"name”:"test"}}], >>> id=5 >>> 2022-11-24T17:42:36Z|00308|jsonrpc|DBG|unix#5: send reply, >>> result=[{"uuid":["uuid","4eb7c407-beec-46ca-b816-19f942e57721"]}], id=5 >>> >>> We checked same with ovn-nbctl running in daemon mode and found that the >>> problem is not reproduced (ovsdb-server after database conversion sends out >>> update3 message to ovn-nbctl daemon process in response to transaction, for >>> example ovs-appctl -t <nbctl daemon socket> run ls-add test-ls): > > If the update3 is not sent, it means that the client doesn't monitor > this table. You need to look at monitor requests sent and replied > to figure out the difference. > > Since the database conversion is involved, server will close all > monitors on schema update and notify all clients that are db_change_aware. > Clients should re-send monitor requests at this point. If clients > are not db_change_aware, server will just disconnect them, so they > can re-connect and see the new schema and send new monitor requests. > > On a quick glance I don't see python idl handling 'monitor_canceled' > notification. That is most likely a root cause. Python IDL claims > to be db_change_aware, but it doesn't seem to be. > > Best regards, Ilya Maximets. > >>> 2022-11-24T17:54:51Z|00623|jsonrpc|DBG|unix#7: received request, >>> method="transact", >>> params=["OVN_Northbound",{"uuid-name":"rowcdb152ce_a9af_4761_b965_708ad300fcb7","table":"Logical_Switch","op":"insert","row":{"name":"test-ls"}},{"comment":"ovn-nbctl: >>> run ls-add test-ls","op":"comment"}], id=5 >>> 2022-11-24T17:54:51Z|00624|jsonrpc|DBG|unix#7: send notification, >>> method="update3", >>> params=[["monid","OVN_Northbound"],"00000000-0000-0000-0000-000000000000",{"Logical_Switch":{"0b147f2c-248d-496a-b718-a5328d3c2995":{"insert":{"name":"test-ls"}}}}] >>> 2022-11-24T17:54:51Z|00625|jsonrpc|DBG|unix#7: send reply, >>> result=[{"uuid":["uuid","0b147f2c-248d-496a-b718-a5328d3c2995"]},{}], id=5 >>> >>> So it seems that the problem is in python-ovs, not in ovsdb-server. >>> We tested with ovsdb-server running version 2.17.3 and python-ovs 2.13.5 >>> and also python-ovs 2.17.3, the behaviour is the same. >>> >>> Do you have any ideas what can be a reason for such behaviour? >>> >>> 0: >>> https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b<https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b> >>> >>> Regards, Anton. >> > > _______________________________________________ > dev mailing list > d...@openvswitch.org <mailto:d...@openvswitch.org> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev