On 1/24/23 14:12, Vladislav Odintsov wrote:
> Hi Ilya,
>
> could you please take a look on this?
> Maybe you can advice any direction how to investigate this issue?
>
> Thanks in advance.
>
> Regards,
> Vladislav Odintsov
>
>> On 24 Nov 2022, at 21:10, Anton Vazhnetsov <[email protected]> wrote:
>>
>> Hi, Terry!
>>
>> In continuation to our yesterday’s conversation [0], we were able to
>> reproduce the issue with KeyError. We found that the problem is not
>> connected with ovsdb-server load but it appears when the ovsdb-server schema
>> is converted online (it even doesn’t matter whether the real ovs schema is
>> changed) while the active connection persists.
>> Please use next commands to reproduce it:
>>
>> # in terminal 1
>>
>> ovsdb-tool create ./ovs.db /usr/share/ovn/ovn-nb.ovsschema
>> ovsdb-server --remote punix://$(pwd)/ovs.sock <punix://$(pwd)/ovs.sock>
>> $(pwd)/ovs.db -vconsole:dbg
>>
>>
>> # in terminal 2. run python shell
>> python3
>> # setup connection
>> import ovsdbapp.schema.ovn_northbound.impl_idl as nb_idl
>> from ovsdbapp.backend.ovs_idl import connection
>>
>> remote = "unix:///<path to ovs.sock>"
>>
>> def get_idl():
>> """Connection getter."""
>>
>> idl = connection.OvsdbIdl.from_server(remote, "OVN_Northbound",
>> leader_only=False)
>> return nb_idl.OvnNbApiIdlImpl(connection.Connection(idl, 100))
>>
>> idl = get_idl()
>>
>>
>> # in terminal 1
>> ovsdb-client convert unix:$(pwd)/ovs.sock /usr/share/ovn/ovn-nb.ovsschema
>>
>> # in terminal 2 python shell:
>> idl.ls_add("test").execute()
>>
>>
>> We get following traceback:
>>
>> Traceback (most recent call last):
>> File
>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
>> line 131, in run
>> txn.results.put(txn.do_commit())
>> File
>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
>> line 143, in do_commit
>> self.post_commit(txn)
>> File
>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py",
>> line 73, in post_commit
>> command.post_commit(txn)
>> File
>> "/usr/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line
>> 94, in post_commit
>> row = self.api.tables[self.table_name].rows[real_uuid]
>> File "/usr/lib64/python3.6/collections/__init__.py", line 991, in
>> __getitem__
>> raise KeyError(key)
>> KeyError: UUID('0256afa4-6dd0-4c2c-b6a2-686a360ab331')
>>
>> In ovsdb-server debug logs we see that update2 or update3 messages are not
>> sent from server in response to client’s transaction, just reply with result
>> UUID:
>> 2022-11-24T17:42:36Z|00306|poll_loop|DBG|wakeup due to [POLLIN] on fd 18
>> (///root/ovsdb-problem/ovs.sock<->) at lib/stream-fd.c:157
>> 2022-11-24T17:42:36Z|00307|jsonrpc|DBG|unix#5: received request,
>> method="transact",
>> params=["OVN_Northbound",{"uuid-name":"row03ef28d6_93f1_43bc_b07a_eae58d4bd1c5","table":"Logical_Switch","op":"insert","row":{"name”:"test"}}],
>> id=5
>> 2022-11-24T17:42:36Z|00308|jsonrpc|DBG|unix#5: send reply,
>> result=[{"uuid":["uuid","4eb7c407-beec-46ca-b816-19f942e57721"]}], id=5
>>
>> We checked same with ovn-nbctl running in daemon mode and found that the
>> problem is not reproduced (ovsdb-server after database conversion sends out
>> update3 message to ovn-nbctl daemon process in response to transaction, for
>> example ovs-appctl -t <nbctl daemon socket> run ls-add test-ls):
If the update3 is not sent, it means that the client doesn't monitor
this table. You need to look at monitor requests sent and replied
to figure out the difference.
Since the database conversion is involved, server will close all
monitors on schema update and notify all clients that are db_change_aware.
Clients should re-send monitor requests at this point. If clients
are not db_change_aware, server will just disconnect them, so they
can re-connect and see the new schema and send new monitor requests.
On a quick glance I don't see python idl handling 'monitor_canceled'
notification. That is most likely a root cause. Python IDL claims
to be db_change_aware, but it doesn't seem to be.
Best regards, Ilya Maximets.
>> 2022-11-24T17:54:51Z|00623|jsonrpc|DBG|unix#7: received request,
>> method="transact",
>> params=["OVN_Northbound",{"uuid-name":"rowcdb152ce_a9af_4761_b965_708ad300fcb7","table":"Logical_Switch","op":"insert","row":{"name":"test-ls"}},{"comment":"ovn-nbctl:
>> run ls-add test-ls","op":"comment"}], id=5
>> 2022-11-24T17:54:51Z|00624|jsonrpc|DBG|unix#7: send notification,
>> method="update3",
>> params=[["monid","OVN_Northbound"],"00000000-0000-0000-0000-000000000000",{"Logical_Switch":{"0b147f2c-248d-496a-b718-a5328d3c2995":{"insert":{"name":"test-ls"}}}}]
>> 2022-11-24T17:54:51Z|00625|jsonrpc|DBG|unix#7: send reply,
>> result=[{"uuid":["uuid","0b147f2c-248d-496a-b718-a5328d3c2995"]},{}], id=5
>>
>> So it seems that the problem is in python-ovs, not in ovsdb-server.
>> We tested with ovsdb-server running version 2.17.3 and python-ovs 2.13.5 and
>> also python-ovs 2.17.3, the behaviour is the same.
>>
>> Do you have any ideas what can be a reason for such behaviour?
>>
>> 0:
>> https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b
>>
>> <https://review.opendev.org/c/openstack/ovsdbapp/+/865454/comments/674c57e6_3849591b>
>>
>> Regards, Anton.
>
_______________________________________________
dev mailing list
[email protected]
https://mail.openvswitch.org/mailman/listinfo/ovs-dev
