Re: [Openstack-operators] Fwd: Nova hypervisor uuid
Many thks Matt. If the issue will happen again I hope openstack commands will show duplicate entries and let me clean them. When happened nova-hypervisor list command did not show duplcated entries but I saw them only in the database. Regards Ignazio Il giorno Gio 29 Nov 2018 17:28 Matt Riedemann ha scritto: > On 11/29/2018 10:27 AM, Ignazio Cassano wrote: > > I did in the DB directly. > > I am using queens now. > > Any python client command to delete hold records or I must use api ? > > You can use the CLI: > > > https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete > > > https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete > > -- > > Thanks, > > Matt > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
Hi Matt, I did in the DB directly. I am using queens now. Any python client command to delete hold records or I must use api ? Thanks & Regards Ignazio Il giorno gio 29 nov 2018 alle ore 16:28 Matt Riedemann ha scritto: > On 11/29/2018 12:49 AM, Ignazio Cassano wrote: > > Hello Mattm > > Yes I mean sometimes I have same host/node names with different uuid in > > compute_nodes table in nova database > > I must delete nodes with uuid those not match with nova-hypervisor list > > command. > > At this time I have the following: > > MariaDB [nova]> select hypervisor_hostname,uuid,deleted from > compute_nodes; > > +-+--+-+ > > | hypervisor_hostname | uuid | deleted | > > +-+--+-+ > > | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | > > | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | > > +-+--+-+ > > 2 rows in set (0,00 sec) > > > > > > But sometimes old uuid are inserted in the table . > > I deleted again them. > > I restarted kvm nodes and now the table is ok. > > I also restarded each controller and the tables is ok. > > I do not know because 3 days ago I had same compute nodes names with > > different uuids. > > > > Thanks and Regards > > Ignazio > > OK I guess if it happens again, please get the > host/hypervisor_hostname/uuid/deleted values from the compute_nodes > table before you cleanup any entries. > > Also, when you're deleting the resources from the DB, are you doing it > in the DB directly or via the DELETE /os-services/{service_id} API? > Because the latter cleans up other related resources to the nova-compute > service (the services table record, the compute_nodes table record, the > related resource_providers table record in placement, and the > host_mappings table record in the nova API DB). The resource > provider/host mappings cleanup when deleting a compute service is a more > recent bug fix though which depending on your release you might not have: > > https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 > > -- > > Thanks, > > Matt > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
On 11/29/2018 10:27 AM, Ignazio Cassano wrote: I did in the DB directly. I am using queens now. Any python client command to delete hold records or I must use api ? You can use the CLI: https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-service-delete https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
On 11/29/2018 12:49 AM, Ignazio Cassano wrote: Hello Mattm Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database I must delete nodes with uuid those not match with nova-hypervisor list command. At this time I have the following: MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes; +-+--+-+ | hypervisor_hostname | uuid | deleted | +-+--+-+ | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | +-+--+-+ 2 rows in set (0,00 sec) But sometimes old uuid are inserted in the table . I deleted again them. I restarted kvm nodes and now the table is ok. I also restarded each controller and the tables is ok. I do not know because 3 days ago I had same compute nodes names with different uuids. Thanks and Regards Ignazio OK I guess if it happens again, please get the host/hypervisor_hostname/uuid/deleted values from the compute_nodes table before you cleanup any entries. Also, when you're deleting the resources from the DB, are you doing it in the DB directly or via the DELETE /os-services/{service_id} API? Because the latter cleans up other related resources to the nova-compute service (the services table record, the compute_nodes table record, the related resource_providers table record in placement, and the host_mappings table record in the nova API DB). The resource provider/host mappings cleanup when deleting a compute service is a more recent bug fix though which depending on your release you might not have: https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
Hello Mattm Yes I mean sometimes I have same host/node names with different uuid in compute_nodes table in nova database I must delete nodes with uuid those not match with nova-hypervisor list command. At this time I have the following: MariaDB [nova]> select hypervisor_hostname,uuid,deleted from compute_nodes; +-+--+-+ | hypervisor_hostname | uuid | deleted | +-+--+-+ | tst2-kvm02 | 802b21c2-11fb-4426-86b9-bf25c8a5ae1d | 0 | | tst2-kvm01 | ce27803b-06cd-44a7-b927-1fa42c813b0f | 0 | +-+--+-+ 2 rows in set (0,00 sec) But sometimes old uuid are inserted in the table . I deleted again them. I restarted kvm nodes and now the table is ok. I also restarded each controller and the tables is ok. I do not know because 3 days ago I had same compute nodes names with different uuids. Thanks and Regards Ignazio Il giorno mer 28 nov 2018 alle ore 17:54 Matt Riedemann ha scritto: > On 11/28/2018 4:19 AM, Ignazio Cassano wrote: > > Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. > > I am sure kvm nodes names are note changed. > > Tables where uuid are duplicated are: > > dataresource_providers in nova_api db > > compute_nodes in nova db > > Regards > > Ignazio > > It would be easier if you simply dumped the result of a select query on > the compute_nodes table where the duplicate nodes exist (you said > duplicate UUIDs but I think you mean duplicate host/node names with > different UUIDs, correct?). > > There is a unique constraint on host/hypervisor_hostname > (nodename)/deleted: > > schema.UniqueConstraint( > 'host', 'hypervisor_hostname', 'deleted', > name="uniq_compute_nodes0host0hypervisor_hostname0deleted"), > > So I'm wondering if the deleted field is not 0 on one of those because > if one is marked as deleted, then the compute service will create a new > compute_nodes table record on startup (and associated resource provider). > > -- > > Thanks, > > Matt > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
On 11/28/2018 4:19 AM, Ignazio Cassano wrote: Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio It would be easier if you simply dumped the result of a select query on the compute_nodes table where the duplicate nodes exist (you said duplicate UUIDs but I think you mean duplicate host/node names with different UUIDs, correct?). There is a unique constraint on host/hypervisor_hostname (nodename)/deleted: schema.UniqueConstraint( 'host', 'hypervisor_hostname', 'deleted', name="uniq_compute_nodes0host0hypervisor_hostname0deleted"), So I'm wondering if the deleted field is not 0 on one of those because if one is marked as deleted, then the compute service will create a new compute_nodes table record on startup (and associated resource provider). -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] Fwd: Nova hypervisor uuid
Hi Matt, sorry but I lost your answer and Gianpiero forwarded it to me. I am sure kvm nodes names are note changed. Tables where uuid are duplicated are: dataresource_providers in nova_api db compute_nodes in nova db Regards Ignazio Il 28/Nov/2018 11:09 AM, "Gianpiero Ardissono" ha scritto: > > -- Forwarded message - > From: Matt Riedemann > Date: mar 27 nov 2018, 19:03 > Subject: Re: [Openstack-operators] Nova hypervisor uuid > To: > > > On 11/27/2018 11:32 AM, Ignazio Cassano wrote: > > Hi All, > > Please anyone know where hypervisor uuid is retrived? > > Sometime updating kmv nodes with yum update it changes and in nova > > database 2 uuids are assigned to the same node. > > regards > > Ignazio > > > > > > > > > > ___ > > OpenStack-operators mailing list > > OpenStack-operators@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > > > > To be clear, do you mean the computes_nodes.uuid column value in the > cell database? Which is also used for the GET /os-hypervisors response > 'id' value if using microversion >= 2.53. If so, that is generated > randomly* when the compute_nodes table record is created: > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L588 > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/objects/compute_node.py#L312 > > When you hit this problem, are you sure the hostname on the compute host > is not changing? Because when nova-compute starts up, it should look for > the existing compute node record by host name and node name, which for > the libvirt driver should be the same. That lookup code is here: > > > https://github.com/openstack/nova/blob/8545ba2af7476e0884b5e7fb90965bef92d605bc/nova/compute/resource_tracker.py#L815 > > So the only way nova-compute should create a new compute_nodes table > record for the same host is if the host/node name changes during the > upgrade. Is the deleted value in the database the same (0) for both of > those records? > > * The exception to this is for the ironic driver which re-uses the > ironic node uuid as of this change: > https://review.openstack.org/#/c/571535/ > > -- > > Thanks, > > Matt > > ___ > OpenStack-operators mailing list > OpenStack-operators@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators > ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators