[ovirt-users] Re: Destroyed VM blocking hosts/filling logs

David Kerry Thu, 08 Apr 2021 06:33:21 -0700

Hi Shani,

These VMs in particular are running just fine on other hosts (and
I'd like to keep them that way, preferably).


It looks like this command would delete the whole VM from the
entire system instead of just removing the stuck/shutdown instances
from the hosts it's not running on any more.

Can you confirm this is what it would do?  If so, is there another
option to remove these stuck "ghost" VM instances from the hosts they are
no longer running on?


Thanks

David


On 2021-04-08 3:20 a.m., Shani Leviim wrote:

Hi David,
You can delete the VM from the DB using this command:
SELECT DeleteVm('<vm-id>');

*Regards,
*
*Shani Leviim
*


On Wed, Apr 7, 2021 at 4:23 PM David Kerry <[email protected] 
<mailto:[email protected]>> wrote:

    Hello,

    This seems to be what the engine is trying to do, and failing at for some 
reason.

    eg:

    [root@ovirt-node217 ~]# vdsm-client Host getVMList fullStatus=True
    [
         "8b3964bc-cd3f-4f13-84c6-1811193c93eb",
         "132668b6-9992-451f-95ac-dbcbeb03f5f1"
    ]

    For reference:

    [root@ovirt-node217 ~]# virsh -r list --all
      Id    Name                           State
    ----------------------------------------------------
      -     vm-s2                          shut off
      -     vm-s1                          shut off

    And in the console, it shows a count of "2" beside this host, but on the 
host detail
    page, under the virtual-machine tab, the list is empty (these VMs are 
actually
    running on a different host).

    [root@ovirt-node217 ~]# vdsm-client VM destroy 
vmID="8b3964bc-cd3f-4f13-84c6-1811193c93eb"
    vdsm-client: Command VM.destroy with args {'vmID': 
'8b3964bc-cd3f-4f13-84c6-1811193c93eb'} failed:
    (code=100, message=General Exception: ("'1048576'",))

    I guess what I need is a way to remove/clean-up these VMs manually since 
ovirt
    does not seem to be able to do it by itself.

    This condition also blocks the host from being put into maintenance mode.

    When I reboot the host manually and "confirm host was rebooted", the VMs
    are still there and still stuck.

    Sincerely,

    David


    On 2021-04-07 6:01 a.m., Shani Leviim wrote:

    Hi,
    You can try with the vdsm-client tool:
    https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html 
<https://www.ovirt.org/develop/developer-guide/vdsm/vdsm-client.html>

    Stopping a VM:
    1) Get the vmId:
    # vdsm-client Host getVMList fullStatus=True

    2) Destroy the VM
    # vdsm-client VM destroy vmID=<vmID>

    *Regards,
    *
    *Shani Leviim
    *


    On Sat, Apr 3, 2021 at 7:50 AM <[email protected] 
<mailto:[email protected]>> wrote:

        Hello,

        I've somehow gotten one of my VMs stuck in a state that ovirt seems to 
be rather confused about its
        existence of now.  I'm running oVirt 4.3.10 and using oVirt Node on all 
the hosts.

        My engine and host event logs are now filling up very rapidly with this 
error:

        VDSM node217 command DestroyVDS failed: General Exception: 
("'1048576'",)

        I was playing with hugetable support, and that error number or string 
looks suspiciously
        like the "hugetable size" custom property I set on the VM.

        This VM was migrated to another host at one point as well, and now that 
host is also
        generating the same error as well.

        When I try to move these hosts to maintenance mode, they get stuck in 
"Preparing for
        Maintenance" while it tries to migrate/deal with the VM that's not 
there any more.

        Forcibly rebooting the hosts does not change anything.  The VM 
state/host seems to be
        captured somewhere persistent in this case.

        The VM in question is not running, and I can start it up on another 
host successfully,
        but ovirt still thinks it exists on the other 2 hosts no matter what I 
do.

        Is there perhaps some way to delete it from the engine database 
directly to straighten
        things out?

        Here's a dump of the vdsm log on one of the hosts.  I haven't been able 
to pinpoint what
        the exact issue is or how to fix it, but hopefully someone here will 
have seen this before?

        2021-04-03 04:40:35,515+0000 INFO  (jsonrpc/1) [api.virt] START 
destroy(gracefulAttempts=1) from=::ffff:10.100.0.210,58150, 
vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:48)
        2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm] 
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Release VM resources (vm:5186)
        2021-04-03 04:40:35,516+0000 WARN  (jsonrpc/1) [virt.vm] 
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') trying to set state to Powering 
down when already Down (vm:626)
        2021-04-03 04:40:35,516+0000 INFO  (jsonrpc/1) [virt.vm] 
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection 
(guestagent:455)
        2021-04-03 04:40:35,517+0000 INFO  (jsonrpc/1) [vdsm.api] START 
teardownImage(sdUUID='a08af6be-3802-4bb1-9fa5-4b6a10227290', 
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', 
imgUUID='9c896907-59b0-4983-9478-b36b2c2eb01e', volUUID=None) 
from=::ffff:10.100.0.210,58150, task_id=fc946d20-126a-4fd0-9078-91
        4b4a64b1d9 (api:48)
        2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [storage.StorageDomain] 
Removing image rundir link 
u'/var/run/vdsm/storage/a08af6be-3802-4bb1-9fa5-4b6a10227290/9c896907-59b0-4983-9478-b36b2c2eb01e'
 (fileSD:592)
        2021-04-03 04:40:35,518+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH 
teardownImage return=None from=::ffff:10.100.0.210,58150, 
task_id=fc946d20-126a-4fd0-9078-914b4a64b1d9 (api:54)
        2021-04-03 04:40:35,519+0000 INFO  (jsonrpc/1) [vdsm.api] START 
teardownImage(sdUUID='b891448d-dd92-4a7b-a51a-22abc3d7da67', 
spUUID='78dc095a-5238-11e8-b8bf-00163e6a7af9', 
imgUUID='c0e95483-35f1-4a61-958e-4e308b70d3f8', volUUID=None) 
from=::ffff:10.100.0.210,58150, task_id=77c0fdca-e13a-44b5-9a00-29
        0522b194b2 (api:48)
        2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [storage.StorageDomain] 
Removing image rundir link 
u'/var/run/vdsm/storage/b891448d-dd92-4a7b-a51a-22abc3d7da67/c0e95483-35f1-4a61-958e-4e308b70d3f8'
 (fileSD:592)
        2021-04-03 04:40:35,520+0000 INFO  (jsonrpc/1) [vdsm.api] FINISH 
teardownImage return=None from=::ffff:10.100.0.210,58150, 
task_id=77c0fdca-e13a-44b5-9a00-290522b194b2 (api:54)
        2021-04-03 04:40:35,521+0000 INFO  (jsonrpc/1) [virt.vm] 
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') Stopping connection 
(guestagent:455)
        2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] File: 
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.ovirt-guest-agent.0
 already removed (fileutils:54)
        2021-04-03 04:40:35,521+0000 WARN  (jsonrpc/1) [root] Attempting to 
remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 
(libvirtnetwork:198)
        2021-04-03 04:40:35,522+0000 WARN  (jsonrpc/1) [root] Attempting to 
remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 
(libvirtnetwork:205)
        2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to 
remove a non existing network: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 
(libvirtnetwork:198)
        2021-04-03 04:40:35,526+0000 WARN  (jsonrpc/1) [root] Attempting to 
remove a non existing net user: ovirtmgmt/58abf0cf-d7b9-4067-a86a-e619928368e7 
(libvirtnetwork:205)
        2021-04-03 04:40:35,527+0000 WARN  (jsonrpc/1) [root] File: 
/var/lib/libvirt/qemu/channels/58abf0cf-d7b9-4067-a86a-e619928368e7.org.qemu.guest_agent.0
 already removed (fileutils:54)
        2021-04-03 04:40:35,528+0000 WARN  (jsonrpc/1) [virt.vm] 
(vmId='58abf0cf-d7b9-4067-a86a-e619928368e7') timestamp already removed from 
stats cache (vm:2445)
        2021-04-03 04:40:35,531+0000 ERROR (jsonrpc/1) [api] FINISH destroy 
error='1048576' (api:134)
        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 124, 
in method
            ret = func(*args, **kwargs)
          File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 333, in 
destroy
            res = self.vm.destroy(gracefulAttempts)
          File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5283, 
in destroy
            result = self.doDestroy(gracefulAttempts, reason)
          File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5302, 
in doDestroy
            return self.releaseVm(gracefulAttempts)
          File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5207, 
in releaseVm
            self._cleanup()
          File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2382, 
in _cleanup
            self._cleanup_hugepages()
          File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2393, 
in _cleanup_hugepages
            self.nr_hugepages, self.hugepagesz
          File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 243, 
in calculate_required_deallocation
            _preallocated_hugepages(vm_hugepagesz))
          File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 262, 
in _preallocated_hugepages
            kernel_args['hugepagesz']
          File "/usr/lib/python2.7/site-packages/vdsm/hugepages.py", line 291, 
in _cmdline_hugepagesz_to_kb
            }[cmdline]
        KeyError: '1048576'
        2021-04-03 04:40:35,531+0000 INFO  (jsonrpc/1) [api.virt] FINISH destroy 
return={'status': {'message': 'General Exception: ("\'1048576\'",)', 'code': 
100}} from=::ffff:10.100.0.210,58150, vmId=58abf0cf-d7b9-4067-a86a-e619928368e7 (api:54)
        2021-04-03 04:40:35,532+0000 INFO  (jsonrpc/1) [jsonrpc.JsonRpcServer] 
RPC call VM.destroy failed (error 100) in 0.02 seconds (__init__:312)
        _______________________________________________
        Users mailing list -- [email protected] <mailto:[email protected]>
        To unsubscribe send an email to [email protected] 
<mailto:[email protected]>
        Privacy Statement: https://www.ovirt.org/privacy-policy.html 
<https://www.ovirt.org/privacy-policy.html>
        oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/ 
<https://www.ovirt.org/community/about/community-guidelines/>
        List Archives: 
https://lists.ovirt.org/archives/list/[email protected]/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/
 
<https://lists.ovirt.org/archives/list/[email protected]/message/KBABFNZIJ6T6QZXGYRSROHCO7LAS4VOM/>

_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/[email protected]/message/TZGZG35HHJRW5CQLVRJO7NDTPV4SFSR7/

[ovirt-users] Re: Destroyed VM blocking hosts/filling logs

Reply via email to