Looking at clientIF.py on the patch Michal pointed to, concretely on:

+    def removeVmFromMonitoredDomains(self, vmId):
+        for dom in self.domainVmIds:
+            #only take lock here to allow runVm to take the lock in-between
+            with self.domainVmIdsLock:
+                try:
+                    self.domainVmIds[dom].remove(vmId)
+                except ValueError:
+                    pass
+                else:
+                    if not self.domainVmIds[dom]:
+                        del self.domainVmIds[dom]


If there is no exception we will be deleting an element from a dictionary while
iterating it. This is dangerous and should be avoided. The fastest alternative
to implement would be that in the "else" we append dom to a list of 
domainsToRemove
and after the iteration we get a lock and remove all of them.

----- Original Message -----
> From: "Michal Skrivanek" <michal.skriva...@redhat.com>
> To: "Sandro Bonazzola" <sbona...@redhat.com>, "Yeela Kaplan" 
> <ykap...@redhat.com>
> Cc: "Oved Ourfalli" <ov...@redhat.com>, "VDSM Project Development" 
> <vdsm-devel@lists.fedorahosted.org>
> Sent: Thursday, August 1, 2013 11:20:07 AM
> Subject: Re: [vdsm] Error while rebooting VM
> 
> Hi,
> the new code from I0f88ebcab07dea20a0288f6b75c5888a74d9e440 broke it…
> 
> Thanks,
> michal
> 
> On Aug 1, 2013, at 10:53 , Sandro Bonazzola <sbona...@redhat.com> wrote:
> 
> > Hi, while rebooting hosted-engine VM I've the following error:
> > 
> > vdsm-4.12.0-10.git295a069.fc19.x86_64 (nightly)
> > libvirt-1.0.5.4-1.fc19.x86_64 (F19)
> > mom-0.3.2-3.fc19.noarch (manually rebuilt from master due to missing
> > package in nightly)
> > 
> > 
> > 
> > Thread-621::DEBUG::2013-08-01
> > 10:42:53,011::BindingXMLRPC::986::vds::(wrapper) return vmGetStats with
> > {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status':
> > 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true',
> > 'guestFQDN': '', 'pid': '26846', 'displayIp': '0', 'displayPort': '-1',
> > 'session': 'Unknown', 'displaySecurePort': u'5900', 'cdrom':
> > '/home/Fedora-19-x86_64-DVD.iso', 'hash': '-6106157858121768929',
> > 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1',
> > 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr':
> > '00:16:3e:4f:10:9a', 'rxDropped': '0', 'rxErrors': '0', 'txDropped':
> > '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state':
> > 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId':
> > 'fd470849-17c9-436e-818d-16232f5b032b', 'monitorResponse': '0',
> > 'cpuUser': '0.69', 'disks': {u'hdc': {'readLatency': '0',
> > 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0',
> > 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}, u'hda':
> > {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '0',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0',
> > 'readRate': '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}},
> > 'boot': 'd', 'statsAge': '0.74', 'elapsedTime': '2834', 'vmType': 'kvm',
> > 'cpuSys': '0.13', 'timeOffset': -500L, 'appsList': [], 'guestIPs': '',
> > 'displayType': 'qxl'}]}
> > Thread-622::DEBUG::2013-08-01
> > 10:42:58,024::BindingXMLRPC::979::vds::(wrapper) client
> > [127.0.0.1]::call vmGetStats with
> > ('fd470849-17c9-436e-818d-16232f5b032b',) {}
> > Thread-622::DEBUG::2013-08-01
> > 10:42:58,024::BindingXMLRPC::986::vds::(wrapper) return vmGetStats with
> > {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status':
> > 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true',
> > 'guestFQDN': '', 'pid': '26846', 'displayIp': '0', 'displayPort': '-1',
> > 'session': 'Unknown', 'displaySecurePort': u'5900', 'cdrom':
> > '/home/Fedora-19-x86_64-DVD.iso', 'hash': '-6106157858121768929',
> > 'balloonInfo': {}, 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1',
> > 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr':
> > '00:16:3e:4f:10:9a', 'rxDropped': '0', 'rxErrors': '0', 'txDropped':
> > '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state':
> > 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId':
> > 'fd470849-17c9-436e-818d-16232f5b032b', 'monitorResponse': '0',
> > 'cpuUser': '0.69', 'disks': {u'hdc': {'readLatency': '0',
> > 'apparentsize': '0', 'writeLatency': '0', 'flushLatency': '0',
> > 'readRate': '0.00', 'truesize': '0', 'writeRate': '0.00'}, u'hda':
> > {'readLatency': '0', 'apparentsize': '26843545600', 'writeLatency': '0',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0',
> > 'readRate': '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}},
> > 'boot': 'd', 'statsAge': '0.76', 'elapsedTime': '2839', 'vmType': 'kvm',
> > 'cpuSys': '0.13', 'timeOffset': -500L, 'appsList': [], 'guestIPs': '',
> > 'displayType': 'qxl'}]}
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,268::vm::4714::vm.Vm::(_onLibvirtLifecycleEvent)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::event Shutdown detail 0
> > opaque None
> > VM Channels Listener::ERROR::2013-08-01
> > 10:42:59,335::vmChannels::53::vds::(_handle_event) Received 00000019 on
> > fileno 32
> > VM Channels Listener::DEBUG::2013-08-01
> > 10:42:59,336::vmChannels::128::vds::(_handle_unconnected) Trying to
> > connect fileno 32.
> > VM Channels Listener::DEBUG::2013-08-01
> > 10:42:59,336::guestIF::147::vm.Vm::(_connect)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Attempting connection to
> > /var/lib/libvirt/qemu/channels/fd470849-17c9-436e-818d-16232f5b032b.com.redhat.rhevm.vdsm
> > VM Channels Listener::DEBUG::2013-08-01
> > 10:42:59,336::guestIF::158::vm.Vm::(_connect)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Failed to connect to
> > /var/lib/libvirt/qemu/channels/fd470849-17c9-436e-818d-16232f5b032b.com.redhat.rhevm.vdsm
> > with 111
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,672::vm::4714::vm.Vm::(_onLibvirtLifecycleEvent)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::event Stopped detail 0
> > opaque None
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,672::vm::2092::vm.Vm::(_onQemuDeath)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::underlying process
> > disconnected
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,672::vm::4214::vm.Vm::(releaseVm)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Release VM resources
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,674::libvirtconnection::101::libvirtconnection::(wrapper)
> > Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Dominio non
> > trovato: no domain with matching uuid
> > 'fd470849-17c9-436e-818d-16232f5b032b' (oVirtHostedEngine)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,675::sampling::292::vm.Vm::(stop)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Stop statistics collection
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,675::vmChannels::205::vds::(unregister) Delete fileno 32 from
> > listener.
> > Thread-44::DEBUG::2013-08-01 10:42:59,675::sampling::323::vm.Vm::(run)
> > vmId=`fd470849-17c9-436e-818d-16232f5b032b`::Stats thread finished
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,675::libvirtconnection::101::libvirtconnection::(wrapper)
> > Unknown libvirterror: ecode: 42 edom: 10 level: 2 message: Dominio non
> > trovato: no domain with matching uuid
> > 'fd470849-17c9-436e-818d-16232f5b032b' (oVirtHostedEngine)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,676::task::579::TaskManager.Task::(_updateState)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::moving from state init ->
> > state preparing
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,676::logUtils::44::dispatcher::(wrapper) Run and protect:
> > teardownImage(sdUUID='ab35a1ff-700b-4354-8539-bc5f0daa6348',
> > spUUID='0c66b59f-bfa5-475d-82c8-ac8878db2565',
> > imgUUID='9ac2ea13-1de5-4a60-83c5-8700a23203b7', volUUID=None)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,676::resourceManager::197::ResourceManager.Request::(__init__)
> > ResName=`Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348`ReqID=`7b99cc2f-21fe-4cd3-9ed3-4be8ffb3b37a`::Request
> > was made in '/usr/share/vdsm/storage/hsm.py' line '3299' at 'teardownImage'
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,676::resourceManager::541::ResourceManager::(registerResource)
> > Trying to register resource
> > 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' for lock type 'shared'
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,676::resourceManager::600::ResourceManager::(registerResource)
> > Resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' is free. Now
> > locking as 'shared' (1 active user)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,676::resourceManager::237::ResourceManager.Request::(grant)
> > ResName=`Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348`ReqID=`7b99cc2f-21fe-4cd3-9ed3-4be8ffb3b37a`::Granted
> > request
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::task::811::TaskManager.Task::(resourceAcquired)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::_resourcesAcquired:
> > Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348 (shared)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::task::974::TaskManager.Task::(_decref)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::ref 1 aborting False
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,677::logUtils::47::dispatcher::(wrapper) Run and protect:
> > teardownImage, Return response: None
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::task::1168::TaskManager.Task::(prepare)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::finished: None
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::task::579::TaskManager.Task::(_updateState)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::moving from state preparing
> > -> state finished
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::939::ResourceManager.Owner::(releaseAll)
> > Owner.releaseAll requests {} resources
> > {'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348': < ResourceRef
> > 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348', isValid: 'True' obj:
> > 'None'>}
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::976::ResourceManager.Owner::(cancelAll)
> > Owner.cancelAll requests {}
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::615::ResourceManager::(releaseResource)
> > Trying to release resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348'
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::634::ResourceManager::(releaseResource)
> > Released resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' (0
> > active users)
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::640::ResourceManager::(releaseResource)
> > Resource 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348' is free, finding
> > out if anyone is waiting for it.
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,677::resourceManager::648::ResourceManager::(releaseResource)
> > No one is waiting for resource
> > 'Storage.ab35a1ff-700b-4354-8539-bc5f0daa6348', Clearing records.
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,678::task::974::TaskManager.Task::(_decref)
> > Task=`9be47a03-b9d9-4686-b43b-9764070a0e4f`::ref 0 aborting False
> > libvirtEventLoop::WARNING::2013-08-01
> > 10:42:59,678::clientIF::384::vds::(teardownVolumePath) Drive is not a
> > vdsm image: VOLWM_CHUNK_MB:1024 VOLWM_CHUNK_REPLICATE_MULT:2
> > VOLWM_FREE_PCT:50 _blockDev:False _checkIoTuneCategories:<bound method
> > Drive._checkIoTuneCategories of <vm.Drive object at 0x2f14210>>
> > _customize:<bound method Drive._customize of <vm.Drive object at
> > 0x2f14210>> _deviceXML:<disk device="cdrom" snapshot="no"
> > type="file"><source file="/home/Fedora-19-x86_64-DVD.iso"
> > startupPolicy="optional"/><target bus="ide"
> > dev="hdc"/><serial></serial></disk> _makeName:<bound method
> > Drive._makeName of <vm.Drive object at 0x2f14210>>
> > _validateIoTuneParams:<bound method Drive._validateIoTuneParams of
> > <vm.Drive object at 0x2f14210>> address:{u'bus': u'1', u'controller':
> > u'0', u'type': u'drive', u'target': u'0', u'unit': u'0'} alias:ide0-1-0
> > apparentsize:0 blockDev:False cache:none conf:{'status': 'Up', 'bridge':
> > 'ovirtmgmt', 'vmId': 'fd470849-17c9-436e-818d-16232f5b032b', 'pid':
> > '26846', 'drives': [{'index': '0', 'domainID':
> > 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'reqsize': '0', 'name': u'hda',
> > 'format': 'raw', 'volumeInfo': {'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'volType': 'path'}, 'address': {u'bus': u'0', u'controller': u'0',
> > u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'volumeID':
> > '1b63a000-a23e-46ad-9283-e7227cca4879', 'apparentsize': '26843545600',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'alias': u'ide0-0-0',
> > 'readonly': 'False', 'iface': 'ide', 'truesize': '0', 'poolID':
> > '0c66b59f-bfa5-475d-82c8-ac8878db2565', 'device': 'disk', 'shared':
> > False, 'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'propagateErrors': 'off', 'type': 'disk', 'volumeChain': [{'domainID':
> > 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'vmVolInfo': {'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'volType': 'path'}, 'leaseOffset': 0, 'volumeID':
> > '1b63a000-a23e-46ad-9283-e7227cca4879', 'leasePath':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879.lease',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'shared': False,
> > 'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879'}]},
> > {'index': 2, 'iface': 'ide', 'name': u'hdc', 'format': 'raw', 'address':
> > {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0',
> > u'unit': u'0'}, 'alias': u'ide0-1-0', 'readonly': 'True',
> > 'propagateErrors': 'off', 'shared': False, 'device': 'cdrom', 'path':
> > '/home/Fedora-19-x86_64-DVD.iso', 'truesize': 0, 'type': 'disk'}],
> > 'cdrom': '/home/Fedora-19-x86_64-DVD.iso', 'displaySecurePort': u'5900',
> > 'displayPort': '-1', 'pauseCode': 'NOERR', 'clientIp': '127.0.0.1',
> > 'nicModel': 'virtio', 'macAddr': '00:16:3e:4f:10:9a', 'vmName':
> > 'oVirtHostedEngine', 'boot': 'd', 'devices': [{'device': 'memballoon',
> > 'specParams': {'model': 'none'}, 'type': 'balloon'}, {'device':
> > 'virtio-serial', 'alias': u'virtio-serial0', 'type': 'controller',
> > 'address': {u'slot': u'0x04', u'bus': u'0x00', u'domain': u'0x0000',
> > u'type': u'pci', u'function': u'0x0'}}, {'device': 'qxl', 'specParams':
> > {'vram': '65536'}, 'alias': u'video0', 'type': 'video', 'address':
> > {u'slot': u'0x02', u'bus': u'0x00', u'domain': u'0x0000', u'type':
> > u'pci', u'function': u'0x0'}}, {'nicModel': 'virtio', 'macAddr':
> > '00:16:3e:4f:10:9a', 'linkActive': True, 'network': 'ovirtmgmt',
> > 'alias': u'net0', 'address': {u'slot': u'0x03', u'bus': u'0x00',
> > u'domain': u'0x0000', u'type': u'pci', u'function': u'0x0'}, 'device':
> > 'bridge', 'type': 'interface', 'name': u'vnet0'}, {'index': '0',
> > 'domainID': 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'reqsize': '0',
> > 'name': u'hda', 'format': 'raw', 'volumeInfo': {'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'volType': 'path'}, 'address': {u'bus': u'0', u'controller': u'0',
> > u'type': u'drive', u'target': u'0', u'unit': u'0'}, 'volumeID':
> > '1b63a000-a23e-46ad-9283-e7227cca4879', 'apparentsize': '26843545600',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'alias': u'ide0-0-0',
> > 'readonly': 'False', 'iface': 'ide', 'truesize': '0', 'poolID':
> > '0c66b59f-bfa5-475d-82c8-ac8878db2565', 'device': 'disk', 'shared':
> > False, 'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'propagateErrors': 'off', 'type': 'disk', 'volumeChain': [{'domainID':
> > 'ab35a1ff-700b-4354-8539-bc5f0daa6348', 'vmVolInfo': {'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879',
> > 'volType': 'path'}, 'leaseOffset': 0, 'volumeID':
> > '1b63a000-a23e-46ad-9283-e7227cca4879', 'leasePath':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879.lease',
> > 'imageID': '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'shared': False,
> > 'path':
> > '/rhev/data-center/0c66b59f-bfa5-475d-82c8-ac8878db2565/ab35a1ff-700b-4354-8539-bc5f0daa6348/images/9ac2ea13-1de5-4a60-83c5-8700a23203b7/1b63a000-a23e-46ad-9283-e7227cca4879'}]},
> > {'index': 2, 'iface': 'ide', 'name': u'hdc', 'format': 'raw', 'address':
> > {u'bus': u'1', u'controller': u'0', u'type': u'drive', u'target': u'0',
> > u'unit': u'0'}, 'alias': u'ide0-1-0', 'readonly': 'True',
> > 'propagateErrors': 'off', 'shared': False, 'device': 'cdrom', 'path':
> > '/home/Fedora-19-x86_64-DVD.iso', 'truesize': 0, 'type': 'disk'},
> > {'device': u'usb', 'alias': u'usb0', 'type': 'controller', 'address':
> > {u'slot': u'0x01', u'bus': u'0x00', u'domain': u'0x0000', u'type':
> > u'pci', u'function': u'0x2'}}, {'device': u'ide', 'alias': u'ide0',
> > 'type': 'controller', 'address': {u'slot': u'0x01', u'bus': u'0x00',
> > u'domain': u'0x0000', u'type': u'pci', u'function': u'0x1'}}, {'device':
> > u'unix', 'alias': u'channel0', 'type': u'channel', 'address': {u'bus':
> > u'0', u'controller': u'0', u'type': u'virtio-serial', u'port': u'1'}},
> > {'device': u'unix', 'alias': u'channel1', 'type': u'channel', 'address':
> > {u'bus': u'0', u'controller': u'0', u'type': u'virtio-serial', u'port':
> > u'2'}}, {'device': u'spicevmc', 'alias': u'channel2', 'type':
> > u'channel', 'address': {u'bus': u'0', u'controller': u'0', u'type':
> > u'virtio-serial', u'port': u'3'}}], 'smp': '2', 'vmType': 'kvm',
> > 'timeOffset': -500L, 'memSize': '4096', 'spiceSecureChannels':
> > 'smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir',
> > 'displayIp': '0', 'display': 'qxl'} createXmlElem:<bound method
> > Drive.createXmlElem of <vm.Drive object at 0x2f14210>> device:cdrom
> > drv:raw format:raw getNextVolumeSize:<bound method
> > Drive.getNextVolumeSize of <vm.Drive object at 0x2f14210>> getXML:<bound
> > method Drive.getXML of <vm.Drive object at 0x2f14210>> iface:ide index:2
> > isDiskReplicationInProgress:<bound method
> > Drive.isDiskReplicationInProgress of <vm.Drive object at 0x2f14210>>
> > isVdsmImage:<bound method Drive.isVdsmImage of <vm.Drive object at
> > 0x2f14210>> log:<logUtils.SimpleLogAdapter object at 0x2f23650> name:hdc
> > networkDev:False path:/home/Fedora-19-x86_64-DVD.iso propagateErrors:off
> > readonly:True reqsize:0 serial: shared:False truesize:0 type:cdrom
> > volExtensionChunk:1024 watermarkLimit:536870912
> > Traceback (most recent call last):
> >  File "/usr/share/vdsm/clientIF.py", line 378, in teardownVolumePath
> >    res = self.irs.teardownImage(drive['domainID'],
> >  File "/usr/share/vdsm/vm.py", line 1343, in __getitem__
> >    raise KeyError(key)
> > KeyError: 'domainID'
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,679::task::579::TaskManager.Task::(_updateState)
> > Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::moving from state init ->
> > state preparing
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,679::logUtils::44::dispatcher::(wrapper) Run and protect:
> > inappropriateDevices(thiefId='fd470849-17c9-436e-818d-16232f5b032b')
> > libvirtEventLoop::INFO::2013-08-01
> > 10:42:59,680::logUtils::47::dispatcher::(wrapper) Run and protect:
> > inappropriateDevices, Return response: None
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,680::task::1168::TaskManager.Task::(prepare)
> > Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::finished: None
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,680::task::579::TaskManager.Task::(_updateState)
> > Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::moving from state preparing
> > -> state finished
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,680::resourceManager::939::ResourceManager.Owner::(releaseAll)
> > Owner.releaseAll requests {} resources {}
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,681::resourceManager::976::ResourceManager.Owner::(cancelAll)
> > Owner.cancelAll requests {}
> > libvirtEventLoop::DEBUG::2013-08-01
> > 10:42:59,681::task::974::TaskManager.Task::(_decref)
> > Task=`23ebd2c4-84a1-4abe-a656-fa27d36d8425`::ref 0 aborting False
> > libvirtEventLoop::ERROR::2013-08-01
> > 10:42:59,681::clientIF::612::vds::(dispatchLibvirtEvents) Error running
> > VM callback
> > Traceback (most recent call last):
> >  File "/usr/share/vdsm/clientIF.py", line 584, in dispatchLibvirtEvents
> >    v._onLibvirtLifecycleEvent(event, detail, None)
> >  File "/usr/share/vdsm/vm.py", line 4728, in _onLibvirtLifecycleEvent
> >    self._onQemuDeath()
> >  File "/usr/share/vdsm/vm.py", line 2095, in _onQemuDeath
> >    response = self.releaseVm()
> >  File "/usr/share/vdsm/vm.py", line 4246, in releaseVm
> >    self.cif.removeVmFromMonitoredDomains(self.id)
> >  File "/usr/share/vdsm/clientIF.py", line 125, in
> > removeVmFromMonitoredDomains
> >    for dom in self.domainVmIds:
> > RuntimeError: dictionary changed size during iteration
> > Thread-24::DEBUG::2013-08-01
> > 10:43:00,274::fileSD::238::Storage.Misc.excCmd::(getReadDelay)
> > '/usr/bin/dd iflag=direct
> > if=/rhev/data-center/mnt/192.168.1.104:_home_images/ab35a1ff-700b-4354-8539-bc5f0daa6348/dom_md/metadata
> > bs=4096 count=1' (cwd None)
> > 
> > 
> > The VM is in the following state:
> > # vdsClient -s localhost getVmStats fd470849-17c9-436e-818d-16232f5b032b
> > 
> > fd470849-17c9-436e-818d-16232f5b032b
> >    Status = Powering down
> >    username = Unknown
> >    memUsage = 0
> >    acpiEnable = true
> >    guestFQDN =
> >    displayPort = -1
> >    session = Unknown
> >    displaySecurePort = 5900
> >    timeOffset = -500
> >    balloonInfo = {}
> >    pauseCode = NOERR
> >    network = {'vnet0': {'macAddr': '00:16:3e:4f:10:9a', 'rxDropped':
> > '0', 'txDropped': '0', 'rxErrors': '0', 'txRate': '0.0', 'rxRate':
> > '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name':
> > 'vnet0'}}
> >    displayType = qxl
> >    cpuUser = 0.69
> >    boot = d
> >    elapsedTime = 3400
> >    vmType = kvm
> >    cpuSys = 0.13
> >    appsList = []
> >    hash = -6106157858121768929
> >    pid = 26846
> >    displayIp = 0
> >    cdrom = /home/Fedora-19-x86_64-DVD.iso
> >    guestIPs =
> >    kvmEnable = true
> >    disks = {'hdc': {'readLatency': '0', 'apparentsize': '0',
> > 'writeLatency': '0', 'flushLatency': '0', 'readRate': '0.00',
> > 'truesize': '0', 'writeRate': '0.00'}, 'hda': {'readLatency': '0',
> > 'apparentsize': '26843545600', 'writeLatency': '0', 'imageID':
> > '9ac2ea13-1de5-4a60-83c5-8700a23203b7', 'flushLatency': '0', 'readRate':
> > '0.00', 'truesize': '1594064896', 'writeRate': '0.00'}}
> >    monitorResponse = -1
> >    statsAge = 559.96
> >    clientIp = 127.0.0.1
> > 
> > 
> > 
> > --
> > Sandro Bonazzola
> > Better technology. Faster innovation. Powered by community collaboration.
> > See how it works at redhat.com
> > 
> > _______________________________________________
> > vdsm-devel mailing list
> > vdsm-devel@lists.fedorahosted.org
> > https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
> _______________________________________________
> vdsm-devel mailing list
> vdsm-devel@lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
> 
_______________________________________________
vdsm-devel mailing list
vdsm-devel@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel

Reply via email to