Hi, yeah both snippets are actually telling you that there was something really unexpected. The first one comes from a call by MOM who is trying to get some IO limiting information about the VM. VDSM responded with "I know about the VM, but I can't find it in libvirt".
The second part is something similar.. VDSM is expecting a VM to be known to libvirt, but there is no sign of it. I am adding some vdsm people who might know more about how to debug this. Best regards Martin Sivak On Fri, Feb 24, 2017 at 6:25 AM, Charles Tassell <[email protected]> wrote: > Hi Martin, > > Thanks for responding, and sorry for the long delay in getting back to > you. Where would I find the libvirt log? I checked on the host and I have > individual log files in /var/log/libvirt/qemu/<vmName>.log, but nothing for > the guest I'm trying to migrate. > > I did some more digging and found these errors, which seem to point to > some sort of issue communicating with the JsonRpcServer module. I've put > the full vdsm.log output from the migration attempt up at > http://krissy.islandadmin.ca/public/vdsm.txt but the interesting parts seems > to be: > > jsonrpc.Executor/3::DEBUG::2017-02-24 > 01:01:36,415::__init__::530::jsonrpc.JsonRpcServer::(_handle_request) > Calling 'Host.getAllVmIoTunePolicies' in bridge with {} > jsonrpc.Executor/3::ERROR::2017-02-24 > 01:01:36,436::__init__::550::jsonrpc.JsonRpcServer::(_handle_request) > Internal server error > Traceback (most recent call last): > File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 545, > in _handle_request > res = method(**params) > File "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 195, in > _dynamicMethod > result = fn(*methodArgs) > File "/usr/share/vdsm/API.py", line 1358, in getAllVmIoTunePolicies > io_tune_policies_dict = self._cif.getAllVmIoTunePolicies() > File "/usr/share/vdsm/clientIF.py", line 447, in getAllVmIoTunePolicies > vm_io_tune_policies[v.id] = {'policy': v.getIoTunePolicy(), > File "/usr/share/vdsm/virt/vm.py", line 2772, in getIoTunePolicy > qos = self._getVmPolicy() > File "/usr/share/vdsm/virt/vm.py", line 2746, in _getVmPolicy > metadata_xml = self._dom.metadata( > File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 47, > in __getattr__ > % self.vmid) > NotConnectedError: VM u'3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' was not > started yet or was shut down > jsonrpc.Executor/3::INFO::2017-02-24 > 01:01:36,437::__init__::513::jsonrpc.JsonRpcServer::(_serveRequest) RPC call > Host.getAllVmIoTunePolicies failed (error -32603) in 0.03 seconds > Thread-351015::DEBUG::2017-02-24 > 01:01:36,471::lvm::288::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> > = 0 > Thread-351015::DEBUG::2017-02-24 > 01:01:36,471::fileUtils::209::Storage.fileUtils::(createdir) Creating > directory: > /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e3045a3c18 > Thread-351015::DEBUG::2017-02-24 > 01:01:36,472::blockSD::1186::Storage.StorageDomain::(createImageLinks) > Creating symlink from > /dev/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/2e00f0ba-5d93-4c3b-af37-b53dcc68a43f > to /v > Thread-351015::DEBUG::2017-02-24 > 01:01:36,472::blockSD::1150::Storage.StorageDomain::(linkBCImage) Creating > symlink from > /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/a8da74c5-24d8-46c1-9a70-92e304 > > ... > > > jsonrpc.Executor/6::DEBUG::2017-02-24 > 01:01:37,645::resourceManager::241::Storage.ResourceManager.Request::(grant) > ResName=`Storage.b22b3666-f8d6-4beb-ac19-f2a6c8f49271`ReqID=`108242ef-f04e-4b9b-8a08-8f9ed1fae9 > jsonrpc.Executor/6::DEBUG::2017-02-24 > 01:01:37,645::task::829::Storage.TaskManager.Task::(resourceAcquired) > Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::_resourcesAcquired: > Storage.b22b3666-f8d6-4beb-ac19-f2a6c > jsonrpc.Executor/6::DEBUG::2017-02-24 > 01:01:37,645::task::995::Storage.TaskManager.Task::(_decref) > Task=`a9e9fcd9-65fe-4652-96d1-8afc2cf47269`::ref 1 aborting False > jsonrpc.Executor/6::DEBUG::2017-02-24 > 01:01:37,645::fileUtils::190::Storage.fileUtils::(cleanupdir) Removing > directory: > /var/run/vdsm/storage/b22b3666-f8d6-4beb-ac19-f2a6c8f49271/9cdfe4ab-c23e-42cd-8f45-2ca573a > jsonrpc.Executor/6::DEBUG::2017-02-24 > 01:01:37,646::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/taskset > --cpu-list 0-15 /usr/bin/sudo -n /usr/sbin/lvm lvs --config ' devices { > preferred_names = ["^/dev/mapper > Thread-351015::ERROR::2017-02-24 > 01:01:37,656::vm::767::virt.vm::(_startUnderlyingVm) > vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Failed to start a migration > destination vm > Traceback (most recent call last): > File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm > self._completeIncomingMigration() > File "/usr/share/vdsm/virt/vm.py", line 3113, in > _completeIncomingMigration > self._incomingMigrationFinished.isSet(), usedTimeout) > File "/usr/share/vdsm/virt/vm.py", line 3196, in > _attachLibvirtDomainAfterMigration > raise MigrationError(e.get_error_message()) > MigrationError: Domain not found: no domain with matching uuid > '3390a529-e0c3-4f1f-9c8f-7595b2b3f38f' > Thread-351015::INFO::2017-02-24 > 01:01:37,658::vm::1330::virt.vm::(setDownStatus) > vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Changed state to Down: VM > failed to migrate (code=8) > Thread-351015::INFO::2017-02-24 > 01:01:37,658::guestagent::430::virt.vm::(stop) > vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Stopping connection > Thread-351015::DEBUG::2017-02-24 > 01:01:37,660::__init__::209::jsonrpc.Notification::(emit) Sending event > {"params": {"notify_time": 5250206560, > "3390a529-e0c3-4f1f-9c8f-7595b2b3f38f": {"status": "Down", "timeOf > Thread-351015::DEBUG::2017-02-24 > 01:01:37,660::vm::777::virt.vm::(_startUnderlyingVm) > vmId=`3390a529-e0c3-4f1f-9c8f-7595b2b3f38f`::Releasing incoming migration > semaphore > > > > On 2017-02-15 05:16 AM, Martin Sivak wrote: >> >> Hi Charles, >> >> I am not sure about the migration issue, we need more logs [1] to be >> able to tell, but the SSL issue is probably just annoying and not a >> functionality bug (see >> https://bugzilla.redhat.com/show_bug.cgi?id=1412583). >> >> Best regards >> >> [1] libvirt log for the domain, the VM process might be crashing for some >> reason >> >> -- >> Martin Sivak >> SLA / oVirt >> >> On Wed, Feb 15, 2017 at 3:45 AM, Charles Tassell <[email protected]> >> wrote: >>> >>> Hi Everyone, >>> >>> I recently installed oVirt 4.0 on two servers with an iSCSI backstore, >>> and >>> I'm having two issues I'm having trouble tracking down. First, the >>> initial >>> datastore that the hosted engine is running on won't connect. It just >>> says >>> "Unattached" in the GUI, and I can't find an error message anywhere. Not >>> a >>> huge deal, it just means I can't see the hosted engine itself in the GUI. >>> >>> More importantly: I can't migrate VMs. They fail with errors like >>> this in >>> the vdsm.log of the destination host: >>> >>> Thread-60381::ERROR::2017-02-14 >>> 21:55:47,921::vm::767::virt.vm::(_startUnderlyingVm) >>> vmId=`59f235cc-3ab2-4dae-b460-62be850e3d5d`::Failed to start a migration >>> destination vm >>> Traceback (most recent call last): >>> File "/usr/share/vdsm/virt/vm.py", line 733, in _startUnderlyingVm >>> self._completeIncomingMigration() >>> File "/usr/share/vdsm/virt/vm.py", line 3113, in >>> _completeIncomingMigration >>> self._incomingMigrationFinished.isSet(), usedTimeout) >>> File "/usr/share/vdsm/virt/vm.py", line 3196, in >>> _attachLibvirtDomainAfterMigration >>> raise MigrationError(e.get_error_message()) >>> MigrationError: Domain not found: no domain with matching uuid >>> '59f235cc-3ab2-4dae-b460-62be850e3d5d' >>> >>> If I shut the VM down and start it on the host it works fine, it just >>> won't migrate. I notice I'm also seeing a lot (like every 7 seconds) of >>> this error: >>> >>> JsonRpc (StompReactor)::ERROR::2017-02-14 >>> 21:57:33,984::betterAsyncore::113::vds.dispatcher::(recv) SSL error >>> during >>> reading data: unexpected eof >>> >>> Which I can't seem to diagnose. Any ideas? >>> >>> >>> _______________________________________________ >>> Users mailing list >>> [email protected] >>> http://lists.ovirt.org/mailman/listinfo/users > > > _______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
