Hi Everyone
I have a two node hosted engine cluster that's been running for a month
or two.
NFS is used for the VM's shared off the nodes on a second network
interface with different hostnames, I hope easier to migrate later on.
NFS 172.16.67.0/24 ov1-nfs.domain.dom on .1 and ov2-nfs.domain.dom
on .2. The NFS shares are working.
Management net is 10.10.10.224/28
Last night the cluster had communication errors, but I could not find
any issues, all nodes can ping & ssh with each other and engine.
Today, it got worse, the engine migrated all but 3 VM's to OV2, the node
with the engine. The VMs still on OV1 are there because the migration
for those failed. I manually can't migrate anything back to ov1. I
eventually shut down the engine and started on OV1, but still no joy.
The VMs are alive, both on OV1 & OV2. OV2 is currently in local
maintenance to stop the engine moving and stop the email alerts.
I have been through the logs, I see be a cert issue in libvirtd.log on
the receiving host?
Any help appreciated.
Mike
[root@ov1 ~]# libvirtd.log
2015-03-18 15:42:17.387+: 3017: error :
virNetTLSContextValidCertificate:1008 : Unable to verify TLS peer: The peer did
not send any certificate.
2015-03-18 15:42:17.387+: 3017: warning :
virNetTLSContextCheckCertificate:1142 : Certificate check failed Unable to
verify TLS peer: The peer did not send any certificate.
2015-03-18 15:42:17.387+: 3017: error :
virNetTLSContextCheckCertificate:1145 : authentication failed: Failed to verify
peer's certificate
[root@ov2 ~]# vdsm.log
Thread-49490::DEBUG::2015-03-18
15:42:17,294::migration::298::vm.Vm::(_startUnderlyingMigration)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::starting migration to
qemu+tls://ov1.domain.dom/system with miguri tcp://10.10.10.227
Thread-49525::DEBUG::2015-03-18 15:42:17,296::migration::361::vm.Vm::(run)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::migration downtime thread started
Thread-49526::DEBUG::2015-03-18
15:42:17,297::migration::410::vm.Vm::(monitor_migration)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::starting migration monitor thread
Thread-49490::DEBUG::2015-03-18
15:42:17,388::libvirtconnection::143::root::(wrapper) Unknown libvirterror:
ecode: 9 edom: 10 level: 2 message: operation failed: Failed to connect to
remote libvirt URI qemu+tls://ov1.domain.dom/system
Thread-49490::DEBUG::2015-03-18 15:42:17,390::migration::376::vm.Vm::(cancel)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::canceling migration downtime thread
Thread-49525::DEBUG::2015-03-18 15:42:17,391::migration::373::vm.Vm::(run)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::migration downtime thread exiting
Thread-49490::DEBUG::2015-03-18 15:42:17,391::migration::470::vm.Vm::(stop)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::stopping migration monitor thread
Thread-49490::ERROR::2015-03-18 15:42:17,393::migration::161::vm.Vm::(_recover)
vmId=`b44b2182-f943-4987-8421-8a98fd2a04d4`::operation failed: Failed to
connect to remote libvirt URI qemu+tls://ov1.domain.dom/system
[root@ov1 ~]# cat /var/log/vdsm/vdsm.log|grep MY_VM
Thread-7589263::DEBUG::2015-03-18
15:22:01,936::BindingXMLRPC::1133::vds::(wrapper) client [10.10.10.228]::call
vmMigrationCreate with ({'status': 'Up', 'acpiEnable': 'true',
'emulatedMachine': 'rhel6.5.0', 'afterMigrationStatus': '', 'tabletEnable':
'true', 'vmId': 'b44b2182-f943-4987-8421-8a98fd2a04d4', 'memGuaranteedSize':
2048, 'transparentHugePages': 'true', 'displayPort': '5929',
'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType':
'SandyBridge', 'smp': '2', 'migrationDest': 'libvirt', 'custom': {}, 'vmType':
'kvm', '_srcDomXML': "\n MY_VM\n
b44b2182-f943-4987-8421-8a98fd2a04d4\n 2097152\n 2097152\n 16\n \n1020\n \n
\n\n oVirt\n oVirt Node<
/entry>\n 6-6.el6.centos.12.2\n 3637-3434-5A43-3234-313130484A52\n b44b2182-f943-4987-8421-8a98fd2a04d4\n\n
\n \nhvm\n
\n \n \n\n \n
\nSandyBridge\n\n \n \n\n\n\n
\n destroy\n
restart\n destroy\n \n
/usr/libexec/qemu-kvm\n\n \n \n
\n \n
\n
6f9a8a62-8419-48ba-9642-0bee23a06d06\n \n \n\n\n
\n \n
\n \n \n \n \n
\n \n \n\n
\n \n \n\n
\n \n \n\n\n \n \n\n
\n \n \n
\n\n \n \n
\n \n \n \n \n \n \n \n\n\n \n \n
\n \n \n \n \n \n \n \n\n
\n \n
\n \n \n
\n \n \n \n \n \n\n\n \n \n \n
\n \n \n
\n \n
\n \n\n\n
\n
\n \n \n\n\n \n
\n \n \n\n\n
\n\n\n
\n \n\n
\n \n \n \n
