I've got 5 Xen servers (xen0..xen4), running stock RHEL 5.3 x86_64, fully
patched, on a variety of Dell server hardware with CPUs ranging from Irwindale
to Nehalem. All guests under test are also RHEL 5.3 x86_64, fully patched. Each
guest has its own SAN LUN, shared among all hosts, addressed by named
multipath.conf entries.
If I "xm create" a guest on a host with Xeon Irwindale, 5060, or 5160 CPUs,
then I am unable to live-migrate that guest to a Dell R710 with two quad-core
Intel L5520 CPUs.
But if I "xm create" the same guest on the L5520, then I am able to
live-migrate to the older systems, and back to the newer system, any number of
times.
Of course it's possible that I've got some configuration difference other that
CPU, but I can't find one after diff -ur'ing /etc/sysconfig and /etc/xen.
User-space observations: The sending host accepts xm migrate -l xen4. Watching
xm list on the source and destination, I see the guest's RAM gradually increase
to the correct value; then the "migrating-guest" entry goes away on the source;
and finally, the guest disappears on the destination.
I'm not entirely sure, but this *might* have worked before, possibly breaking
with 2.6.18-128.1.16.el5xen. Upgrading source, destination, and guests to
2.6.18-128.2.1.el5xen did not help.
This seems to be the first hard error in xend.log:
(XendCheckpoint:351) ERROR Internal error: Failed to pin batch of 65 page
tables
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:281)
XendDomainInfo.restore(['domain', ['domid', '33'], ['uuid',
'0513217c-e0bc-4893-bb04-29114018cd62'], ['vcpus', '1'], ['vcpu_avail', '1'],
['cpu_weight', '1.0'], ['memory', '600'], ['shadow_memory', '0'], ['maxmem',
'2000'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'xen'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'],
['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:65:a1:6d']]], ['device',
['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname',
'phy:/dev/mapper/xen'], ['mode', 'w']]], ['device', ['vkbd', ['backend',
'0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused',
'1'], ['xauthority', '/root/.Xauthority']]], ['s
tate', '--p---'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.0'],
['online_vcpus', '1'], ['up_time', '58.9367001057'], ['start_time',
'1247855358.21'], ['store_mfn', '573954'], ['console_mfn', '573953']])
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:312)
parseConfig: config is ['domain', ['domid', '33'], ['uuid',
'0513217c-e0bc-4893-bb04-29114018cd62'], ['vcpus', '1'], ['vcpu_avail', '1'],
['cpu_weight', '1.0'], ['memory', '600'], ['shadow_memory', '0'], ['maxmem',
'2000'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'xen'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'],
['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:65:a1:6d']]], ['device',
['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname',
'phy:/dev/mapper/xen'], ['mode', 'w']]], ['device', ['vkbd', ['backend',
'0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused',
'1'], ['xauthority', '/root/.Xauthority']]], ['s
tate', '--p---'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.0'],
['online_vcpus', '1'], ['up_time', '58.9367001057'], ['start_time',
'1247855358.21'], ['store_mfn', '573954'], ['console_mfn', '573953']]
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:417)
parseConfig: result is {'shadow_memory': 0, 'start_time': 1247855358.21,
'uuid': '6b25d1b9-d648-48e2-a051-13151265a16d', 'on_crash': 'restart',
'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['ramdisk',
'/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']], 'on_poweroff': 'destroy', 'bootloader_args': None, 'cpus':
None, 'name': 'xen', 'backend': [], 'vcpus': 1, 'cpu_weight': 1.0, 'features':
None, 'vcpu_avail': 1, 'memory': 600, 'device': [('vif', ['vif', ['backend',
'0'], ['script', 'vif-bridge'], ['bridge', 'xenbr0'], ['mac',
'00:16:3e:65:a1:6d']]), ('vbd', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'],
['uname', 'phy:/dev/mapper/xen'], ['mode', 'w']]), ('vkbd', ['vkbd',
['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type', 'vnc'],
['vncunused', '1'], ['xauthority', '/root/.Xauthority']])], 'bootloade
r': '/usr/bin/pygrub', 'cpu': None, 'maxmem': 2000}
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:1427)
XendDomainInfo.construct: None
[2009-07-17 13:30:17 xend 29619] DEBUG (balloon:143) Balloon: 47438784 KiB
free; need 2048; done.
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:797)
Storing VM details: {'shadow_memory': '0', 'uuid':
'6b25d1b9-d648-48e2-a051-13151265a16d', 'on_reboot': 'restart', 'start_time':
'1247855358.21', 'on_poweroff': 'destroy', 'name': 'xen', 'xend/restart_count':
'0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '600', 'on_crash': 'restart',
'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.GV9qmc) (kernel
/var/lib/xen/boot_kernel.CqKqLQ) (args 'ro root=/dev/xvda2 console=xvc0'))",
'maxmem': '2000'}
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110) DevController:
writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vkbd/12/0'} to /local/domain/12/device/vkbd/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112) DevController:
writing {'frontend-id': '12', 'domain': 'xen', 'frontend':
'/local/domain/12/device/vkbd/0', 'state': '1', 'online': '1'} to
/local/domain/0/backend/vkbd/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110) DevController:
writing {'backend-id': '0', 'mac': '00:16:3e:65:a1:6d', 'handle': '0', 'state':
'1', 'backend': '/local/domain/0/backend/vif/12/0'} to
/local/domain/12/device/vif/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112) DevController:
writing {'bridge': 'xenbr0', 'domain': 'xen', 'handle': '0', 'script':
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/12/device/vif/0', 'mac': '00:16:3e:65:a1:6d', 'online': '1',
'frontend-id': '12'} to /local/domain/0/backend/vif/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (blkif:27) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110) DevController:
writing {'backend-id': '0', 'virtual-device': '51712', 'device-type': 'disk',
'state': '1', 'backend': '/local/domain/0/backend/vbd/12/51712'} to
/local/domain/12/device/vbd/51712.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112) DevController:
writing {'domain': 'xen', 'frontend': '/local/domain/12/device/vbd/51712',
'format': 'raw', 'dev': 'xvda', 'state': '1', 'params': '/dev/mapper/xen',
'mode': 'w', 'online': '1', 'frontend-id': '12', 'type': 'phy'} to
/local/domain/0/backend/vbd/12/51712.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110) DevController:
writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vfb/12/0'} to /local/domain/12/device/vfb/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112) DevController:
writing {'vncunused': '1', 'domain': 'xen', 'frontend':
'/local/domain/12/device/vfb/0', 'xauthority': '/root/.Xauthority', 'state':
'1', 'online': '1', 'frontend-id': '12', 'type': 'vnc'} to
/local/domain/0/backend/vfb/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (vfbif:70) No VNC passwd configured for
vfb access
[2009-07-17 13:30:17 xend 29619] DEBUG (vfbif:11) Spawn:
['/usr/lib64/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '12', '-domain-name',
'xen', '-vnc', '127.0.0.1:0', '-vncunused']
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:832)
Storing domain details: {'console/port': '2', 'name': 'xen', 'console/limit':
'1048576', 'vm': '/vm/6b25d1b9-d648-48e2-a051-13151265a16d', 'domid': '12',
'cpu/0/availability': 'online', 'memory/target': '614400', 'store/port': '1'}
[2009-07-17 13:30:17 xend 29619] DEBUG (XendCheckpoint:198) restore:shadow=0x0,
_static_max=0x7d0, _static_min=0x258,
[2009-07-17 13:30:17 xend 29619] DEBUG (balloon:143) Balloon: 47438776 KiB
free; need 614400; done.
[2009-07-17 13:30:17 xend 29619] DEBUG (XendCheckpoint:215) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 15 12 1 2 0 0 0
[2009-07-17 13:30:17 xend 29619] INFO (XendCheckpoint:351) xc_domain_restore
start: p2m_size = 7d800
[2009-07-17 13:30:17 xend 29619] INFO (XendCheckpoint:351) Reloading memory
pages: 0%
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) Received all pages
(0 races)
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) ERROR Internal
error: Failed to pin batch of 65 page tables
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) Restore exit with
rc=1
[2009-07-17 13:30:27 xend.XendDomainInfo 29619] DEBUG (XendDomainInfo:1637)
XendDomainInfo.destroy: domid=12
[2009-07-17 13:30:27 xend.XendDomainInfo 29619] ERROR (XendDomainInfo:1645)
XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", line
1643, in destroy
xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2009-07-17 13:30:27 xend 29619] ERROR (XendDomain:278) Restore failed
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line 273,
in domain_restore_fd
return XendCheckpoint.restore(self, fd)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line
219, in restore
forkHelper(cmd, fd, handler.handler, True)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py", line
339, in forkHelper
raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 15 12 1 2 0 0 0 failed
_______________________________________________
rhelv5-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhelv5-list
