Hello,
Try to enable Xen DomU crash dumps - the setting to enable crash
dumps is in /etc/xen/xend-config.sxp. Maybe the guest crashes for some
reason right after being unpaused on the target machine and this would
give you more information about what happened. But from the xend log
messages you provided it is also possible that Xen itself is unable to
unpause the DomU after migration...
Regards,
Daniel
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Rich Graves
Sent: Friday, July 17, 2009 8:50 PM
To: rhelv5-list
Subject: [rhelv5-list] Can't xm migrate to new Nehalem server
I've got 5 Xen servers (xen0..xen4), running stock RHEL 5.3 x86_64,
fully patched, on a variety of Dell server hardware with CPUs ranging
from Irwindale to Nehalem. All guests under test are also RHEL 5.3
x86_64, fully patched. Each guest has its own SAN LUN, shared among all
hosts, addressed by named multipath.conf entries.
If I "xm create" a guest on a host with Xeon Irwindale, 5060, or 5160
CPUs, then I am unable to live-migrate that guest to a Dell R710 with
two quad-core Intel L5520 CPUs.
But if I "xm create" the same guest on the L5520, then I am able to
live-migrate to the older systems, and back to the newer system, any
number of times.
Of course it's possible that I've got some configuration difference
other that CPU, but I can't find one after diff -ur'ing /etc/sysconfig
and /etc/xen.
User-space observations: The sending host accepts xm migrate -l xen4.
Watching xm list on the source and destination, I see the guest's RAM
gradually increase to the correct value; then the "migrating-guest"
entry goes away on the source; and finally, the guest disappears on the
destination.
I'm not entirely sure, but this *might* have worked before, possibly
breaking with 2.6.18-128.1.16.el5xen. Upgrading source, destination, and
guests to 2.6.18-128.2.1.el5xen did not help.
This seems to be the first hard error in xend.log:
(XendCheckpoint:351) ERROR Internal error: Failed to pin batch of 65
page tables
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:281) XendDomainInfo.restore(['domain', ['domid', '33'],
['uuid', '0513217c-e0bc-4893-bb04-29114018cd62'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '600'],
['shadow_memory', '0'], ['maxmem', '2000'], ['bootloader',
'/usr/bin/pygrub'], ['features'], ['name', 'xen'], ['on_poweroff',
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image',
['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:65:a1:6d']]],
['device', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname',
'phy:/dev/mapper/xen'], ['mode', 'w']]], ['device', ['vkbd', ['backend',
'0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'],
['vncunused', '1'], ['xauthority', '/root/.Xauthority']]], ['s tate',
'--p---'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.0'],
['online_vcpus', '1'], ['up_time', '58.9367001057'], ['start_time',
'1247855358.21'], ['store_mfn', '573954'], ['console_mfn', '573953']])
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:312) parseConfig: config is ['domain', ['domid', '33'],
['uuid', '0513217c-e0bc-4893-bb04-29114018cd62'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '600'],
['shadow_memory', '0'], ['maxmem', '2000'], ['bootloader',
'/usr/bin/pygrub'], ['features'], ['name', 'xen'], ['on_poweroff',
'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image',
['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']]], ['device', ['vif', ['backend', '0'], ['script',
'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:65:a1:6d']]],
['device', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname',
'phy:/dev/mapper/xen'], ['mode', 'w']]], ['device', ['vkbd', ['backend',
'0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'],
['vncunused', '1'], ['xauthority', '/root/.Xauthority']]], ['s tate',
'--p---'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.0'],
['online_vcpus', '1'], ['up_time', '58.9367001057'], ['start_time',
'1247855358.21'], ['store_mfn', '573954'], ['console_mfn', '573953']]
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:417) parseConfig: result is {'shadow_memory': 0,
'start_time': 1247855358.21, 'uuid':
'6b25d1b9-d648-48e2-a051-13151265a16d', 'on_crash': 'restart',
'on_reboot': 'restart', 'localtime': None, 'image': ['linux',
['ramdisk', '/var/lib/xen/boot_ramdisk.GV9qmc'], ['kernel',
'/var/lib/xen/boot_kernel.CqKqLQ'], ['args', 'ro root=/dev/xvda2
console=xvc0']], 'on_poweroff': 'destroy', 'bootloader_args': None,
'cpus': None, 'name': 'xen', 'backend': [], 'vcpus': 1, 'cpu_weight':
1.0, 'features': None, 'vcpu_avail': 1, 'memory': 600, 'device':
[('vif', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge',
'xenbr0'], ['mac', '00:16:3e:65:a1:6d']]), ('vbd', ['vbd', ['backend',
'0'], ['dev', 'xvda:disk'], ['uname', 'phy:/dev/mapper/xen'], ['mode',
'w']]), ('vkbd', ['vkbd', ['backend', '0']]), ('vfb', ['vfb',
['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority',
'/root/.Xauthority']])], 'bootloade
r': '/usr/bin/pygrub', 'cpu': None, 'maxmem': 2000}
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:1427) XendDomainInfo.construct: None
[2009-07-17 13:30:17 xend 29619] DEBUG (balloon:143) Balloon: 47438784
KiB free; need 2048; done.
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:797) Storing VM details: {'shadow_memory': '0', 'uuid':
'6b25d1b9-d648-48e2-a051-13151265a16d', 'on_reboot': 'restart',
'start_time': '1247855358.21', 'on_poweroff': 'destroy', 'name': 'xen',
'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory':
'600', 'on_crash': 'restart', 'image': "(linux (ramdisk
/var/lib/xen/boot_ramdisk.GV9qmc) (kernel
/var/lib/xen/boot_kernel.CqKqLQ) (args 'ro root=/dev/xvda2
console=xvc0'))", 'maxmem': '2000'}
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110)
DevController: writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vkbd/12/0'} to /local/domain/12/device/vkbd/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112)
DevController: writing {'frontend-id': '12', 'domain': 'xen',
'frontend': '/local/domain/12/device/vkbd/0', 'state': '1', 'online':
'1'} to /local/domain/0/backend/vkbd/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110)
DevController: writing {'backend-id': '0', 'mac': '00:16:3e:65:a1:6d',
'handle': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/12/0'} to /local/domain/12/device/vif/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112)
DevController: writing {'bridge': 'xenbr0', 'domain': 'xen', 'handle':
'0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/12/device/vif/0', 'mac': '00:16:3e:65:a1:6d', 'online':
'1', 'frontend-id': '12'} to /local/domain/0/backend/vif/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (blkif:27) exception looking up
device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110)
DevController: writing {'backend-id': '0', 'virtual-device': '51712',
'device-type': 'disk', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/12/51712'} to
/local/domain/12/device/vbd/51712.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112)
DevController: writing {'domain': 'xen', 'frontend':
'/local/domain/12/device/vbd/51712', 'format': 'raw', 'dev': 'xvda',
'state': '1', 'params': '/dev/mapper/xen', 'mode': 'w', 'online': '1',
'frontend-id': '12', 'type': 'phy'} to
/local/domain/0/backend/vbd/12/51712.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:110)
DevController: writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vfb/12/0'} to /local/domain/12/device/vfb/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (DevController:112)
DevController: writing {'vncunused': '1', 'domain': 'xen', 'frontend':
'/local/domain/12/device/vfb/0', 'xauthority': '/root/.Xauthority',
'state': '1', 'online': '1', 'frontend-id': '12', 'type': 'vnc'} to
/local/domain/0/backend/vfb/12/0.
[2009-07-17 13:30:17 xend 29619] DEBUG (vfbif:70) No VNC passwd
configured for vfb access
[2009-07-17 13:30:17 xend 29619] DEBUG (vfbif:11) Spawn:
['/usr/lib64/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '12',
'-domain-name', 'xen', '-vnc', '127.0.0.1:0', '-vncunused']
[2009-07-17 13:30:17 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:832) Storing domain details: {'console/port': '2',
'name': 'xen', 'console/limit': '1048576', 'vm':
'/vm/6b25d1b9-d648-48e2-a051-13151265a16d', 'domid': '12',
'cpu/0/availability': 'online', 'memory/target': '614400', 'store/port':
'1'}
[2009-07-17 13:30:17 xend 29619] DEBUG (XendCheckpoint:198)
restore:shadow=0x0, _static_max=0x7d0, _static_min=0x258,
[2009-07-17 13:30:17 xend 29619] DEBUG (balloon:143) Balloon: 47438776
KiB free; need 614400; done.
[2009-07-17 13:30:17 xend 29619] DEBUG (XendCheckpoint:215)
[xc_restore]: /usr/lib64/xen/bin/xc_restore 15 12 1 2 0 0 0
[2009-07-17 13:30:17 xend 29619] INFO (XendCheckpoint:351)
xc_domain_restore start: p2m_size = 7d800
[2009-07-17 13:30:17 xend 29619] INFO (XendCheckpoint:351) Reloading
memory pages: 0%
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) Received all
pages (0 races)
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) ERROR
Internal error: Failed to pin batch of 65 page tables
[2009-07-17 13:30:27 xend 29619] INFO (XendCheckpoint:351) Restore exit
with rc=1
[2009-07-17 13:30:27 xend.XendDomainInfo 29619] DEBUG
(XendDomainInfo:1637) XendDomainInfo.destroy: domid=12
[2009-07-17 13:30:27 xend.XendDomainInfo 29619] ERROR
(XendDomainInfo:1645) XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 1643, in destroy
xc.domain_destroy(self.domid)
Error: (3, 'No such process')
[2009-07-17 13:30:27 xend 29619] ERROR (XendDomain:278) Restore failed
Traceback (most recent call last):
File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line
273, in domain_restore_fd
return XendCheckpoint.restore(self, fd)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 219, in restore
forkHelper(cmd, fd, handler.handler, True)
File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 339, in forkHelper
raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 15 12 1 2 0 0 0 failed
_______________________________________________
rhelv5-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhelv5-list
_______________________________________________
rhelv5-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhelv5-list
