Hi there,
You may remember me from such emails as "Issues with RHEL5 and PV Xen RHEL4"
and "Fast reaping, slow processes!". Now I'm back with something new!
The issue is that whenever we do any kind of IO-intensive operations with
our Xen DomU instances (for example 'scp' copies to the filesystems of the
DomU from a remote machine) the IOwait rapidly rises to 100% and pretty much
pegs there with occasional drops to 97% or so before returning to 100%
again.
Console/SSHD'd shell response is almost nil but sometimes works. 'ls' or
other commands which require disk access are fine unless the result hasn't
been cached in which case response can be anything from one minute to ten.
When pegged at 100% IOwait the load average (reported by 'top') spirals
slowly but inexorably upwards (on the 1 VCPU instance we got as high as
12.45).
If left long enough the 'scp' (copy to a DomU in this instance, initiated on
the remote (push) machine) _will_ complete. After the prompt is return on
the remote machine the IOwait on the DomU remains at 100% for another five
to ten minutes before dropping back down to pretty much 0% (idle). The load
average also returns to ~0.00 too. Only then is console/SSHD'd shell access
properly responsive again.
During these IOwait slowdowns I've managed to get a 'ps -auxwww' output on a
DomU which shows between four and six '[pdflush]' processes as well as sadc
running in the D state. I'm not clever enough to know if this is important.
Also of note is that IOwait on the Dom0 never rises above 25% ever, despite
having stuff happening on the other DomU (potentially that too being at 100%
IOwait due to a similar copy to its filesystem, this time a straight 'cp'
from an NFS share to the local disk) at the same time.
Can someone tell me what's going on please? I'd be hugely grateful.
Thanks!
Setup
-----
SunFire x4200 M2, 16 GB RAM, 4 cores, RAID 1 (x2 136GB)
Dom0 - RHEL 5.1 32-bit (kernel 2.6.18-53.1.4.el5xen)
DomU - RHEL 4 Update 6 32-bit (kernel 2.6.9-67.0.1.ELxenU)
1 VCPU, 4G, 41GB, 1 physical chassis NIC (dedicated)
No LVM, plain ext3 (/boot (100MB), swap (2GB), / (39GB)
DomU - RHEL 4 Update 6 32-bit (kernel 2.6.9-67.0.1.ELxenU)
3 VCPUs, 8G, 81GB, 1 NIC bridged/shared with Dom0
No LVM, plain ext3 (/boot (100MB), swap (2GB), / (79GB)
Dom0:
# rpm -qa | egrep 'xen|virt' | grep -v rhn
xen-3.0.3-41.el5
kernel-xen-2.6.18-53.el5
python-virtinst-0.103.0-3.el5_1.1
kernel-xen-2.6.18-53.1.4.el5
xen-libs-3.0.3-41.el5
libvirt-0.2.3-9.el5
kernel-xen-devel-2.6.18-53.1.4.el5
libvirt-python-0.2.3-9.el5
DomU (both):
# rpm -qa | egrep 'xen|virt'
kernel-xenU-2.6.9-67.0.1.EL
kernel-xenU-devel-2.6.9-67.0.1.EL
Tools used:
iostat
top
mpstat
Ben
--
Unix Support, MISD, University of Cambridge, England
Plugger of wire, typer of keyboard, imparter of Clue
Life Is Short. It's All Good.
_______________________________________________
rhelv5-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhelv5-list
