Happy to share what i can.
I should have mentioned that the backup script goes through all my VM's, and my
ambiguous comment mentions that it went through 3 of the VM's before stalling
on this, the fourth. System is low utilisation for RAM CPU and disk. Proliant
G5 dual chip quad core (no HT) using P400 RAID CARD (transparent to system)
with /images being on a RAID-1'd SATA spindle, and /images2 being on a RAID-1'd
SATA SSD. While there are some i/o wait indicators in my hypervisor (very low
though), there's no steal time recorded on any of my VM's.
The ten or so VM's are low utilisation, administrative (my mail server, a
zabbix server, a landscape server, etc). System is LTS with no tweaks, up to
date on a regular basis.
/backup has been an NFS mount point and an external USB drive, witnessed
failure condition on both.
the failing VM is on my SSD raid drive at /images2
Smart Array P400 in Slot 0 (Embedded)
Bus Interface: PCI
Slot: 0
Serial Number: PA2240J9SU5360
Cache Serial Number: PA2270D9SU21FK
Controller Status: OK
Hardware Revision: B
Firmware Version: 1.18
Rebuild Priority: Low
Surface Scan Delay: 15 secs
Surface Scan Mode: Idle
Parallel Surface Scan Supported: No
Elevator Sort: Enabled
Post Prompt Timeout: 0 secs
Cache Board Present: True
Cache Status: OK
Cache Ratio: 100% Read / 0% Write
Drive Write Cache: Disabled
Total Cache Size: 512 MB
Total Cache Memory Available: 464 MB
No-Battery Write Cache: Disabled
Battery/Capacitor Count: 0
SATA NCQ Supported: False
Number of Ports: 2 Internal only
Driver Name: cciss
Driver Version: 3.6.26
PCI Address (Domain:Bus:Device.Function): 0000:06:00.0
Host Serial Number: 2UX70501S6
Sanitize Erase Supported: False
Array: A
Interface Type: SATA
Unused Space: 0 MB (0.0%)
Used Space: 931.5 GB (100.0%)
Status: OK
Array Type: Data
Array: B
Interface Type: SATA
Unused Space: 0 MB (0.0%)
Used Space: 447.1 GB (100.0%)
Status: OK
Array Type: Data
logicaldrive 1 (465.7 GB, RAID 1, OK)
logicaldrive 2 (223.5 GB, RAID 1, OK)
physicaldrive 2I:1:1 (port 2I:box 1:bay 1, SATA, 500 GB, OK)
physicaldrive 2I:1:2 (port 2I:box 1:bay 2, SATA, 500 GB, OK)
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SATA, 240.0 GB, OK)
physicaldrive 2I:1:4 (port 2I:box 1:bay 4, SATA, 250 GB, OK)
root@thewind:~#
root@thewind:~# top
top - 09:10:18 up 38 days, 14:09, 1 user, load average: 4.81, 4.53, 4.60
Tasks: 280 total, 1 running, 279 sleeping, 0 stopped, 0 zombie
%Cpu(s): 10.4 us, 13.1 sy, 0.0 ni, 75.5 id, 1.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 64943112 total, 394580 free, 25359596 used, 39188936 buff/cache
KiB Swap: 66056188 total, 64023212 free, 2032976 used. 37839620 avail Mem
root@thewind:~# cat /proc/cpuinfo | grep Xe
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz
root@thewind:~# cat /home/bestpa/scripts/backup_thewind.candidate.sh
#!/bin/bash
set -e
# gleaned good stuff from https://www.gonzalomarcote.com/2014/kvm-live-
backups-with-qcow2/
MOUNTPOINT=/backup
date
echo "Beginning backup of virtual machines"
iptables --list
free -m
virsh list --all
set +e
echo "mounting drive"
# use for external USB drives
mount $MOUNTPOINT
set -e
# Check if mounted
echo "checking to see if $MOUNTPOINT is mounted"
if ! mountpoint -q $MOUNTPOINT/
then
{ echo "$MOUNTPOINT not mounted!"; exit; }
else
{
echo "filesystem $MOUNTPOINT mounted"
logger "thewind_backup : $MOUNTPOINT confirmed mounted"
df -h
du -sh $MOUNTPOINT
# copy the following
#
# /home/bestpa/ only if new file
# /images
# /etc/libvirt
logger "Starting bestpa backups"
echo "Starting bestpa backups"
# rsync cause only copy if more recent timestamp.
# Can't do -rav because NFS mount doesn't let you change
ownership
# exludes (o)wner (g)roup from (a)rchive
# rsync -rav /home/bestpa $MOUNTPOINT
rsync -rlptDv /home/bestpa $MOUNTPOINT
logger "Starting libvirt xml backups"
echo "Starting libvirt xml backups"
#Only copy if more recent timestamp.
# Can't do -rav because NFS mount doesn't let you change
ownership
# rsync -rav /etc/libvirt $MOUNTPOINT
rsync -rlptDv /etc/libvirt $MOUNTPOINT
logger "Starting virsh backups"
echo "Starting virsh backups"
# splunk is a volumegroup logicalvolume, so not here.
# palo also needs attention
# canary is a fucked up one too - it's got multiple disks, one VG and
one QCOW2
for i in cacti landscape mail realworldnumbers tenantinvoice pfsense ;
do
echo "-------------BEGIN backup for VM called $i"
date
#sanity check - we don't want to see a current snapshot in
domblklist
string=`virsh domblklist $i | sed -n '3,3p'`
if [[ $string == *"snap"* ]]; then
echo "Can't Proceed - The current image is a
snapshot."
exit;
fi
#sanity check - we don't want to see a current snapshot in
snapshot-list
echo "current snapshots $i - should be empty"
virsh snapshot-list $i
string=`virsh snapshot-list $i | sed -n '3,3p'`
if [[ $string == *"snap"* ]]; then
echo "Can't Proceed - There is already a snapshot
listed."
exit;
fi
echo "initial blklist and snapshot list $i"
virsh domblklist $i
virsh snapshot-list $i
BLK_TYPE=`virsh domblklist $i | sed -n '3,3p' | awk {'print
$1'}`
IMG_LOC=`virsh domblklist $i | sed -n '3,3p' | awk {'print $2'}`
echo "block type is $BLK_TYPE"
echo "image location is $IMG_LOC"
echo creating snapshot for $i
virsh snapshot-create-as --domain $i $i-snap1 --disk-only
--atomic
sleep 10
echo "current snapshots for $i"
virsh snapshot-list $i
## if this is the first time we are copying the file, we copy
it sparse
## otherwise, we only copy changed blocks with --inplace
##
## First lets get the proper directory
IMAGE_DIR=`virsh domblklist mail | sed -n '3,3p' | awk {'print
$2'} | awk -F '/' {'print $2'}`
if [ -e $MOUNTPOINT/$IMAGE_DIR/$i.img ]; then
echo " performng INPLACE rsync for $i"
rsync -vh --inplace $IMG_LOC
$MOUNTPOINT/$IMAGE_DIR/$i.img
else
echo " performing FIRST TIME SPARSE rsync for $i"
rsync -vh --sparse $IMG_LOC
$MOUNTPOINT/$IMAGE_DIR/$i.img
fi
echo "I am done with the rsync."
echo "current blklist $i"
virsh domblklist $i
echo "blockcommit $i"
virsh blockcommit $i $BLK_TYPE --active --verbose --pivot
echo "current blklist $i"
virsh domblklist $i
echo "current snapshots $i"
virsh snapshot-list $i
## NO DELETES YET, TESTING MODE, check every morning and
kill any -snap
#echo "deleting snapshot"
#virsh snapshot-delete $i $i-snap1 --metadata
#echo "current snapshots"
#virsh snapshot-list $i
#echo "deleting old snapshot file"
#rm -v /images/$i-snap1
echo "DONE backup $i"
echo
done
virsh list --all
logger "Finished virsh backups"
echo "Finished virsh backups"
du -sh $MOUNTPOINT
df -h
umount $MOUNTPOINT
logger "unmounting $MOUNTPOINT drive"
echo "unmounting $MOUNTPOINT drive"
date
}
fi
root@thewind:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/";
SUPPORT_URL="http://help.ubuntu.com/";
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/";
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
root@thewind:~#
root@thewind:~#
root@thewind:~#
root@thewind:~# virsh list mail
error: unexpected data 'mail'
root@thewind:~# cat /etc/libvirt/qemu/mail.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
virsh edit mail
or other application using the libvirt API.
-->
<domain type='kvm'>
<name>mail</name>
<uuid>e58f22b6-a3b7-4763-af47-b1cc36fb8df3</uuid>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<vcpu placement='static'>2</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64' machine='pc-i440fx-wily'>hvm</type>
<boot dev='hd'/>
<boot dev='cdrom'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'>Penryn</model>
<vendor>Intel</vendor>
<feature policy='require' name='osxsave'/>
<feature policy='require' name='xsave'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='monitor'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='vme'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/kvm-spice</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/images2/mail.mail-snap1'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/home/bestpa/iso/ubuntu-14.04.5-server-amd64.iso'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'
multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x2'/>
</controller>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<interface type='network'>
<mac address='52:54:00:4f:fe:5c'/>
<source network='private'/>
<model type='rtl8139'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='10.0.1.12'>
<listen type='address' address='10.0.1.12'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</memballoon>
</devices>
<seclabel type='dynamic' model='apparmor' relabel='yes'/>
</domain>
root@thewind:~#
root@thewind:~# iostat 5
Linux 4.4.0-66-generic (thewind) 04/24/2017 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
10.35 0.00 13.11 1.08 0.00 75.45
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
cciss/c0d0 5.80 52.99 169.64 176686197 565655689
cciss/c0d1 23.76 87.58 440.35 292034952 1468321408
sda 0.89 3.07 15.72 10246856 52418340
dm-0 18.46 2.56 77.79 8547523 259397184
dm-1 0.54 0.40 1.77 1320142 5887228
dm-2 24.06 87.58 440.35 292030859 1468321408
dm-3 0.81 46.98 22.49 156654007 74979028
dm-4 3.17 2.93 67.48 9759267 224998111
dm-5 0.00 0.01 0.00 44468 220
dm-6 0.00 0.01 0.00 43767 0
avg-cpu: %user %nice %system %iowait %steal %idle
34.91 0.00 35.21 0.53 0.00 29.35
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
cciss/c0d0 4.80 2.40 96.80 12 484
cciss/c0d1 12.80 0.00 301.60 0 1508
sda 0.00 0.00 0.00 0 0
dm-0 11.00 0.00 44.00 0 220
dm-1 0.00 0.00 0.00 0 0
dm-2 13.20 0.00 301.60 0 1508
dm-3 0.40 0.00 8.00 0 40
dm-4 1.80 0.00 44.80 0 224
dm-5 0.00 0.00 0.00 0 0
dm-6 0.00 0.00 0.00 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
35.06 0.00 17.93 2.04 0.00 44.97
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
cciss/c0d0 18.80 0.00 624.80 0 3124
cciss/c0d1 35.40 0.00 1389.60 0 6948
sda 0.00 0.00 0.00 0 0
dm-0 113.40 0.00 460.00 0 2300
dm-1 0.00 0.00 0.00 0 0
dm-2 37.00 0.00 1402.40 0 7012
dm-3 0.00 0.00 0.00 0 0
dm-4 13.20 0.00 164.80 0 824
dm-5 0.00 0.00 0.00 0 0
dm-6 0.00 0.00 0.00 0 0
^C
root@thewind:~#
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1681839
Title:
libvirt - disk not ready for pivot yet
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1681839/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
