Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
Hi list, I just tried this again with preallocated disk, otherwise the exact same procedure as described in my original post. No problems at all so far with regards, -- Morten A. Middelthon Email: mor...@flipp.net Phone: +47 907 83 708 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
so there is a problem extending... perhaps I missed it, but can you attach the vm qemu log, libvirt and vdsm logs? On 05/16/2014 02:05 PM, Morten A. Middelthon wrote: On 16. mai 2014 15:04, Dafna Ron wrote: adding federico since I think he can probably add more info here. when we use thin provisionining we have to extend the disk during writing (it's set to do that every 2GB I think). during extend the vm pauses and resumes. However, this action should not be noticeable by the vm user. Does the vm pause and resumes or pauses completely? is it noticeable by the vm user? Hi, the vm pauses completely, and I can't resume it without deactivating/disconnecting the new disk Morten -- Dafna Ron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
On 16. mai 2014 15:04, Dafna Ron wrote: adding federico since I think he can probably add more info here. when we use thin provisionining we have to extend the disk during writing (it's set to do that every 2GB I think). during extend the vm pauses and resumes. However, this action should not be noticeable by the vm user. Does the vm pause and resumes or pauses completely? is it noticeable by the vm user? Hi, the vm pauses completely, and I can't resume it without deactivating/disconnecting the new disk Morten ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
adding federico since I think he can probably add more info here. when we use thin provisionining we have to extend the disk during writing (it's set to do that every 2GB I think). during extend the vm pauses and resumes. However, this action should not be noticeable by the vm user. Does the vm pause and resumes or pauses completely? is it noticeable by the vm user? Thanks, Dafna On 05/16/2014 01:40 PM, John Taylor wrote: Hi Morten, My understanding of thin disks on a block domain is that vdsm traps ENOSPC on the thin lv and uses the mailbox to get the SPM to extend it. See a presentation by Nir http://www.ovirt.org/File:Storage-mailbox.odp I thought I saw somewhere there were some changes/bugs around that for allowing mixed data centers (both block and file domains). ...just looked now and this bz looks relevant https://bugzilla.redhat.com/show_bug.cgi?id=1083476 So maybe you could check the engine logs and spm logs for that flow (sorry I can't tell you any specifics about what you should look for ) -John On Fri, May 16, 2014 at 1:54 AM, Morten A. Middelthon wrote: Hi, I just re-ran the test with adding a preallocated disk, and the problem did _not_ appear. I tried a few times to write large files with dd, but the vm continued to run without problems with regards, -- Morten A. Middelthon Email: mor...@flipp.net Phone: +47 907 83 708 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users -- Dafna Ron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
Hi Morten, My understanding of thin disks on a block domain is that vdsm traps ENOSPC on the thin lv and uses the mailbox to get the SPM to extend it. See a presentation by Nir http://www.ovirt.org/File:Storage-mailbox.odp I thought I saw somewhere there were some changes/bugs around that for allowing mixed data centers (both block and file domains). ...just looked now and this bz looks relevant https://bugzilla.redhat.com/show_bug.cgi?id=1083476 So maybe you could check the engine logs and spm logs for that flow (sorry I can't tell you any specifics about what you should look for ) -John On Fri, May 16, 2014 at 1:54 AM, Morten A. Middelthon wrote: > Hi, > > I just re-ran the test with adding a preallocated disk, and the problem did > _not_ appear. I tried a few times to write large files with dd, but the vm > continued to run without problems > > with regards, > > > -- > Morten A. Middelthon > Email: mor...@flipp.net > Phone: +47 907 83 708 > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
Hi, I just re-ran the test with adding a preallocated disk, and the problem did _not_ appear. I tried a few times to write large files with dd, but the vm continued to run without problems with regards, -- Morten A. Middelthon Email: mor...@flipp.net Phone: +47 907 83 708 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
On 15. mai 2014 11:58, Dafna Ron wrote: ovirt does not pause the vm for ENOSpace, if a vm is pause for this reason it would be done by qemu. The logs you attached show no issues. can you please attach the vm log and libvirt logs? Also, can you please try the same test on a prealloacted disk? Thanks, I will first try to add a preallocated disk. Syslog on the vm did not show anything related, but I included the dmesg output in my original post with regards, -- Morten A. Middelthon Email: mor...@flipp.net Phone: +47 907 83 708 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
On 16. mai 2014 04:45, Trey Dockendorf wrote: A colleague of mine has run into issues doing Linux (EL6.5) iSCSI over bonded interfaces. The solution he found was to abandon bonding and instead use multiple interfaces and multipath on the iscsi initiators. I can't provide details yet as this is all second hand knowledge. The errors you posted are almost identical to what he faced. Thanks for your input. The strange thing is that the existing iSCSI disks on the very same LUN works without any issues. The problem only seems to appear when I add a new disk to a running VM. To properly test this I probably have to add another hypervisor host without bonding, as my current hypervisor hosts are in a production environment with regards, -- Morten A. Middelthon Email: mor...@flipp.net Phone: +47 907 83 708 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
A colleague of mine has run into issues doing Linux (EL6.5) iSCSI over bonded interfaces. The solution he found was to abandon bonding and instead use multiple interfaces and multipath on the iscsi initiators. I can't provide details yet as this is all second hand knowledge. The errors you posted are almost identical to what he faced. - Trey On May 13, 2014 3:10 AM, "Morten A. Middelthon" wrote: > Hi, > > I have a new oVirt setup which is giving me some headache. I have one > administration server running on CentOS 6.5, and two hosts also running > CentOS 6.5. Storage is running on both NFS and iSCSI, but iSCSI is the > preferred storage, running on a HP LeftHand server. > > oVirt Engine Version: 3.4.0-1.el6 > > Administration server and hosts are all updated with latest packages both > for CentOS and oVirt. > > Both hosts are attached to a dedicated storage network with bonded > interfaces: > > # ifconfig bond1 > bond1 Link encap:Ethernet HWaddr AC:16:2D:8B:90:5E > inet addr:192.168.40.160 Bcast:192.168.40.255 Mask:255.255.255.0 > inet6 addr: fe80::ae16:2dff:fe8b:905e/64 Scope:Link > UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 > RX packets:4149464 errors:0 dropped:0 overruns:0 frame:0 > TX packets:5590278 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:5387975600 (5.0 GiB) TX bytes:6981018034 (6.5 GiB) > > On the hosts the LUN is seen as: > > scsi3 : iSCSI Initiator over TCP/IP > connection1:0: detected conn error (1020) > scsi 3:0:0:0: Direct-Access LEFTHAND iSCSIDiska500 PQ: 0 > ANSI: 5 > sd 3:0:0:0: Attached scsi generic sg2 type 0 > sd 3:0:0:0: [sdb] 8589934592 512-byte logical blocks: (4.39 TB/4.00 TiB) > sd 3:0:0:0: [sdb] Write Protect is off > sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08 > sd 3:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't > support DPO or FUA > sdb: unknown partition table > sd 3:0:0:0: [sdb] Attached SCSI disk > > The iSCSI LUN I'm using is 4TB large, with 2.5TB free, according to the > oVirt admin interface: > > Size: 4095 GB > Available: 2551 GB > Used: 1544 GB > Allocated: 1540 GB > Over Allocation Ratio: 0% > > Over to the problem: > > On the virtual machine I'm testing on, I'm adding a new disk on the iSCSI > storage server. The virtual machine is Debian 7.4 amd64. In this case 20GB > large with thin provisioning. Then on the virtual machine I create a new > logical volume group and disk: > > Create a new partition with type 8e (Linux LVM): > fdisk /dev/vdc > > Setup LVM: > pvcreate /dev/vdc1 > vgcreate VGTEST /dev/vdc1 > lvcreate -n lv_test -l 100%FREE VGTEST > mkfs.ext4 /dev/mapper/VGTEST-lv_test > mkdir /mnt/test > mount /dev/mapper/VGTEST-lv_test /mnt/test > > Then I try writing to the new disk with f.ex dd: > dd if=/dev/zero of=/mnt/test/storfil bs=64k count=20480 > > Almost immediately oVirt will pause the machine with the following error > message: > > VM INT-KVM-SSM02 has paused due to no Storage space error. > > dmesg on the virtual machine gives to following errors: > > [353290.564507] Buffer I/O error on device dm-7, logical block 294774 > [353290.564509] Buffer I/O error on device dm-7, logical block 294775 > [353290.564511] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error > writing to inode 12 (offset 1039114240 size 520192 starting block 294650) > [353290.564514] end_request: I/O error, dev vdc, sector 2362304 > [353290.564516] Buffer I/O error on device dm-7, logical block 294776 > [353290.564518] Buffer I/O error on device dm-7, logical block 294777 > [353290.564519] Buffer I/O error on device dm-7, logical block 294778 > [353290.564521] Buffer I/O error on device dm-7, logical block 294779 > [353290.564522] Buffer I/O error on device dm-7, logical block 294780 > [353290.564524] Buffer I/O error on device dm-7, logical block 294781 > [353290.564525] Buffer I/O error on device dm-7, logical block 294782 > [353290.564744] Buffer I/O error on device dm-7, logical block 294910 > [353290.564746] Buffer I/O error on device dm-7, logical block 294911 > [353290.564748] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error > writing to inode 12 (offset 1040154624 size 32768 starting block 294904) > [353291.035703] EXT4-fs (dm-7): This should not happen!! Data will be lost > [353291.035705] > [353291.036280] JBD2: Detected IO errors while flushing file data on dm-7-8 > [353291.036506] end_request: I/O error, dev vdc, sector 17043640 > [353291.036509] end_request: I/O error, dev vdc, sector 17043640 > [353291.036518] Aborting journal on device dm-7-8. > [353291.036734] end_request: I/O error, dev vdc, sector 17043456 > [353291.036737] Buffer I/O error on device dm-7, logical block 2129920 > [353291.036739] lost page write due to I/O error on dm-7 > etc etc > > If I try to do large writes on the existing disk on either the iSCSI or > NFS storage there are no problems at all. >
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
On 13. mai 2014 10:09, Morten A. Middelthon wrote: Hi, I have a new oVirt setup which is giving me some headache. I have one administration server running on CentOS 6.5, and two hosts also running CentOS 6.5. Storage is running on both NFS and iSCSI, but iSCSI is the preferred storage, running on a HP LeftHand server. Anyone with any suggestions or input on this problem? with regards, Morten ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
ovirt does not pause the vm for ENOSpace, if a vm is pause for this reason it would be done by qemu. The logs you attached show no issues. can you please attach the vm log and libvirt logs? Also, can you please try the same test on a prealloacted disk? Thanks, Dafna On 05/13/2014 09:09 AM, Morten A. Middelthon wrote: Hi, I have a new oVirt setup which is giving me some headache. I have one administration server running on CentOS 6.5, and two hosts also running CentOS 6.5. Storage is running on both NFS and iSCSI, but iSCSI is the preferred storage, running on a HP LeftHand server. oVirt Engine Version: 3.4.0-1.el6 Administration server and hosts are all updated with latest packages both for CentOS and oVirt. Both hosts are attached to a dedicated storage network with bonded interfaces: # ifconfig bond1 bond1 Link encap:Ethernet HWaddr AC:16:2D:8B:90:5E inet addr:192.168.40.160 Bcast:192.168.40.255 Mask:255.255.255.0 inet6 addr: fe80::ae16:2dff:fe8b:905e/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:4149464 errors:0 dropped:0 overruns:0 frame:0 TX packets:5590278 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5387975600 (5.0 GiB) TX bytes:6981018034 (6.5 GiB) On the hosts the LUN is seen as: scsi3 : iSCSI Initiator over TCP/IP connection1:0: detected conn error (1020) scsi 3:0:0:0: Direct-Access LEFTHAND iSCSIDiska500 PQ: 0 ANSI: 5 sd 3:0:0:0: Attached scsi generic sg2 type 0 sd 3:0:0:0: [sdb] 8589934592 512-byte logical blocks: (4.39 TB/4.00 TiB) sd 3:0:0:0: [sdb] Write Protect is off sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08 sd 3:0:0:0: [sdb] Write cache: disabled, read cache: disabled, doesn't support DPO or FUA sdb: unknown partition table sd 3:0:0:0: [sdb] Attached SCSI disk The iSCSI LUN I'm using is 4TB large, with 2.5TB free, according to the oVirt admin interface: Size:4095 GB Available:2551 GB Used:1544 GB Allocated:1540 GB Over Allocation Ratio:0% Over to the problem: On the virtual machine I'm testing on, I'm adding a new disk on the iSCSI storage server. The virtual machine is Debian 7.4 amd64. In this case 20GB large with thin provisioning. Then on the virtual machine I create a new logical volume group and disk: Create a new partition with type 8e (Linux LVM): fdisk /dev/vdc Setup LVM: pvcreate /dev/vdc1 vgcreate VGTEST /dev/vdc1 lvcreate -n lv_test -l 100%FREE VGTEST mkfs.ext4 /dev/mapper/VGTEST-lv_test mkdir /mnt/test mount /dev/mapper/VGTEST-lv_test /mnt/test Then I try writing to the new disk with f.ex dd: dd if=/dev/zero of=/mnt/test/storfil bs=64k count=20480 Almost immediately oVirt will pause the machine with the following error message: VM INT-KVM-SSM02 has paused due to no Storage space error. dmesg on the virtual machine gives to following errors: [353290.564507] Buffer I/O error on device dm-7, logical block 294774 [353290.564509] Buffer I/O error on device dm-7, logical block 294775 [353290.564511] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error writing to inode 12 (offset 1039114240 size 520192 starting block 294650) [353290.564514] end_request: I/O error, dev vdc, sector 2362304 [353290.564516] Buffer I/O error on device dm-7, logical block 294776 [353290.564518] Buffer I/O error on device dm-7, logical block 294777 [353290.564519] Buffer I/O error on device dm-7, logical block 294778 [353290.564521] Buffer I/O error on device dm-7, logical block 294779 [353290.564522] Buffer I/O error on device dm-7, logical block 294780 [353290.564524] Buffer I/O error on device dm-7, logical block 294781 [353290.564525] Buffer I/O error on device dm-7, logical block 294782 [353290.564744] Buffer I/O error on device dm-7, logical block 294910 [353290.564746] Buffer I/O error on device dm-7, logical block 294911 [353290.564748] EXT4-fs warning (device dm-7): ext4_end_bio:250: I/O error writing to inode 12 (offset 1040154624 size 32768 starting block 294904) [353291.035703] EXT4-fs (dm-7): This should not happen!! Data will be lost [353291.035705] [353291.036280] JBD2: Detected IO errors while flushing file data on dm-7-8 [353291.036506] end_request: I/O error, dev vdc, sector 17043640 [353291.036509] end_request: I/O error, dev vdc, sector 17043640 [353291.036518] Aborting journal on device dm-7-8. [353291.036734] end_request: I/O error, dev vdc, sector 17043456 [353291.036737] Buffer I/O error on device dm-7, logical block 2129920 [353291.036739] lost page write due to I/O error on dm-7 etc etc If I try to do large writes on the existing disk on either the iSCSI or NFS storage there are no problems at all. I have attached the engine.log from the administration server, and also the vdsm.log from the host ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users --
Re: [ovirt-users] IO errors when adding new disk on iSCSI storage
On 13. mai 2014 10:09, Morten A. Middelthon wrote: Hi, I have a new oVirt setup which is giving me some headache. I have one administration server running on CentOS 6.5, and two hosts also running CentOS 6.5. Storage is running on both NFS and iSCSI, but iSCSI is the preferred storage, running on a HP LeftHand server. Anyone with suggestions or ideas on this problems? with regards, Morten ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
