Hello If you use Gluster as FUSE mount it's always slower than you expect it to be. If you want to get better performance out of your oVirt/Gluster storage, try the following:
- create a Linux VM in your oVirt environment, assign 4/8/12 virtual disks (Virtual disks are located on your Gluster storage volume). - Boot/configure the VM, then use LVM to create VG/LV with 4 stripes (lvcreate -i 4) and use all 4/8/12 virtual disks as PVS. - then install NFS server and export LV you created in previous step, use the NFS export as export domain in oVirt/RHEV. You should get wire speed when you use multiple stripes on Gluster storage, FUSE mount on oVirt host will fan out requests to all 4 servers. Gluster is very good at distributed/parallel workloads, but when you use direct Gluster FUSE mount for Export domain you only have one data stream, which is fragmented even more my multiple writes/reads that Gluster needs to do to save your data on all member servers. On Mon, Nov 27, 2017 at 8:41 PM, Donny Davis <do...@fortnebula.com> wrote: > What about mounting over nfs instead of the fuse client. Or maybe > libgfapi. Is that available for export domains > > On Fri, Nov 24, 2017 at 3:48 AM Jiří Sléžka <jiri.sle...@slu.cz> wrote: > >> On 11/24/2017 06:41 AM, Sahina Bose wrote: >> > >> > >> > On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <jiri.sle...@slu.cz >> > <mailto:jiri.sle...@slu.cz>> wrote: >> > >> > Hi, >> > >> > On 11/22/2017 07:30 PM, Nir Soffer wrote: >> > > On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <jiri.sle...@slu.cz >> <mailto:jiri.sle...@slu.cz> >> > > <mailto:jiri.sle...@slu.cz <mailto:jiri.sle...@slu.cz>>> wrote: >> > > >> > > Hi, >> > > >> > > I am trying realize why is exporting of vm to export storage >> on >> > > glusterfs such slow. >> > > >> > > I am using oVirt and RHV, both instalations on version 4.1.7. >> > > >> > > Hosts have dedicated nics for rhevm network - 1gbps, data >> > storage itself >> > > is on FC. >> > > >> > > GlusterFS cluster lives separate on 4 dedicated hosts. It has >> > slow disks >> > > but I can achieve about 200-400mbit throughput in other >> > applications (we >> > > are using it for "cold" data, backups mostly). >> > > >> > > I am using this glusterfs cluster as backend for export >> > storage. When I >> > > am exporting vm I can see only about 60-80mbit throughput. >> > > >> > > What could be the bottleneck here? >> > > >> > > Could it be qemu-img utility? >> > > >> > > vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 >> 0:06 >> > > /usr/bin/qemu-img convert -p -t none -T none -f raw >> > > >> > /rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ >> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- >> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e >> > > -O raw >> > > >> > /rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ >> export/81094499-a392-4ea2-b081-7c6288fbb636/images/ >> ba42cbcc-c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e >> > > >> > > Any idea how to make it work faster or what throughput should >> I >> > > expected? >> > > >> > > >> > > gluster storage operations are using fuse mount - so every write: >> > > - travel to the kernel >> > > - travel back to the gluster fuse helper process >> > > - travel to all 3 replicas - replication is done on client side >> > > - return to kernel when all writes succeeded >> > > - return to caller >> > > >> > > So gluster will never set any speed record. >> > > >> > > Additionally, you are copying from raw lv on FC - qemu-img cannot >> do >> > > anything >> > > smart and avoid copying unused clusters. Instead if copies >> > gigabytes of >> > > zeros >> > > from FC. >> > >> > ok, it does make sense >> > >> > > However 7.5-10 MiB/s sounds too slow. >> > > >> > > I would try to test with dd - how much time it takes to copy >> > > the same image from FC to your gluster storage? >> > > >> > > dd >> > > if=/rhev/data-center/2ff6d0ee-a10b-473d-b77c-be9149945f5f/ >> ff3cd56a-1005-4426-8137-8f422c0b47c1/images/ba42cbcc- >> c068-4df8-af3d-00f2077b1e27/c57acd5f-d6cf-48cc-ad0c-4a7d979c0c1e >> > > of=/rhev/data-center/mnt/glusterSD/10.20.30.41:_rhv__ >> export/81094499-a392-4ea2-b081-7c6288fbb636/__test__ >> > > bs=8M oflag=direct status=progress >> > >> > unfrotunately dd performs the same >> > >> > 1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s >> > >> > >> > > If dd can do this faster, please ask on qemu-discuss mailing list: >> > > https://lists.nongnu.org/mailman/listinfo/qemu-discuss >> > <https://lists.nongnu.org/mailman/listinfo/qemu-discuss> >> > > >> > > If both give similar results, I think asking in gluster mailing >> list >> > > about this can help. Maybe your gluster setup can be optimized. >> > >> > ok, this is definitly on the gluster side. Thanks for your guidance. >> > >> > I will investigate the gluster side and also will try Export on NFS >> > share. >> > >> > >> > [Adding gluster users ml] >> > >> > Please provide "gluster volume info" output for the rhv_export gluster >> > volume and also volume profile details (refer to earlier mail from Shani >> > on how to run this) while performing the dd operation above. >> >> you can find all this output on https://pastebin.com/sBK01VS8 >> >> as mentioned in other posts. Gluster cluster uses really slow (green) >> disks but without direct io it can achieve throughput around 400mbit/s. >> >> This storage is used mostly for backup purposes. It is not used as a vm >> storage. >> >> In my case it would be nice not to use direct io in export case but I >> understand why it might not be wise. >> >> Cheers, >> >> Jiri >> >> > >> > >> > >> > >> > Cheers, >> > >> > Jiri >> > >> > >> > > >> > > Nir >> > > >> > > >> > > >> > > Cheers, >> > > >> > > Jiri >> > > >> > > >> > > _______________________________________________ >> > > Users mailing list >> > > Users@ovirt.org <mailto:Users@ovirt.org> >> > <mailto:Users@ovirt.org <mailto:Users@ovirt.org>> >> > > http://lists.ovirt.org/mailman/listinfo/users >> > <http://lists.ovirt.org/mailman/listinfo/users> >> > > >> > >> > >> > >> > _______________________________________________ >> > Users mailing list >> > Users@ovirt.org <mailto:Users@ovirt.org> >> > http://lists.ovirt.org/mailman/listinfo/users >> > <http://lists.ovirt.org/mailman/listinfo/users> >> > >> > >> >> >> _______________________________________________ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> > > _______________________________________________ > Gluster-users mailing list > gluster-us...@gluster.org > http://lists.gluster.org/mailman/listinfo/gluster-users >
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users