[ovirt-users] Re: i/o wait and slow system
See below: > On Aug 27, 2020, at 3:19 PM, info--- via Users wrote: > > Thank you. Reboot of the engine and afterwards the backup server helped :-) Good deal. > Should I revert some of my previous changes? Reduce the write window size? > - gluster volume set vmstore performance.read-ahead on > - gluster volume set vmstore performance.stat-prefetch on > - gluster volume set vmstore performance.write-behind-window-size 64MB > - gluster volume set vmstore performance.flush-behind on > - gluster volume set vmstore cluster.choose-local on I find these tend to be workload & system dependent, you’re best bet is to benchmark and test yourself on your system. I’m using these on my volumes: performance.stat-prefetch: on performance.read-ahead: off performance.write-behind-window-size: 64MB___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YTGSW23P4Z36UMRU26PZCRATRLP75XHC/
[ovirt-users] Re: i/o wait and slow system
Thank you. Reboot of the engine and afterwards the backup server helped :-) Before the change -> 53 minutes for 10% and the restore broke at 98% ... - [2020-08-26:20:45:30] Restore Imaging: 1/1 - 10% Imaging in progress - [2020-08-26:20:41:22] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-26:20:37:33] Restore Imaging: 1/1 - 8% Imaging in progress. - [2020-08-26:20:32:04] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-26:20:25:38] Restore Imaging: 1/1 - 6% Imaging in progress... - [2020-08-26:20:21:34] Restore Imaging: 1/1 - 5% Imaging in progress... - [2020-08-26:20:18:57] Restore Imaging: 1/1 - 3% Imaging in progress. - [2020-08-26:20:03:15] Restore Imaging: 1/1 - 2% Imaging in progress - [2020-08-26:19:57:10] Restore Imaging: 1/1 - 1% Imaging in progress. - [2020-08-26:19:53:20] Restore Imaging: 1/1 - 0% Imaging in progress... - [2020-08-26:19:53:17] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb After the change -> 10 minutes for 10% - [2020-08-27:22:08:59] Restore Imaging: 1/1 - 10% Imaging in progress... - [2020-08-27:22:08:02] Restore Imaging: 1/1 - 9% Imaging in progress... - [2020-08-27:22:07:05] Restore Imaging: 1/1 - 8% Imaging in progress... - [2020-08-27:22:05:34] Restore Imaging: 1/1 - 7% Imaging in progress... - [2020-08-27:22:04:43] Restore Imaging: 1/1 - 6% Imaging in progress.. - [2020-08-27:22:03:34] Restore Imaging: 1/1 - 5% Imaging in progress.. - [2020-08-27:22:02:27] Restore Imaging: 1/1 - 4% Imaging in progress. - [2020-08-27:22:01:39] Restore Imaging: 1/1 - 3% Imaging in progress - [2020-08-27:22:00:16] Restore Imaging: 1/1 - 2% Imaging in progress... - [2020-08-27:21:59:24] Restore Imaging: 1/1 - 1% Imaging in progress - [2020-08-27:21:58:16] Restore Imaging: 1/1 - 0% Imaging in progress.. - [2020-08-27:21:58:13] Restore Imaging: 1/1 - 0% Started Imaging Disk1.img.lzo to /dev/sdb Should I revert some of my previous changes? Reduce the write window size? - gluster volume set vmstore performance.read-ahead on - gluster volume set vmstore performance.stat-prefetch on - gluster volume set vmstore performance.write-behind-window-size 64MB - gluster volume set vmstore performance.flush-behind on - gluster volume set vmstore cluster.choose-local on Now I will reboot the other VMs and go on with moving from thin provisioned disk to preallocated. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4NYU6QTVX2LESDTEYO5FHASS43RQE7RG/
[ovirt-users] Re: i/o wait and slow system
Looks like you’ve got a posix or nfs mount there? Is your gluster storage domain of type GlusterFS? And make sure you restarted the ovirt-engine after enabling LibfgApiSupported, before stopping and restarting the vm. An active libgf mount looks like: > On Aug 26, 2020, at 1:12 PM, info--- via Users wrote: > > I enabled libgfapi and powered off / on the VM. > > - engine-config --all > - LibgfApiSupported: true version: 4.3 > > How can I see that this is active on the VM? The disk looks the same like > before. > > - virsh dumpxml 15 > > io='threads'/> > file='/rhev/data-center/mnt/glusterSD/10.9.9.101:_vmstore/f2c621de-42bf-4dbf-920c-adf4506b786d/images/1e231e3e-d98c-491a-9236-907814d4837/c755aaa3-7d3d-4c0d-8184-c6aae37229ba'> > > > > > > > > Here is the Volume setup: > > Volume Name: vmstore > Type: Distributed-Replicate > Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 > Status: Started > Snapshot Count: 0 > Number of Bricks: 2 x 3 = 6 > Transport-type: tcp > Bricks: > Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore > Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore > Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore > Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L > Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L > Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y > Options Reconfigured: > performance.write-behind-window-size: 64MB > performance.flush-behind: on > performance.stat-prefetch: on > performance.client-io-threads: on > nfs.disable: on > transport.address-family: inet > performance.strict-o-direct: on > performance.quick-read: off > performance.read-ahead: on > performance.io-cache: off > performance.low-prio-threads: 32 > network.remote-dio: off > cluster.eager-lock: enable > cluster.quorum-type: auto > cluster.server-quorum-type: server > cluster.data-self-heal-algorithm: full > cluster.locking-scheme: granular > cluster.shd-max-threads: 8 > cluster.shd-wait-qlength: 1 > features.shard: on > user.cifs: off > cluster.choose-local: on > client.event-threads: 4 > server.event-threads: 4 > network.ping-timeout: 30 > storage.owner-uid: 36 > storage.owner-gid: 36 > cluster.granular-entry-heal: enable > > Thank you for your support. > ___ > Users mailing list -- users@ovirt.org > To unsubscribe send an email to users-le...@ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z3JD7MQV2PIQZJSMW6NKPL4W7JLBGPKN/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OMVWGGWHNP33GNYFMLUYYFS32BL3FHQC/
[ovirt-users] Re: i/o wait and slow system
On Wed, Aug 26, 2020 at 8:19 PM info--- via Users wrote: > I enabled libgfapi and powered off / on the VM. > > - engine-config --all > - LibgfApiSupported: true version: 4.3 > > How can I see that this is active on the VM? The disk looks the same like > before. > > - virsh dumpxml 15 > >io='threads'/> > > > > > > > > > > Latest status I remember was this one reported by me at 4.3 RC2 time last year: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KHWQA72JZVGLVXCIQPU2FDFRS73BW4LY/ and I think nothing changed in the mean time, neither for 4.3 nor for 4.4. But not directly tested. Someone reported 4x-5x improvements using libgfapi but initially there were some blockers related to snapshots and HA if I remember correctly and so disabled. My idea is that all previous problems in upstream qemu and libvirtd pieces are now solved since many months but developers didn't consider implementing them due to not so big improvements detected in their tests and/or sufficient time to dedicate to fix and/or other priorities. Some bugzilla referred, to dig into if interested: https://bugzilla.redhat.com/show_bug.cgi?id=1484227 https://bugzilla.redhat.com/show_bug.cgi?id=1465810 https://bugzilla.redhat.com/show_bug.cgi?id=1633642 HIH revamping the interesting topic. Gianluca ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/IEP5T3AZLLRDMAW2CCQPQEXYC5PD62LZ/
[ovirt-users] Re: i/o wait and slow system
I enabled libgfapi and powered off / on the VM. - engine-config --all - LibgfApiSupported: true version: 4.3 How can I see that this is active on the VM? The disk looks the same like before. - virsh dumpxml 15 Here is the Volume setup: Volume Name: vmstore Type: Distributed-Replicate Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 3 = 6 Transport-type: tcp Bricks: Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y Options Reconfigured: performance.write-behind-window-size: 64MB performance.flush-behind: on performance.stat-prefetch: on performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: on performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 1 features.shard: on user.cifs: off cluster.choose-local: on client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enable Thank you for your support. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z3JD7MQV2PIQZJSMW6NKPL4W7JLBGPKN/
[ovirt-users] Re: i/o wait and slow system
Libgfapi bypasses the context switching from User space to kernel to user space (FUSE) , so it gets better performance. I can't find the previous communication ,so can you share your volume settings again ? Best Regards, Strahil Nikolov В неделя, 23 август 2020 г., 21:45:22 Гринуич+3, info--- via Users написа: Setting cluster.choose-local to on helped a lot to improve the read performance. Write performance still bad. Am I right that this look then more like a glusterfs issue and not something what need to be changed on ovirt (libgfapi) or on VMs. Changing tcp offloading did not make any difference. - ethtool --offload enp1s0f0 rx off tx off - ethtool --offload enp1s0f0 rx on tx on MTU 9000 is fine and ping is working. - ping -M do -s 8972 another-gluster-node ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/23FIY4DLT5CBZFJV7TTDHHLI5SAO3JWL/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/EN57VNXQ4M3QXLPACINEU67QS52G7JSN/
[ovirt-users] Re: i/o wait and slow system
Setting cluster.choose-local to on helped a lot to improve the read performance. Write performance still bad. Am I right that this look then more like a glusterfs issue and not something what need to be changed on ovirt (libgfapi) or on VMs. Changing tcp offloading did not make any difference. - ethtool --offload enp1s0f0 rx off tx off - ethtool --offload enp1s0f0 rx on tx on MTU 9000 is fine and ping is working. - ping -M do -s 8972 another-gluster-node ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/23FIY4DLT5CBZFJV7TTDHHLI5SAO3JWL/
[ovirt-users] Re: i/o wait and slow system
Thank you. ping -M do -s 8972 another-gluster-node is working. The rest I'll check on the weekend. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/FFBX4LBEMELPAJUGJLZJAPKYQQT4NWDH/
[ovirt-users] Re: i/o wait and slow system
Thank you. They are all Thin. I have something to do at the weekend and change them to Preallocated. Looks like stopping the VM. Downloading and uploading the disk to change the format. ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/65JQW62GKXJXHD6MCQVPKZXTVODBXGHM/
[ovirt-users] Re: i/o wait and slow system
Are you using fully allocated VM disks ? На 20 август 2020 г. 0:53:40 GMT+03:00, info--- via Users написа: >Additional Info: I've running Nextcloud on a VM. >- to sync (download) 1 file to the client with 300 MB is fast >- to sync (download) 300 files to the client with in total 1 MB is very >slow >___ >Users mailing list -- users@ovirt.org >To unsubscribe send an email to users-le...@ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/LZRN2NRC7Y57XQ6GV2OJT4NY7Q74N6OE/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/566DKGVEH3XRVP2LTF5OWDJHXUZWGTJ4/
[ovirt-users] Re: i/o wait and slow system
На 19 август 2020 г. 22:39:22 GMT+03:00, info--- via Users написа: >Thank you for the quick reply. > >- I/O scheduler hosts -> changed >echo noop > /sys/block/sdb/queue/scheduler >echo noop > /sys/block/sdc/queue/scheduler On reboot it will be reverted. Test this way and if you notice improvement use udev rules. Keep in mind that multipath devices also have a scheduler. >- CPU states -> can you explain this a bit more? On Intel CPUs you can use these kernel parameters (check them online as I type them by memory) processor.max_cstate=1 intel_idle.max_cstate=0 >cat /dev/cpu_dma_latency >F >hexdump -C /dev/cpu_dma_latency > 46 00 00 00 |F...| >0004 > >- Tuned profile -> can you explain this a bit more? Download ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-6.el7rhgs.src.rpm and extract the srpm. Inside it you will find 2 dirs: 'rhgs-random-io' &'rhgs-sequential-io'. Open the tuned.conf files inside them and check the 'dirty' sysctl tunings. One is for random I/O (usually the Vm workload is random I/O) and another for sequential. You can set the 'dirty' settings in your sysctl.conf or your custom tuned profile. >- MTU was already set to 9000 -> can you explain a bit more Have you tested via : ping -M do -s 8972 another-gluster-node >tcp-offloading and how I change the settings? You can google it. Usually ethtool is your best friend. >ethtool -k enp1s0f0 | grep offload >tcp-segmentation-offload: on >udp-fragmentation-offload: off [fixed] >generic-segmentation-offload: on >generic-receive-offload: on >large-receive-offload: off [fixed] >rx-vlan-offload: on >tx-vlan-offload: on >l2-fwd-offload: off [fixed] >hw-tc-offload: off >rx-udp_tunnel-port-offload: on > >- I/O scheduler VM had already none >cat /sys/block/vda/queue/scheduler >none Consider using udev here too. >- oVirt 4.4 -> thanks. Was not aware that the new version is final. I >will check and upgrade. Upgrades are not so easy as you need to: A) Upgrade OS to EL8 B) Upgrade ovirt Test in advance before proceeding on prod ! >- Use High Performance VMs -> I got following message -> So I need to >do changes also on the VM? >The following recommended settings for running the High Performance >type with the optimal configuration were not detected. Please consider >manually changing of the following before applying: >CPU PINNING: >VIRTUAL NUMA AND NUMA PINNING: >HUGE PAGES: >KERNEL SAME PAGE MERGING (KSM): > >- libgfapi -> I will execute on the engine following command and then >it will change all VMs after reboot? >engine-config -s LibgfApiSupported=true --cver4.2 VM reboot won't reread the VM config. You will need an 'off' and 'on' action for each VM. I'm not sure about the command. I haven't used libgfapi for eons. >- changed values (from your other threat): >gluster volume set vmstore performance.read-ahead on >gluster volume set vmstore performance.stat-prefetch on >gluster volume set vmstore performance.write-behind-window-size 64MB >gluster volume set vmstore performance.flush-behind on > >- already configured: >performance.client-io-threads: on It's easier to use: gluster volume set VOLUME group virt WARNING: This will enable sharding and SHARDING cannot be disabled in an easy way. NEVER EVER DISABLE SHARDING. You can check the 'virt' group of settings on your gluster nodes in /var/lib/glusterd/group. >Best Regards >Dirk >___ >Users mailing list -- users@ovirt.org >To unsubscribe send an email to users-le...@ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/VNE4O46SKA5KALKNW77MQ6UL3V2MJWWI/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/GRBRHCNPVQQAEZP4VVKCGNSARTNNVYG3/
[ovirt-users] Re: i/o wait and slow system
Additional Info: I've running Nextcloud on a VM. - to sync (download) 1 file to the client with 300 MB is fast - to sync (download) 300 files to the client with in total 1 MB is very slow ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LZRN2NRC7Y57XQ6GV2OJT4NY7Q74N6OE/
[ovirt-users] Re: i/o wait and slow system
Thank you for the quick reply. - I/O scheduler hosts -> changed echo noop > /sys/block/sdb/queue/scheduler echo noop > /sys/block/sdc/queue/scheduler - CPU states -> can you explain this a bit more? cat /dev/cpu_dma_latency F hexdump -C /dev/cpu_dma_latency 46 00 00 00 |F...| 0004 - Tuned profile -> can you explain this a bit more? - MTU was already set to 9000 -> can you explain a bit more tcp-offloading and how I change the settings? ethtool -k enp1s0f0 | grep offload tcp-segmentation-offload: on udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: on tx-vlan-offload: on l2-fwd-offload: off [fixed] hw-tc-offload: off rx-udp_tunnel-port-offload: on - I/O scheduler VM had already none cat /sys/block/vda/queue/scheduler none - oVirt 4.4 -> thanks. Was not aware that the new version is final. I will check and upgrade. - Use High Performance VMs -> I got following message -> So I need to do changes also on the VM? The following recommended settings for running the High Performance type with the optimal configuration were not detected. Please consider manually changing of the following before applying: CPU PINNING: VIRTUAL NUMA AND NUMA PINNING: HUGE PAGES: KERNEL SAME PAGE MERGING (KSM): - libgfapi -> I will execute on the engine following command and then it will change all VMs after reboot? engine-config -s LibgfApiSupported=true --cver4.2 - changed values (from your other threat): gluster volume set vmstore performance.read-ahead on gluster volume set vmstore performance.stat-prefetch on gluster volume set vmstore performance.write-behind-window-size 64MB gluster volume set vmstore performance.flush-behind on - already configured: performance.client-io-threads: on Best Regards Dirk ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/VNE4O46SKA5KALKNW77MQ6UL3V2MJWWI/
[ovirt-users] Re: i/o wait and slow system
Check and tune the following: - I/O scheduler on the host (usually noop/none are good for writes, (mq-)deadline for reads) - CPU cstates - Tuned profile , there are some 'dirty' settings that will avoid I/O locks - MTU size and tcp offloading (some users report enabled is better, others feel disabled brings better performance) -I/O scheduler in VMs is best to be noop/none as we already reorder our I/O on Hypervisour level - with oVirt 4.4 Hugepages for VMs should be fixed, so you can try - Use high-performance VMs - Using libgfapi is better than FUSE, so if you are OK with the limitations that it imposes -> you can swith to libgfapi instead of FUSE (requires VM power off and power on to switch) Best Regards, Strahil Nikolov На 19 август 2020 г. 21:26:44 GMT+03:00, info--- via Users написа: >Hello, > >I'm running a home setup with 3 nodes and 2 SATA SSDs per node. >As storage I'm running glusterfs and 40GBit/s links. > >Software Version:4.3.9.4-1.el7 > >I've a lot of I/O Wait on the nodes (20%) and on the VMs (50%). > >gluster volume top vmstore write-perf bs 2014 count 1024 | grep Through >Throughput 635.54 MBps time 0.0032 secs >Throughput 614.89 MBps time 0.0034 secs >Throughput 622.31 MBps time 0.0033 secs >Throughput 643.07 MBps time 0.0032 secs >Throughput 621.75 MBps time 0.0033 secs >Throughput 609.26 MBps time 0.0034 secs > >gluster volume top vmstore read-perf bs 2014 count 1024 | grep Through >Throughput 1274.62 MBps time 0.0016 secs >Throughput 1320.32 MBps time 0.0016 secs >Throughput 1203.93 MBps time 0.0017 secs >Throughput 1293.81 MBps time 0.0016 secs >Throughput 1213.14 MBps time 0.0017 secs >Throughput 1193.48 MBps time 0.0017 secs > >Volume Name: vmstore >Type: Distributed-Replicate >Volume ID: 195e2a05-9667-4b8b-b0b7-82294631de50 >Status: Started >Snapshot Count: 0 >Number of Bricks: 2 x 3 = 6 >Transport-type: tcp >Bricks: >Brick1: 10.9.9.101:/gluster_bricks/vmstore/vmstore >Brick2: 10.9.9.102:/gluster_bricks/vmstore/vmstore >Brick3: 10.9.9.103:/gluster_bricks/vmstore/vmstore >Brick4: 10.9.9.101:/gluster_bricks/S4CYNF0M219849L/S4CYNF0M219849L >Brick5: 10.9.9.102:/gluster_bricks/S4CYNF0M219836L/S4CYNF0M219836L >Brick6: 10.9.9.103:/gluster_bricks/S4CYNF0M219801Y/S4CYNF0M219801Y >Options Reconfigured: >performance.client-io-threads: on >nfs.disable: on >transport.address-family: inet >performance.strict-o-direct: on >performance.quick-read: off >performance.read-ahead: off >performance.io-cache: off >performance.low-prio-threads: 32 >network.remote-dio: off >cluster.eager-lock: enable >cluster.quorum-type: auto >cluster.server-quorum-type: server >cluster.data-self-heal-algorithm: full >cluster.locking-scheme: granular >cluster.shd-max-threads: 8 >cluster.shd-wait-qlength: 1 >features.shard: on >user.cifs: off >cluster.choose-local: off >client.event-threads: 4 >server.event-threads: 4 >network.ping-timeout: 30 >storage.owner-uid: 36 >storage.owner-gid: 36 >cluster.granular-entry-heal: enable > >Please help me to analyse the root cause. > >Many thanks >Metz >___ >Users mailing list -- users@ovirt.org >To unsubscribe send an email to users-le...@ovirt.org >Privacy Statement: https://www.ovirt.org/privacy-policy.html >oVirt Code of Conduct: >https://www.ovirt.org/community/about/community-guidelines/ >List Archives: >https://lists.ovirt.org/archives/list/users@ovirt.org/message/TSACLDXFZXD5XV6WQ5GPJDBJQBNAUC7P/ ___ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-le...@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ARGQLJSGND5T37WIZC7RF6YD6PAVKTVF/
