First of all,thank you for answering. the result is : it is not vm's problem,it is raid card problem,raid card without cache. I test again ,the data as below: dd if=/dev/zero of=dd.img bs=1M count=2000 ------------------------------------------------------------------------- use single thread: /root/fio -filename=/mnt/dd.img -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=1G -numjobs=1 -runtime=60 -group_reporting -name=mytest R330(H330 0MB-cache,SAS 10K 600G * 6, Raid-10) : bw=3028.7KB/s, iops=189 NX3200 (H710 Mini 512MB-cache,SAS 7.2K 4T * 10,Raid-5) : bw=14555KB/s, iops=909 MD3200i (2GB-cache,SAS 15K 600G * 12 ,Raid-5) : bw=24120KB/s, iops=1507 R710 (PERC 6/i Integrated RAID Controller 256MB-cache,SAS 15K 600G * 2, Raid-1) : bw=16183KB/s, iops=1011 R410 (SAS 6/iR Integrated Controller 0MB-cache,SAS 15K 300G * 2,Raid-1) : bw=3397.6KB/s, iops=212
At 2016-06-15 12:22:35, "Alexandre DERUMIER" <[email protected]> wrote: >raid10 is physical host's Raid-10 with 6 disks(SAS 10k 600G) > > >>>local-highspeed is LogicalVolume on Raid-10 > >lvm volume ? classic lvm or thin lvm ? > > >>>raid10 is physical host's Raid-10 with 6 disks(SAS 10k 600G) >>>"do you have cache on your raid controller ?" I don't know what's means? > >Well, generally (for hdd mainly), hardware raid controller have memory to >handle writes fast, then flush them to disk. >what is your model ? > > > >>>fio command is : ./fio -filename=/dev/vda -direct=1 -iodepth 1 -thread >>>-rw=randwrite -ioengine=psync -bs=16k -size=4G -numjobs=30 -runtime=100 >>>-group_reporting -name=mytest > >So you are doing sync write, >that mean that it should bypass all caches (including raid controller cache, >disk cache) to be sure that data are correctly written to the disk platters. > >I'm surprised than you can reach iops=1344 with 6 SAS 10K in raid10. (should >be 250 iops by disk , x3 because of raid10, so around 750iops). > > >Now, some raid hardware controller, can handle the sync writes, without >transmit them to disk platter (They need to have a batery, or persistent >memory). >Could explain why you have better performance with your san. > > >can you do fio benchmark with ioengin : libaio , and increase iodepth to 32 > >./fio -filename=/dev/vda -direct=1 -iodepth 32 -rw=randwrite -ioengine=libaiO >-bs=16k -size=4G -numjobs=30 -runtime=100 -group_reporting -name=mytest > > >then compare with --sync=1 option > >./fio -filename=/dev/vda -direct=1 -iodepth 32 -thread -rw=randwrite >-ioengine=libaiO -bs=16k -size=4G -numjobs=30 -runtime=100 -group_reporting >-name=mytest > >----- Mail original ----- >De: "haoyun" <[email protected]> >À: "proxmoxve" <[email protected]> >Envoyé: Mercredi 15 Juin 2016 03:51:03 >Objet: Re: [PVE-User] vm's disk io lower with local highspeed disk? > >fio command is : ./fio -filename=/dev/vda -direct=1 -iodepth 1 -thread >-rw=randwrite -ioengine=psync -bs=16k -size=4G -numjobs=30 -runtime=100 >-group_reporting -name=mytest > > >vmconfig: >balloon: 1024 >bootdisk: virtio2 >cores: 2 >ide2: none,media=cdrom >memory: 4096 >name: tt >net0: bridge=vmbr0,e1000=66:37:31:32:34:37 >numa: 0 >ostype: l26 >smbios1: uuid=ce775a6f-6652-43c4-aa92-69ffd3d20013 >sockets: 1 >virtio0: local-highspeed:vm-255-disk-2,size=60G >virtio2: local-highspeed:vm-255-disk-1,size=8G > > >raid10 is physical host's Raid-10 with 6 disks(SAS 10k 600G) > > >local-highspeed is LogicalVolume on Raid-10 > >"do you have cache on your raid controller ?" I don't know what's means? > > > >At 2016-06-14 17:59:17, "Alexandre DERUMIER" <[email protected]> wrote: >>>>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm >>>>%util >>>>vda 0.00 71.00 8.00 226.00 64.00 2368.00 10.39 1.37 5.85 4.24 99.10 >>>>but I test disk random write is better with fio tools, >>>>bw=21518KB/s iops=1344 >> >> >>do you have cache on your raid controller ? >> >>can you send the fio job config? >> >>can you send vmconfig ? >> >>what is the local raid10 filesystem ? >> >> >> >> >>----- Mail original ----- >>De: "haoyun" <[email protected]> >>À: "proxmoxve" <[email protected]> >>Envoyé: Mardi 14 Juin 2016 07:53:29 >>Objet: [PVE-User] vm's disk io lower with local highspeed disk? >> >>hello everyone~ >> >> >>I have a problem, >>I have a new host is Dell R330, >>run pve4.2 os >>disk with 2 raid,one is raid1,another is raid10(sas 10k 600G * 6) >>raid1 as system disk,raid10 as vm images >>my mysql slave run on this vm with raid10 local disk(lvm). >>the question is this vm's disk io is very slow, >>that's why? >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm >>%util >>vda 0.00 71.00 8.00 226.00 64.00 2368.00 10.39 1.37 5.85 4.24 99.10 >>but I test disk random write is better with fio tools, >>bw=21518KB/s iops=1344 >>same vm,I use iscsi remote storage better than it, >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm >>%util >>vda 0.00 0.00 0.00 3.00 0.00 24.00 8.00 0.00 0.33 0.33 0.10 >>that's why? >>who can help me? >>What information is needed？please tell me? >>thanks! >>_______________________________________________ >>pve-user mailing list >>[email protected] >>http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>_______________________________________________ >>pve-user mailing list >>[email protected] >>http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >_______________________________________________ >pve-user mailing list >[email protected] >http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >_______________________________________________ >pve-user mailing list >[email protected] >http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list [email protected] http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
