[ovirt-users] Re: Storage IO

2018-06-05 Thread Yaniv Kaul 
On Mon, Jun 4, 2018, 11:20 PM Thomas Fecke  wrote:

> Hey Juan,
>
>
>
> That would be perfect. Ill searching for weeks now and can´t find the
> Bottleneck
>
>
>
> The Storage is attached to an 10 gig switch
>
> Storage:
>
>
>
> 1HE 19'' Chassis mit 4 hot-swap Einschüben
>
> 600W Platinum PSU
>
> Intel Xeon E5-2620 v4 CPU
>
> 16GB DDR4-2400 regECC RAM (2x8)
>
> 4x 1TB SATA3 SSD (Samsung Pro)
>
> LSI3108 Raidcontroller
>
> 2x Intel 10G-BaseT LAN
>
> 1x ded. KVM Port (IPMI2.0)
>
>
>
>
>
> The Storage is attached to an 10 gig Switch. Just our Hypervisors are
> connected aswell to that switch. I don’t know the Switch Model, its rented
> from our Hoster ( like the Hardware ).
>
>
>
> The Data Domain is shared via NFS.
>
>
>
> We are working a lot with Templates – so a single Template gets deployed
> like 20 Times. Dunno if its important. The Guests running Win 10 and Win
> Server 2016 – Guest tools are installed.
>
>
>
> “Iotop” show about 100 MB/Sec
>
>
>
> “Io sta”t show Storage Timeouts
>
>
>
> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=sync
>

oflag=direct , bs=1m and bs=1000 makes a lot more sense.
/root/testfile is on that storage? So where's the network medium?

1+0 Datensätze ein
>
> 1+0 Datensätze aus
>
> 1073741824 Bytes (1,1 GB) kopiert, 10,7527 s, 99,9 MB/s
>

Can you verify your 10g did not by mistake auto-negotiated to 1g?
Y.


>
> DD ist way to low. When I reboot the Server its up to 800-900 MB/s. It
> drops slowly to under 100 in about 5 Minutes. Like a Cache that is filling
> up.
>
>
>
> RAM and CPU are fine ( maximum 50% system load – average 30% ).
>
>
>
> File system is XFS – Raid 4 is used
>
>
>
>
>
>
>
> *From:* Juan Pablo 
> *Sent:* Montag, 4. Juni 2018 21:10
> *To:* Thomas Fecke 
> *Cc:* users@ovirt.org
> *Subject:* Re: [ovirt-users] Storage IO
>
>
>
> Hi Thomas, so you are seeing high load on your storage and you are asking
> 'why'? an answer with the facts you give would be: you are using your
> storage, so, you have storage IO.
>
>
>
> so, if you want to dive deeper:
>
> -which storage are you using, specs would be nice.
>
> -which host model are you using?
>
> -network specs? card model, etc. switch model, etc.
>
>
>
> hows your setup made? iscsi? nfs? gluster?
>
>
>
> based on the former, we might get a better idea and after this some tests
> could be made if needed to find if there's a bottleneck or if the
> environment is working as expected..
>
>
>
> regards,
>
>
>
>
>
> 2018-06-04 14:29 GMT-03:00 Thomas Fecke :
>
> Hey Guys,
>
>
>
> sorry i need to ask again.
>
>
>
> We got 2 Hypervisor with about 50 running VM´s and a single Storage with
> 10 Gig connection.
>
>
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   3,00   694,00 1627,00  947,00 103812,00 61208,00
> 128,22 6,782,632,133,49   0,39  99,70
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,70   31,370,00   64,93
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   1,00   805,00  836,00  997,00 43916,00 57900,00
> 111,09 6,003,271,874,44   0,54  99,30
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,54   29,960,00   66,50
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   2,00   822,00 1160,00 1170,00 46700,00 52176,00
> 84,87 5,682,441,573,30   0,43  99,50
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,005,05   31,460,00   63,50
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   3,00  1248,00 2337,00 1502,00 134932,00 48536,00
> 95,58 6,591,721,532,01   0,26  99,30
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,95   31,790,00   64,26
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0,00   704,00  556,00 1292,00 19908,00 72600,00
> 100,12 5,502,991,833,48   0,54  99,50
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,03   28,900,00   68,07
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0,00   544,00  278,00 1095,00  7848,00 66124,00
> 107,75 5,313,871,494,47   0,72  99,10
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,03   29,320,00   67,65
>
>
>
> Device: rrqm/s   wrqm/s r/s w/s

[ovirt-users] Re: Storage IO

2018-06-05 Thread Gianluca Cecchi 
On Mon, Jun 4, 2018 at 10:11 PM, Thomas Fecke  wrote:

> Hey Juan,
>
>
>
> That would be perfect. Ill searching for weeks now and can´t find the
> Bottleneck
>
>
>
> The Storage is attached to an 10 gig switch
>
> Storage:
>
>
>
> 1HE 19'' Chassis mit 4 hot-swap Einschüben
>
> 600W Platinum PSU
>
> Intel Xeon E5-2620 v4 CPU
>
> 16GB DDR4-2400 regECC RAM (2x8)
>
> 4x 1TB SATA3 SSD (Samsung Pro)
>
> LSI3108 Raidcontroller
>
> 2x Intel 10G-BaseT LAN
>
> 1x ded. KVM Port (IPMI2.0)
>
>
>

[snip]


>
>
> File system is XFS – Raid 4 is used
>
>
>

Hello,
but which kind of software does it run the storage system?
You say filesystem formatted as XFS and serviced as NFS, so this seems to
imply an OS on it: what kind?
Possibly this information could provide more hints to help you.

Gianluca
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/L6773XW3M4V5TYW4PGUTMFH6CXJ7NAYD/

[ovirt-users] Re: Storage IO

2018-06-05 Thread Christophe TREFOIS 
Hi Thomas,

I am by no means expert, but here couple of thoughts.

The cache filling up could be raid controller cache filling up and not emptying 
fast enough.
To have valid results, you could use another tool than dd, or run dd multiple 
times after another.

I don’t think, network have anything to do here with disk load, it would rather 
create system load as writes get queued.

Raid 4 seems strange to run VMs. Any reason why you pick this? The way I 
understand is that although you have 3 disks for data, all parity bits from the 
3 disks write to the 4th disk. This seems to be create bottlenecks.
Or you use Raid 4 only for OS?

Also, what are the VMs doing? Are they doing a lot of IO inside? 50 VMs on 4 
disks seems a little much also if each VM do many IO. (If you use the 4TB SSD 
for storage)
Lastly, are the disk volumes thin provision or thick?

All this could help you figure out some more things I think.

Kind regards,
T

> On 4 Jun 2018, at 22:11, Thomas Fecke  wrote:
> 
> Hey Juan,
>  
> That would be perfect. Ill searching for weeks now and can´t find the 
> Bottleneck
>  
> The Storage is attached to an 10 gig switch
> Storage:
>  
> 1HE 19'' Chassis mit 4 hot-swap Einschüben
> 600W Platinum PSU
> Intel Xeon E5-2620 v4 CPU
> 16GB DDR4-2400 regECC RAM (2x8)
> 4x 1TB SATA3 SSD (Samsung Pro)
> LSI3108 Raidcontroller
> 2x Intel 10G-BaseT LAN
> 1x ded. KVM Port (IPMI2.0)
>  
>  
> The Storage is attached to an 10 gig Switch. Just our Hypervisors are 
> connected aswell to that switch. I don’t know the Switch Model, its rented 
> from our Hoster ( like the Hardware ).
>  
> The Data Domain is shared via NFS.
>  
> We are working a lot with Templates – so a single Template gets deployed like 
> 20 Times. Dunno if its important. The Guests running Win 10 and Win Server 
> 2016 – Guest tools are installed. 
>  
> “Iotop” show about 100 MB/Sec
>  
> “Io sta”t show Storage Timeouts
>  
> dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=sync
> 1+0 Datensätze ein
> 1+0 Datensätze aus
> 1073741824 Bytes (1,1 GB) kopiert, 10,7527 s, 99,9 MB/s
>  
> DD ist way to low. When I reboot the Server its up to 800-900 MB/s. It drops 
> slowly to under 100 in about 5 Minutes. Like a Cache that is filling up.
>  
> RAM and CPU are fine ( maximum 50% system load – average 30% ).
>  
> File system is XFS – Raid 4 is used
>  
>  
>  
> From: Juan Pablo  > 
> Sent: Montag, 4. Juni 2018 21:10
> To: Thomas Fecke mailto:thomas.fe...@eset.de>>
> Cc: users@ovirt.org 
> Subject: Re: [ovirt-users] Storage IO
>  
> Hi Thomas, so you are seeing high load on your storage and you are asking 
> 'why'? an answer with the facts you give would be: you are using your 
> storage, so, you have storage IO.
>  
> so, if you want to dive deeper:
> -which storage are you using, specs would be nice.
> -which host model are you using?
> -network specs? card model, etc. switch model, etc.
>  
> hows your setup made? iscsi? nfs? gluster?
>  
> based on the former, we might get a better idea and after this some tests 
> could be made if needed to find if there's a bottleneck or if the environment 
> is working as expected..
>  
> regards,  
>  
>  
> 2018-06-04 14:29 GMT-03:00 Thomas Fecke  >:
> Hey Guys,
>  
> sorry i need to ask again.
>  
> We got 2 Hypervisor with about 50 running VM´s and a single Storage with 10 
> Gig connection.
>  
>  
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
> avgqu-sz   await r_await w_await  svctm  %util
> sda   3,00   694,00 1627,00  947,00 103812,00 61208,00   128,22   
>   6,782,632,133,49   0,39  99,70
>  
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>0,000,003,70   31,370,00   64,93
>  
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
> avgqu-sz   await r_await w_await  svctm  %util
> sda   1,00   805,00  836,00  997,00 43916,00 57900,00   111,09
>  6,003,271,874,44   0,54  99,30
>  
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>0,000,003,54   29,960,00   66,50
>  
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
> avgqu-sz   await r_await w_await  svctm  %util
> sda   2,00   822,00 1160,00 1170,00 46700,00 52176,0084,87
>  5,682,441,573,30   0,43  99,50
>  
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>0,000,005,05   31,460,00   63,50
>  
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
> avgqu-sz   await r_await w_await  svctm  %util
> sda   3,00  1248,00 2337,00 1502,00 134932,00 48536,0095,58   
>   6,591,721,532,01   0,26  99,30
>  
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>0,000,003,95   31,790,00   64,26
>  
> Device: rrqm/s

[ovirt-users] Re: Storage IO

2018-06-04 Thread Thomas Fecke 
Hey Juan,

That would be perfect. Ill searching for weeks now and can´t find the Bottleneck

The Storage is attached to an 10 gig switch
Storage:

1HE 19'' Chassis mit 4 hot-swap Einschüben
600W Platinum PSU
Intel Xeon E5-2620 v4 CPU
16GB DDR4-2400 regECC RAM (2x8)
4x 1TB SATA3 SSD (Samsung Pro)
LSI3108 Raidcontroller
2x Intel 10G-BaseT LAN
1x ded. KVM Port (IPMI2.0)


The Storage is attached to an 10 gig Switch. Just our Hypervisors are connected 
aswell to that switch. I don't know the Switch Model, its rented from our 
Hoster ( like the Hardware ).

The Data Domain is shared via NFS.

We are working a lot with Templates - so a single Template gets deployed like 
20 Times. Dunno if its important. The Guests running Win 10 and Win Server 2016 
- Guest tools are installed.

"Iotop" show about 100 MB/Sec

"Io sta"t show Storage Timeouts

dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=sync
1+0 Datensätze ein
1+0 Datensätze aus
1073741824 Bytes (1,1 GB) kopiert, 10,7527 s, 99,9 MB/s

DD ist way to low. When I reboot the Server its up to 800-900 MB/s. It drops 
slowly to under 100 in about 5 Minutes. Like a Cache that is filling up.

RAM and CPU are fine ( maximum 50% system load - average 30% ).

File system is XFS - Raid 4 is used



From: Juan Pablo 
Sent: Montag, 4. Juni 2018 21:10
To: Thomas Fecke 
Cc: users@ovirt.org
Subject: Re: [ovirt-users] Storage IO

Hi Thomas, so you are seeing high load on your storage and you are asking 
'why'? an answer with the facts you give would be: you are using your storage, 
so, you have storage IO.

so, if you want to dive deeper:
-which storage are you using, specs would be nice.
-which host model are you using?
-network specs? card model, etc. switch model, etc.

hows your setup made? iscsi? nfs? gluster?

based on the former, we might get a better idea and after this some tests could 
be made if needed to find if there's a bottleneck or if the environment is 
working as expected..

regards,


2018-06-04 14:29 GMT-03:00 Thomas Fecke 
mailto:thomas.fe...@eset.de>>:
Hey Guys,

sorry i need to ask again.

We got 2 Hypervisor with about 50 running VM´s and a single Storage with 10 Gig 
connection.


Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   3,00   694,00 1627,00  947,00 103812,00 61208,00   128,22 
6,782,632,133,49   0,39  99,70

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,003,70   31,370,00   64,93

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   1,00   805,00  836,00  997,00 43916,00 57900,00   111,09 
6,003,271,874,44   0,54  99,30

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,003,54   29,960,00   66,50

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   2,00   822,00 1160,00 1170,00 46700,00 52176,0084,87 
5,682,441,573,30   0,43  99,50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,005,05   31,460,00   63,50

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   3,00  1248,00 2337,00 1502,00 134932,00 48536,0095,58 
6,591,721,532,01   0,26  99,30

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,003,95   31,790,00   64,26

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   0,00   704,00  556,00 1292,00 19908,00 72600,00   100,12 
5,502,991,833,48   0,54  99,50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,003,03   28,900,00   68,07

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   0,00   544,00  278,00 1095,00  7848,00 66124,00   107,75 
5,313,871,494,47   0,72  99,10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   0,000,003,03   29,320,00   67,65

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz 
avgqu-sz   await r_await w_await  svctm  %util
sda   0,00   464,00  229,00 1172,00  6588,00 72384,00   112,74 
5,443,881,674,31   0,71  99,50




and this is our Problem. Anyone know why our Storage recive that much of 
Precesses?

Thanks in advance

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to 
users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:

[ovirt-users] Re: Storage IO

2018-06-04 Thread Juan Pablo 
Hi Thomas, so you are seeing high load on your storage and you are asking
'why'? an answer with the facts you give would be: you are using your
storage, so, you have storage IO.

so, if you want to dive deeper:
-which storage are you using, specs would be nice.
-which host model are you using?
-network specs? card model, etc. switch model, etc.

hows your setup made? iscsi? nfs? gluster?

based on the former, we might get a better idea and after this some tests
could be made if needed to find if there's a bottleneck or if the
environment is working as expected..

regards,


2018-06-04 14:29 GMT-03:00 Thomas Fecke :

> Hey Guys,
>
>
>
> sorry i need to ask again.
>
>
>
> We got 2 Hypervisor with about 50 running VM´s and a single Storage with
> 10 Gig connection.
>
>
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   3,00   694,00 1627,00  947,00 103812,00 61208,00
> 128,22 6,782,632,133,49   0,39  99,70
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,70   31,370,00   64,93
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   1,00   805,00  836,00  997,00 43916,00 57900,00
> 111,09 6,003,271,874,44   0,54  99,30
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,54   29,960,00   66,50
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   2,00   822,00 1160,00 1170,00 46700,00 52176,00
> 84,87 5,682,441,573,30   0,43  99,50
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,005,05   31,460,00   63,50
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   3,00  1248,00 2337,00 1502,00 134932,00 48536,00
> 95,58 6,591,721,532,01   0,26  99,30
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,95   31,790,00   64,26
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0,00   704,00  556,00 1292,00 19908,00 72600,00
> 100,12 5,502,991,833,48   0,54  99,50
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,03   28,900,00   68,07
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0,00   544,00  278,00 1095,00  7848,00 66124,00
> 107,75 5,313,871,494,47   0,72  99,10
>
>
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle
>
>0,000,003,03   29,320,00   67,65
>
>
>
> Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
> avgqu-sz   await r_await w_await  svctm  %util
>
> sda   0,00   464,00  229,00 1172,00  6588,00 72384,00
> 112,74 5,443,881,674,31   0,71  99,50
>
>
>
>
>
>
>
>
>
> and this is our Problem. Anyone know why our Storage recive that much of
> Precesses?
>
>
>
> Thanks in advance
>
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-
> guidelines/
> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/
> message/YENR6R4ESX3JCOS7DYA354EOPNL6WGUN/
>
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/P2YH5SLSFSHA6BNZHSIIJUUTZLUOOMGK/