Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-08 Thread Abi Askushi
Hi all,

Seems that I am seeing light in he tunnel!

I have done the below:

Added the option
/etc/glusterfs/glusterd.vol :
option rpc-auth-allow-insecure on

restarted glusterd

Then set the volume option:
gluster volume set vms server.allow-insecure on

Then disabled the option:
gluster volume set vms performance.readdir-ahead off

which seems to have been enabled when desperately testing gluster options.

then I added again the storage domain with glusterfs (not NFS).
This lead me to have really high performance boost on writes of VMs.

The dd went from 10MB/s to 60MB/s!
And the VM disk benchmarks from few MB to 80 - 100MB/s!

Seems that the above two options which were missing did the trick for the
gluster.

When checking the VM XML I still see the disk defines as file and not
network. I am not sure that ligfapi is used from ovirt.
I will set also a new VM just in case to check the XML again.
Is there any way to confirm use of ligfapi?

On Fri, Sep 8, 2017 at 10:08 AM, Abi Askushi 
wrote:

> I don't see any other bottleneck. CPUs are quite idle. Seems that the load
> is mostly due to high latency on IO.
>
> Reading further the gluster docs:
>
> https://github.com/gluster/glusterfs-specs/blob/master/
> done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md
>
> I see that I am missing the following options:
>
> /etc/glusterfs/glusterd.vol :
> option rpc-auth-allow-insecure on
>
> gluster volume set vms gluster server.allow-insecure on
>
> It says that the above allow qemu to use libgfapi.
> When checking the VM XML, I don't see any gluster protocol at the disk
> drive:
>
> 
>io='threads'/>
>   
>   
>   222a1312-5efa-4731-8914-9a9d24dccba5
>   
>   
> 
>
>
>
> While at gluster docs it mentions the below type of disk:
>
> 
>
>
> 
> 
>
> function='0x0'/>
> 
>
>
> Does the above indicate that ovirt/qemu is not using libgfapi but FUSE only?
> This could be the reason of such slow perf.
>
>
> On Thu, Sep 7, 2017 at 1:47 PM, Yaniv Kaul  wrote:
>
>>
>>
>> On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi 
>> wrote:
>>
>>>
>>>
>>> On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul  wrote:
>>>


 On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul  wrote:

>
>
> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi 
> wrote:
>
>> For a first idea I use:
>>
>> dd if=/dev/zero of=testfile bs=1GB count=1
>>
>
> This is an incorrect way to test performance, for various reasons:
> 1. You are not using oflag=direct , thus not using DirectIO, but using
> cache.
> 2. It's unrealistic - it is very uncommon to write large blocks of
> zeros (sometimes during FS creation or wiping). Certainly not 1GB
> 3. It is a single thread of IO - again, unrealistic for VM's IO.
>
> I forgot to mention that I include oflag=direct in my tests. I agree
>>> though that dd is not the correct way to test, hence I mentioned I just use
>>> it to get a first feel. More tests are done within the VM benchmarking its
>>> disk IO (with tools like IOmeter).
>>>
>>> I suggest using fio and such. See https://github.com/pcuzner/fio-tools
> for example.
>
 Do you have any recommended config file to use for VM workload?
>>>
>>
>> Desktops and Servers VMs behave quite differently, so not really. But the
>> 70/30 job is typically a good baseline.
>>
>>
>>>
>>>
>
>>
>> When testing on the gluster mount point using above command I hardly
>> get 10MB/s. (On the same time the network traffic hardly reaches 
>> 100Mbit).
>>
>> When testing our of the gluster (for example at /root) I get 600 -
>> 700MB/s.
>>
>
> That's very fast - from 4 disks doing RAID5? Impressive (unless you
> use caching!). Are those HDDs or SSDs/NVMe?
>
>
 These are SAS disks. But there is also a RAID controller with 1GB cache.
>>>
>>>
>> When I mount the gluster volume with NFS and test on it I get 90 -
>> 100 MB/s, (almost 10x from gluster results) which is the max I can get
>> considering I have only 1 Gbit network for the storage.
>>
>> Also, when using glusterfs the general VM performance is very poor
>> and disk write benchmarks show that is it at least 4 times slower then 
>> when
>> the VM is hosted on the same data store when NFS mounted.
>>
>> I don't know why I hitting such a significant performance penalty,
>> and every possible tweak that I was able to find out there did not make 
>> any
>> difference on the performance.
>>
>> The hardware I am using is pretty decent for the purposes intended:
>> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2
>> TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
>> store of ovirt where 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-08 Thread Abi Askushi
I don't see any other bottleneck. CPUs are quite idle. Seems that the load
is mostly due to high latency on IO.

Reading further the gluster docs:

https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.5/libgfapi%20with%20qemu%20libvirt.md

I see that I am missing the following options:

/etc/glusterfs/glusterd.vol :
option rpc-auth-allow-insecure on

gluster volume set vms gluster server.allow-insecure on

It says that the above allow qemu to use libgfapi.
When checking the VM XML, I don't see any gluster protocol at the disk
drive:


  
  
  
  222a1312-5efa-4731-8914-9a9d24dccba5
  
  




While at gluster docs it mentions the below type of disk:


   
   


   
   



Does the above indicate that ovirt/qemu is not using libgfapi but FUSE only?
This could be the reason of such slow perf.


On Thu, Sep 7, 2017 at 1:47 PM, Yaniv Kaul  wrote:

>
>
> On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi 
> wrote:
>
>>
>>
>> On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul  wrote:
>>
>>>
>>>
>>> On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul  wrote:
>>>


 On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi 
 wrote:

> For a first idea I use:
>
> dd if=/dev/zero of=testfile bs=1GB count=1
>

 This is an incorrect way to test performance, for various reasons:
 1. You are not using oflag=direct , thus not using DirectIO, but using
 cache.
 2. It's unrealistic - it is very uncommon to write large blocks of
 zeros (sometimes during FS creation or wiping). Certainly not 1GB
 3. It is a single thread of IO - again, unrealistic for VM's IO.

 I forgot to mention that I include oflag=direct in my tests. I agree
>> though that dd is not the correct way to test, hence I mentioned I just use
>> it to get a first feel. More tests are done within the VM benchmarking its
>> disk IO (with tools like IOmeter).
>>
>> I suggest using fio and such. See https://github.com/pcuzner/fio-tools
 for example.

>>> Do you have any recommended config file to use for VM workload?
>>
>
> Desktops and Servers VMs behave quite differently, so not really. But the
> 70/30 job is typically a good baseline.
>
>
>>
>>

>
> When testing on the gluster mount point using above command I hardly
> get 10MB/s. (On the same time the network traffic hardly reaches 100Mbit).
>
> When testing our of the gluster (for example at /root) I get 600 -
> 700MB/s.
>

 That's very fast - from 4 disks doing RAID5? Impressive (unless you use
 caching!). Are those HDDs or SSDs/NVMe?


>>> These are SAS disks. But there is also a RAID controller with 1GB cache.
>>
>>
> When I mount the gluster volume with NFS and test on it I get 90 - 100
> MB/s, (almost 10x from gluster results) which is the max I can get
> considering I have only 1 Gbit network for the storage.
>
> Also, when using glusterfs the general VM performance is very poor and
> disk write benchmarks show that is it at least 4 times slower then when 
> the
> VM is hosted on the same data store when NFS mounted.
>
> I don't know why I hitting such a significant performance penalty, and
> every possible tweak that I was able to find out there did not make any
> difference on the performance.
>
> The hardware I am using is pretty decent for the purposes intended:
> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2
> TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
> store of ovirt where VMs are stored.
>

>>> I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10?
>>> Same usable capacity, higher performance, same protection and faster
>>> recovery, I believe.
>>>
>> Correction: there are 5 disks of 600GB each. The main reason going with
>> RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only
>> 1.1 TB usable, with RAID 5 I get 2.2 TB usable. I agree going with RAID 10
>> (+ one additional drive to go with 6 drives) would be better but this is
>> what I have now.
>>
>> Y.
>>>
>>>
 You have not mentioned your NIC speeds. Please ensure all work well,
 with 10g.
 Is the network dedicated for Gluster traffic? How are they connected?


>>> I have mentioned that I have 1 Gbit dedicated for the storage. A
>> different network is used for this and a dedicated 1Gbit switch. The
>> throughput has been confirmed between all nodes with iperf.
>>
>
> Oh With 1Gb, you can't get more than 100+MBps...
>
>
>> I know 10Gbit would be better, but when using native gluster at ovirt the
>> network pipe was hardly reaching 100Mbps thus the bottleneck was gluster
>> and not the network. If I can saturate 1Gbit and I still have performance
>> issues then I may 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-07 Thread Yaniv Kaul
On Thu, Sep 7, 2017 at 12:52 PM, Abi Askushi 
wrote:

>
>
> On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul  wrote:
>
>>
>>
>> On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul  wrote:
>>
>>>
>>>
>>> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi 
>>> wrote:
>>>
 For a first idea I use:

 dd if=/dev/zero of=testfile bs=1GB count=1

>>>
>>> This is an incorrect way to test performance, for various reasons:
>>> 1. You are not using oflag=direct , thus not using DirectIO, but using
>>> cache.
>>> 2. It's unrealistic - it is very uncommon to write large blocks of zeros
>>> (sometimes during FS creation or wiping). Certainly not 1GB
>>> 3. It is a single thread of IO - again, unrealistic for VM's IO.
>>>
>>> I forgot to mention that I include oflag=direct in my tests. I agree
> though that dd is not the correct way to test, hence I mentioned I just use
> it to get a first feel. More tests are done within the VM benchmarking its
> disk IO (with tools like IOmeter).
>
> I suggest using fio and such. See https://github.com/pcuzner/fio-tools
>>> for example.
>>>
>> Do you have any recommended config file to use for VM workload?
>

Desktops and Servers VMs behave quite differently, so not really. But the
70/30 job is typically a good baseline.


>
>
>>>

 When testing on the gluster mount point using above command I hardly
 get 10MB/s. (On the same time the network traffic hardly reaches 100Mbit).

 When testing our of the gluster (for example at /root) I get 600 -
 700MB/s.

>>>
>>> That's very fast - from 4 disks doing RAID5? Impressive (unless you use
>>> caching!). Are those HDDs or SSDs/NVMe?
>>>
>>>
>> These are SAS disks. But there is also a RAID controller with 1GB cache.
>
>
 When I mount the gluster volume with NFS and test on it I get 90 - 100
 MB/s, (almost 10x from gluster results) which is the max I can get
 considering I have only 1 Gbit network for the storage.

 Also, when using glusterfs the general VM performance is very poor and
 disk write benchmarks show that is it at least 4 times slower then when the
 VM is hosted on the same data store when NFS mounted.

 I don't know why I hitting such a significant performance penalty, and
 every possible tweak that I was able to find out there did not make any
 difference on the performance.

 The hardware I am using is pretty decent for the purposes intended:
 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2
 TB of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
 store of ovirt where VMs are stored.

>>>
>> I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10?
>> Same usable capacity, higher performance, same protection and faster
>> recovery, I believe.
>>
> Correction: there are 5 disks of 600GB each. The main reason going with
> RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only
> 1.1 TB usable, with RAID 5 I get 2.2 TB usable. I agree going with RAID 10
> (+ one additional drive to go with 6 drives) would be better but this is
> what I have now.
>
> Y.
>>
>>
>>> You have not mentioned your NIC speeds. Please ensure all work well,
>>> with 10g.
>>> Is the network dedicated for Gluster traffic? How are they connected?
>>>
>>>
>> I have mentioned that I have 1 Gbit dedicated for the storage. A
> different network is used for this and a dedicated 1Gbit switch. The
> throughput has been confirmed between all nodes with iperf.
>

Oh With 1Gb, you can't get more than 100+MBps...


> I know 10Gbit would be better, but when using native gluster at ovirt the
> network pipe was hardly reaching 100Mbps thus the bottleneck was gluster
> and not the network. If I can saturate 1Gbit and I still have performance
> issues then I may think to go with 10Gbit. With NFS on top gluster I see
> traffic reaching 800Mbit when testing with dd which is much better.
>

Agreed. Do you see the bottleneck elsewhere? CPU?


>
>
 The gluster configuration is the following:

>>>
>>> Which version of Gluster are you using?
>>>
>>>
>> The version is  3.8.12
>

I think it's a very old release (near end of life?). I warmly suggest
3.10.x or 3.12.
There are performance improvements (AFAIR) in both.
Y.


>
 Volume Name: vms
 Type: Replicate
 Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 1 x (2 + 1) = 3
 Transport-type: tcp
 Bricks:
 Brick1: gluster0:/gluster/vms/brick
 Brick2: gluster1:/gluster/vms/brick
 Brick3: gluster2:/gluster/vms/brick (arbiter)
 Options Reconfigured:
 nfs.export-volumes: on
 nfs.disable: off
 performance.readdir-ahead: on
 transport.address-family: inet
 performance.quick-read: off
 performance.read-ahead: off
 performance.io-cache: off
 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-07 Thread Abi Askushi
On Thu, Sep 7, 2017 at 10:30 AM, Yaniv Kaul  wrote:

>
>
> On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul  wrote:
>
>>
>>
>> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi 
>> wrote:
>>
>>> For a first idea I use:
>>>
>>> dd if=/dev/zero of=testfile bs=1GB count=1
>>>
>>
>> This is an incorrect way to test performance, for various reasons:
>> 1. You are not using oflag=direct , thus not using DirectIO, but using
>> cache.
>> 2. It's unrealistic - it is very uncommon to write large blocks of zeros
>> (sometimes during FS creation or wiping). Certainly not 1GB
>> 3. It is a single thread of IO - again, unrealistic for VM's IO.
>>
>> I forgot to mention that I include oflag=direct in my tests. I agree
though that dd is not the correct way to test, hence I mentioned I just use
it to get a first feel. More tests are done within the VM benchmarking its
disk IO (with tools like IOmeter).

I suggest using fio and such. See https://github.com/pcuzner/fio-tools for
>> example.
>>
> Do you have any recommended config file to use for VM workload?


>>
>>>
>>> When testing on the gluster mount point using above command I hardly get
>>> 10MB/s. (On the same time the network traffic hardly reaches 100Mbit).
>>>
>>> When testing our of the gluster (for example at /root) I get 600 -
>>> 700MB/s.
>>>
>>
>> That's very fast - from 4 disks doing RAID5? Impressive (unless you use
>> caching!). Are those HDDs or SSDs/NVMe?
>>
>>
> These are SAS disks. But there is also a RAID controller with 1GB cache.


>>> When I mount the gluster volume with NFS and test on it I get 90 - 100
>>> MB/s, (almost 10x from gluster results) which is the max I can get
>>> considering I have only 1 Gbit network for the storage.
>>>
>>> Also, when using glusterfs the general VM performance is very poor and
>>> disk write benchmarks show that is it at least 4 times slower then when the
>>> VM is hosted on the same data store when NFS mounted.
>>>
>>> I don't know why I hitting such a significant performance penalty, and
>>> every possible tweak that I was able to find out there did not make any
>>> difference on the performance.
>>>
>>> The hardware I am using is pretty decent for the purposes intended:
>>> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB
>>> of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
>>> store of ovirt where VMs are stored.
>>>
>>
> I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10?
> Same usable capacity, higher performance, same protection and faster
> recovery, I believe.
>
Correction: there are 5 disks of 600GB each. The main reason going with
RAID 5 was the capacity. With RAID 10 I can use only 4 of them and get only
1.1 TB usable, with RAID 5 I get 2.2 TB usable. I agree going with RAID 10
(+ one additional drive to go with 6 drives) would be better but this is
what I have now.

Y.
>
>
>> You have not mentioned your NIC speeds. Please ensure all work well, with
>> 10g.
>> Is the network dedicated for Gluster traffic? How are they connected?
>>
>>
> I have mentioned that I have 1 Gbit dedicated for the storage. A different
network is used for this and a dedicated 1Gbit switch. The throughput has
been confirmed between all nodes with iperf.
I know 10Gbit would be better, but when using native gluster at ovirt the
network pipe was hardly reaching 100Mbps thus the bottleneck was gluster
and not the network. If I can saturate 1Gbit and I still have performance
issues then I may think to go with 10Gbit. With NFS on top gluster I see
traffic reaching 800Mbit when testing with dd which is much better.


>>> The gluster configuration is the following:
>>>
>>
>> Which version of Gluster are you using?
>>
>>
> The version is  3.8.12

>
>>> Volume Name: vms
>>> Type: Replicate
>>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gluster0:/gluster/vms/brick
>>> Brick2: gluster1:/gluster/vms/brick
>>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>>> Options Reconfigured:
>>> nfs.export-volumes: on
>>> nfs.disable: off
>>> performance.readdir-ahead: on
>>> transport.address-family: inet
>>> performance.quick-read: off
>>> performance.read-ahead: off
>>> performance.io-cache: off
>>> performance.stat-prefetch: on
>>>
>>
>> I think this should be off.
>>
>>
>>> performance.low-prio-threads: 32
>>> network.remote-dio: off
>>>
>>
>> I think this should be enabled.
>>
>>
>>> cluster.eager-lock: off
>>> cluster.quorum-type: auto
>>> cluster.server-quorum-type: server
>>> cluster.data-self-heal-algorithm: full
>>> cluster.locking-scheme: granular
>>> cluster.shd-max-threads: 8
>>> cluster.shd-wait-qlength: 1
>>> features.shard: on
>>> user.cifs: off
>>> storage.owner-uid: 36
>>> storage.owner-gid: 36
>>> network.ping-timeout: 30
>>> performance.strict-o-direct: on
>>> 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-07 Thread Yaniv Kaul
On Thu, Sep 7, 2017 at 10:06 AM, Yaniv Kaul  wrote:

>
>
> On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi 
> wrote:
>
>> For a first idea I use:
>>
>> dd if=/dev/zero of=testfile bs=1GB count=1
>>
>
> This is an incorrect way to test performance, for various reasons:
> 1. You are not using oflag=direct , thus not using DirectIO, but using
> cache.
> 2. It's unrealistic - it is very uncommon to write large blocks of zeros
> (sometimes during FS creation or wiping). Certainly not 1GB
> 3. It is a single thread of IO - again, unrealistic for VM's IO.
>
> I suggest using fio and such. See https://github.com/pcuzner/fio-tools
> for example.
>
>
>>
>> When testing on the gluster mount point using above command I hardly get
>> 10MB/s. (On the same time the network traffic hardly reaches 100Mbit).
>>
>> When testing our of the gluster (for example at /root) I get 600 -
>> 700MB/s.
>>
>
> That's very fast - from 4 disks doing RAID5? Impressive (unless you use
> caching!). Are those HDDs or SSDs/NVMe?
>
>
>>
>> When I mount the gluster volume with NFS and test on it I get 90 - 100
>> MB/s, (almost 10x from gluster results) which is the max I can get
>> considering I have only 1 Gbit network for the storage.
>>
>> Also, when using glusterfs the general VM performance is very poor and
>> disk write benchmarks show that is it at least 4 times slower then when the
>> VM is hosted on the same data store when NFS mounted.
>>
>> I don't know why I hitting such a significant performance penalty, and
>> every possible tweak that I was able to find out there did not make any
>> difference on the performance.
>>
>> The hardware I am using is pretty decent for the purposes intended:
>> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB
>> of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
>> store of ovirt where VMs are stored.
>>
>
I forgot to ask why are you using RAID 5 with 4 disks and not RAID 10? Same
usable capacity, higher performance, same protection and faster recovery, I
believe.
Y.


> You have not mentioned your NIC speeds. Please ensure all work well, with
> 10g.
> Is the network dedicated for Gluster traffic? How are they connected?
>
>
>>
>> The gluster configuration is the following:
>>
>
> Which version of Gluster are you using?
>
>
>>
>> Volume Name: vms
>> Type: Replicate
>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster0:/gluster/vms/brick
>> Brick2: gluster1:/gluster/vms/brick
>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>> Options Reconfigured:
>> nfs.export-volumes: on
>> nfs.disable: off
>> performance.readdir-ahead: on
>> transport.address-family: inet
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: on
>>
>
> I think this should be off.
>
>
>> performance.low-prio-threads: 32
>> network.remote-dio: off
>>
>
> I think this should be enabled.
>
>
>> cluster.eager-lock: off
>> cluster.quorum-type: auto
>> cluster.server-quorum-type: server
>> cluster.data-self-heal-algorithm: full
>> cluster.locking-scheme: granular
>> cluster.shd-max-threads: 8
>> cluster.shd-wait-qlength: 1
>> features.shard: on
>> user.cifs: off
>> storage.owner-uid: 36
>> storage.owner-gid: 36
>> network.ping-timeout: 30
>> performance.strict-o-direct: on
>> cluster.granular-entry-heal: enable
>> features.shard-block-size: 64MB
>>
>
> I'm not sure if this should not be 512MB.  I don't remember the last
> resolution on this.
> Y.
>
>
>> performance.client-io-threads: on
>> client.event-threads: 4
>> server.event-threads: 4
>> performance.write-behind-window-size: 4MB
>> performance.cache-size: 1GB
>>
>> In case I can provide any other details let me know.
>>
>
> What is your tuned profile?
>
>
>> At the moment I already switched to gluster based NFS but I have a
>> similar setup with 2 nodes  where the data store is mounted through gluster
>> (and again relatively good hardware) where I might check any tweaks or
>> improvements on this setup.
>>
>> Thanx
>>
>>
>> On Wed, Sep 6, 2017 at 5:32 PM, Yaniv Kaul  wrote:
>>
>>>
>>>
>>> On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi 
>>> wrote:
>>>
 Hi All,

 I've playing with ovirt self hosted engine setup and I even use it to
 production for several VM. The setup I have is 3 server with gluster
 storage in replica 2+1 (1 arbiter).
 The data storage domain where VMs are stored is mounted with gluster
 through ovirt. The performance I get for the VMs is very low and I was
 thinking to switch and mount the same storage through NFS instead of
 glusterfs.

>>>
>>> I don't see how it'll improve performance.
>>> I suggest you share the gluster configuration (as well as the storage
>>> HW) so we can understand why the performance is 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-07 Thread Yaniv Kaul
On Wed, Sep 6, 2017 at 6:08 PM, Abi Askushi  wrote:

> For a first idea I use:
>
> dd if=/dev/zero of=testfile bs=1GB count=1
>

This is an incorrect way to test performance, for various reasons:
1. You are not using oflag=direct , thus not using DirectIO, but using
cache.
2. It's unrealistic - it is very uncommon to write large blocks of zeros
(sometimes during FS creation or wiping). Certainly not 1GB
3. It is a single thread of IO - again, unrealistic for VM's IO.

I suggest using fio and such. See https://github.com/pcuzner/fio-tools for
example.


>
> When testing on the gluster mount point using above command I hardly get
> 10MB/s. (On the same time the network traffic hardly reaches 100Mbit).
>
> When testing our of the gluster (for example at /root) I get 600 -
> 700MB/s.
>

That's very fast - from 4 disks doing RAID5? Impressive (unless you use
caching!). Are those HDDs or SSDs/NVMe?


>
> When I mount the gluster volume with NFS and test on it I get 90 - 100
> MB/s, (almost 10x from gluster results) which is the max I can get
> considering I have only 1 Gbit network for the storage.
>
> Also, when using glusterfs the general VM performance is very poor and
> disk write benchmarks show that is it at least 4 times slower then when the
> VM is hosted on the same data store when NFS mounted.
>
> I don't know why I hitting such a significant performance penalty, and
> every possible tweak that I was able to find out there did not make any
> difference on the performance.
>
> The hardware I am using is pretty decent for the purposes intended:
> 3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB
> of storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data
> store of ovirt where VMs are stored.
>

You have not mentioned your NIC speeds. Please ensure all work well, with
10g.
Is the network dedicated for Gluster traffic? How are they connected?


>
> The gluster configuration is the following:
>

Which version of Gluster are you using?


>
> Volume Name: vms
> Type: Replicate
> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster0:/gluster/vms/brick
> Brick2: gluster1:/gluster/vms/brick
> Brick3: gluster2:/gluster/vms/brick (arbiter)
> Options Reconfigured:
> nfs.export-volumes: on
> nfs.disable: off
> performance.readdir-ahead: on
> transport.address-family: inet
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.stat-prefetch: on
>

I think this should be off.


> performance.low-prio-threads: 32
> network.remote-dio: off
>

I think this should be enabled.


> cluster.eager-lock: off
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 1
> features.shard: on
> user.cifs: off
> storage.owner-uid: 36
> storage.owner-gid: 36
> network.ping-timeout: 30
> performance.strict-o-direct: on
> cluster.granular-entry-heal: enable
> features.shard-block-size: 64MB
>

I'm not sure if this should not be 512MB.  I don't remember the last
resolution on this.
Y.


> performance.client-io-threads: on
> client.event-threads: 4
> server.event-threads: 4
> performance.write-behind-window-size: 4MB
> performance.cache-size: 1GB
>
> In case I can provide any other details let me know.
>

What is your tuned profile?


> At the moment I already switched to gluster based NFS but I have a similar
> setup with 2 nodes  where the data store is mounted through gluster (and
> again relatively good hardware) where I might check any tweaks or
> improvements on this setup.
>
> Thanx
>
>
> On Wed, Sep 6, 2017 at 5:32 PM, Yaniv Kaul  wrote:
>
>>
>>
>> On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi 
>> wrote:
>>
>>> Hi All,
>>>
>>> I've playing with ovirt self hosted engine setup and I even use it to
>>> production for several VM. The setup I have is 3 server with gluster
>>> storage in replica 2+1 (1 arbiter).
>>> The data storage domain where VMs are stored is mounted with gluster
>>> through ovirt. The performance I get for the VMs is very low and I was
>>> thinking to switch and mount the same storage through NFS instead of
>>> glusterfs.
>>>
>>
>> I don't see how it'll improve performance.
>> I suggest you share the gluster configuration (as well as the storage HW)
>> so we can understand why the performance is low.
>> Y.
>>
>>
>>>
>>> The only think I am hesitant is how can I ensure high availability of
>>> the storage when I loose one server? I was thinking to have at /etc/hosts
>>> sth like below:
>>>
>>> 10.100.100.1 nfsmount
>>> 10.100.100.2 nfsmount
>>> 10.100.100.3 nfsmount
>>>
>>> then use nfsmount as the server name when adding this domain through
>>> ovirt GUI.
>>> Are there any other more elegant solutions? What 

Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-06 Thread Abi Askushi
For a first idea I use:

dd if=/dev/zero of=testfile bs=1GB count=1

When testing on the gluster mount point using above command I hardly get
10MB/s. (On the same time the network traffic hardly reaches 100Mbit).

When testing our of the gluster (for example at /root) I get 600 - 700MB/s.

When I mount the gluster volume with NFS and test on it I get 90 - 100
MB/s, (almost 10x from gluster results) which is the max I can get
considering I have only 1 Gbit network for the storage.

Also, when using glusterfs the general VM performance is very poor and disk
write benchmarks show that is it at least 4 times slower then when the VM
is hosted on the same data store when NFS mounted.

I don't know why I hitting such a significant performance penalty, and
every possible tweak that I was able to find out there did not make any
difference on the performance.

The hardware I am using is pretty decent for the purposes intended:
3 nodes, each node having with 32 MB of RAM, 16 physical CPU cores, 2 TB of
storage in RAID5 (4 disks), of which 1.5 TB are sliced for the data store
of ovirt where VMs are stored.

The gluster configuration is the following:

Volume Name: vms
Type: Replicate
Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: gluster0:/gluster/vms/brick
Brick2: gluster1:/gluster/vms/brick
Brick3: gluster2:/gluster/vms/brick (arbiter)
Options Reconfigured:
nfs.export-volumes: on
nfs.disable: off
performance.readdir-ahead: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: on
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
network.ping-timeout: 30
performance.strict-o-direct: on
cluster.granular-entry-heal: enable
features.shard-block-size: 64MB
performance.client-io-threads: on
client.event-threads: 4
server.event-threads: 4
performance.write-behind-window-size: 4MB
performance.cache-size: 1GB

In case I can provide any other details let me know.
At the moment I already switched to gluster based NFS but I have a similar
setup with 2 nodes  where the data store is mounted through gluster (and
again relatively good hardware) where I might check any tweaks or
improvements on this setup.

Thanx


On Wed, Sep 6, 2017 at 5:32 PM, Yaniv Kaul  wrote:

>
>
> On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi 
> wrote:
>
>> Hi All,
>>
>> I've playing with ovirt self hosted engine setup and I even use it to
>> production for several VM. The setup I have is 3 server with gluster
>> storage in replica 2+1 (1 arbiter).
>> The data storage domain where VMs are stored is mounted with gluster
>> through ovirt. The performance I get for the VMs is very low and I was
>> thinking to switch and mount the same storage through NFS instead of
>> glusterfs.
>>
>
> I don't see how it'll improve performance.
> I suggest you share the gluster configuration (as well as the storage HW)
> so we can understand why the performance is low.
> Y.
>
>
>>
>> The only think I am hesitant is how can I ensure high availability of the
>> storage when I loose one server? I was thinking to have at /etc/hosts sth
>> like below:
>>
>> 10.100.100.1 nfsmount
>> 10.100.100.2 nfsmount
>> 10.100.100.3 nfsmount
>>
>> then use nfsmount as the server name when adding this domain through
>> ovirt GUI.
>> Are there any other more elegant solutions? What do you do for such cases?
>> Note: gluster has the back-vol-file option which provides a lean way to
>> have redundancy on the mount point and I am using this when mounting with
>> glusterfs.
>>
>> Thanx
>>
>> ___
>> Users mailing list
>> Users@ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users


Re: [ovirt-users] oVirt selft-hosted with NFS on top gluster

2017-09-06 Thread Yaniv Kaul
On Wed, Sep 6, 2017 at 3:32 PM, Abi Askushi  wrote:

> Hi All,
>
> I've playing with ovirt self hosted engine setup and I even use it to
> production for several VM. The setup I have is 3 server with gluster
> storage in replica 2+1 (1 arbiter).
> The data storage domain where VMs are stored is mounted with gluster
> through ovirt. The performance I get for the VMs is very low and I was
> thinking to switch and mount the same storage through NFS instead of
> glusterfs.
>

I don't see how it'll improve performance.
I suggest you share the gluster configuration (as well as the storage HW)
so we can understand why the performance is low.
Y.


>
> The only think I am hesitant is how can I ensure high availability of the
> storage when I loose one server? I was thinking to have at /etc/hosts sth
> like below:
>
> 10.100.100.1 nfsmount
> 10.100.100.2 nfsmount
> 10.100.100.3 nfsmount
>
> then use nfsmount as the server name when adding this domain through ovirt
> GUI.
> Are there any other more elegant solutions? What do you do for such cases?
> Note: gluster has the back-vol-file option which provides a lean way to
> have redundancy on the mount point and I am using this when mounting with
> glusterfs.
>
> Thanx
>
> ___
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
___
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users