[ovirt-users] Re: Gluster Performance issues

2022-02-25 Thread Sunil Kumar Heggodu Gopala Acharya 
Regards,
Sunil


On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins  wrote:

> Have you verified that you're actually getting 10Gbps between the hosts?
>
> -derek
>
> On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
> > Hello Derek,
> >
> > We have a 10Gig connection dedicated to the storage network, nothing else
> > is on that switch.
> >
> > On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins  wrote:
> >
> >> Hi,
> >>
> >> Another question which I don't see answered:   What is the underlying
> >> connectivity between the Gluster hosts?
> >>
> >> -derek
> >>
> >> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
> >> > Hello Sunil,
> >> >
> >> > [root@ovirt1 ~]# gluster --version
> >> > glusterfs 8.6
> >> >
> >> > same on all hosts
>
Latest Release-10.1(
https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html)
has some performance fixes which should help in this situation compared to
the older gluster bits.

> >> >
> >> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
> >> > shegg...@redhat.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Which version of gluster is in use?
> >> >>
> >> >> Regards,
> >> >>
> >> >> Sunil kumar Acharya
> >> >>
> >> >> Red Hat
> >> >>
> >> >> 
> >> >>
> >> >> T: +91-8067935170
> >> >> 
> >> >>
> >> >> 
> >> >> TRIED. TESTED. TRUSTED. 
> >> >>
> >> >>
> >> >>
> >> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
> >> 
> >> >> wrote:
> >> >>
> >> >>> Hello All,
> >> >>>
> >> >>> We have 3 servers with a raid 50 array each, we are having extreme
> >> >>> performance issues with our gluster, writes on gluster seem to take
> >> at
> >> >>> least 3 times longer than on the raid directly. Can this be
> >> improved?
> >> >>> I've
> >> >>> read through several other performance issues threads but have been
> >> >>> unable
> >> >>> to make any improvements
> >> >>>
> >> >>> "gluster volume info" and "gluster volume profile vmstore info" is
> >> >>> below
> >> >>>
> >> >>>
> >> >>>
> >>
> =
> >> >>>
> >> >>> -Inside Gluster - test took 35+ hours:
> >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d .
> -s
> >> >>> 600G -n 0 -m TEST -f -b -u root
> >> >>> Using uid:0, gid:0.
> >> >>> Writing intelligently...done
> >> >>> Rewriting...done
> >> >>> Reading intelligently...done
> >> >>> start 'em...done...done...done...done...done...
> >> >>> Version  1.98   --Sequential Output-- --Sequential
> >> Input-
> >> >>> --Random-
> >> >>> -Per Chr- --Block-- -Rewrite- -Per Chr-
> >> --Block--
> >> >>> --Seeks--
> >> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> >> %CP
> >> >>>  /sec %CP
> >> >>> TEST   600G   35.7m  17 5824k   7112m
> >> 13
> >> >>> 182.7   6
> >> >>> Latency5466ms   12754ms  3499ms
> >> >>>  1589ms
> >> >>>
> >> >>>
> >> >>>
> >>
> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
> >> >>>
> >> >>>
> >> >>>
> >>
> =
> >> >>>
> >> >>> -Outside Gluster - test took 18 minutes:
> >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d .
> -s
> >> >>> 600G -n 0 -m TEST -f -b -u root
> >> >>> Using uid:0, gid:0.
> >> >>> Writing intelligently...done
> >> >>> Rewriting...done
> >> >>> Reading intelligently...done
> >> >>> start 'em...done...done...done...done...done...
> >> >>> Version  1.98   --Sequential Output-- --Sequential
> >> Input-
> >> >>> --Random-
> >> >>> -Per Chr- --Block-- -Rewrite- -Per Chr-
> >> --Block--
> >> >>> --Seeks--
> >> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
> >> %CP
> >> >>>  /sec %CP
> >> >>> TEST   600G567m  78  149m  30307m
> >> 37
> >> >>>  83.0  57
> >> >>> Latency 205ms4630ms  1450ms
> >> >>> 679ms
> >> >>>
> >> >>>
> >> >>>
> >>
> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,,
> >> >>>
> >> >>>
> >> >>>
> >>
> =
> >> >>>
> >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
> >> info
> >> >>> Volume Name: engine
> >> >>> Type: Replicate
> >> >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
> >> >>> Status: Started
> >> >>> Snapshot Count: 0
> >> >>> Number of Bricks: 1 x 3 = 3
> >> >>> Transport-type: tcp
> >> >>> Bricks:
> >> >>> Brick1:

[ovirt-users] Re: Gluster Performance issues

2022-02-24 Thread Dhanaraj Ramesh via Users 
Since RAID config add additional io penalties, the RAID should be removed and 
have to add each of the disks directly to GLUSTER  configs.

We had similar issues with RAID, after eliminating that we are good with 
performances

Get Outlook for iOS<https://aka.ms/o0ukef>

From: Strahil Nikolov via Users 
Sent: Thursday, February 24, 2022 3:43:39 AM
To: Alex Morrison ; Sunil Kumar Heggodu Gopala Acharya 

Cc: users@ovirt.org 
Subject: [ovirt-users] Re: Gluster Performance issues

You can try to play a little bit with the I/O threads (but don't jump too fast).

What is your I/O scheduler and mount options.
You can reduce I/O lookups if you specify the 'noatime' and the selinux context 
on the mount options.

A real killer of performance is the lattency. What is the lattency between all 
nodes ?


Best Regards,
Strahil Nikolov

On Wed, Feb 23, 2022 at 20:26, Alex Morrison
 wrote:
Hello All,

I believe the network is performing as expected, I did an iperf test:

[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2
Connecting to host 10.10.1.2, port 5201
[  5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201
[ ID] Interval   Transfer Bitrate Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.24 Gbits/sec0   2.96 MBytes
[  5]   1.00-2.00   sec  1.03 GBytes  8.81 Gbits/sec0   2.96 MBytes
[  5]   2.00-3.00   sec  1006 MBytes  8.44 Gbits/sec  101   1.45 MBytes
[  5]   3.00-4.00   sec  1.04 GBytes  8.92 Gbits/sec5901 KBytes
[  5]   4.00-5.00   sec  1.05 GBytes  9.01 Gbits/sec0957 KBytes
[  5]   5.00-6.00   sec  1.08 GBytes  9.23 Gbits/sec0990 KBytes
[  5]   6.00-7.00   sec  1008 MBytes  8.46 Gbits/sec  159655 KBytes
[  5]   7.00-8.00   sec  1.06 GBytes  9.11 Gbits/sec0970 KBytes
[  5]   8.00-9.00   sec  1.03 GBytes  8.85 Gbits/sec2829 KBytes
[  5]   9.00-10.00  sec  1.04 GBytes  8.96 Gbits/sec0947 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.4 GBytes  8.90 Gbits/sec  267 sender
[  5]   0.00-10.04  sec  10.4 GBytes  8.87 Gbits/sec  receiver

iperf Done.

On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya 
mailto:shegg...@redhat.com>> wrote:

Regards,
Sunil


On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins 
mailto:de...@ihtfp.com>> wrote:
Have you verified that you're actually getting 10Gbps between the hosts?

-derek

On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
> Hello Derek,
>
> We have a 10Gig connection dedicated to the storage network, nothing else
> is on that switch.
>
> On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins 
> mailto:de...@ihtfp.com>> wrote:
>
>> Hi,
>>
>> Another question which I don't see answered:   What is the underlying
>> connectivity between the Gluster hosts?
>>
>> -derek
>>
>> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
>> > Hello Sunil,
>> >
>> > [root@ovirt1 ~]# gluster --version
>> > glusterfs 8.6
>> >
>> > same on all hosts
Latest 
Release-10.1(https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html)
 has some performance fixes which should help in this situation compared to the 
older gluster bits.
>> >
>> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
>> > shegg...@redhat.com<mailto:shegg...@redhat.com>> wrote:
>> >
>> >> Hi,
>> >>
>> >> Which version of gluster is in use?
>> >>
>> >> Regards,
>> >>
>> >> Sunil kumar Acharya
>> >>
>> >> Red Hat
>> >>
>> >> <https://www.redhat.com>
>> >>
>> >> T: +91-8067935170
>> >> <http://redhatemailsignature-marketing.itos.redhat.com/>
>> >>
>> >> <https://red.ht/sig>
>> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> >>
>> >>
>> >>
>> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
>> mailto:a...@discoverygarden.ca>>
>> >> wrote:
>> >>
>> >>> Hello All,
>> >>>
>> >>> We have 3 servers with a raid 50 array each, we are having extreme
>> >>> performance issues with our gluster, writes on gluster seem to take
>> at
>> >>> least 3 times longer than on the raid directly. Can this be
>> improved?
>> >>> I've
>> >>> read through several other performance issues threads but have been
>> >>> unable
>> >>> to make any improvements
>> >>>
&g

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Strahil Nikolov via Users 
You can try to play a little bit with the I/O threads (but don't jump too fast).
What is your I/O scheduler and mount options.You can reduce I/O lookups if you 
specify the 'noatime' and the selinux context on the mount options.
A real killer of performance is the lattency. What is the lattency between all 
nodes ?

Best Regards,Strahil Nikolov
 
 
  On Wed, Feb 23, 2022 at 20:26, Alex Morrison wrote:  
 Hello All,
I believe the network is performing as expected, I did an iperf test:

[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2
Connecting to host 10.10.1.2, port 5201
[  5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.24 Gbits/sec    0   2.96 MBytes
[  5]   1.00-2.00   sec  1.03 GBytes  8.81 Gbits/sec    0   2.96 MBytes
[  5]   2.00-3.00   sec  1006 MBytes  8.44 Gbits/sec  101   1.45 MBytes
[  5]   3.00-4.00   sec  1.04 GBytes  8.92 Gbits/sec    5    901 KBytes
[  5]   4.00-5.00   sec  1.05 GBytes  9.01 Gbits/sec    0    957 KBytes
[  5]   5.00-6.00   sec  1.08 GBytes  9.23 Gbits/sec    0    990 KBytes
[  5]   6.00-7.00   sec  1008 MBytes  8.46 Gbits/sec  159    655 KBytes
[  5]   7.00-8.00   sec  1.06 GBytes  9.11 Gbits/sec    0    970 KBytes
[  5]   8.00-9.00   sec  1.03 GBytes  8.85 Gbits/sec    2    829 KBytes
[  5]   9.00-10.00  sec  1.04 GBytes  8.96 Gbits/sec    0    947 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.4 GBytes  8.90 Gbits/sec  267             sender
[  5]   0.00-10.04  sec  10.4 GBytes  8.87 Gbits/sec                  receiver

iperf Done.

On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya 
 wrote:


Regards,
Sunil


On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins  wrote:

Have you verified that you're actually getting 10Gbps between the hosts?

-derek

On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
> Hello Derek,
>
> We have a 10Gig connection dedicated to the storage network, nothing else
> is on that switch.
>
> On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins  wrote:
>
>> Hi,
>>
>> Another question which I don't see answered:   What is the underlying
>> connectivity between the Gluster hosts?
>>
>> -derek
>>
>> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
>> > Hello Sunil,
>> >
>> > [root@ovirt1 ~]# gluster --version
>> > glusterfs 8.6
>> >
>> > same on all hosts

Latest 
Release-10.1(https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html)
 has some performance fixes which should help in this situation compared to the 
older gluster bits. 
>> >
>> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
>> > shegg...@redhat.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Which version of gluster is in use?
>> >>
>> >> Regards,
>> >>
>> >> Sunil kumar Acharya
>> >>
>> >> Red Hat
>> >>
>> >> 
>> >>
>> >> T: +91-8067935170
>> >> 
>> >>
>> >> 
>> >> TRIED. TESTED. TRUSTED. 
>> >>
>> >>
>> >>
>> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
>> 
>> >> wrote:
>> >>
>> >>> Hello All,
>> >>>
>> >>> We have 3 servers with a raid 50 array each, we are having extreme
>> >>> performance issues with our gluster, writes on gluster seem to take
>> at
>> >>> least 3 times longer than on the raid directly. Can this be
>> improved?
>> >>> I've
>> >>> read through several other performance issues threads but have been
>> >>> unable
>> >>> to make any improvements
>> >>>
>> >>> "gluster volume info" and "gluster volume profile vmstore info" is
>> >>> below
>> >>>
>> >>>
>> >>>
>> =
>> >>>
>> >>> -Inside Gluster - test took 35+ hours:
>> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>> >>> 600G -n 0 -m TEST -f -b -u root
>> >>> Using uid:0, gid:0.
>> >>> Writing intelligently...done
>> >>> Rewriting...done
>> >>> Reading intelligently...done
>> >>> start 'em...done...done...done...done...done...
>> >>> Version  1.98       --Sequential Output-- --Sequential
>> Input-
>> >>> --Random-
>> >>>                     -Per Chr- --Block-- -Rewrite- -Per Chr-
>> --Block--
>> >>> --Seeks--
>> >>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
>> %CP
>> >>>  /sec %CP
>> >>> TEST           600G           35.7m  17 5824k   7            112m
>> 13
>> >>> 182.7   6
>> >>> Latency                        5466ms   12754ms              3499ms
>> >>>  1589ms
>> >>>
>> >>>
>> >>>
>> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
>> >>>
>> >>>
>> >>>
>>

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Alex Morrison 
Hello All,

I believe the network is performing as expected, I did an iperf test:

[root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2
Connecting to host 10.10.1.2, port 5201
[  5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201
[ ID] Interval   Transfer Bitrate Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.24 Gbits/sec0   2.96 MBytes
[  5]   1.00-2.00   sec  1.03 GBytes  8.81 Gbits/sec0   2.96 MBytes
[  5]   2.00-3.00   sec  1006 MBytes  8.44 Gbits/sec  101   1.45 MBytes
[  5]   3.00-4.00   sec  1.04 GBytes  8.92 Gbits/sec5901 KBytes
[  5]   4.00-5.00   sec  1.05 GBytes  9.01 Gbits/sec0957 KBytes
[  5]   5.00-6.00   sec  1.08 GBytes  9.23 Gbits/sec0990 KBytes
[  5]   6.00-7.00   sec  1008 MBytes  8.46 Gbits/sec  159655 KBytes
[  5]   7.00-8.00   sec  1.06 GBytes  9.11 Gbits/sec0970 KBytes
[  5]   8.00-9.00   sec  1.03 GBytes  8.85 Gbits/sec2829 KBytes
[  5]   9.00-10.00  sec  1.04 GBytes  8.96 Gbits/sec0947 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.4 GBytes  8.90 Gbits/sec  267 sender
[  5]   0.00-10.04  sec  10.4 GBytes  8.87 Gbits/sec
 receiver

iperf Done.

On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya <
shegg...@redhat.com> wrote:

>
> Regards,
> Sunil
>
>
> On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins  wrote:
>
>> Have you verified that you're actually getting 10Gbps between the hosts?
>>
>> -derek
>>
>> On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
>> > Hello Derek,
>> >
>> > We have a 10Gig connection dedicated to the storage network, nothing
>> else
>> > is on that switch.
>> >
>> > On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins  wrote:
>> >
>> >> Hi,
>> >>
>> >> Another question which I don't see answered:   What is the underlying
>> >> connectivity between the Gluster hosts?
>> >>
>> >> -derek
>> >>
>> >> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
>> >> > Hello Sunil,
>> >> >
>> >> > [root@ovirt1 ~]# gluster --version
>> >> > glusterfs 8.6
>> >> >
>> >> > same on all hosts
>>
> Latest Release-10.1(
> https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html)
> has some performance fixes which should help in this situation compared to
> the older gluster bits.
>
>> >> >
>> >> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
>> >> > shegg...@redhat.com> wrote:
>> >> >
>> >> >> Hi,
>> >> >>
>> >> >> Which version of gluster is in use?
>> >> >>
>> >> >> Regards,
>> >> >>
>> >> >> Sunil kumar Acharya
>> >> >>
>> >> >> Red Hat
>> >> >>
>> >> >> 
>> >> >>
>> >> >> T: +91-8067935170
>> >> >> 
>> >> >>
>> >> >> 
>> >> >> TRIED. TESTED. TRUSTED. 
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
>> >> 
>> >> >> wrote:
>> >> >>
>> >> >>> Hello All,
>> >> >>>
>> >> >>> We have 3 servers with a raid 50 array each, we are having extreme
>> >> >>> performance issues with our gluster, writes on gluster seem to take
>> >> at
>> >> >>> least 3 times longer than on the raid directly. Can this be
>> >> improved?
>> >> >>> I've
>> >> >>> read through several other performance issues threads but have been
>> >> >>> unable
>> >> >>> to make any improvements
>> >> >>>
>> >> >>> "gluster volume info" and "gluster volume profile vmstore info" is
>> >> >>> below
>> >> >>>
>> >> >>>
>> >> >>>
>> >>
>> =
>> >> >>>
>> >> >>> -Inside Gluster - test took 35+ hours:
>> >> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d .
>> -s
>> >> >>> 600G -n 0 -m TEST -f -b -u root
>> >> >>> Using uid:0, gid:0.
>> >> >>> Writing intelligently...done
>> >> >>> Rewriting...done
>> >> >>> Reading intelligently...done
>> >> >>> start 'em...done...done...done...done...done...
>> >> >>> Version  1.98   --Sequential Output-- --Sequential
>> >> Input-
>> >> >>> --Random-
>> >> >>> -Per Chr- --Block-- -Rewrite- -Per Chr-
>> >> --Block--
>> >> >>> --Seeks--
>> >> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
>> >> %CP
>> >> >>>  /sec %CP
>> >> >>> TEST   600G   35.7m  17 5824k   7112m
>> >> 13
>> >> >>> 182.7   6
>> >> >>> Latency5466ms   12754ms  3499ms
>> >> >>>  1589ms
>> >> >>>
>> >> >>>
>> >> >>>
>> >>
>> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
>> >> >>>
>> >> >>>
>> >> >>>
>> >>
>> =
>> >> >>>
>> >> >>> -Outside

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Derek Atkins 
Have you verified that you're actually getting 10Gbps between the hosts?

-derek

On Wed, February 23, 2022 9:02 am, Alex Morrison wrote:
> Hello Derek,
>
> We have a 10Gig connection dedicated to the storage network, nothing else
> is on that switch.
>
> On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins  wrote:
>
>> Hi,
>>
>> Another question which I don't see answered:   What is the underlying
>> connectivity between the Gluster hosts?
>>
>> -derek
>>
>> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
>> > Hello Sunil,
>> >
>> > [root@ovirt1 ~]# gluster --version
>> > glusterfs 8.6
>> >
>> > same on all hosts
>> >
>> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
>> > shegg...@redhat.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> Which version of gluster is in use?
>> >>
>> >> Regards,
>> >>
>> >> Sunil kumar Acharya
>> >>
>> >> Red Hat
>> >>
>> >> 
>> >>
>> >> T: +91-8067935170
>> >> 
>> >>
>> >> 
>> >> TRIED. TESTED. TRUSTED. 
>> >>
>> >>
>> >>
>> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison
>> 
>> >> wrote:
>> >>
>> >>> Hello All,
>> >>>
>> >>> We have 3 servers with a raid 50 array each, we are having extreme
>> >>> performance issues with our gluster, writes on gluster seem to take
>> at
>> >>> least 3 times longer than on the raid directly. Can this be
>> improved?
>> >>> I've
>> >>> read through several other performance issues threads but have been
>> >>> unable
>> >>> to make any improvements
>> >>>
>> >>> "gluster volume info" and "gluster volume profile vmstore info" is
>> >>> below
>> >>>
>> >>>
>> >>>
>> =
>> >>>
>> >>> -Inside Gluster - test took 35+ hours:
>> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>> >>> 600G -n 0 -m TEST -f -b -u root
>> >>> Using uid:0, gid:0.
>> >>> Writing intelligently...done
>> >>> Rewriting...done
>> >>> Reading intelligently...done
>> >>> start 'em...done...done...done...done...done...
>> >>> Version  1.98   --Sequential Output-- --Sequential
>> Input-
>> >>> --Random-
>> >>> -Per Chr- --Block-- -Rewrite- -Per Chr-
>> --Block--
>> >>> --Seeks--
>> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
>> %CP
>> >>>  /sec %CP
>> >>> TEST   600G   35.7m  17 5824k   7112m
>> 13
>> >>> 182.7   6
>> >>> Latency5466ms   12754ms  3499ms
>> >>>  1589ms
>> >>>
>> >>>
>> >>>
>> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
>> >>>
>> >>>
>> >>>
>> =
>> >>>
>> >>> -Outside Gluster - test took 18 minutes:
>> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>> >>> 600G -n 0 -m TEST -f -b -u root
>> >>> Using uid:0, gid:0.
>> >>> Writing intelligently...done
>> >>> Rewriting...done
>> >>> Reading intelligently...done
>> >>> start 'em...done...done...done...done...done...
>> >>> Version  1.98   --Sequential Output-- --Sequential
>> Input-
>> >>> --Random-
>> >>> -Per Chr- --Block-- -Rewrite- -Per Chr-
>> --Block--
>> >>> --Seeks--
>> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
>> %CP
>> >>>  /sec %CP
>> >>> TEST   600G567m  78  149m  30307m
>> 37
>> >>>  83.0  57
>> >>> Latency 205ms4630ms  1450ms
>> >>> 679ms
>> >>>
>> >>>
>> >>>
>> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,,
>> >>>
>> >>>
>> >>>
>> =
>> >>>
>> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
>> info
>> >>> Volume Name: engine
>> >>> Type: Replicate
>> >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
>> >>> Status: Started
>> >>> Snapshot Count: 0
>> >>> Number of Bricks: 1 x 3 = 3
>> >>> Transport-type: tcp
>> >>> Bricks:
>> >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
>> >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
>> >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
>> >>> Options Reconfigured:
>> >>> cluster.granular-entry-heal: enable
>> >>> performance.strict-o-direct: on
>> >>> network.ping-timeout: 30
>> >>> storage.owner-gid: 36
>> >>> storage.owner-uid: 36
>> >>> server.event-threads: 4
>> >>> client.event-threads: 4
>> >>> cluster.choose-local: off
>> >>> user.cifs: off
>> >>> features.shard: on
>> >>> cluster.shd-wait-qlength: 1
>> >>> cluster.shd-max-threads: 8

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Alex Morrison 
Hello Derek,

We have a 10Gig connection dedicated to the storage network, nothing else
is on that switch.

On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins  wrote:

> Hi,
>
> Another question which I don't see answered:   What is the underlying
> connectivity between the Gluster hosts?
>
> -derek
>
> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
> > Hello Sunil,
> >
> > [root@ovirt1 ~]# gluster --version
> > glusterfs 8.6
> >
> > same on all hosts
> >
> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
> > shegg...@redhat.com> wrote:
> >
> >> Hi,
> >>
> >> Which version of gluster is in use?
> >>
> >> Regards,
> >>
> >> Sunil kumar Acharya
> >>
> >> Red Hat
> >>
> >> 
> >>
> >> T: +91-8067935170
> >> 
> >>
> >> 
> >> TRIED. TESTED. TRUSTED. 
> >>
> >>
> >>
> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison 
> >> wrote:
> >>
> >>> Hello All,
> >>>
> >>> We have 3 servers with a raid 50 array each, we are having extreme
> >>> performance issues with our gluster, writes on gluster seem to take at
> >>> least 3 times longer than on the raid directly. Can this be improved?
> >>> I've
> >>> read through several other performance issues threads but have been
> >>> unable
> >>> to make any improvements
> >>>
> >>> "gluster volume info" and "gluster volume profile vmstore info" is
> >>> below
> >>>
> >>>
> >>>
> =
> >>>
> >>> -Inside Gluster - test took 35+ hours:
> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
> >>> 600G -n 0 -m TEST -f -b -u root
> >>> Using uid:0, gid:0.
> >>> Writing intelligently...done
> >>> Rewriting...done
> >>> Reading intelligently...done
> >>> start 'em...done...done...done...done...done...
> >>> Version  1.98   --Sequential Output-- --Sequential Input-
> >>> --Random-
> >>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> >>> --Seeks--
> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> >>>  /sec %CP
> >>> TEST   600G   35.7m  17 5824k   7112m  13
> >>> 182.7   6
> >>> Latency5466ms   12754ms  3499ms
> >>>  1589ms
> >>>
> >>>
> >>>
> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
> >>>
> >>>
> >>>
> =
> >>>
> >>> -Outside Gluster - test took 18 minutes:
> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
> >>> 600G -n 0 -m TEST -f -b -u root
> >>> Using uid:0, gid:0.
> >>> Writing intelligently...done
> >>> Rewriting...done
> >>> Reading intelligently...done
> >>> start 'em...done...done...done...done...done...
> >>> Version  1.98   --Sequential Output-- --Sequential Input-
> >>> --Random-
> >>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> >>> --Seeks--
> >>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> >>>  /sec %CP
> >>> TEST   600G567m  78  149m  30307m  37
> >>>  83.0  57
> >>> Latency 205ms4630ms  1450ms
> >>> 679ms
> >>>
> >>>
> >>>
> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,,
> >>>
> >>>
> >>>
> =
> >>>
> >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume
> info
> >>> Volume Name: engine
> >>> Type: Replicate
> >>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
> >>> Status: Started
> >>> Snapshot Count: 0
> >>> Number of Bricks: 1 x 3 = 3
> >>> Transport-type: tcp
> >>> Bricks:
> >>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
> >>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
> >>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
> >>> Options Reconfigured:
> >>> cluster.granular-entry-heal: enable
> >>> performance.strict-o-direct: on
> >>> network.ping-timeout: 30
> >>> storage.owner-gid: 36
> >>> storage.owner-uid: 36
> >>> server.event-threads: 4
> >>> client.event-threads: 4
> >>> cluster.choose-local: off
> >>> user.cifs: off
> >>> features.shard: on
> >>> cluster.shd-wait-qlength: 1
> >>> cluster.shd-max-threads: 8
> >>> cluster.locking-scheme: granular
> >>> cluster.data-self-heal-algorithm: full
> >>> cluster.server-quorum-type: server
> >>> cluster.quorum-type: auto
> >>> cluster.eager-lock: enable
> >>> network.remote-dio: off
> >>> performance.low-prio-threads: 32
> >>> performance.io-cache: off
> >>>

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Derek Atkins 
Hi,

Another question which I don't see answered:   What is the underlying
connectivity between the Gluster hosts?

-derek

On Wed, February 23, 2022 8:39 am, Alex Morrison wrote:
> Hello Sunil,
>
> [root@ovirt1 ~]# gluster --version
> glusterfs 8.6
>
> same on all hosts
>
> On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
> shegg...@redhat.com> wrote:
>
>> Hi,
>>
>> Which version of gluster is in use?
>>
>> Regards,
>>
>> Sunil kumar Acharya
>>
>> Red Hat
>>
>> 
>>
>> T: +91-8067935170
>> 
>>
>> 
>> TRIED. TESTED. TRUSTED. 
>>
>>
>>
>> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison 
>> wrote:
>>
>>> Hello All,
>>>
>>> We have 3 servers with a raid 50 array each, we are having extreme
>>> performance issues with our gluster, writes on gluster seem to take at
>>> least 3 times longer than on the raid directly. Can this be improved?
>>> I've
>>> read through several other performance issues threads but have been
>>> unable
>>> to make any improvements
>>>
>>> "gluster volume info" and "gluster volume profile vmstore info" is
>>> below
>>>
>>>
>>> =
>>>
>>> -Inside Gluster - test took 35+ hours:
>>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>>> 600G -n 0 -m TEST -f -b -u root
>>> Using uid:0, gid:0.
>>> Writing intelligently...done
>>> Rewriting...done
>>> Reading intelligently...done
>>> start 'em...done...done...done...done...done...
>>> Version  1.98   --Sequential Output-- --Sequential Input-
>>> --Random-
>>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
>>> --Seeks--
>>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>>  /sec %CP
>>> TEST   600G   35.7m  17 5824k   7112m  13
>>> 182.7   6
>>> Latency5466ms   12754ms  3499ms
>>>  1589ms
>>>
>>>
>>> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
>>>
>>>
>>> =
>>>
>>> -Outside Gluster - test took 18 minutes:
>>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>>> 600G -n 0 -m TEST -f -b -u root
>>> Using uid:0, gid:0.
>>> Writing intelligently...done
>>> Rewriting...done
>>> Reading intelligently...done
>>> start 'em...done...done...done...done...done...
>>> Version  1.98   --Sequential Output-- --Sequential Input-
>>> --Random-
>>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
>>> --Seeks--
>>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>>  /sec %CP
>>> TEST   600G567m  78  149m  30307m  37
>>>  83.0  57
>>> Latency 205ms4630ms  1450ms
>>> 679ms
>>>
>>>
>>> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,,
>>>
>>>
>>> =
>>>
>>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info
>>> Volume Name: engine
>>> Type: Replicate
>>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
>>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
>>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
>>> Options Reconfigured:
>>> cluster.granular-entry-heal: enable
>>> performance.strict-o-direct: on
>>> network.ping-timeout: 30
>>> storage.owner-gid: 36
>>> storage.owner-uid: 36
>>> server.event-threads: 4
>>> client.event-threads: 4
>>> cluster.choose-local: off
>>> user.cifs: off
>>> features.shard: on
>>> cluster.shd-wait-qlength: 1
>>> cluster.shd-max-threads: 8
>>> cluster.locking-scheme: granular
>>> cluster.data-self-heal-algorithm: full
>>> cluster.server-quorum-type: server
>>> cluster.quorum-type: auto
>>> cluster.eager-lock: enable
>>> network.remote-dio: off
>>> performance.low-prio-threads: 32
>>> performance.io-cache: off
>>> performance.read-ahead: off
>>> performance.quick-read: off
>>> transport.address-family: inet
>>> storage.fips-mode-rchecksum: on
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>> diagnostics.latency-measurement: on
>>> diagnostics.count-fop-hits: on
>>>
>>> Volume Name: vmstore
>>> Type: Replicate
>>> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>>

[ovirt-users] Re: Gluster Performance issues

2022-02-23 Thread Alex Morrison 
Hello Sunil,

[root@ovirt1 ~]# gluster --version
glusterfs 8.6

same on all hosts

On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya <
shegg...@redhat.com> wrote:

> Hi,
>
> Which version of gluster is in use?
>
> Regards,
>
> Sunil kumar Acharya
>
> Red Hat
>
> 
>
> T: +91-8067935170 
>
> 
> TRIED. TESTED. TRUSTED. 
>
>
>
> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison 
> wrote:
>
>> Hello All,
>>
>> We have 3 servers with a raid 50 array each, we are having extreme
>> performance issues with our gluster, writes on gluster seem to take at
>> least 3 times longer than on the raid directly. Can this be improved? I've
>> read through several other performance issues threads but have been unable
>> to make any improvements
>>
>> "gluster volume info" and "gluster volume profile vmstore info" is below
>>
>>
>> =
>>
>> -Inside Gluster - test took 35+ hours:
>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>> 600G -n 0 -m TEST -f -b -u root
>> Using uid:0, gid:0.
>> Writing intelligently...done
>> Rewriting...done
>> Reading intelligently...done
>> start 'em...done...done...done...done...done...
>> Version  1.98   --Sequential Output-- --Sequential Input-
>> --Random-
>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
>> --Seeks--
>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>  /sec %CP
>> TEST   600G   35.7m  17 5824k   7112m  13
>> 182.7   6
>> Latency5466ms   12754ms  3499ms
>>  1589ms
>>
>>
>> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,,
>>
>>
>> =
>>
>> -Outside Gluster - test took 18 minutes:
>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s
>> 600G -n 0 -m TEST -f -b -u root
>> Using uid:0, gid:0.
>> Writing intelligently...done
>> Rewriting...done
>> Reading intelligently...done
>> start 'em...done...done...done...done...done...
>> Version  1.98   --Sequential Output-- --Sequential Input-
>> --Random-
>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
>> --Seeks--
>> Name:Size etc/sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>>  /sec %CP
>> TEST   600G567m  78  149m  30307m  37
>>  83.0  57
>> Latency 205ms4630ms  1450ms
>> 679ms
>>
>>
>> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,,
>>
>>
>> =
>>
>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info
>> Volume Name: engine
>> Type: Replicate
>> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine
>> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine
>> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine
>> Options Reconfigured:
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> network.ping-timeout: 30
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> server.event-threads: 4
>> client.event-threads: 4
>> cluster.choose-local: off
>> user.cifs: off
>> features.shard: on
>> cluster.shd-wait-qlength: 1
>> cluster.shd-max-threads: 8
>> cluster.locking-scheme: granular
>> cluster.data-self-heal-algorithm: full
>> cluster.server-quorum-type: server
>> cluster.quorum-type: auto
>> cluster.eager-lock: enable
>> network.remote-dio: off
>> performance.low-prio-threads: 32
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> transport.address-family: inet
>> storage.fips-mode-rchecksum: on
>> nfs.disable: on
>> performance.client-io-threads: on
>> diagnostics.latency-measurement: on
>> diagnostics.count-fop-hits: on
>>
>> Volume Name: vmstore
>> Type: Replicate
>> Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore
>> Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore
>> Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore
>> Options Reconfigured:
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> network.ping-timeout: 20
>> storage.owner-gid: 36
>>