[ovirt-users] Re: Gluster Performance issues
Hello All, I believe the network is performing as expected, I did an iperf test: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# iperf3 -c 10.10.1.2 Connecting to host 10.10.1.2, port 5201 [ 5] local 10.10.1.1 port 38422 connected to 10.10.1.2 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.08 GBytes 9.24 Gbits/sec0 2.96 MBytes [ 5] 1.00-2.00 sec 1.03 GBytes 8.81 Gbits/sec0 2.96 MBytes [ 5] 2.00-3.00 sec 1006 MBytes 8.44 Gbits/sec 101 1.45 MBytes [ 5] 3.00-4.00 sec 1.04 GBytes 8.92 Gbits/sec5901 KBytes [ 5] 4.00-5.00 sec 1.05 GBytes 9.01 Gbits/sec0957 KBytes [ 5] 5.00-6.00 sec 1.08 GBytes 9.23 Gbits/sec0990 KBytes [ 5] 6.00-7.00 sec 1008 MBytes 8.46 Gbits/sec 159655 KBytes [ 5] 7.00-8.00 sec 1.06 GBytes 9.11 Gbits/sec0970 KBytes [ 5] 8.00-9.00 sec 1.03 GBytes 8.85 Gbits/sec2829 KBytes [ 5] 9.00-10.00 sec 1.04 GBytes 8.96 Gbits/sec0947 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 10.4 GBytes 8.90 Gbits/sec 267 sender [ 5] 0.00-10.04 sec 10.4 GBytes 8.87 Gbits/sec receiver iperf Done. On Wed, Feb 23, 2022 at 11:45 AM Sunil Kumar Heggodu Gopala Acharya < shegg...@redhat.com> wrote: > > Regards, > Sunil > > > On Wed, Feb 23, 2022 at 7:34 PM Derek Atkins wrote: > >> Have you verified that you're actually getting 10Gbps between the hosts? >> >> -derek >> >> On Wed, February 23, 2022 9:02 am, Alex Morrison wrote: >> > Hello Derek, >> > >> > We have a 10Gig connection dedicated to the storage network, nothing >> else >> > is on that switch. >> > >> > On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins wrote: >> > >> >> Hi, >> >> >> >> Another question which I don't see answered: What is the underlying >> >> connectivity between the Gluster hosts? >> >> >> >> -derek >> >> >> >> On Wed, February 23, 2022 8:39 am, Alex Morrison wrote: >> >> > Hello Sunil, >> >> > >> >> > [root@ovirt1 ~]# gluster --version >> >> > glusterfs 8.6 >> >> > >> >> > same on all hosts >> > Latest Release-10.1( > https://lists.gluster.org/pipermail/gluster-users/2022-February/039761.html) > has some performance fixes which should help in this situation compared to > the older gluster bits. > >> >> > >> >> > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < >> >> > shegg...@redhat.com> wrote: >> >> > >> >> >> Hi, >> >> >> >> >> >> Which version of gluster is in use? >> >> >> >> >> >> Regards, >> >> >> >> >> >> Sunil kumar Acharya >> >> >> >> >> >> Red Hat >> >> >> >> >> >> <https://www.redhat.com> >> >> >> >> >> >> T: +91-8067935170 >> >> >> <http://redhatemailsignature-marketing.itos.redhat.com/> >> >> >> >> >> >> <https://red.ht/sig> >> >> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> >> >> >> >> >> >> >> >> >> >> >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison >> >> >> >> >> wrote: >> >> >> >> >> >>> Hello All, >> >> >>> >> >> >>> We have 3 servers with a raid 50 array each, we are having extreme >> >> >>> performance issues with our gluster, writes on gluster seem to take >> >> at >> >> >>> least 3 times longer than on the raid directly. Can this be >> >> improved? >> >> >>> I've >> >> >>> read through several other performance issues threads but have been >> >> >>> unable >> >> >>> to make any improvements >> >> >>> >> >> >>> "gluster volume info" and "gluster volume profile vmstore info" is >> >> >>> below >> >> >>> >> >> >>> >> >> >>> >> >> >> = >> >> >>>
[ovirt-users] Re: Gluster Performance issues
Hello Derek, We have a 10Gig connection dedicated to the storage network, nothing else is on that switch. On Wed, Feb 23, 2022 at 9:49 AM Derek Atkins wrote: > Hi, > > Another question which I don't see answered: What is the underlying > connectivity between the Gluster hosts? > > -derek > > On Wed, February 23, 2022 8:39 am, Alex Morrison wrote: > > Hello Sunil, > > > > [root@ovirt1 ~]# gluster --version > > glusterfs 8.6 > > > > same on all hosts > > > > On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < > > shegg...@redhat.com> wrote: > > > >> Hi, > >> > >> Which version of gluster is in use? > >> > >> Regards, > >> > >> Sunil kumar Acharya > >> > >> Red Hat > >> > >> <https://www.redhat.com> > >> > >> T: +91-8067935170 > >> <http://redhatemailsignature-marketing.itos.redhat.com/> > >> > >> <https://red.ht/sig> > >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > >> > >> > >> > >> On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison > >> wrote: > >> > >>> Hello All, > >>> > >>> We have 3 servers with a raid 50 array each, we are having extreme > >>> performance issues with our gluster, writes on gluster seem to take at > >>> least 3 times longer than on the raid directly. Can this be improved? > >>> I've > >>> read through several other performance issues threads but have been > >>> unable > >>> to make any improvements > >>> > >>> "gluster volume info" and "gluster volume profile vmstore info" is > >>> below > >>> > >>> > >>> > = > >>> > >>> -Inside Gluster - test took 35+ hours: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 --Sequential Output-- --Sequential Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > >>> --Seeks-- > >>> Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > >>> /sec %CP > >>> TEST 600G 35.7m 17 5824k 7112m 13 > >>> 182.7 6 > >>> Latency5466ms 12754ms 3499ms > >>> 1589ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,, > >>> > >>> > >>> > = > >>> > >>> -Outside Gluster - test took 18 minutes: > >>> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s > >>> 600G -n 0 -m TEST -f -b -u root > >>> Using uid:0, gid:0. > >>> Writing intelligently...done > >>> Rewriting...done > >>> Reading intelligently...done > >>> start 'em...done...done...done...done...done... > >>> Version 1.98 --Sequential Output-- --Sequential Input- > >>> --Random- > >>> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > >>> --Seeks-- > >>> Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP > >>> /sec %CP > >>> TEST 600G567m 78 149m 30307m 37 > >>> 83.0 57 > >>> Latency 205ms4630ms 1450ms > >>> 679ms > >>> > >>> > >>> > 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,, > >>> > >>> > >>> > = > >>> > >>> [root@ovirt1 1801ed24-5b55-4431-9813-49614
[ovirt-users] Re: Gluster Performance issues
Hello Sunil, [root@ovirt1 ~]# gluster --version glusterfs 8.6 same on all hosts On Wed, Feb 23, 2022 at 5:24 AM Sunil Kumar Heggodu Gopala Acharya < shegg...@redhat.com> wrote: > Hi, > > Which version of gluster is in use? > > Regards, > > Sunil kumar Acharya > > Red Hat > > <https://www.redhat.com> > > T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/> > > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > > > > On Wed, Feb 23, 2022 at 2:17 PM Alex Morrison > wrote: > >> Hello All, >> >> We have 3 servers with a raid 50 array each, we are having extreme >> performance issues with our gluster, writes on gluster seem to take at >> least 3 times longer than on the raid directly. Can this be improved? I've >> read through several other performance issues threads but have been unable >> to make any improvements >> >> "gluster volume info" and "gluster volume profile vmstore info" is below >> >> >> = >> >> -Inside Gluster - test took 35+ hours: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 --Sequential Output-- --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G 35.7m 17 5824k 7112m 13 >> 182.7 6 >> Latency5466ms 12754ms 3499ms >> 1589ms >> >> >> 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,, >> >> >> = >> >> -Outside Gluster - test took 18 minutes: >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s >> 600G -n 0 -m TEST -f -b -u root >> Using uid:0, gid:0. >> Writing intelligently...done >> Rewriting...done >> Reading intelligently...done >> start 'em...done...done...done...done...done... >> Version 1.98 --Sequential Output-- --Sequential Input- >> --Random- >> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP >> /sec %CP >> TEST 600G567m 78 149m 30307m 37 >> 83.0 57 >> Latency 205ms4630ms 1450ms >> 679ms >> >> >> 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,, >> >> >> = >> >> [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info >> Volume Name: engine >> Type: Replicate >> Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine >> Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine >> Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine >> Options Reconfigured: >> cluster.granular-entry-heal: enable >> performance.strict-o-direct: on >> network.ping-timeout: 30 >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> server.event-threads: 4 >> client.event-threads: 4 >> cluster.choose-local: off >> user.cifs: off >> features.shard: on >> cluster.shd-wait-qlength: 1 >> cluster.shd-max-threads: 8 >> cluster.locking-scheme: granular >> cluster.data-self-heal-algorithm: full >> cluster.server-quorum-type: server >> cluster.quorum-type: auto >> cluster.eager-lock: enable >> network.remote-dio: off >> performance.low-prio-threads: 32 >> performance.io-cache: off >> performance.read-ahead: off >> performance.quick-read: off >> transport.address-family
[ovirt-users] Gluster Performance issues
Hello All, We have 3 servers with a raid 50 array each, we are having extreme performance issues with our gluster, writes on gluster seem to take at least 3 times longer than on the raid directly. Can this be improved? I've read through several other performance issues threads but have been unable to make any improvements "gluster volume info" and "gluster volume profile vmstore info" is below = -Inside Gluster - test took 35+ hours: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G 35.7m 17 5824k 7112m 13 182.7 6 Latency5466ms 12754ms 3499ms 1589ms 1.98,1.98,TEST,1,1644359706,600G,,8192,5,,,36598,17,5824,7,,,114950,13,182.7,6,,,5466ms,12754ms,,3499ms,1589ms,, = -Outside Gluster - test took 18 minutes: [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# bonnie++ -d . -s 600G -n 0 -m TEST -f -b -u root Using uid:0, gid:0. Writing intelligently...done Rewriting...done Reading intelligently...done start 'em...done...done...done...done...done... Version 1.98 --Sequential Output-- --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc/sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP TEST 600G567m 78 149m 30307m 37 83.0 57 Latency 205ms4630ms 1450ms 679ms 1.98,1.98,TEST,1,1648288012,600G,,8192,5,,,580384,78,152597,30,,,314533,37,83.0,57,,,205ms,4630ms,,1450ms,679ms,, = [root@ovirt1 1801ed24-5b55-4431-9813-496143367f66]# gluster volume info Volume Name: engine Type: Replicate Volume ID: 7ed15c5a-f054-450c-bac9-3ad1b4e5931b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/engine/engine Brick2: ovirt2-storage.dgi:/gluster_bricks/engine/engine Brick3: ovirt3-storage.dgi:/gluster_bricks/engine/engine Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 30 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on Volume Name: vmstore Type: Replicate Volume ID: 2670ff29-8d43-4610-a437-c6ec2c235753 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: ovirt1-storage.dgi:/gluster_bricks/vmstore/vmstore Brick2: ovirt2-storage.dgi:/gluster_bricks/vmstore/vmstore Brick3: ovirt3-storage.dgi:/gluster_bricks/vmstore/vmstore Options Reconfigured: cluster.granular-entry-heal: enable performance.strict-o-direct: on network.ping-timeout: 20 storage.owner-gid: 36 storage.owner-uid: 36 server.event-threads: 4 client.event-threads: 4 cluster.choose-local: off user.cifs: off features.shard: on cluster.shd-wait-qlength: 1 cluster.shd-max-threads: 8 cluster.locking-scheme: granular cluster.data-self-heal-algorithm: full cluster.server-quorum-type: server cluster.quorum-type: auto cluster.eager-lock: enable network.remote-dio: off performance.low-prio-threads: 32 performance.io-cache: off performance.read-ahead: off performance.quick-read: off transport.address-family: inet storage.fips-mode-rchecksum: on nfs.disable: on performance.client-io-threads: on diagnostics.latency-measurement: on diagnostics.count-fop-hits: on server.tcp-user-timeout: 20 server.keepalive-time: 10 server.keepalive-interval: 2 server.keepalive-count: 5 cluster.lookup-optimize: off
