Hi Pascal, Sorry for complete delay in this one. And thanks for testing out in different scenarios. Few questions before others can have a look and advice you.
1. What is the volume info output ? 2. Do you see any concerning logs in glusterfs log files? 3. Please use `gluster volume profile` while running the tests, and that gives a lot of information. 4. Considering you are using glusterfs-6.0, please take statedump of client process (on any node) before and after the test, so we can analyze the latency information of each translators. With these information, I hope we will be in a better state to answer the questions. On Wed, Apr 10, 2019 at 3:45 PM Pascal Suter <[email protected]> wrote: > i continued my testing with 5 clients, all attached over 100Gbit/s > omni-path via IP over IB. when i run the same iozone benchmark across > all 5 clients where gluster is mounted using the glusterfs client, i get > an aggretated write throughput of only about 400GB/s and an aggregated > read throughput of 1.5GB/s. Each node was writing a single 200Gb file in > 16MB chunks and the files where distributed across all three bricks on > the server. > > the connection was established over Omnipath for sure, as there is no > other link between the nodes and server. > > i have no clue what i'm doing wrong here. i can't believe that this is a > normal performance people would expect to see from gluster. i guess > nobody would be using it if it was this slow. > > again, when written dreictly to the xfs filesystem on the bricks, i get > over 6GB/s read and write throughput using the same benchmark. > > any advise is appreciated > > cheers > > Pascal > > On 04.04.19 12:03, Pascal Suter wrote: > > I just noticed i left the most important parameters out :) > > > > here's the write command with filesize and recordsize in it as well :) > > > > ./iozone -i 0 -t 1 -F /mnt/gluster/storage/thread1 -+n -c -C -e -I -w > > -+S 0 -s 200G -r 16384k > > > > also i ran the benchmark without direct_io which resulted in an even > > worse performance. > > > > i also tried to mount the gluster volume via nfs-ganesha which further > > reduced throughput down to about 450MB/s > > > > if i run the iozone benchmark with 3 threads writing to all three > > bricks directly (from the xfs filesystem) i get throughputs of around > > 6GB/s .. if I run the same benchmark through gluster mounted locally > > using the fuse client and with enough threads so that each brick gets > > at least one file written to it, i end up seing throughputs around > > 1.5GB/s .. that's a 4x decrease in performance. at it actually is the > > same if i run the benchmark with less threads and files only get > > written to two out of three bricks. > > > > cpu load on the server is around 25% by the way, nicely distributed > > across all available cores. > > > > i can't believe that gluster should really be so slow and everybody is > > just happily using it. any hints on what i'm doing wrong are very > > welcome. > > > > i'm using gluster 6.0 by the way. > > > > regards > > > > Pascal > > > > On 03.04.19 12:28, Pascal Suter wrote: > >> Hi all > >> > >> I am currently testing gluster on a single server. I have three > >> bricks, each a hardware RAID6 volume with thin provisioned LVM that > >> was aligned to the RAID and then formatted with xfs. > >> > >> i've created a distributed volume so that entire files get > >> distributed across my three bricks. > >> > >> first I ran a iozone benchmark across each brick testing the read and > >> write perofrmance of a single large file per brick > >> > >> i then mounted my gluster volume locally and ran another iozone run > >> with the same parameters writing a single file. the file went to > >> brick 1 which, when used driectly, would write with 2.3GB/s and read > >> with 1.5GB/s. however, through gluster i got only 800MB/s read and > >> 750MB/s write throughput > >> > >> another run with two processes each writing a file, where one file > >> went to the first brick and the other file to the second brick (which > >> by itself when directly accessed wrote at 2.8GB/s and read at > >> 2.7GB/s) resulted in 1.2GB/s of aggregated write and also aggregated > >> read throughput. > >> > >> Is this a normal performance i can expect out of a glusterfs or is it > >> worth tuning in order to really get closer to the actual brick > >> filesystem performance? > >> > >> here are the iozone commands i use for writing and reading.. note > >> that i am using directIO in order to make sure i don't get fooled by > >> cache :) > >> > >> ./iozone -i 0 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > >> -s $filesize -r $recordsize > iozone-brick${b}-write.txt > >> > >> ./iozone -i 1 -t 1 -F /mnt/brick${b}/thread1 -+n -c -C -e -I -w -+S 0 > >> -s $filesize -r $recordsize > iozone-brick${b}-read.txt > >> > >> cheers > >> > >> Pascal > >> > >> _______________________________________________ > >> Gluster-users mailing list > >> [email protected] > >> https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > > Gluster-users mailing list > > [email protected] > > https://lists.gluster.org/mailman/listinfo/gluster-users > _______________________________________________ > Gluster-users mailing list > [email protected] > https://lists.gluster.org/mailman/listinfo/gluster-users > > > -- Amar Tumballi (amarts)
_______________________________________________ Gluster-users mailing list [email protected] https://lists.gluster.org/mailman/listinfo/gluster-users
