Thanks a lot for your reply! You guessed it right though - mailing lists, various blogs, documentation, videos and even source code at this point. Changing some off the options does make performance slightly better, but nothing particularly groundbreaking.
So, if I understand you correctly, no one has yet managed to get acceptable performance (relative to underlying hardware capabilities) with smaller block sizes? Is there an explanation for this? 2018-04-13 1:57 GMT+03:00 Vlad Kopylov <[email protected]>: > Guess you went through user lists and tried something like this already > http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html > I have a same exact setup and below is as far as it went after months of > trail and error. > We all have somewhat same setup and same issue with this - you can find > same post as yours on the daily basis. > > On Wed, Apr 11, 2018 at 3:03 PM, Anastasia Belyaeva < > [email protected]> wrote: > >> Hello everybody! >> >> I have 3 gluster servers (*gluster 3.12.6, Centos 7.2*; those are >> actually virtual machines located on 3 separate physical XenServer7.1 >> servers) >> >> They are all connected via infiniband network. Iperf3 shows around *23 >> Gbit/s network bandwidth *between each 2 of them. >> >> Each server has 3 HDD put into a *stripe*3 thin pool (LVM2) *with >> logical volume created on top of it, formatted with *xfs*. Gluster top >> reports the following throughput: >> >> root@fsnode2 ~ $ gluster volume top r3vol write-perf bs 4096 count >>> 524288 list-cnt 0 >>> Brick: fsnode2.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Throughput *631.82 MBps *time 3.3989 secs >>> Brick: fsnode6.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Throughput *566.96 MBps *time 3.7877 secs >>> Brick: fsnode4.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Throughput *546.65 MBps *time 3.9285 secs >> >> >> root@fsnode2 ~ $ gluster volume top r2vol write-perf bs 4096 count >>> 524288 list-cnt 0 >>> Brick: fsnode2.ibnet:/data/glusterfs/r2vol/brick1/brick >>> Throughput *539.60 MBps *time 3.9798 secs >>> Brick: fsnode4.ibnet:/data/glusterfs/r2vol/brick1/brick >>> Throughput *580.07 MBps *time 3.7021 secs >> >> >> And two *pure replicated ('replica 2' and 'replica 3')* volumes. *The >> 'replica 2' volume is for testing purpose only. >> >>> Volume Name: r2vol >>> Type: Replicate >>> Volume ID: 4748d0c0-6bef-40d5-b1ec-d30e10cfddd9 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x 2 = 2 >>> Transport-type: tcp >>> Bricks: >>> Brick1: fsnode2.ibnet:/data/glusterfs/r2vol/brick1/brick >>> Brick2: fsnode4.ibnet:/data/glusterfs/r2vol/brick1/brick >>> Options Reconfigured: >>> nfs.disable: on >>> >> >> >>> Volume Name: r3vol >>> Type: Replicate >>> Volume ID: b0f64c28-57e1-4b9d-946b-26ed6b499f29 >>> Status: Started >>> Snapshot Count: 0 >>> Number of Bricks: 1 x 3 = 3 >>> Transport-type: tcp >>> Bricks: >>> Brick1: fsnode2.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Brick2: fsnode4.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Brick3: fsnode6.ibnet:/data/glusterfs/r3vol/brick1/brick >>> Options Reconfigured: >>> nfs.disable: on >> >> >> >> *Client *is also gluster 3.12.6, Centos 7.3 virtual machine, *FUSE mount* >> >> >>> root@centos7u3-nogdesktop2 ~ $ mount |grep gluster >>> gluster-host.ibnet:/r2vol on /mnt/gluster/r2 type fuse.glusterfs >>> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_ >>> other,max_read=131072) >>> gluster-host.ibnet:/r3vol on /mnt/gluster/r3 type fuse.glusterfs >>> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_ >>> other,max_read=131072) >> >> >> >> *The problem *is that there is a significant performance loss with >> smaller block sizes. For example: >> >> *4K block size* >> [replica 3 volume] >> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero >> of=/mnt/gluster/r3/file$RANDOM bs=4096 count=262144 >> 262144+0 records in >> 262144+0 records out >> 1073741824 bytes (1.1 GB) copied, 11.2207 s, *95.7 MB/s* >> >> [replica 2 volume] >> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero >> of=/mnt/gluster/r2/file$RANDOM bs=4096 count=262144 >> 262144+0 records in >> 262144+0 records out >> 1073741824 bytes (1.1 GB) copied, 12.0149 s, *89.4 MB/s* >> >> *512K block size* >> [replica 3 volume] >> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero >> of=/mnt/gluster/r3/file$RANDOM bs=512K count=2048 >> 2048+0 records in >> 2048+0 records out >> 1073741824 bytes (1.1 GB) copied, 5.27207 s, *204 MB/s* >> >> [replica 2 volume] >> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero >> of=/mnt/gluster/r2/file$RANDOM bs=512K count=2048 >> 2048+0 records in >> 2048+0 records out >> 1073741824 bytes (1.1 GB) copied, 4.22321 s, *254 MB/s* >> >> With bigger block size It's still not where I expect it to be, but at >> least it starts to make some sense. >> >> I've been trying to solve this for a very long time with no luck. >> I've already tried both kernel tuning (different 'tuned' profiles and the >> ones recommended in the "Linux Kernel Tuning" section) and tweaking gluster >> volume options, including write-behind/flush-behind/writ >> e-behind-window-size. >> The latter, to my surprise, didn't make any difference. 'Cause at first I >> thought it was the buffering issue but it turns out it does buffer writes, >> just not very efficient (well at least what it looks like in the *gluster >> profile output*) >> >> root@fsnode2 ~ $ gluster volume profile r3vol info clear >>> ... >>> Cleared stats. >> >> >> root@centos7u3-nogdesktop2 ~ $ dd if=/dev/zero >>> of=/mnt/gluster/r3/file$RANDOM bs=4096 count=262144 >>> 262144+0 records in >>> 262144+0 records out >>> 1073741824 bytes (1.1 GB) copied, 10.9743 s, 97.8 MB/s >> >> >> >>> root@fsnode2 ~ $ gluster volume profile r3vol info >>> Brick: fsnode2.ibnet:/data/glusterfs/r3vol/brick1/brick >>> ------------------------------------------------------- >>> Cumulative Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 18.00 us 18.00 us 18.00 us 1 >>> STATFS >>> 0.00 20.50 us 11.00 us 30.00 us 2 >>> FLUSH >>> 0.00 22.50 us 17.00 us 28.00 us 2 >>> FINODELK >>> 0.01 76.50 us 65.00 us 88.00 us 2 >>> FXATTROP >>> 0.01 177.00 us 177.00 us 177.00 us 1 >>> CREATE >>> 0.02 56.14 us 23.00 us 128.00 us 7 >>> LOOKUP >>> 0.02 259.00 us 20.00 us 498.00 us 2 >>> ENTRYLK >>> 99.94 59.23 us 17.00 us 10914.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >>> Interval 0 Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 18.00 us 18.00 us 18.00 us 1 >>> STATFS >>> 0.00 20.50 us 11.00 us 30.00 us 2 >>> FLUSH >>> 0.00 22.50 us 17.00 us 28.00 us 2 >>> FINODELK >>> 0.01 76.50 us 65.00 us 88.00 us 2 >>> FXATTROP >>> 0.01 177.00 us 177.00 us 177.00 us 1 >>> CREATE >>> 0.02 56.14 us 23.00 us 128.00 us 7 >>> LOOKUP >>> 0.02 259.00 us 20.00 us 498.00 us 2 >>> ENTRYLK >>> 99.94 59.23 us 17.00 us 10914.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >>> Brick: fsnode6.ibnet:/data/glusterfs/r3vol/brick1/brick >>> ------------------------------------------------------- >>> Cumulative Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 33.00 us 33.00 us 33.00 us 1 >>> STATFS >>> 0.00 22.50 us 13.00 us 32.00 us 2 >>> ENTRYLK >>> 0.00 32.00 us 26.00 us 38.00 us 2 >>> FLUSH >>> 0.01 47.50 us 16.00 us 79.00 us 2 >>> FINODELK >>> 0.01 157.00 us 157.00 us 157.00 us 1 >>> CREATE >>> 0.01 92.00 us 70.00 us 114.00 us 2 >>> FXATTROP >>> 0.03 72.57 us 39.00 us 121.00 us 7 >>> LOOKUP >>> 99.94 47.97 us 15.00 us 1598.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >>> Interval 0 Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 33.00 us 33.00 us 33.00 us 1 >>> STATFS >>> 0.00 22.50 us 13.00 us 32.00 us 2 >>> ENTRYLK >>> 0.00 32.00 us 26.00 us 38.00 us 2 >>> FLUSH >>> 0.01 47.50 us 16.00 us 79.00 us 2 >>> FINODELK >>> 0.01 157.00 us 157.00 us 157.00 us 1 >>> CREATE >>> 0.01 92.00 us 70.00 us 114.00 us 2 >>> FXATTROP >>> 0.03 72.57 us 39.00 us 121.00 us 7 >>> LOOKUP >>> 99.94 47.97 us 15.00 us 1598.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >>> Brick: fsnode4.ibnet:/data/glusterfs/r3vol/brick1/brick >>> ------------------------------------------------------- >>> Cumulative Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 58.00 us 58.00 us 58.00 us 1 >>> STATFS >>> 0.00 38.00 us 38.00 us 38.00 us 2 >>> ENTRYLK >>> 0.01 59.00 us 32.00 us 86.00 us 2 >>> FLUSH >>> 0.01 81.00 us 33.00 us 129.00 us 2 >>> FINODELK >>> 0.01 91.50 us 73.00 us 110.00 us 2 >>> FXATTROP >>> 0.01 239.00 us 239.00 us 239.00 us 1 >>> CREATE >>> 0.04 103.14 us 63.00 us 210.00 us 7 >>> LOOKUP >>> 99.92 52.99 us 16.00 us 11289.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >>> Interval 0 Stats: >>> Block Size: 4096b+ 8192b+ >>> 16384b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 1576 4173 >>> 19605 >>> Block Size: 32768b+ 65536b+ >>> 131072b+ >>> No. of Reads: 0 0 >>> 0 >>> No. of Writes: 7777 1847 >>> 657 >>> %-latency Avg-latency Min-Latency Max-Latency No. of calls >>> Fop >>> --------- ----------- ----------- ----------- ------------ >>> ---- >>> 0.00 0.00 us 0.00 us 0.00 us 1 >>> RELEASE >>> 0.00 58.00 us 58.00 us 58.00 us 1 >>> STATFS >>> 0.00 38.00 us 38.00 us 38.00 us 2 >>> ENTRYLK >>> 0.01 59.00 us 32.00 us 86.00 us 2 >>> FLUSH >>> 0.01 81.00 us 33.00 us 129.00 us 2 >>> FINODELK >>> 0.01 91.50 us 73.00 us 110.00 us 2 >>> FXATTROP >>> 0.01 239.00 us 239.00 us 239.00 us 1 >>> CREATE >>> 0.04 103.14 us 63.00 us 210.00 us 7 >>> LOOKUP >>> 99.92 52.99 us 16.00 us 11289.00 us 35635 >>> WRITE >>> Duration: 38 seconds >>> Data Read: 0 bytes >>> Data Written: 1073741824 bytes >> >> >> >> At this point I'm officially run out of idea where to look next. So any >> help, suggestions or pointers are highly appreciated! >> >> -- >> Best regards, >> Anastasia Belyaeva >> >> >> >> >> >> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://lists.gluster.org/mailman/listinfo/gluster-users >> > > -- Best regards, Anastasia Belyaeva С уважением, Анастасия Беляева
_______________________________________________ Gluster-users mailing list [email protected] http://lists.gluster.org/mailman/listinfo/gluster-users
