Hi,
Same problem on 3.8.1. Even on loopback interface (traffic not leaves gluster
node):
Writing locally to replica 2 volume (each brick is separate local RAID6): 613
MB/sec
Writing locally to 1-brick volume: 877 MB/sec
Writing locally to the brick itself (directly to XFS): 1400 MB/sec
Tests were performed using fio with following settings:
bs=4096k
ioengine=libaio
iodepth=32
direct=0
runtime=600
directory=/R1
numjobs=1
rw=write
size=40g
Even with direct=1 the brick itself gives 1400 MB/sec.
1-brick volume profiling below:
# gluster volume profile test-data-03 info
Brick: gluster-01:/R1/test-data-03
-----------------------------------------------
Cumulative Stats:
Block Size: 131072b+ 262144b+
No. of Reads: 0 0
No. of Writes: 889072 20
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
100.00 122.96 us 67.00 us 42493.00 us 208598 WRITE
Duration: 1605 seconds
Data Read: 0 bytes
Data Written: 116537688064 bytes
Interval 0 Stats:
Block Size: 131072b+ 262144b+
No. of Reads: 0 0
No. of Writes: 889072 20
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
100.00 122.96 us 67.00 us 42493.00 us 208598 WRITE
Duration: 1605 seconds
Data Read: 0 bytes
Data Written: 116537688064 bytes
#
As you can see all writes are performed using 128 KB block size. And it looks
like a bottleneck. Which was discussed previously btw:
http://www.gluster.org/pipermail/gluster-devel/2013-March/038821.html
Using GFAPI to access the volume shows better speed, but still far from raw
brick. fio tests with ioengine=gfapi gives following:
Writing locally to replica 2 volume (each brick is separate local RAID6): 680
MB/sec
Writing locally to 1-brick volume: 960 MB/sec
Accorging to 1-brick volume profile 128 KB blocks no more used:
# gluster volume profile tzk-data-03 info
Brick: j-gluster-01.vcod.jet.su:/R1/tzk-data-03
-----------------------------------------------
Cumulative Stats:
Block Size: 4194304b+
No. of Reads: 0
No. of Writes: 9211
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
100.00 2237.67 us 1880.00 us 5785.00 us 8701 WRITE
Duration: 49 seconds
Data Read: 0 bytes
Data Written: 38633734144 bytes
Interval 0 Stats:
Block Size: 4194304b+
No. of Reads: 0
No. of Writes: 9211
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
100.00 2237.67 us 1880.00 us 5785.00 us 8701 WRITE
Duration: 49 seconds
Data Read: 0 bytes
Data Written: 38633734144 bytes
[root@j-gluster-01 ~]#
So, it may be worth to try using NFS Ganesha with GFAPI plugin.
> 3 авг. 2016 г., в 9:40, Kaamesh Kamalaaharan <[email protected]>
> написал(а):
>
> Hi ,
> I have gluster 3.6.2 installed on my server network. Due to internal issues
> we are not allowed to upgrade the gluster version. All the clients are on the
> same version of gluster. When transferring files to/from the clients or
> between my nodes over the 10gb network, the transfer rate is capped at
> 450Mb/s .Is there any way to increase the transfer speeds for gluster mounts?
>
> Our server setup is as following:
>
> 2 gluster servers -gfs1 and gfs2
> volume name : gfsvolume
> 3 clients - hpc1, hpc2,hpc3
> gluster volume mounted on /export/gfsmount/
>
>
>
> The following is the average results what i did so far:
>
> 1) test bandwith with iperf between all machines - 9.4 GiB/s
> 2) test write speed with dd
> dd if=/dev/zero of=/export/gfsmount/testfile bs=1G count=1
>
> result=399Mb/s
>
> 3) test read speed with dd
> dd if=/export/gfsmount/testfile of=/dev/zero bs=1G count=1
>
> result=284MB/s
>
> My gluster volume configuration:
>
> Volume Name: gfsvolume
> Type: Replicate
> Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: gfs1:/export/sda/brick
> Brick2: gfs2:/export/sda/brick
> Options Reconfigured:
> performance.quick-read: off
> network.ping-timeout: 30
> network.frame-timeout: 90
> performance.cache-max-file-size: 2MB
> cluster.server-quorum-type: none
> nfs.addr-namelookup: off
> nfs.trusted-write: off
> performance.write-behind-window-size: 4MB
> cluster.data-self-heal-algorithm: diff
> performance.cache-refresh-timeout: 60
> performance.cache-size: 1GB
> cluster.quorum-type: fixed
> auth.allow: 172.*
> cluster.quorum-count: 1
> diagnostics.latency-measurement: on
> diagnostics.count-fop-hits: on
> cluster.server-quorum-ratio: 50%
>
> Any help would be appreciated.
> Thanks,
> Kaamesh
>
> _______________________________________________
> Gluster-users mailing list
> [email protected]
> http://www.gluster.org/mailman/listinfo/gluster-users
--
Dmitry Glushenok
Jet Infosystems
_______________________________________________
Gluster-users mailing list
[email protected]
http://www.gluster.org/mailman/listinfo/gluster-users
