Hi all, We did some more tests and analysis yesterday. It looks like 50MB/s is the top theoretical speed in replica 1 volume over 1Gbps network. GlusterFS write 128KB data once a block, then wait for return. The 128KB data would cost about 1ms in 1Gbps network. And in the server-side, it took about 800us to 1000us to write 128KB to the HDD and return. Plus some other 100us to 200us time elapsed. GlusterFS would take about 2ms-2.2ms to finish a 128KB block data writing, which is about 50MB/s.
The question is that why don't glusterfs use pipeline writting or reading to speed up this chatty process? On Tuesday, September 2, 2014, Jaden Liang <[email protected]> wrote: > Hello, gluster-devel and gluster-users team, > > We are running a performance test in a replica 1 volume and find out the > single file sequence writing performance only get about 50MB/s in a 1Gbps > Ethernet. However, if we test multiple files sequence writing, the writing > performance can go up to 120MB/s which is the top speed of network. > > We also tried to use the stat xlator to find out where is the bottleneck > of single file write performance. Here is the stat data: > > Client-side: > ...... > vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us, > mean:2665.328491us, count:8192, max:4063475, min:1849 > ...... > > Server-side: > ...... > /data/sdb1/brick1.latency.WRITE=total:6156857.000000us, mean:751.569458us, > count:8192, max:230864, min:611 > ...... > > Note that the test is write a 1GB single file sequentially to a replica 1 > volume through 1Gbps Ethernet network. > > On the client-side, we can see there are 8192 write requests totally. > Every request will write 128KB data. Total eclipsed time is 21834371us, > about 21 seconds. The mean time of request is 2665us, about 2.6ms which > means it could only serves about 380 requests in 1 seconds. Plus there are > other time consuming like statfs, lookup, but those are not major reasons. > > On the server-side, the mean time of request is 751us include write data > to HDD disk. So we think that is not the major reason. > > And we also modify some codes to do the statistic of system epoll elapsed > time. It only took about 20us from enqueue data to finish sent-out. > > Now we are heading to the rpc mechanism in glusterfs. Still, we think this > issue maybe encountered in gluster-devel or gluster-users teams. Therefor, > any suggestions would be grateful. Or have anyone know such issue? > > Best regards, > Jaden Liang > 9/2/2014 > > > -- > Best regards, > Jaden Liang > >
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
