I am interested in this also, so I'll add myself here too :) 3.5.2, same speeds for replica 1 volume: near 50 MB/s or 400 Mbps. Servers are connected using 10g interfaces, clients using 1gbps.
2014-09-03 8:02 GMT+03:00 Jaden Liang <[email protected]>: > Hi all, > > We did some more tests and analysis yesterday. It looks like 50MB/s is the > top theoretical speed in replica 1 volume over 1Gbps network. GlusterFS > write 128KB data once a block, then wait for return. The 128KB data would > cost about 1ms in 1Gbps network. And in the server-side, it took about > 800us to 1000us to write 128KB to the HDD and return. Plus some other 100us > to 200us time elapsed. GlusterFS would take about 2ms-2.2ms to finish a > 128KB block data writing, which is about 50MB/s. > > The question is that why don't glusterfs use pipeline writting or reading > to speed up this chatty process? > > > On Tuesday, September 2, 2014, Jaden Liang <[email protected]> wrote: > >> Hello, gluster-devel and gluster-users team, >> >> We are running a performance test in a replica 1 volume and find out the >> single file sequence writing performance only get about 50MB/s in a 1Gbps >> Ethernet. However, if we test multiple files sequence writing, the writing >> performance can go up to 120MB/s which is the top speed of network. >> >> We also tried to use the stat xlator to find out where is the bottleneck >> of single file write performance. Here is the stat data: >> >> Client-side: >> ...... >> vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us, >> mean:2665.328491us, count:8192, max:4063475, min:1849 >> ...... >> >> Server-side: >> ...... >> /data/sdb1/brick1.latency.WRITE=total:6156857.000000us, >> mean:751.569458us, count:8192, max:230864, min:611 >> ...... >> >> Note that the test is write a 1GB single file sequentially to a replica 1 >> volume through 1Gbps Ethernet network. >> >> On the client-side, we can see there are 8192 write requests totally. >> Every request will write 128KB data. Total eclipsed time is 21834371us, >> about 21 seconds. The mean time of request is 2665us, about 2.6ms which >> means it could only serves about 380 requests in 1 seconds. Plus there are >> other time consuming like statfs, lookup, but those are not major reasons. >> >> On the server-side, the mean time of request is 751us include write data >> to HDD disk. So we think that is not the major reason. >> >> And we also modify some codes to do the statistic of system epoll elapsed >> time. It only took about 20us from enqueue data to finish sent-out. >> >> Now we are heading to the rpc mechanism in glusterfs. Still, we think >> this issue maybe encountered in gluster-devel or gluster-users teams. >> Therefor, any suggestions would be grateful. Or have anyone know such issue? >> >> Best regards, >> Jaden Liang >> 9/2/2014 >> >> >> -- >> Best regards, >> Jaden Liang >> >> > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users > -- Best regards, Roman.
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
