-----Original Mail----- From: Jung Young Seok [[email protected]] Sent: 22.08.13 - 19:09:32 To: Maik Kulbe [[email protected]] Cc: [email protected] Subject: Re: [Gluster-users] Writing performance gets slower when writethreadsare many
2013/8/23 Maik Kulbe <[email protected]> -----Original Mail----- From: Jung Young Seok [[email protected]] Sent: 22.08.13 - 17:19:32 To: [email protected] Subject: [Gluster-users] Writing performance gets slower when write threadsare many Dear gluster users, We have set up glusterfs 3.4 on cent 6. 2 Gluster FS servers are replicated each (2brick x 2). Glusterfs is mounted on WAS as fuse. We realized when there are more thread to write files on gluster fs, it gets slower on writing. We've set 250 thread to write 1M file each repeatedly. In this case we get under 30 TPS. No NW bandwidth problem. Is there any way to avoid performance degradation when there are lots of write threads? That depends on a lot of factors - what interconnect are you using? 1GE? 10GE? RDMA? What are your bricks built on? SATA? SSD? RAID? My first guess would be that interconnect latency and a kernel maximum for some connection count could be the problem. Thank you for the comments, We use 1GE for storage network. When I checked the net stream using iftop, it never reached 1G. (only was under 500Mbps. Since we have 2 glusterfs servers, 250Mb for first brick 250Mb for second brick) When I scp a file to Gluster servers from WAS directly , it reached 80MB/s each. (used less encrypted option) About HDD, First server is on SAS 10k RAID1. Second server is on SATA 10k RAID1.
Have you benchmarked both network and harddisks without gluster? I would especially look at the network timings and there behaviour with multiple connections between clients and gluster servers. From my experience I would say too high latencies could very quickly become a bottleneck for the whole system.
Those are for test. We'll use SATA RAID1 both finally to reduce cost. We are using CentOS 6.4. Since we use vanilla installation, most of kernel parameters are defaults. If you give me any option that I can test, it would be great.
There is an article on that in the Gluster docs: http://www.gluster.org/community/documentation/index.php/Linux_Kernel_Tuning You can also look at the performance options that Gluster provides. Those are in the Gluster administration guide: http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf - there are some caching mechanisms, etc. that might increase your write rate.
Thank you. Look forward any help. Thanks. Regards, Youngseok jung
_______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
