Dan, Thanks very much for this (and to all that replied!). I took my test system down to 80% full and the system went back to the expect performance rate. There must be some high(90%)/(80%)low water triggers in the code to tell me to buy more storage (which I have started to do :-).
Thanks again. Bob -----Original Message----- From: Dan Mons [mailto:[email protected]] Sent: Monday, October 06, 2014 6:41 PM To: Ellison, Bob Cc: [email protected] Subject: Re: [Gluster-users] Perfomance issue on a 90+% full file system Yup, pretty common for us. Once we hit ~90% on either of our two production clusters (107 TB usable each), performance takes a beating. I don't consider this a problem, per se. Most file systems (clustered or otherwise) are the same. I consider a high water mark for any production file system to be 80% (and I consider that vendor agnostic), at which time action should be taken to begin clean up. That's good sysadminning 101. -Dan ---------------- Dan Mons Unbreaker of broken things Cutting Edge http://cuttingedge.com.au On 7 October 2014 08:36, Ellison, Bob <[email protected]> wrote: > Hello, > > My glusterfs-3.4.2-1.el6 is having a performance issue. It was working > fine until the 100TB file system hit ~90% full. I was seeing around > 90Mb/s for the last 10 months. This then dropped to 40Mb/s. Since > nothing changed on the system, I focused on the transition to the 90% > full file system. I also found that the 6 undelaying XFS files systems were > pretty fragmented (~56%). > > > > We are using gluster to achieve a large flat file system. This is a > single server/node configuration, so no network issues are involved. > > > > As the problem is on a production system, I setup a smaller test > system. I monitored performance and was able to duplicate the problem > (90MB/s up to 90% full, then a drop off in performance thereafter). > The closer to 100% full, the lower the throughput. > > > > I then started deleting content from the test server. I was surprised > to find that the performance did NOT increase – it stayed the same. I > took the test system to file system 50% full but still saw 40Mb/s! > > > > The test I set up was designed to fragment the XFS partitions (to > mimic the production system state). I defragmented online > successfully, however performance did not increase. > > > > I am currently trying a rebalance across the 6 XFS partitions to see > if that helps. > > > > I was wondering if anyone remembers a problem like this? Is there a > chance that the rebalance will get me back to the normally seen > performance? Would upgrading gluster fix this? > > > > Thanks, > > Bob > > > _______________________________________________ > Gluster-users mailing list > [email protected] > http://supercolony.gluster.org/mailman/listinfo/gluster-users _______________________________________________ Gluster-users mailing list [email protected] http://supercolony.gluster.org/mailman/listinfo/gluster-users
