Thanks for sharing. It looks like 3.5 million files are directly
placed under /export/read-write/g0{1,2} on 2 servers. I am assuming
performance stays the same as no. of files increase.Couple of more questions: Did you also consider using RAID 0 over these 2 disks on the same host? Do you know what happens when a new node is added. Since gluster uses elastic hashing does it still work or does new node throws off the hashing algorithm. I am wondering how gluster is able to retrieve old files after adding new node because now it's hashing accross new nodes. Thanks again and keep us posted! I will be starting similar testing in next 2 weeks also. On Wed, Mar 9, 2011 at 10:53 AM, Burnash, James <[email protected]> wrote: > I'm running ext4 on CentOS 5.5. Each storage server has two bricks, each in > their own directory, mounted as straight partitions. No LVM used in this > config. > > I did find that I had to keep the ext4 format and mount options simple to > avoid crashes that I encountered with more "tuned" configs. > > # create ext4 read-write storage > > yum -y install e4fsprogs e4fsprogs-libs e4fsprogs-devel dmapi > > create a single partition on each disk, maximum size, primary type > > time mke4fs -F -L g01 -v -j /dev/cciss/c1d0p1 > time mke4fs -F -L g02 -v -j /dev/cciss/c2d0p1 > > mkdir -p /export/read-write/g0{1,2} > mount -t ext4 /dev/cciss/c1d0p1 /export/read-write/g01 > mount > mount -t ext4 /dev/cciss/c2d0p1 /export/read-write/g02 > > fgrep read-write /etc/fstab > /dev/cciss/c1d0p1 /export/read-write/g01 ext4 defaults 0 0 > /dev/cciss/c2d0p1 /export/read-write/g02 ext4 defaults 0 0 > > James Burnash, Unix Engineering > > -----Original Message----- > From: Mohit Anchlia [mailto:[email protected]] > Sent: Wednesday, March 09, 2011 1:15 PM > To: Burnash, James; [email protected] > Subject: Re: [Gluster-users] Files per directory > > Thanks! Is this on ext3 or ext4? Are all these files in mount > directory or they are in sub directories. On glusterfs does it matter > if all the files are placed in same directory? Generally from what > I've seen in the past is that multiple no. of subdirs are recommended > to improve performance. > > On Wed, Mar 9, 2011 at 9:59 AM, Burnash, James <[email protected]> wrote: >> I'm going through a rebalance operation now on my "small" Glusterfs storage >> pool - 2 servers, 4 bricks, 30TB of total storage, 175 native Glusterfs >> clients. >> >> Current files checked is at 3.5 million - a lot of those are in the 1-1.5GB >> size range. >> >> Hopefully that is of some help - more stats to follow as I get a chance to >> document them ... >> >> James Burnash, Unix Engineering >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of Mohit Anchlia >> Sent: Wednesday, March 09, 2011 12:47 PM >> To: [email protected] >> Subject: Re: [Gluster-users] Files per directory >> >> It will be good if I can get some suggestion from people who already >> have millions of files on glusterFS. >> >> On Mon, Mar 7, 2011 at 4:11 PM, Mohit Anchlia <[email protected]> wrote: >>> Is there any recommendation about how many files should be stored in >>> one directory in glusterFS? In my experience I've seen spreading files >>> accross many directories helps, but I am not sure if it's same with >>> glusterFS. >>> >> _______________________________________________ >> Gluster-users mailing list >> [email protected] >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users >> >> >> DISCLAIMER: >> This e-mail, and any attachments thereto, is intended only for use by the >> addressee(s) named herein and may contain legally privileged and/or >> confidential information. If you are not the intended recipient of this >> e-mail, you are hereby notified that any dissemination, distribution or >> copying of this e-mail, and any attachments thereto, is strictly prohibited. >> If you have received this in error, please immediately notify me and >> permanently delete the original and any copy of any e-mail and any printout >> thereof. E-mail transmission cannot be guaranteed to be secure or >> error-free. The sender therefore does not accept liability for any errors or >> omissions in the contents of this message which arise as a result of e-mail >> transmission. >> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at >> its discretion, monitor and review the content of all e-mail communications. >> http://www.knight.com >> > _______________________________________________ Gluster-users mailing list [email protected] http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
