Roger: Here are some different approaches:
1. XFS, as you know, would work for you. It has a 16EB filesystem limit, way beyond what you are looking for on each server. 2. Using LVM2 where you combine smaller filesystems into one. For 64-bit CPU's on 2.6 kernels, the maximum logical volume is 8EB. 3. Run a separate server process for each LUN. OrangeFS will aggregate them together. I am curious. On these 720's, are you planning to run some sort of RAID? If so, then you won't really have 48TB of storage and ext4 may work, just depends. Also, what is your interconnect between the I/O nodes and between the I/O nodes and your compute nodes? The OrangeFS server process does not require a lot of memory. For example, one of our OrangeFS filesystems at Clemson University has 32 servers (Dell R510's), each with 12GB, and we have never had a memory issue with them. So, the 32GB should be sufficient. We have a white paper describing performance and hardware setup in the Dell environment that may help you determine your own needs. Below is the URL where it is located. http://content.dell.com/us/en/enterprise/d/business~solutions~engineering-docs~en/Documents~orange-fs-reference-architecture.pdf.aspx?c=us&l=en&s=biz&preview=true Hope this helps. Let us know if we can answer any other questions. Becky On Tue, Feb 19, 2013 at 12:07 PM, Moye,Roger V <[email protected]>wrote: > Becky,**** > > ** ** > > We do not have a configuration yet but are considering purchasing four > Dell PowerEdge R720xd servers, each fully populated with twelve 4TB > drives. So this would be 48TB per I/O server or 192TB total for the > entire PVFS system. ** ** > > ** ** > > In addition to knowing how to breach the 16TB limit (we are looking at > XFS), we wonder if there is a rule of thumb to follow on the amount of > memory required per server?**** > > Each server is currently configured for 32GB of memory. **** > > ** ** > > Thanks so much for your help.**** > > ** ** > > -Roger**** > > -----------------------------------------------------------**** > > Roger V. Moye**** > > Systems Analyst III**** > > University of Texas MD Anderson Cancer Center**** > > Division of Quantitative Sciences**** > > FCT4.6109**** > > Houston, Texas**** > > -----------------------------------------------------------**** > > ** ** > > *From:* Becky Ligon [mailto:[email protected]] > *Sent:* Monday, February 18, 2013 5:01 PM > *To:* Moye,Roger V > *Cc:* [email protected] > *Subject:* Re: [Pvfs2-users] How to breach 16TB ext4 limit?**** > > ** ** > > Roger: > > I am consulting with our hardware experts to see what they suggest. Could > you tell me more about your storage configuration? > > Becky**** > > ** ** > -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
