Thanks Becky! This will help a lot. I want to split the data and metadata handle ranges across 40 servers which will give me plenty of room to grow until OrangeFS-Next comes out. I am assuming that if I reserve handle ranges for 40 servers in my config file, there will not be any issues if there are only 10 servers online when I initially deploy?
Cheers, --Scott From: Becky Ligon <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Thursday, January 3, 2013 6:18 PM To: Scott Roberts <[email protected]<mailto:[email protected]>> Cc: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Elaine Quarles <[email protected]<mailto:[email protected]>> Subject: Re: [Pvfs2-users] How do I back up the metadata? Scott: I have attached some details that I wrote when I configured the hardware for our OrangeFS servers. Some of it is outdated, referring to RHEL 5, but the basic content is still applicable. We are using 32 Dell R510s right now running Scientific Linux 6.1, kernel 2.6.32-220.17.1 I have also attached an example configuration file representing our filesystem when we were running only 16 servers. Please note that we created the handle ranges manually (split into approximately 256 ranges), instead of using the ranges provided by pvfs2-genconfig. Reason being is that pvfs2-genconfig will split up the handle ranges amongst the servers listed, consuming the entire handle range space. Thus, you can't EASILY add more servers without some detective work or physically copying your data out of the old set of servers and into the new. By manually creating the ranges, you can reserve some handles for future servers and avoid physical copies. Becky On Thu, Jan 3, 2013 at 5:59 PM, Becky Ligon <[email protected]<mailto:[email protected]>> wrote: Scott: I'm happy to answer any questions that you have about our setup. Really, the only difference between Clemson's setup and yours is the amount of disks and the interconnect. We are using a RAID5 configuration with 4-2T disks for our data on each of 32 servers, and we are using Myrinet as the interconnect between nodes and servers. As you are planning, we have 2-50GB SSDs that are divided into 2 ~25GB partitions, one for the OS and one for metadata, and are mirrored. I have been following your emails and have interjected my thoughts when necessary. Boyd has then passed on that information to you. Sounds like you are well on your way. Becky On Thu, Jan 3, 2013 at 4:11 PM, Scott Roberts <[email protected]<mailto:[email protected]>> wrote: Hi Becky, Is there a place I can find more information about your configuration at Clemson? I'm trying to get a feel for "best practices" for our storage project here at JHU. I know Omnibond is working on the OFS Administrator Guide, but I'm going to be deploying these servers in February. The storage nodes I'm evaluating are 1U boxes which can hold (12) 3.5" drives and (2) 2.5" drives and has one or two 10Gb SFP+ ports. I'm planning to use RAID1 SSDs for the O/S and OFS Metadata, and RAID10 for the data drives. Thanks! --Scott From: Becky Ligon <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday, January 2, 2013 2:23 PM To: Elaine Quarles <[email protected]<mailto:[email protected]>> Cc: Scott Roberts <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [Pvfs2-users] How do I back up the metadata? Scott: In addition to scheduled backups, here at Clemson University, we have the metadata mirrored (RAID-1). So, for hardware failures, we have the mirrors. For software failures (or corruption), we have the backups. We run backups every 4 hours and keep the backups for a short period of time. Becky On Wed, Jan 2, 2013 at 12:37 PM, Elaine Quarles <[email protected]<mailto:[email protected]>> wrote: Scott, Having scheduled times for the backup would be great. You can also make copies while the file system is running; you are just not guaranteed that all the files will be accessible after a restore. Depending on your use case, the hot backup may or may not be a desirable thing. -- Elaine -----Original Message----- From: Scott Roberts [mailto:[email protected]<mailto:[email protected]>] Sent: Monday, December 31, 2012 9:24 AM To: Elaine Quarles Cc: [email protected]<mailto:[email protected]> Subject: Re: [Pvfs2-users] How do I back up the metadata? Thanks, Elaine. I suppose I'll need to schedule periodic maintenance with the end users so I can take the filesystem offline for backups. At least, until the next version of OrangeFS comes out :-) Happy New Years, --Scott On 12/27/12 2:22 PM, "Elaine Quarles" <[email protected]<mailto:[email protected]>> wrote: > > > >Hi Scott, > >In order to back up the metadata you need to copy the Berkeley db files. >You can just zip the directory where they reside. Note that it is best >to have the file system quiesced during the copy. > >-- Elaine > >On Thu, Dec 27, 2012 at 8:57 AM, Scott Roberts ><[email protected]<mailto:[email protected]>> wrote: > >Good morning all, > >I'd like to know how to safely back up my metadata. I've looked >through the pvfs2-users archive and while I can find numerous folks >mentioning it's a good thing to do, I don't see a set of instructions >on how to actually do it. I suppose this might be a topic in the >upcoming OrangeFS Administrator Manual? > >Thanks in advance, and Happy Holidays! > >--Scott > > >_______________________________________________ >Pvfs2-users mailing list >[email protected]<mailto:[email protected]> >http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users > > > > _______________________________________________ Pvfs2-users mailing list [email protected]<mailto:[email protected]> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina -- Becky Ligon OrangeFS Support and Development Omnibond Systems Anderson, South Carolina
_______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
