I'm able to backup several systems with 100's of millions of files with the diskcache method.
If your backup is running that slow on a gpfs filesystem, you probably have a performance problem with the filesystem. Can you give a quick over view of the gpfs environment? ie sata, sas, or are they fc attached? in what topology? what interconnect on the nodes? ---- Daniel Murphy-Olson Systems Administrator Mathematics & Computer Science Division Argonne National Laboratory 630-252-0055 ----- Original Message ----- From: "Howard Coles" <[email protected]> To: [email protected] Sent: Thursday, February 2, 2012 11:11:07 AM Subject: Re: [ADSM-L] million files backup Have you tried using the memoryefficient=diskcache method? I'm assuming you have. Takes a while, but on fast systems I've had pretty decent results. You may need to go the virtual mount point route though with a file system that large. See Ya' Howard Coles Jr. John 3:16! -----Original Message----- From: ADSM: Dist Stor Manager [mailto:[email protected]] On Behalf Of Jorge Amil Sent: Thursday, February 02, 2012 9:06 AM To: [email protected] Subject: Re: [ADSM-L] million files backup Hi Jim, thank you very much for your answer. Actually we are doing what you say. Filesystem .tar. It was a great solution when the filesystem was 500Gb-1Tb but actually our filesystem is 14Tb. The tar/gzip command took 10-12 days... :( So we need another aproach Thanks Jorge > Date: Thu, 2 Feb 2012 08:47:24 -0600 > From: [email protected] > Subject: Re: [ADSM-L] million files backup > To: [email protected] > > Jorge, > > On Unix systems: > I've done it in two steps. Create a tar file of the file system and zip > it. Create a second file listing all the files in the tarred directory. > The tar extract command allows single files to be recalled if the > absolute path name is available. > > I've used this to backup a Sterling Commerce flat file database with > multiple subdirectories holding more than 1.5 million files. The > tar/gzip command took 5 or 6 hours for a 500 GB file system. > > I have not tried to do this on a Windows system. > > Jim Schneider > > -----Original Message----- > From: ADSM: Dist Stor Manager [mailto:[email protected]] On Behalf Of > Jorge Amil > Sent: Thursday, February 02, 2012 8:30 AM > To: [email protected] > Subject: [ADSM-L] million files backup > > Hi everybody, > > Does anyone know what is the best way to make a filesystem backup than > contains million files? > > Backup image is not posible because is a GPFS filesystem and is not > supported. > > Thanks in advance > > Jorge > DISCLAIMER: This communication, along with any documents, files or attachments, is intended only for the use of the addressee and may contain legally privileged and confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of any information contained in or attached to this communication is strictly prohibited. If you have received this message in error, please notify the sender immediately and destroy the original communication and its attachments without reading, printing or saving in any manner. Please consider the environment before printing this e-mail.
