[BackupPC-users] Copying the pool / filesystem migration
Greetings, I'm in the process of migrating to a new BackupPC server, my old machine having software RAID5 on SATA; it was just getting a bit outdated and more than a bit starved for IOPS. The new machine (a VM, though the host is dedicated) is RAID10 SAS MDL (7.2krpm) across 8 spindles on a P410/512MB FBWC - a far superior build for IOPS. The old machine is using ext3 on its' filesystem, as it was a direct filesystem move from the machine before that (which was CentOS6). So, I'm stuck with ext3 on slow hardware; trying to move to xfs on the new faster hardware. Getting the data to the new machine is easy enough - I've done it twice already; once with an intermediary disk physically moving it between machines, and once over the network. The network is just as fast as a physical disk, as the decrease in speed still outweighs having to copy the data twice. The real problem I have is in converting the ext3 filesystem to xfs. I've staged the copy as two different disks in the guest, one containing the ext3 filesystem (which i can later dispose of), and one containing the xfs filesystem. Using rsync -aH, the copy went to about 950/1200GB then slowed to a crawl, getting perhaps 2-4GB per day; because it's in the hardlink territory of the backuppc store. I tried using the BackupPC_tarPCCopy instead of rsync, but the command refused to work. It stated an error about the pool root configuration, even though the configuration was correct. I was unable to resolve the error. What strategies or suggestions could the community make? At this rate, it's going to take another THREE MONTHS to copy the pool between filesystems, a time during which this server isn't making backups. The old server is, but at the end of it all i'm faced with trying to merge the pools (probably functionally impossible given the performance issues), or having a substantial gap in my backups. Neither option is appealing. I'm OK with a gap in backups, but I'd like to contain it to a week or two, not an entire quarter. -- --- - Nick Bright- - Vice President of Technology - - Valnet -=- We Connect You -=- - - Tel 888-332-1616 x 315 / Fax 620-331-0789 - - Web http://www.valnet.net/ - --- - Are your files safe?- - Valnet Vault - Secure Cloud Backup - - More information & 30 day free trial at - - http://www.valnet.net/services/valnet-vault - --- This email message and any attachments are intended solely for the use of the addressees hereof. This message and any attachments may contain information that is confidential, privileged and exempt from disclosure under applicable law. If you are not the intended recipient of this message, you are prohibited from reading, disclosing, reproducing, distributing, disseminating or otherwise using this transmission. If you have received this message in error, please promptly notify the sender by reply E-mail and immediately delete this message from your system. -- Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] copying the pool to a new filesystem?!
Hello, I'm a happy user of BackupPC since a few years, running an old installation of backuppc that was created on some version of SuSE linux, then ported over to debian lenny. The pool is a reiserfs3 on LVM, about 300GB size, but with a lot of hardlinks... Now I'm trying to put the pool onto a new filesystem, so I created an XFS on a striped RAID0 of 3 disks (to speed up copying), and use rsync -aHv to copy everything including the hardlinks. The cpool itself took about a day, and now it is running for 6 days and maybe it has done 70% of the work. BTW, a copy with dd takes about 2 hours. I've tried to do this with BackupPC_TarPCCopy, but it does not seem to be any faster. Anybody else has seen this? Is it generally useful to move the pool to xfs, or should I just stay with the old filesystem? Anyway, keep up the good work... Thanks, Oliver -- Dr. Oliver Freyd Software Developer Tel: +49 251 1622 231 Fax: +49 251 1622 199 email: oliver.fr...@iontof.com ION-TOF GmbH Heisenbergstr. 15 48149 Muenster Germany www.iontof.com Registergericht: Amtsgericht Muenster, HRB 3077 Geschaeftsfuehrer: Prof. A. Benninghoven T. Heller Dr. E. Niehuis -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] copying the pool to a new filesystem?!
On Thu, 2010-12-02 at 13:27 +0100, Oliver Freyd wrote: The pool is a reiserfs3 on LVM, about 300GB size, but with a lot of hardlinks... Now I'm trying to put the pool onto a new filesystem, so I created an XFS on a striped RAID0 of 3 disks (to speed up copying), and use rsync -aHv to copy everything including the hardlinks. The cpool itself took about a day, and now it is running for 6 days and maybe it has done 70% of the work. BTW, a copy with dd takes about 2 hours. That span of time is about my experience. I estimated it would take 10 days to move my 1 TB ext3 filesystem to ext4 (with extents), with rsync. It takes 13 hours with dd over the gigabit LAN. Regards, Tyler -- Cheops' Law: Nothing ever gets built on schedule or within budget. -- Lazarus Long, Time Enough for Love, by Robert A. Heinlein -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] copying the pool to a new filesystem?!
On 12/02 01:27 , Oliver Freyd wrote: The pool is a reiserfs3 on LVM, about 300GB size, but with a lot of hardlinks... Now I'm trying to put the pool onto a new filesystem, so I created an XFS on a striped RAID0 of 3 disks (to speed up copying), and use rsync -aHv to copy everything including the hardlinks. The cpool itself took about a day, and now it is running for 6 days and maybe it has done 70% of the work. BTW, a copy with dd takes about 2 hours. Yep. this is normal. you've discovered why backing up the BackupPC data pool with rsync is a non-scaleable solution. As you noted, dd is the way to go if at all possible. If you want to change the filesystem (i.e. make a file-level copy); best thing is to use tar. Presuming that your old data pool is under /var/lib/backuppc and the new filesystem is mounted on /mnt/newfilesystem: cd /var/lib/backuppc; tar cpf - . | tar xpv -C /mnt/newfilesystem Is the use of RAID0 on 3 disks a temporary thing? Keep in mind that the more disks you have the greater the likelihood that one of them will die. With RAID0 if one dies you lose all your data. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] copying the pool to a new filesystem?!
Am 02.12.2010 14:05, schrieb Carl Wilhelm Soderstrom: On 12/02 01:27 , Oliver Freyd wrote: The pool is a reiserfs3 on LVM, about 300GB size, but with a lot of hardlinks... Now I'm trying to put the pool onto a new filesystem, so I created an XFS on a striped RAID0 of 3 disks (to speed up copying), and use rsync -aHv to copy everything including the hardlinks. The cpool itself took about a day, and now it is running for 6 days and maybe it has done 70% of the work. BTW, a copy with dd takes about 2 hours. Yep. this is normal. you've discovered why backing up the BackupPC data pool with rsync is a non-scaleable solution. As you noted, dd is the way to go if at all possible. If you want to change the filesystem (i.e. make a file-level copy); best thing is to use tar. Presuming that your old data pool is under /var/lib/backuppc and the new filesystem is mounted on /mnt/newfilesystem: cd /var/lib/backuppc; tar cpf - . | tar xpv -C /mnt/newfilesystem Is the use of RAID0 on 3 disks a temporary thing? Keep in mind that the more disks you have the greater the likelihood that one of them will die. With RAID0 if one dies you lose all your data. Thank you for the quick answer, I knew that rsync would need a lot of memory to keep track of the hardlinks, but that it would be that slow... Maybe I'll try the tar approach, for now I'll just let it run on. The RAID0 is just temporary, I tried the rsync before onto a single disk, and that was even slower, but definitely the gain is not much, especially in the hardlink-copying phase. When the rsync has succeeded, I'll dd the pool back into an LVM volume, which lives on a RAID1 (2 disks mirrored). Regards, Oliver -- Dr. Oliver Freyd Software Developer Tel: +49 251 1622 231 Fax: +49 251 1622 199 email: oliver.fr...@iontof.com ION-TOF GmbH Heisenbergstr. 15 48149 Muenster Germany www.iontof.com Registergericht: Amtsgericht Muenster, HRB 3077 Geschaeftsfuehrer: Prof. A. Benninghoven T. Heller Dr. E. Niehuis -- Increase Visibility of Your 3D Game App Earn a Chance To Win $500! Tap into the largest installed PC base get more eyes on your game by optimizing for Intel(R) Graphics Technology. Get started today with the Intel(R) Software Partner Program. Five $500 cash prizes are up for grabs. http://p.sf.net/sfu/intelisp-dev2dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] copying the pool
I recently copied the pool to a new hard disk following the Copying the pool instructions from the main documentation. The documentation says to copy the 'cpool', 'log', and 'conf' directories using any technique and the 'pc' directory using BackupPC_tarPCCopy; however, there is no mention of what to do with the 'pool' directory. I thought it might be created automatically when the nightly cleanup runs, but three days later and still no 'pool' directory. Is this an oversight in the documentation or is the 'pool' directory not needed? I am using BackupPC 3.1.0. Thanks. -- Chris Purves I've seen the look of a fat man having dinner. - Frank Sinatra -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] copying the pool
On Mon, Oct 04, 2010 at 08:56:49AM -0400, Chris Purves wrote: I recently copied the pool to a new hard disk following the Copying the pool instructions from the main documentation. The documentation says to copy the 'cpool', 'log', and 'conf' directories using any technique and the 'pc' directory using BackupPC_tarPCCopy; however, there is no mention of what to do with the 'pool' directory. I thought it might be created automatically when the nightly cleanup runs, but three days later and still no 'pool' directory. Is this an oversight in the documentation or is the 'pool' directory not needed? I am using BackupPC 3.1.0. Unless you have compression turned off, the pool directory should be totally empty. If you have compression turned off, the cpool directory should be totally empty. Whichever one is empty can be ignored. -Robin -- http://singinst.org/ : Our last, best hope for a fantastic future. Lojban (http://www.lojban.org/): The language in which this parrot is dead is ti poi spitaki cu morsi, but this sentence is false is na nei. My personal page: http://www.digitalkingdom.org/rlp/ -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] copying the pool
On 04/10/2010 2:20 PM, Robin Lee Powell wrote: On Mon, Oct 04, 2010 at 08:56:49AM -0400, Chris Purves wrote: I recently copied the pool to a new hard disk following the Copying the pool instructions from the main documentation. The documentation says to copy the 'cpool', 'log', and 'conf' directories using any technique and the 'pc' directory using BackupPC_tarPCCopy; however, there is no mention of what to do with the 'pool' directory. I thought it might be created automatically when the nightly cleanup runs, but three days later and still no 'pool' directory. Is this an oversight in the documentation or is the 'pool' directory not needed? I am using BackupPC 3.1.0. Unless you have compression turned off, the pool directory should be totally empty. If you have compression turned off, the cpool directory should be totally empty. Whichever one is empty can be ignored. So it is. I reconnected and mounted the old drive and the pool directory is indeed empty. Thanks for your reply. -- Chris Purves What we observe is not nature itself, but nature exposed to our method of questioning. - Werner Heisenberg -- Virtualization is moving to the mainstream and overtaking non-virtualized environment for deploying applications. Does it make network security easier or more difficult to achieve? Read this whitepaper to separate the two and get a better understanding. http://p.sf.net/sfu/hp-phase2-d2d ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
