snackup (was: Re: [gentoo-user] backup to a cold-swap drive)
Iain Buchanan writes: On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote: using this script, adapted to their needs, I started to rewrite it in a way that it reads a config file, and no modification of the script itself is necessary. If anyone is interested, send me an email. interested! So is it on sourceforge yet ;) Sorry, it took a little longer. I was ill, and then these scripts tend to grow and grow until I am satisfied with them. It's still not perfect, but I think at the moment it does what it should do. Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup The name is silly, but I kinda like it now. A backup utilizing LVM snapshots... snapshot backup... snackup! It needs a config file that is looked up in some default locations, or can be specified with the -c option. Use option -T to generate a template that has information on the syntax and some examples. In short: The config file defines targets you give as arguments to snackup. So 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz file of your /etc. The config file is sourced, so you can put any bash stuff you want into there, like some shell functions you want to have executed. For example, I want my /var/log/portage/*.log files to be compressed before I backup my /var partition. A target is started by a colon, followed by name, type, source and destination (and optionally more), separated by one or more tab characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only tested dd, tar and rdiff so far, though. The source directory may be prefixed by the volume group and LVM of that partition in order to create an lvm snapshot first. You may add a LUKS key if the partition is encrypted. The source may contain wildcards. In that case, all matching files are backed up to the destination. If the type has a * appended, all matching files are backed up individually. Some examples: : tar kde .kde* /backup/kde.%s Target 'kde' will backup all your .kde* directories to /backup/kde.tar (the %s is replaced by a suffix, usually tar). You need to be in your home directory, otherwise the path will not be found. This is not necessary if you use ~//.kde* instead. : kernels tar*/usr/src//linux-* /backup/src/%f.%s -j - lsf Target 'kernels' will backup your /usr/src/linux-* directories individually - one tar file for each (note the * appendended to the tar target type). The -z option will be passed to the tar command, and the suffix is changed to tgz accordingly. The // denotes that the backup should be done locally from the directory left to the //. Otherwise, the full path would be included in the tar file. : tar etc /etc/backup/etc_%d.%s -j Target 'etc' will tar your /etc directory, compressed with bzip2. The %d will be replaced by the current date. You can change the date format by re-defining a date() shell function. You will get a message about tar removing the leading /, you can avoid it by replacing /etc by ///etc, meaning that your local directroy will be /. : bootdd /dev/sda5 /dev/sdb5 bs=32M Target 'boot' will backup a boot partition with dd. Options like the bs=32M are added directly to the dd command. : homersync system/home::. /backup/home/ Target 'home' will create a LVM snapshot of the /dev/system/home logical volume, mount it and back it up with rdiff-backup. The '.' could also be a / or left off. : var rsync system/var::/ /backup/var/- ziplog Similar, but the command ziplog will be executed before the backup, compressing some log files. It must be defined in the config file. The script has some options: -c file location of config file -C clear destination first -d dummy mode, just show what would be done -f force backup (the initial rdiff-backup may need this) -h show this help -l output to log file, too -L log file (default:/home/wonko/log/snackup.log); may be a directory (add a trailing slash) to create target-specific logs -n luse nice level l (default:10, 0 to turn off) -N luse ionice level l (default:3, 0 to turn off) -o extra options that will added to the actual backup command -s size size of LVM snapshot (default:2G) -S char replace tab as delimiter for targets in config file -T output config file template -v verbose output; may be given multiple times 1: some extra output; 2: add -v to commands; 3: set -xv -V output version information and exit Most important are: -c to specify the location of the config file -d to output what would be done, you will see the final tar / rdiff- backup / whatever command with all its options. -T to see a template. Save it with snackup
Re: snackup (was: Re: [gentoo-user] backup to a cold-swap drive)
Alex Schuster wo...@wonkology.org wrote: Iain Buchanan writes: On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote: using this script, adapted to their needs, I started to rewrite it in a way that it reads a config file, and no modification of the script itself is necessary. If anyone is interested, send me an email. interested! So is it on sourceforge yet ;) Sorry, it took a little longer. I was ill, and then these scripts tend to grow and grow until I am satisfied with them. It's still not perfect, but I think at the moment it does what it should do. Still not on sourceforge, but here: http://www.wonkology.org/utils/snackup The name is silly, but I kinda like it now. A backup utilizing LVM snapshots... snapshot backup... snackup! It needs a config file that is looked up in some default locations, or can be specified with the -c option. Use option -T to generate a template that has information on the syntax and some examples. In short: The config file defines targets you give as arguments to snackup. So 'snackup home' backs up your /home partition, 'snackup etc' creates a .tgz file of your /etc. The config file is sourced, so you can put any bash stuff you want into there, like some shell functions you want to have executed. For example, I want my /var/log/portage/*.log files to be compressed before I backup my /var partition. A target is started by a colon, followed by name, type, source and destination (and optionally more), separated by one or more tab characters. Type can be 'cp', 'dd', 'tar', 'rsync' or 'rdiff'. I only tested dd, tar and rdiff so far, though. The source directory may be prefixed by the volume group and LVM of that partition in order to create an lvm snapshot first. You may add a LUKS key if the partition is encrypted. The source may contain wildcards. In that case, all matching files are backed up to the destination. If the type has a * appended, all matching files are backed up individually. Some examples: : tar kde .kde* /backup/kde.%s Target 'kde' will backup all your .kde* directories to /backup/kde.tar (the %s is replaced by a suffix, usually tar). You need to be in your home directory, otherwise the path will not be found. This is not necessary if you use ~//.kde* instead. : kernels tar*/usr/src//linux-* /backup/src/%f.%s -j - lsf Target 'kernels' will backup your /usr/src/linux-* directories individually - one tar file for each (note the * appendended to the tar target type). The -z option will be passed to the tar command, and the suffix is changed to tgz accordingly. The // denotes that the backup should be done locally from the directory left to the //. Otherwise, the full path would be included in the tar file. : tar etc /etc/backup/etc_%d.%s -j Target 'etc' will tar your /etc directory, compressed with bzip2. The %d will be replaced by the current date. You can change the date format by re-defining a date() shell function. You will get a message about tar removing the leading /, you can avoid it by replacing /etc by ///etc, meaning that your local directroy will be /. : bootdd /dev/sda5 /dev/sdb5 bs=32M Target 'boot' will backup a boot partition with dd. Options like the bs=32M are added directly to the dd command. : homersync system/home::. /backup/home/ Target 'home' will create a LVM snapshot of the /dev/system/home logical volume, mount it and back it up with rdiff-backup. The '.' could also be a / or left off. : var rsync system/var::/ /backup/var/- ziplog Similar, but the command ziplog will be executed before the backup, compressing some log files. It must be defined in the config file. The script has some options: -c file location of config file -C clear destination first -d dummy mode, just show what would be done -f force backup (the initial rdiff-backup may need this) -h show this help -l output to log file, too -L log file (default:/home/wonko/log/snackup.log); may be a directory (add a trailing slash) to create target-specific logs -n luse nice level l (default:10, 0 to turn off) -N luse ionice level l (default:3, 0 to turn off) -o extra options that will added to the actual backup command -s size size of LVM snapshot (default:2G) -S char replace tab as delimiter for targets in config file -T output config file template -v verbose output; may be given multiple times 1: some extra output; 2: add -v to commands; 3: set -xv -V output version information and exit Most important are: -c to specify the location of the config file -d to output what would be
Re: [gentoo-user] backup to a cold-swap drive
Iain Buchanan writes: On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote: [snip] All my partitions are LVM volumes, so before the backup starts, I make a LVM snapshot of the partition. This way I can modify it while the backup is still in progress. hmm, never got into LVM. Sounds interesting though... It is. it gives you great flexibility. Your /home is becoming too small? So just add a Gigabyte: lvresize -L +1G /dev/system/home resize2fs /dev/system/home Takes half a minute, and you don't need to unmount anything before that. You want to backup your large /home partition, and want to continue working meanwhile? Create a snapshot, mount it, and backup this: lvcreate -s -L 1G -n home-snapshot /dev/system/home mount /dev/system/home-snapshot /mnt [snip] I wrote a shell script to do this, so I do not have to issue a lot of commands every time I want to do the backup. I don't use too many commands, something like this in /etc/cron.daily/custom-backup: sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file --delete-delay --delete-excluded --partial --human-readable / /unique-mount-of-external-drive || echo external backup failed! That's okay, I did it in this way, too. Well, the backup may be inconsistent when you are modifying the root partition in the meantime, so I would not emerge things when the backup is running. But now I am using LVM for everything, and so I just create a snapshot. And I prefer to use rdiff-backup, so I can have incremental backups. As there are now some others using this script, adapted to their needs, I started to rewrite it in a way that it reads a config file, and no modification of the script itself is necessary. If anyone is interested, send me an email. interested! So is it on sourceforge yet ;) Um, no... not yet. Not really sure if it belongs there... The script has become a little large, but I think it's working now. I will do some testing, and inform you when it's done. I'm a little ill at the moment, so it may take 1-2 more days. Wonko
Re: [gentoo-user] backup to a cold-swap drive
On Thu, 2010-04-29 at 16:44 +0200, Alex Schuster wrote: [snip] All my partitions are LVM volumes, so before the backup starts, I make a LVM snapshot of the partition. This way I can modify it while the backup is still in progress. hmm, never got into LVM. Sounds interesting though... [snip] I wrote a shell script to do this, so I do not have to issue a lot of commands every time I want to do the backup. I don't use too many commands, something like this in /etc/cron.daily/custom-backup: sudo /usr/bin/ionice -c 3 /usr/bin/rsync -aAx --exclude suspend_file --delete-delay --delete-excluded --partial --human-readable / /unique-mount-of-external-drive || echo external backup failed! As there are now some others using this script, adapted to their needs, I started to rewrite it in a way that it reads a config file, and no modification of the script itself is necessary. If anyone is interested, send me an email. interested! So is it on sourceforge yet ;) thanks, -- Iain Buchanan iaindb at netspace dot net dot au BOFH Excuse #418: Sysadmins busy fighting SPAM.
Re: [gentoo-user] backup to a cold-swap drive
On Fri, 2010-04-30 at 16:24 +0200, Florian Philipp wrote: Am 29.04.2010 02:38, schrieb Iain Buchanan: Hi thanks, On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote: [...] If you can live with just one big partition as a backup (probably with separate /boot), you should replace fstab and grub.conf on the backup medium and blacklist them from the files which you want to back up. why wouldn't I backup fstab and grub.conf as well? If my internal disk dies, I assume I'll swap them over, meaning grub and fstab will have to be the same. I think you misunderstood me or I didn't explain it correctly. I try it again: [snip] ah, NOW I get it :) [snip] Ah, I see what you mean. I've never worked with the file alteration monitor (FAM) but once evaluated inotify for some administrative purposes. AFAIK they are not scalable good enough to work on a system wide basis. For example, I think the default limit of observable files with inotify is 8192. hm, there goes that idea! Is there any kernel based watch on all file based I/O that I could queue up somehow? Just thinking aloud here. I know everything is a file, but no doubt I could watch all write operations; filter out /dev and put the rest into a file; and then use it like an rsync file-list... thanks for the tips :) rsync will at least get me going quickly. Yesterday I tried iotop to with dd - some slowness but otherwise quite nice. To reduce the performance impact, you can also use the ionice command. whoops, that's what I meant. Even with ionice, there was some noticeable delay when switching screens, opening programs, etc. More so when my RAM had been swapped from the large amount of I/O (I assume). I didn't try ionice with nice. thanks, -- Iain Buchanan iaindb at netspace dot net dot au One family builds a wall, two families enjoy it.
Re: [gentoo-user] backup to a cold-swap drive
Am 29.04.2010 02:38, schrieb Iain Buchanan: Hi thanks, On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote: [...] If you can live with just one big partition as a backup (probably with separate /boot), you should replace fstab and grub.conf on the backup medium and blacklist them from the files which you want to back up. why wouldn't I backup fstab and grub.conf as well? If my internal disk dies, I assume I'll swap them over, meaning grub and fstab will have to be the same. I think you misunderstood me or I didn't explain it correctly. I try it again: You probably have a lot of partitions on your internal disk; one for /, /usr and /home, for example. My reasoning was that it is probably easier to have just one big partition on your backup medium instead of many small ones. If you would then swap your backup medium in, with exactly the same fstab as in your original installation, your system would try to mount partitions which are simply not there. Therefore you need a different fstab and most likely also another grub.conf. Concerning the backup tool, I would use `rsync --delete` plus all relevant switches for permissions, times, acls, etc. If you use another tool, just make sure it doesn't put some metadata onto the backup medium and that it can delete files which no longer exist on the original medium. I was thinking of rsync, but I didn't want to do it in an hourly cron fashion, I was hoping for some gamin alteration-triggered idea. Ah, I see what you mean. I've never worked with the file alteration monitor (FAM) but once evaluated inotify for some administrative purposes. AFAIK they are not scalable good enough to work on a system wide basis. For example, I think the default limit of observable files with inotify is 8192. With regard to your requirement to just 'pull the cord' without umounting it: I wasn't thinking of pulling the chord without unmounting, I was thinking of the machine dying, hence leaving the disk in a non-shutdown state. Okay, I thought you meant the unreliable power at those weird locations you were talking about. Such a black-out or brown-out is basically the same as pulling the cord. thanks for the tips :) rsync will at least get me going quickly. Yesterday I tried iotop to with dd - some slowness but otherwise quite nice. To reduce the performance impact, you can also use the ionice command. Hope this helps, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] backup to a cold-swap drive
Iain Buchanan writes: A winblows colleague said he uses a utility to backup his internal hard drive to an external disk, such that if his internal disk fails he can replace it with the external disk and continue straight away. I do the same, but with a 2nd internal drive. The drive is partitioned similar, with some partitions being a bit larger so I can do incremental backups, too. I am using rdiff-backup, which makes use of rsync. The backup partition has exactly the same contents as the source partition, except for an additional 'rdiff-backup' directory that contains incremental backups of files that were modified from backup to backup, gzipped. Some other partitions are handled differently: /boot is just being dd'ed, contents of /usr/src are tarred each, and /var/portage/packages/ is just plain rsynced. Some unnecessary stuff like .ccache and /var/tmp/portage is excluded. All my partitions are LVM volumes, so before the backup starts, I make a LVM snapshot of the partition. This way I can modify it while the backup is still in progress. I wrote a shell script to do this, so I do not have to issue a lot of commands every time I want to do the backup. As there are now some others using this script, adapted to their needs, I started to rewrite it in a way that it reads a config file, and no modification of the script itself is necessary. If anyone is interested, send me an email. Some time ago my first drive started having bad blocks. Without LVM, I could just have swapped the disks, but so I had to rename the backup volume group to the original volume group from a live cd. And the system was running from the new drive as it was before - only that I no longer had a backup until the new drive arrived. This makes an uneasy feeling with these 1.5 TB drives. Wonko
Re: [gentoo-user] backup to a cold-swap drive
Am 28.04.2010 03:41, schrieb Iain Buchanan: Hi, A winblows colleague said he uses a utility to backup his internal hard drive to an external disk, such that if his internal disk fails he can replace it with the external disk and continue straight away. Since I go to weird locations with unreliable power and sometimes drop my laptop I thought it should be simple to do the same in Linux. I have an external disk the same size, but now what? * I want to copy changes intelligently (ie. no dd, gparted, or Ghost4Linux). * I want to copy a specific device only (no usb keys, etc) to a specific external device. * Windows partitions can be ignored. * It doesn't matter if the copy is not unmounted properly, eg. if power is shut of without shutting down. * The external disk must be able to be absent Can md use one internal and one external disk in a RAID 1 setup, with the external disk not always there? Any other suggestions? thanks :) md would be extremely slow because it has to rebuild/resync the complete array. I suggest you manually recreate the partitioning scheme, install grub, then mount it with some little script and call some backup tool to do the actual copying. If you can live with just one big partition as a backup (probably with separate /boot), you should replace fstab and grub.conf on the backup medium and blacklist them from the files which you want to back up. Concerning the backup tool, I would use `rsync --delete` plus all relevant switches for permissions, times, acls, etc. If you use another tool, just make sure it doesn't put some metadata onto the backup medium and that it can delete files which no longer exist on the original medium. With regard to your requirement to just 'pull the cord' without umounting it: Better mount it with '-o sync' to increase your chance that everything works fine afterwards. But in reality, nothing can really protect you from filesystem corruption in this situation. If you can afford, better keep two backup media which you round-robin. Hope this helps, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] backup to a cold-swap drive
Am Mittwoch, 28. April 2010 schrieb Iain Buchanan: Hi, A winblows colleague said he uses a utility to backup his internal hard drive to an external disk, such that if his internal disk fails he can replace it with the external disk and continue straight away. Since I go to weird locations with unreliable power and sometimes drop my laptop I thought it should be simple to do the same in Linux. I have an external disk the same size, but now what? * I want to copy changes intelligently (ie. no dd, gparted, or Ghost4Linux). * I want to copy a specific device only (no usb keys, etc) to a specific external device. * Windows partitions can be ignored. * It doesn't matter if the copy is not unmounted properly, eg. if power is shut of without shutting down. * The external disk must be able to be absent Can md use one internal and one external disk in a RAID 1 setup, with the external disk not always there? Any other suggestions? After I upgraded my laptop with an internal HDD of 500 GB, I started using my old external 500 GB drive as backup. Though of different dimensions and makers, they both have the same number of sectors. So I dd'ed the entire disk first, which gave me an exact mirror of the internal disk. I though this would be faster, because I have lots of small files in some places. But now I can update the backup by a simple call to rsync: rsync -aX --delete / /dev/backup root partition/ -a (archive) copies permissions, ownerships and the likes -X stops at file system boundaries, i.e. it will only backup the actual root partition, without other mounted file systems such as /proc, /dev and /home. -- Gruß | Greetings | Qapla' Why did the tachyon cross the road? Because it was on the other side. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] backup to a cold-swap drive
Hi thanks, On Wed, 2010-04-28 at 17:31 +0200, Florian Philipp wrote: Can md use one internal and one external disk in a RAID 1 setup, with the external disk not always there? Any other suggestions? thanks :) md would be extremely slow because it has to rebuild/resync the complete array. so every time you unplug and re-plug the external disk, it will essentially re-copy everything? Damn, there goes that fine idea! If you can live with just one big partition as a backup (probably with separate /boot), you should replace fstab and grub.conf on the backup medium and blacklist them from the files which you want to back up. why wouldn't I backup fstab and grub.conf as well? If my internal disk dies, I assume I'll swap them over, meaning grub and fstab will have to be the same. Concerning the backup tool, I would use `rsync --delete` plus all relevant switches for permissions, times, acls, etc. If you use another tool, just make sure it doesn't put some metadata onto the backup medium and that it can delete files which no longer exist on the original medium. I was thinking of rsync, but I didn't want to do it in an hourly cron fashion, I was hoping for some gamin alteration-triggered idea. With regard to your requirement to just 'pull the cord' without umounting it: I wasn't thinking of pulling the chord without unmounting, I was thinking of the machine dying, hence leaving the disk in a non-shutdown state. thanks for the tips :) rsync will at least get me going quickly. Yesterday I tried iotop to with dd - some slowness but otherwise quite nice. -- Iain Buchanan iaindb at netspace dot net dot au Real computer scientists don't comment their code. The identifiers are so long they can't afford the disk space.
Re: [gentoo-user] backup to a cold-swap drive
Hi, On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote: After I upgraded my laptop with an internal HDD of 500 GB, I started using my old external 500 GB drive as backup. Though of different dimensions and makers, they both have the same number of sectors. So I dd'ed the entire disk first, which gave me an exact mirror of the internal disk. I though this would be faster, because I have lots of small files in some places. I tried that, but after dd finished, I was left with strange partitions and id's that I couldn't mount. The two disks are both 160Gb with the same sector size... It might be easier to do the fdisk-ing by hand. thanks, -- Iain Buchanan iaindb at netspace dot net dot au Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves. -- William Pitt, 1783
Re: [gentoo-user] backup to a cold-swap drive
On Wed, 2010-04-28 at 23:16 +0200, Frank Steinmetzger wrote: rsync -aX --delete / /dev/backup root partition/ -a (archive) copies permissions, ownerships and the likes -X stops at file system boundaries, i.e. it will only backup the actual root partition, without other mounted file systems such as /proc, /dev and /home. actually, lower case x is --one-file-system or don't cross filesystem boundaries. Upper case X is --xattrs or preserve extended attributes :) -- Iain Buchanan iaindb at netspace dot net dot au When you don't know what to do, walk fast and look worried.
[gentoo-user] backup to a cold-swap drive
Hi, A winblows colleague said he uses a utility to backup his internal hard drive to an external disk, such that if his internal disk fails he can replace it with the external disk and continue straight away. Since I go to weird locations with unreliable power and sometimes drop my laptop I thought it should be simple to do the same in Linux. I have an external disk the same size, but now what? * I want to copy changes intelligently (ie. no dd, gparted, or Ghost4Linux). * I want to copy a specific device only (no usb keys, etc) to a specific external device. * Windows partitions can be ignored. * It doesn't matter if the copy is not unmounted properly, eg. if power is shut of without shutting down. * The external disk must be able to be absent Can md use one internal and one external disk in a RAID 1 setup, with the external disk not always there? Any other suggestions? thanks :) -- Iain Buchanan iaindb at netspace dot net dot au Better tried by twelve than carried by six. -- Jeff Cooper
