Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
On Thu, 30 Apr 2009 10:52:42 +0600, Mike Kazantsev wrote: So I thought there's gotta be something that fits these criterias, but so far I've found only dar and it seems quite slow and a bit unsuited for these needs. What backup medium are you using? If hard disks, do you have a separate machine for storing them? If so, BackupPC may suit your needs. The ebuild in Portage is out of date but the latest version is available from Bugzilla - http://bugs.gentoo.org/show_bug.cgi?id=141018 It allows restoration of individual files or directories, from any backup point, and doesn't require any special software on the machines being backed up, only SSH access. It can backup Windows and *nix machines. -- Neil Bothwick Waiter! There's a fly in my qagh! signature.asc Description: PGP signature
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
On Thu, 30 Apr 2009 08:25:38 +0100 Neil Bothwick n...@digimed.co.uk wrote: What backup medium are you using? Oh my, I've managed to forget about it! The medium is regular sata2 hard drives with ext3 filesystem on a dedicated backup machine with quite rusty debian (etch) linux. Most backed-up systems (that I care about) are actually freebsd 6, the rest are linux. Most stored backups are 20-60 GB. Main bottleneck here is the network - quite laggy 100 Mbps link, because this backup server is quite far and isolated from the rest. Also it's completely inaccessible from backed-up machines, aside from reverse tunnels, which I rarely use as a dirty hacks. And this link has tendency to go down every once in a while, interrupting ongoing transfers. That said, nightly backup should always be available, so the backups are actually created on hotswap sata2 drives of each individual machine and grabbed by backup server over ssh and, in some cases, nfs. These days the scripts on the backup server quite frequently (10-50 times a day) connect to the other hosts and receive requests for certain paths from stored backups. So they parse backups with tar picking out given paths and pushes them back, as requested. Needless to say, it is slow, hence the need. Well, that's probably a bit more verbose than necessary, but the point is that (I believe) the backups should be created right on backed-up systems' hard disks, so: 1. Have random access to backup storage. 2. Prolonged io/cpu load is a bad thing. 3. Compression (at least of gzip ratio) is a must, because of limited storage on backed-up machines. 4. In-backup seek times should be lower than tar (which is scanning the whole file). 5. I write py scripts for a living, so the question is really in a backup format - transfer and storage structure is not the issue. Easiest thing I've thought of is just to generate tar index on first archive pass and then just skip to the recorded point in ungzipped stream, feeding the rest to tar, stopping when necessary, but there's no point to debug and maintain this system if there are better solutions already. If hard disks, do you have a separate machine for storing them? If so, BackupPC may suit your needs. The ebuild in Portage is out of date but the latest version is available from Bugzilla - http://bugs.gentoo.org/show_bug.cgi?id=141018 It allows restoration of individual files or directories, from any backup point, and doesn't require any special software on the machines being backed up, only SSH access. It can backup Windows and *nix machines. Thanks, will check it out, but I'm afraid that live network backups aren't the best solution in my case. -- Mike Kazantsev // fraggod.net signature.asc Description: PGP signature
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
Mike Kazantsev mike_kazant...@fraggod.net wrote: It seems that tar/gzip/bzip2 are almost universal solutions for unix-like system backups and we're using tar/gz combo to create backups from the dawn of times. But as the time goes by I stumble upon two misfits of such a combination more and more: People on Linux who use the term tar are usually not talking about tar bug about gtar which is not 100% tar compatible and thus creates problems with archive interchange. So I thought there's gotta be something that fits these criterias, but so far I've found only dar and it seems quite slow and a bit unsuited for these needs. dar is using a nonstandard and proprietary archive format. Did you look at star? Star is the oldest free tar implementation. It is 100% compatible to the standard and allows you to do incremental backups based on the POSIX.1-2001 archive format. Star includes support for all additional meta data. Any at least POSIX.1-2001 compliant archiver is able to read the archives written by star and if you are ever need to restore a star based backup with a different program, you only loose the ability to do incremental restores that deal with renamed/removed files. ftp://ftp.berlios.de/pub/star/ The latest development source is in the schily source bundle at: ftp://ftp.berlios.de/pub/schily/ Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
Neil Bothwick n...@digimed.co.uk wrote: BackupPC should cope with this. It uses rsync over SSH, so only needs to transfer new/changed files, and will restart where it left off if the connection fails (this happens to me sometimes when I switch off my laptop while it is backing up and the backup just restarts the next morning). What rsync does can also be done by star by running something like: umount /mnt fssnap -F ufs -d /export/nfs rm /export/home/EXPORT-NFS.snap sync sleep 10 echo /tmp/S.$$ svcadm disable svc:/network/nfs/server mount -r `fssnap -F ufs -o backing-store=/export/home/EXPORT-NFS.snap /export/nfs` /mnt svcadm enable svc:/network/nfs/server star bs=1m fs=256m -c -xdev -sparse -acl -link-dirs level=1 -cumulative dumpdate=/tmp/S.$$ fs-name=/export/nfs - wtardumps tardumps=/etc/td-copy -C /mnt . | \ star bs=1m fs=256m -xpU -no-fsync -restore -time -C /export/nfs2 star supports enhanced file meta data, what does rsync? Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
* Joerg Schilling (joerg.schill...@fokus.fraunhofer.de) [30.04.09 12:31]: Neil Bothwick n...@digimed.co.uk wrote: BackupPC should cope with this. It uses rsync over SSH, so only needs to transfer new/changed files, and will restart where it left off if the connection fails (this happens to me sometimes when I switch off my laptop while it is backing up and the backup just restarts the next morning). What rsync does can also be done by star by running something like: umount /mnt fssnap -F ufs -d /export/nfs rm /export/home/EXPORT-NFS.snap sync sleep 10 echo /tmp/S.$$ svcadm disable svc:/network/nfs/server mount -r `fssnap -F ufs -o backing-store=/export/home/EXPORT-NFS.snap /export/nfs` /mnt svcadm enable svc:/network/nfs/server star bs=1m fs=256m -c -xdev -sparse -acl -link-dirs level=1 -cumulative dumpdate=/tmp/S.$$ fs-name=/export/nfs - wtardumps tardumps=/etc/td-copy -C /mnt . | \ star bs=1m fs=256m -xpU -no-fsync -restore -time -C /export/nfs2 10 lines where 1 is sufficient? Not so userfriendly, and what is the benefit? If space is not a problem the main benefit of rsync is that you have your backup simply on a filesystem, that you can mount anywhere you want it. And the plus is no programm needed to restore the data other than cp... With rsnapshot it get even easier to configure your backup. star supports enhanced file meta data, what does rsync? -A, --acls preserve ACLs (implies -p) -X, --xattrspreserve extended attributes is all file meta data I have, so it is sufficient. Jörg Sebastian -- Religion ist das Opium des Volkes. | _ ASCII ribbon campaign Karl Marx | ( ) against HTML e-mail s...@sti@N GÜNTHER | X against M$ attachments mailto:sam...@guenther-roetgen.de | / \ www.asciiribbon.org pgpJCGuuTu1bG.pgp Description: PGP signature
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
On Thu, 30 Apr 2009 14:18:33 +0600, Mike Kazantsev wrote: The medium is regular sata2 hard drives with ext3 filesystem on a dedicated backup machine with quite rusty debian (etch) linux. Most backed-up systems (that I care about) are actually freebsd 6, the rest are linux. Most stored backups are 20-60 GB. Main bottleneck here is the network - quite laggy 100 Mbps link, because this backup server is quite far and isolated from the rest. Also it's completely inaccessible from backed-up machines, aside from reverse tunnels, which I rarely use as a dirty hacks. And this link has tendency to go down every once in a while, interrupting ongoing transfers. BackupPC should cope with this. It uses rsync over SSH, so only needs to transfer new/changed files, and will restart where it left off if the connection fails (this happens to me sometimes when I switch off my laptop while it is backing up and the backup just restarts the next morning). It is also space-efficient when backing up multiple machines, it uses hard links to store only one copy of each file, no matter how many machines have the same file. The lack of network access from the clients to the server would mean you couldn't access the web interface from the client you wished to restore to, but you could do that on the backup server if necessary. -- Neil Bothwick The mechanic said I had blown a seal. I said, `Just fix the damn thing and leave my private life out of it, OK?' signature.asc Description: PGP signature
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
On Thu, 30 Apr 2009 13:23:19 +0200 Sebastian Günther sam...@guenther-roetgen.de wrote: * Joerg Schilling (joerg.schill...@fokus.fraunhofer.de) [30.04.09 12:31]: What rsync does can also be done by star by running something like: ... 10 lines where 1 is sufficient? Not so userfriendly, and what is the benefit? Most lines are used to freeze ufs as a snapshot, so they should be there with rsync, as well. Obvious benefit of a snapshot is consistency, and, luckily for me, no users are involved in this process ;) If space is not a problem the main benefit of rsync is that you have your backup simply on a filesystem, that you can mount anywhere you want it. And the plus is no programm needed to restore the data other than cp... Sync is indeed a great idea, which should save tons of time and resources, but some compression should still be necessary, even on backup server, since most content should occupy 2x-10x space when unpacked. In fact, compressed write-enabled FS with snapshot capability plus rsync is the closest thing to ideal backup as I can think of. And in fact, it's called ZFS! ;) Another sad fact is that the server is shared with a few other people, so I can't just roll freebsd7 or solaris onto it, but I guess fuse-zfs and compressed fuse filesystems should be worth a try indeed. Besides, I've got some strange idea that to make squashfs you don't really need its support in kernel... All in all, more things to test out and think over, thanks. -- Mike Kazantsev // fraggod.net signature.asc Description: PGP signature
Re: [gentoo-user] Random-access cross-platform FS_backup tool suggestions
On Thu, 30 Apr 2009 10:22:00 +0200 joerg.schill...@fokus.fraunhofer.de (Joerg Schilling) wrote: People on Linux who use the term tar are usually not talking about tar bug about gtar which is not 100% tar compatible and thus creates problems with archive interchange. In fact, I'm more used to refer to freebsd tar as 'bsdtar', treating GNU as standard ;) dar is using a nonstandard and proprietary archive format. Did you look at star? Actually that's the first thing I did, since it's closest (to tar) implementation I know, but I haven't seen there the main reason why I've decided to ditch tar - random access to files inside the archive (which is stored on random-access media). I don't have to pipe the archives sequentially, but if not for the ever-increasing demand to read the contents, I'd have actually been okay with tar. Incremental backups are certainly handy feature to have, but I'm relucant to use it, since it makes backups dependant on one another, requiring additional logic for their storage. When I think about that it seem like a great idea, as long as you bundle them together all the time, so no increment gets lost, but then my laziness and certain relucance to complicate things (so no one else will be cursing me under his breath, sorting out why it lost some data) always seem to get the upper hand :( -- Mike Kazantsev // fraggod.net signature.asc Description: PGP signature
[gentoo-user] Random-access cross-platform FS_backup tool suggestions
Hello, It seems that tar/gzip/bzip2 are almost universal solutions for unix-like system backups and we're using tar/gz combo to create backups from the dawn of times. But as the time goes by I stumble upon two misfits of such a combination more and more: 1. It's quite inpractical to keep tens of tarballs for one backup. 2. Seeking within single tens-hundreds-of-gigs tarball is suboptimal, at least. 3. Single thread operation. At home, being a gentoo-only user (w/ gentoo-patched kernels), I've solved the problem with squashfs - it keeps all the necessary attributes, hardlinks, boasts multi-threaded creation and instant access to any file within. Alas, I can't use it on a production servers due to compatibility issues - not a single linux here have support for it and changing / patching kernels is a bit of nuisance. Even worse, many systems that need to be backed-up are FreeBSD. So I thought there's gotta be something that fits these criterias, but so far I've found only dar and it seems quite slow and a bit unsuited for these needs. Any suggestions? Thanks. -- Mike Kazantsev // fraggod.net signature.asc Description: PGP signature
