Yeah I know, there's more I need to do to optimize. That script is probably nearing ten years old. It's been running without a single problem so I never bothered to revisit it, even after updating machines, I just copy it over and keep in trucking.
On Tue, Jan 22, 2013 at 4:14 PM, Kevin Korb <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Also, if you put dates and times in the file names instead of .01, > .02, etc you don't have to do any mv's, you can easily tell when each > backup was run, and ls can tell you which the newest and oldest are. > > On 01/22/13 18:12, Kevin Korb wrote: > > That is the old way that pre-dates --link-dest. Instead of cp -al > > daily.02 daily.01 you can do a mkdir daily.01 then an rsync ... > > --link-dest=../daily.02 daily.01 > > > > Rsync then doesn't need any --delete and you don't bother making > > any hard links that will eventually be replaced. Plus the linking > > happens while rsync is running so it is usually much faster. > > > > On 01/22/13 18:10, Ashley M. Kirchner wrote: > >> Joe, this is specific to having a backup with rsync. The way I > >> use links for rsync is by not using the link (ln) command at > >> all, but instead using cp's build-in -l (link) option. It looks > >> something like this: > > > >> 1) delete the oldest backup (simple 'rm' command) 2) shift the > >> rest (with 'rm') by 1, so if I have a 7 day daily backup, it > >> moves daily.06 -> 07, daily.05 -> 06, etc., etc. This > >> effectively removed daily.01 as that is now .02. 3) make a linked > >> copy of .02 back into .01, so cp -al daily.02 daily.01 - this > >> links only, it doesn't physically copy anything. 4) run rsync > >> against that .01 backup. I run rsync with --delete and > >> --delete-excluded options in case something got removed from the > >> live system, it gets deleted, but only from this newest backup. > >> If I need to recover it 5 days from now, I still can by looking > >> in daily.05. > > > >> This setup works well in that only one folder will have actual > >> physical files (and thus the largest of them all.) The rest of > >> them are links back to it and are very small. The neat thing > >> is, when a file gets deleted from the newest backup, .01, it > >> breaks the link from the others, and the physical file remains in > >> the next older backup, in this case .02, (till it "expires" out > >> of the cycle.) > > > >> I have 12 servers backing up gigabytes worth of data every night > >> like this, maintaining anywhere between 3 to 6 weeks worth for > >> each one. They're all going to a single 1 terabyte drive. > > > >> A > > > > > > > >> On Tue, Jan 22, 2013 at 3:31 PM, Franois <[email protected] > >> <mailto:[email protected]>> wrote: > > > >> Hi Joe, > > > >> If you want to understand hard-links, just take a look at > >> Wikipedia : http://en.wikipedia.org/wiki/Hard_link#Example > > > >> I think it's pretty easy to understand. > > > >> To understand how hard-links (and rsync) can help you make strong > >> incremental backups, head over > >> http://blog.interlinked.org/tutorials/rsync_time_machine.html > > > >> Cheers, > > > >> -- Franois > > > > > >> Le mardi 22 janvier 2013 23:12:38, Joe a ￩crit : > >>> Thanks for the reply. I know what hard and soft links are and > >> have some > >>> idea of how they relate to backup. > >>> > >>> What I need is a tutorial on how all of that works with rsync. > >>> I can see that there are a lot of considerations as to which > >>> options to use for different situations and maybe some general > >>> strategies on how to build something like an incremental or > >>> differential backup. > >>> > >>> I use rsync now on a one directory tree to one directory tree > >> basis now, > >>> but I'd like to have more than one backup on my backup device > >>> without doubling the storage which is what hard links will > >>> help me do once I know how they work in more detail. > >>> > >>> Joe > >>> > >>> On 01/22/2013 02:48 AM, Tony Abernethy wrote: > >>>> This may help: (man ln) > >>>> > >>>> A hard link to a file is indistinguishable from the original > >>>> directory entry; any changes > >> to a > >>>> file are effectively independent of the name used to > >>>> reference > >> the file. > >>>> Hard links may not normally refer to directories and may not > >> span file > >>>> systems. > >>>> > >>>> Assuming you do many backups and many of the files do not > >>>> change, hard links are your friend. > >>>> > >>>> Backing up soft links: Do you back up the link or what the > >>>> link points to? (Even that simple thing has interesting ways > >>>> to get complicated.) > >>>> > >>>> -----Original Message----- From: > >>>> [email protected] > >> <mailto:[email protected]> > >>>> [mailto:[email protected] > >> <mailto:[email protected]>] On Behalf Of Joe Sent: > >> Tuesday, > >>>> January 22, 2013 1:32 AM To: [email protected] > >>>> <mailto:[email protected]> Subject: Is there a > >>>> howto/tutorial on backups/rsync that covers > >> the use > >>>> of hard and soft links? > >>>> > >>>> There have been a lot of posts on the list lately about > >>>> issues > >> with hard > >>>> links. It has been very interesting, but I don't understand > >>>> it very thoroughly. I haven't used hard links for anything > >>>> yet. I've used symlinks - not for backups, of course - and > >>>> have seen them get > >> broken or > >>>> deleted in backups. > >>>> > >>>> Is there a tutorial anywhere that will explain how this > >>>> works > >> (assuming > >>>> that the reader understands the basic concepts of backups > >>>> and > >> knows how > >>>> to program, but doesn't really understand how to use links > >>>> to create things like incremental or differential backups)? > >>>> It seems like > >> there > >>>> are a lot of fine points to consider, some of which can > >>>> really > >> bite you > >>>> if you don't take them into account. > >>>> > >>>> I'm working on my own personal backup system using bash and > >> rsync. When > >>>> it's done it will be pretty good, but it would be *much* > >>>> better if I rewrote it to have more backup versions using > >>>> hard links to save > >> space. > >>>> > >>>> I am writing my own because (aside from learning a lot) I > >>>> have > >> only seen > >>>> two types of backup utilities - those that are very > >>>> simplistic > >> and won't > >>>> let me do what I want and those which are enterprise level > >>>> and I > >> can't > >>>> figure out how to get them to do anything without extensive > >> study. (I > >>>> did experiment with areca (I think it uses rsync libraries > >>>> under the hood) which would probably do everything I want, > >>>> but I got stuck too many times and couldn't get enough > >>>> support on their forum to keep going.) > >>>> > >>>> TIA > >>>> > >>>> Joe > >>>> > >>>> -- Please use reply-all for most replies to avoid omitting > >>>> the > >> mailing list. > >>>> To unsubscribe or change options: > >>>> https://lists.samba.org/mailman/listinfo/rsync Before > >>>> posting, read: > >>>> http://www.catb.org/~esr/faqs/smart-questions.html > >> -- Please use reply-all for most replies to avoid omitting the > >> mailing list. To unsubscribe or change options: > >> https://lists.samba.org/mailman/listinfo/rsync Before posting, > >> read: http://www.catb.org/~esr/faqs/smart-questions.html > > > > > > > > > > > > > > - -- > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ > Kevin Korb Phone: (407) 252-6853 > Systems Administrator Internet: > FutureQuest, Inc. [email protected] (work) > Orlando, Florida [email protected] (personal) > Web page: http://www.sanitarium.net/ > PGP public key available on web site. > ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.19 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iEYEARECAAYFAlD/HVIACgkQVKC1jlbQAQd2OwCgwF8H2EjiCmKcTboG5rK8w7Vm > Sv4AoIIknAaOcuMLOj9dKlykKV1Wvu1F > =kLZP > -----END PGP SIGNATURE----- > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html > >
-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
