RE: General backup advice help?
> -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] > On Behalf Of Jon LaBadie > Sent: Friday, October 20, 2006 7:50 PM > To: [email protected] > Subject: Re: General backup advice help? > > > On Fri, Oct 20, 2006 at 03:50:25PM -0700, Don Murray wrote: > > > > Hello list, > > > > I've been lurking on the list a few days (not really long enough to be > > opening my mouth, but here it goes). > > > > I am a relatively novice sysadmin looking for a better backup setup than > > our handcrafted scripts. Amanda looks interesting but I am worried that > > it will not necessarily solve my problem. So I was hoping to beg your > > indulgence for a moment and see if anyone has any advice for me. > > > > My main concern is that we have to be "up" round the clock these days as > > we have people connecting during their work day, 9 hours different from > > ours. So having major services disrupted in the middle of the night for > > backups doesn't seem a viable solution anymore, as it hits the other > > people in the middle of the day. [McGraw, Robert P.] The solution to this is to take a snapshot of the data you want to back up and back up the snapshot. Users continue to work without interruption and you get a good backup without any file changes during the backup. > > "Up" is less a concern to me than "connected". Are some of these > people working on laptops or other computers that are not accessible > to your backup server except at certain times? > > Let's assume not, that all are connected when you want to do backups. > > > > > We are a small business. We have 2 main linux servers each with about > > 200G to back up. Backing up a full 200G volume across the network takes > > about 7 hours currently. > > > > I understand from the documentation that Amanda will attempt to balance > > the time taken for backups through its scheduling process. However, it > > seems to me that when you have to do a level 0 of size X, you have to do > > it, and so you will take the full 7 hours. > > > > The bad news is that currently these systems are set up with the full > > 200G on single partitions. > > > > If you are willing to use tar as as your backup program (many, many > amanda sites do that) then there is not problem. You say you have > basically 400GB of data. The basic unit of backup for amanda is > called a "DiskList Entry" (DLE) after the name of a config file. > With tar as the backup program you can specify directory tree's > as a DLE rather than filesystems. So you could have a DLE for > /boot, /var, /home, /usr/local, /usr (without local), /opt, > /database, and one for / without all the others. This is done > by "including" or "excluding" files and directories in the DLE > config. Now with 16 DLE's from two systems (or even more is you > wish), amanda can spread the level 0's of those 16 items over > your dumpcycle (typically a week). So on a typical day, only > 2 or 3 of the DLEs will get a level 0 and on average your daily > backups will be 1/7th of your 400GB or about 60GB/day. > > -- > Jon H. LaBadie [EMAIL PROTECTED] > JG Computing > 4455 Province Line Road(609) 252-0159 > Princeton, NJ 08540-4322 (609) 683-7220 (fax) smime.p7s Description: S/MIME cryptographic signature
Re: General backup advice help?
On Fri, Oct 20, 2006 at 05:29:05PM -0700, Don Murray wrote: > > Hi Jon, thanks for the reply! > > Jon LaBadie wrote: > > > >"Up" is less a concern to me than "connected". > > Actually for those people the main issue is that some key services (like > the email server) are shut down during backups. This is a tangential > issue I guess because by getting smoother backups I'm just going to make > their outages shorter, but the outage will still be there. I'm not > backing up from Europe, no. So connectivity is not an issue. Does the service really need to be shutdown? You might solicit opinion and experience from others on the list about this specific thing. Personally I don't shut things down. Occasionally a file fails during backup, but it affects only the individual file, not the backup. And lots of times the "failure" is not really a failure, more a change between generating the list to backup and actually backing up the file. Many times those are simply temporary files that disappeared in the interim. But there are some "file changed while being backed up". Those probably would not recover to a consistant state, but I've not explored further. You could experiment and If it doesn't happen too often, perhaps you don't need to shutdown. Another possibility is check if your file system supports "filesystem snapshots". If so, generate a snapshot (readonly and takes little space), backup that static image while the rest of the world continues using the active filesystem. I'm not sure, but I think on linux only lvm partitions support snapshots. > >If you are willing to use tar as as your backup program (many, many > >amanda sites do that) then there is not problem. You say you have > >basically 400GB of data. ... > >... . So on a typical day, only > >2 or 3 of the DLEs will get a level 0 and on average your daily > >backups will be 1/7th of your 400GB or about 60GB/day. > > I have heard of this but I thought that you can only use a directory > rather than a partition for a level 0 backup. If I need daily L1 > backups, is it still possible with the above scheme? If it is, then I > think thats definitely the way I will go! You are thinking dump, not tar. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
Re: General backup advice help?
Hi Jon, thanks for the reply! Jon LaBadie wrote: My main concern is that we have to be "up" round the clock these days as we have people connecting during their work day, 9 hours different from ours. So having major services disrupted in the middle of the night for backups doesn't seem a viable solution anymore, as it hits the other people in the middle of the day. "Up" is less a concern to me than "connected". Are some of these people working on laptops or other computers that are not accessible to your backup server except at certain times? Actually for those people the main issue is that some key services (like the email server) are shut down during backups. This is a tangential issue I guess because by getting smoother backups I'm just going to make their outages shorter, but the outage will still be there. I'm not backing up from Europe, no. So connectivity is not an issue. If you are willing to use tar as as your backup program (many, many amanda sites do that) then there is not problem. You say you have basically 400GB of data. The basic unit of backup for amanda is called a "DiskList Entry" (DLE) after the name of a config file. With tar as the backup program you can specify directory tree's as a DLE rather than filesystems. So you could have a DLE for /boot, /var, /home, /usr/local, /usr (without local), /opt, /database, and one for / without all the others. This is done by "including" or "excluding" files and directories in the DLE config. Now with 16 DLE's from two systems (or even more is you wish), amanda can spread the level 0's of those 16 items over your dumpcycle (typically a week). So on a typical day, only 2 or 3 of the DLEs will get a level 0 and on average your daily backups will be 1/7th of your 400GB or about 60GB/day. I have heard of this but I thought that you can only use a directory rather than a partition for a level 0 backup. If I need daily L1 backups, is it still possible with the above scheme? If it is, then I think thats definitely the way I will go! Thanks again, Don
Re: General backup advice help?
On Fri, Oct 20, 2006 at 03:50:25PM -0700, Don Murray wrote: > > Hello list, > > I've been lurking on the list a few days (not really long enough to be > opening my mouth, but here it goes). > > I am a relatively novice sysadmin looking for a better backup setup than > our handcrafted scripts. Amanda looks interesting but I am worried that > it will not necessarily solve my problem. So I was hoping to beg your > indulgence for a moment and see if anyone has any advice for me. > > My main concern is that we have to be "up" round the clock these days as > we have people connecting during their work day, 9 hours different from > ours. So having major services disrupted in the middle of the night for > backups doesn't seem a viable solution anymore, as it hits the other > people in the middle of the day. "Up" is less a concern to me than "connected". Are some of these people working on laptops or other computers that are not accessible to your backup server except at certain times? Let's assume not, that all are connected when you want to do backups. > > We are a small business. We have 2 main linux servers each with about > 200G to back up. Backing up a full 200G volume across the network takes > about 7 hours currently. > > I understand from the documentation that Amanda will attempt to balance > the time taken for backups through its scheduling process. However, it > seems to me that when you have to do a level 0 of size X, you have to do > it, and so you will take the full 7 hours. > > The bad news is that currently these systems are set up with the full > 200G on single partitions. > If you are willing to use tar as as your backup program (many, many amanda sites do that) then there is not problem. You say you have basically 400GB of data. The basic unit of backup for amanda is called a "DiskList Entry" (DLE) after the name of a config file. With tar as the backup program you can specify directory tree's as a DLE rather than filesystems. So you could have a DLE for /boot, /var, /home, /usr/local, /usr (without local), /opt, /database, and one for / without all the others. This is done by "including" or "excluding" files and directories in the DLE config. Now with 16 DLE's from two systems (or even more is you wish), amanda can spread the level 0's of those 16 items over your dumpcycle (typically a week). So on a typical day, only 2 or 3 of the DLEs will get a level 0 and on average your daily backups will be 1/7th of your 400GB or about 60GB/day. -- Jon H. LaBadie [EMAIL PROTECTED] JG Computing 4455 Province Line Road(609) 252-0159 Princeton, NJ 08540-4322 (609) 683-7220 (fax)
