Re: How Long does this take for you?
OK, I promised an update on this... Things are OK on this issue. I had to define subvolumes in order to make this folder back up, and that was a pain, but seems to work OK. The biggest problem is that if I add new folders I have to then also add them as defined subvolumes in Retrospect and edit the script. That leads to more administration than I would like, but Dantz says the fix is on the way... The full backup takes the better part of the weekend, and the dailies go into the middle of the morning, but by then they are just doing our less busy servers. I had hoped to add some of the desktops to the script, a few at a time, to see how much more time it would take to back up the whole office, but I have to get the servers running well first. I have also run into problems with St Bernards OFM, which I use to backup open files. It is incompatible with NT 4.0 services for Macintosh, and It caused Retrospect to hang pretty tight during the backup last night. There is a soon to be released patch from microsoft, but it isn't available yet at least not for free download. Does anyone know if Win2k suffers the same problem? (info here: http://support.microsoft.com/support/kb/articles/q242/8/96.ASP ) Another question for the group: I have an Exabyte 230D tape library, which has 30 DLT slots and one DLT drive. There is an option to add a second DLT drive to the box, and it can be on the same or a different SCSI bus. Can someone explain the advantages of having a second drive in the library? would it make it possible to backup two clients at once? Of maybe do a backup to one drive while doing the compares and verification to the other? Does Retrospect support that? Lots to read, and more questions than answers, sorry! --Ben Ben Liberman [EMAIL PROTECTED] on 12/06/2000 10:59:31 AM Please respond to "retro-talk" [EMAIL PROTECTED] To: "retro-talk" [EMAIL PROTECTED] cc:(bcc: Ben Eastwood/HMG/Wilson Learning/US) Subject: Re: How Long does this take for you? At 10:05 -0800 12/6/00, Ben Eastwood wrote: AHHH HA! Thanks Eric, that explains alot... the folder I backed up had well over 300K files and the memory usage was pretty high during some portions of the process. Thanks for the workaround. Any ETA on the fix? Hey Ben, if it's not too much trouble, I'm sure that some of us would like to know how your backups go after you make the changes. Thanks, -- -- [EMAIL PROTECTED] Ben Liberman --- -- -- To subscribe:[EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives:http://list.working-dogs.com/lists/retro-talk/ For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050.
Re: How Long does this take for you?
At 10:05 -0800 12/6/00, Ben Eastwood wrote: AHHH HA! Thanks Eric, that explains alot... the folder I backed up had well over 300K files and the memory usage was pretty high during some portions of the process. Thanks for the workaround. Any ETA on the fix? Hey Ben, if it's not too much trouble, I'm sure that some of us would like to know how your backups go after you make the changes. Thanks, -- -- [EMAIL PROTECTED] Ben Liberman -- -- -- To subscribe:[EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives:http://list.working-dogs.com/lists/retro-talk/ For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050.
Re: How Long does this take for you?
It's not a stupid question at all, Ben! I just wish I could be more precise with an answer. Best regards, Eric Ullman Dantz Development Ben Eastwood [EMAIL PROTECTED] wrote: Yeah, I know. Stupid question... You guys work so many miracles already, I guess I was just hoping for one more! Thanks for the best guess and of course I won't call on april first saying "Eric Ullman daid it would be ready by now..." --Ben E Eric Ullman [EMAIL PROTECTED] on 12/06/2000 11:10:42 AM ETA? On software? Ha ha ha! That's funny, Ben! You crack me up, man... ;-) Sorry. Couldn't help myself. I could say early Q2, 2001, but we _are_ talking about backup software here. That means that any bug that causes data loss is unacceptable. So, whenever coding and testing is complete. Still, that's my best guess. Eric Ullman Dantz Development Ben Eastwood [EMAIL PROTECTED] wrote: AHHH HA! Thanks Eric, that explains alot... the folder I backed up had well over 300K files and the memory usage was pretty high during some portions of the process. Thanks for the workaround. Any ETA on the fix? -- -- To subscribe:[EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives:http://list.working-dogs.com/lists/retro-talk/ For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050.
Re: How Long does this take for you? (long)
Wow. There's a lot to respond to, here! I'll start with IncrementalPLUS, Snapshot, and the scanning and matching process. Retrospect's normal backup operation is something we call IncrementalPLUS. Essentially, it's an incremental backup, but with one major difference from the way that all other file-based backup software works: Retrospect's IncrementalPLUS backups are incremental to the backup set currently in use, not to the last full or incremental backup performed. In other words, when you run a backup to a specific set of media, Retrospect will automatically complete that set, giving it one copy of every unique file from the source volume(s). To do this, Retrospect employs Snapshots. For each backup operation, it scans the source volume and saves a listing of all the files, folders, and (for Windows computers) registry information. That's the Snapshot. Then, to determine what files need to be copied to the backup set, Retrospect matches that Snapshot against each session previously run to this set. Every time Retrospect runs a backup of a volume, it generates a session. So, if you have 30 drives to back up, and you've backed up each, four times to the same backup set, you'll have 120 sessions. This matching process can take some time. It's dependent upon the total number of files in those sessions and the speed of the backup computer. At a certain point, the scanning and matching process will take longer than actually copying the files to the backup set. Scanning and matching performance is dependent on the total number of files; copy performance is dependent on the amount of data being moved. It is because Retrospect functions this way that it is able to ensure each backup set is complete. This is how Retrospect is able to restore a volume exactly, with only a single pass over the backup media, from any backup set. Each "Normal" (IncrementalPLUS) backup performed to the same backup set increases both the time needed for scanning and matching and the time it will take to perform a restore. When the scanning and matching process is preventing your backups from completing in the period allotted, then it's time to perform a "Recycle" or "New Media" backup, resetting the session count to zero for that set. Adjusting your media use in this fashion is not an expensive ordeal. There are other options, but they cost more money (getting a faster computer to run the backups or adding another backup computer and splitting the clients between them). So, for Ben Mihailescu, 15-20 days of Normal backups to the same set is too many. At Dantz, we have to recycle or archive backup sets after five Normal backups. Regarding performance when you're dealing with a large number of relatively small files, as Ben Eastwood commented on: fast backup devices will only achieve their rated performance level when backing up large files. Small files result in decreased performance, as shown in this table. Drive model: IBM Ultrium LTO 3580 (Non-Compressible Data) Data Set BackupCopyCompare 1K files7.2 10.3 5.5 10K files 110.8173.4 81.4 100K files285.0362.4 234.8 1MB files 716.4857.1 618.5 10MB files800.6870.2 741.3 100MB Files 792.4870.0 727.6 500MB Files 840.7882.8 802.5 (With larger files, we've seen 1.2 GB/min.) Finally, to answer Ben Eastwood's question about the time required to update the Catalog and build the Snapshot, it is during those processes that Retrospect updates NTFS permissions and performs catalog compression, both of which may take some time with large file sets. I hope this helps everyone understand a little better how Retrospect works and how to manage backup sets. Certainly respond to the list with additional questions or comments. Best regards, Eric Ullman Dantz Development Ben Mihailescu [EMAIL PROTECTED] wrote: I'm totally with you here. My full backup takes about 30 hours on a weekend. Fast and straight forward, no problem. After 15-20 days of incremental backup to the same set, it is totally impossible to collect all the clients in one night. I have posted at length about this before, yet no real solution - other than spending a ton of money - has been found. I gott'a say, Dantz should realize that disk space is so cheap and easy to add this days, that backup software and hardware should start matching the price and efficiency. "Douglas B. McKay" wrote: I have found that Retrospect spends most of its time in my nightly backups processing files and catalogs, not actually backing up data. I have 15 clients which are backed up by Retrospect from a machine with 4 OnStream ADR 50 drives. The full backups on the weekend take about 26 hours (~120GB). Nightly backups take almost 12 hours (usually less than 5GB). Be careful about the half-million file limit. If you get around 500,000 files, Retrospect has a problem with memory and dies. Dantz knows
RE: How Long does this take for you?
Ben, It is a limit with one volume. Having more files than that across multiple volumes (as we currently do) is not a problem. ...Doug -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Ben Eastwood Sent: Tuesday, December 05, 2000 11:01 AM To: retro-talk Subject: RE: How Long does this take for you? Hmmm, I wasn't aware of the half million file limit... that is a _major_ concern for me as I am backing up lots of html files and the like. Is is a half million files from one source? What are the details of this problem? Also, how soon are we going to see a fix for this? Anyone from Dantz care to comment on that? --Ben -- -- To subscribe:[EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives:http://list.working-dogs.com/lists/retro-talk/ For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050.
