First, I want to say thanks to Eric Ullman for the details. It is enlightnening indeed! After using Arcserve for a while I was happy to get back to Retrospect because I feel it provides an easier way to find out what's really going on. With Arcserve I was never really sure I had gotten good backups, and setting it up was truly a pain. Restoring was VERY counter intuitive. That seems to be part of what Retrospect does best. But all of that scanning and matching does take time, and that is a trade-off, I guess. It would be nice if the "Time remaining" could be made a little more accureate, as it seems to just take what's left and compare to the current MB/ minute, which changes dramatically. I wasn't sure whether my 36 GB folder would take 2 hours or two days. It wound up taking about 8.5 hours, which is acceptable given the extra checking that Retrospect does. Of coure I won't be looking at it whe it's working in the middle of the night (I hope), so maybe that doesn't matter.... I am still interested in imformation about the 500,000 file limit that Doug McKay wrote about. What Eric U. says makes me think it may be "per session," which would be OK, if not ideal. I definately have over 500,000 files total that need to be backed up in a full backup of my server. Any info would be appreciated! --Ben Eric Ullman <[EMAIL PROTECTED]> on 12/05/2000 09:04:45 AM Please respond to "retro-talk" <[EMAIL PROTECTED]> To: retro-talk <[EMAIL PROTECTED]> cc: (bcc: Ben Eastwood/HMG/Wilson Learning/US) Subject: Re: How Long does this take for you? (long) Wow. There's a lot to respond to, here! I'll start with IncrementalPLUS, Snapshot, and the scanning and matching process. Retrospect's normal backup operation is something we call IncrementalPLUS. Essentially, it's an incremental backup, but with one major difference from the way that all other file-based backup software works: Retrospect's IncrementalPLUS backups are incremental to the backup set currently in use, not to the last full or incremental backup performed. In other words, when you run a backup to a specific set of media, Retrospect will automatically complete that set, giving it one copy of every unique file from the source volume(s). To do this, Retrospect employs Snapshots. For each backup operation, it scans the source volume and saves a listing of all the files, folders, and (for Windows computers) registry information. That's the Snapshot. Then, to determine what files need to be copied to the backup set, Retrospect matches that Snapshot against each session previously run to this set. Every time Retrospect runs a backup of a volume, it generates a session. So, if you have 30 drives to back up, and you've backed up each, four times to the same backup set, you'll have 120 sessions. This matching process can take some time. It's dependent upon the total number of files in those sessions and the speed of the backup computer. At a certain point, the scanning and matching process will take longer than actually copying the files to the backup set. Scanning and matching performance is dependent on the total number of files; copy performance is dependent on the amount of data being moved. It is because Retrospect functions this way that it is able to ensure each backup set is complete. This is how Retrospect is able to restore a volume exactly, with only a single pass over the backup media, from any backup set. Each "Normal" (IncrementalPLUS) backup performed to the same backup set increases both the time needed for scanning and matching and the time it will take to perform a restore. When the scanning and matching process is preventing your backups from completing in the period allotted, then it's time to perform a "Recycle" or "New Media" backup, resetting the session count to zero for that set. Adjusting your media use in this fashion is not an expensive ordeal. There are other options, but they cost more money (getting a faster computer to run the backups or adding another backup computer and splitting the clients between them). So, for Ben Mihailescu, 15-20 days of Normal backups to the same set is too many. At Dantz, we have to recycle or archive backup sets after five Normal backups. Regarding performance when you're dealing with a large number of relatively small files, as Ben Eastwood commented on: fast backup devices will only achieve their rated performance level when backing up large files. Small files result in decreased performance, as shown in this table. Drive model: IBM Ultrium LTO 3580 (Non-Compressible Data) Data Set Backup Copy Compare 1K files 7.2 10.3 5.5 10K files 110.8 173.4 81.4 100K files 285.0 362.4 234.8 1MB files 716.4 857.1 618.5 10MB files 800.6 870.2 741.3 100MB Files 792.4 870.0 727.6 500MB Files 840.7 882.8 802.5 (With larger files, we've seen 1.2 GB/min.) Finally, to answer Ben Eastwood's question about the time required to update the Catalog and build the Snapshot, it is during those processes that Retrospect updates NTFS permissions and performs catalog compression, both of which may take some time with large file sets. I hope this helps everyone understand a little better how Retrospect works and how to manage backup sets. Certainly respond to the list with additional questions or comments. Best regards, Eric Ullman Dantz Development Ben Mihailescu <[EMAIL PROTECTED]> wrote: > I'm totally with you here. My full backup takes about 30 hours on a > weekend. Fast and straight forward, no problem. After 15-20 days of > incremental backup to the same set, it is totally impossible to collect > all the clients in one night. I have posted at length about this before, > yet no real solution - other than spending a ton of money - has been > found. I gott'a say, Dantz should realize that disk space is so cheap and > easy to add this days, that backup software and hardware should start > matching the price and efficiency. > > "Douglas B. McKay" wrote: > >> I have found that Retrospect spends most of its time in my nightly >> backups processing files and catalogs, not actually backing up data. >> I have 15 clients which are backed up by Retrospect from a machine >> with 4 OnStream ADR 50 drives. The full backups on the weekend take >> about 26 hours (~120GB). Nightly backups take almost 12 hours >> (usually less than 5GB). >> >> Be careful about the half-million file limit. If you get around >> 500,000 files, Retrospect has a problem with memory and dies. Dantz >> knows about the issue and is working on ways to eliminate the memory >> problem (at least that's what I was told several months ago). >> >> Anyway, the bottom line is that what you are experiencing has been my >> experience as well (long periods "building" catalogs and things). >> >> It sure would be nice if my backup time could shrink and allow the >> file copying to take place at full speed, but perhaps have the catalog >> processing, etc. take place separately (perhaps on the client using >> its CPU! - my backups happen after hours). >> >> ...Doug >> >>> Ben Eastwood wrote: >>> >>> When running a backup operation of a LOCAL disk (RAID 5 Array, actually)to a >>> DLT7000 drive, I noticed that the Performance reported varied widely, from a >>> high of 450MB/min to a low of 13 MB/min, and the "Time Remaining" would jump >>> around pretty much based on this. Is that normal? The folder I backed up had >>> over 36 GB in it, mostly little files. In fact there were over 300,000 files >>> in about 10,000 folders, if that matters. Also the "Scanning" before the >>> backup took a long time. >>> >>> The actual backup took about 6 hours, and then it went into "Updating >>> catalog" for about 45 minutes and then on to "building snapshot," where it >>> is >>> now. During these last two sections, it has said "time remaining 00:00:00," >>> but it's not really done... and the progress bar is only about halfway >>> across. Is that any real indication of how much time I have left? >>> >>> More on this: >>> Retrospect is still "building snapshot" and appears to be hogging about >>> 95-99% CPU, but not stuck, really because it varies... I also notice that >>> taskman reports memory usage of 86716K, which seems like a lot on a machine >>> with 196 MB of RAM where the System takes up only 6364K... Any hints? -- ---------------------------------------------------------- To subscribe: [EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives: <http://list.working-dogs.com/lists/retro-talk/> For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050. -- ---------------------------------------------------------- To subscribe: [EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives: <http://list.working-dogs.com/lists/retro-talk/> For urgent issues, please contact Dantz technical support directly at [EMAIL PROTECTED] or 925.253.3050.