Wow. There's a lot to respond to, here! I'll start with IncrementalPLUS,
Snapshot, and the scanning and matching process.
Retrospect's normal backup operation is something we call IncrementalPLUS.
Essentially, it's an incremental backup, but with one major difference from
the way that all other file-based backup software works: Retrospect's
IncrementalPLUS backups are incremental to the backup set currently in use,
not to the last full or incremental backup performed.
In other words, when you run a backup to a specific set of media, Retrospect
will automatically complete that set, giving it one copy of every unique
file from the source volume(s).
To do this, Retrospect employs Snapshots. For each backup operation, it
scans the source volume and saves a listing of all the files, folders, and
(for Windows computers) registry information. That's the Snapshot. Then, to
determine what files need to be copied to the backup set, Retrospect matches
that Snapshot against each session previously run to this set. Every time
Retrospect runs a backup of a volume, it generates a session. So, if you
have 30 drives to back up, and you've backed up each, four times to the same
backup set, you'll have 120 sessions.
This matching process can take some time. It's dependent upon the total
number of files in those sessions and the speed of the backup computer. At a
certain point, the scanning and matching process will take longer than
actually copying the files to the backup set. Scanning and matching
performance is dependent on the total number of files; copy performance is
dependent on the amount of data being moved.
It is because Retrospect functions this way that it is able to ensure each
backup set is complete. This is how Retrospect is able to restore a volume
exactly, with only a single pass over the backup media, from any backup set.
Each "Normal" (IncrementalPLUS) backup performed to the same backup set
increases both the time needed for scanning and matching and the time it
will take to perform a restore. When the scanning and matching process is
preventing your backups from completing in the period allotted, then it's
time to perform a "Recycle" or "New Media" backup, resetting the session
count to zero for that set. Adjusting your media use in this fashion is not
an expensive ordeal. There are other options, but they cost more money
(getting a faster computer to run the backups or adding another backup
computer and splitting the clients between them).
So, for Ben Mihailescu, 15-20 days of Normal backups to the same set is too
many. At Dantz, we have to recycle or archive backup sets after five Normal
Regarding performance when you're dealing with a large number of relatively
small files, as Ben Eastwood commented on: fast backup devices will only
achieve their rated performance level when backing up large files. Small
files result in decreased performance, as shown in this table.
Drive model: IBM Ultrium LTO 3580 (Non-Compressible Data)
Data Set Backup Copy Compare
1K files 7.2 10.3 5.5
10K files 110.8 173.4 81.4
100K files 285.0 362.4 234.8
1MB files 716.4 857.1 618.5
10MB files 800.6 870.2 741.3
100MB Files 792.4 870.0 727.6
500MB Files 840.7 882.8 802.5
(With larger files, we've seen 1.2 GB/min.)
Finally, to answer Ben Eastwood's question about the time required to update
the Catalog and build the Snapshot, it is during those processes that
Retrospect updates NTFS permissions and performs catalog compression, both
of which may take some time with large file sets.
I hope this helps everyone understand a little better how Retrospect works
and how to manage backup sets. Certainly respond to the list with additional
questions or comments.
Ben Mihailescu <[EMAIL PROTECTED]> wrote:
> I'm totally with you here. My full backup takes about 30 hours on a
> weekend. Fast and straight forward, no problem. After 15-20 days of
> incremental backup to the same set, it is totally impossible to collect
> all the clients in one night. I have posted at length about this before,
> yet no real solution - other than spending a ton of money - has been
> found. I gott'a say, Dantz should realize that disk space is so cheap and
> easy to add this days, that backup software and hardware should start
> matching the price and efficiency.
> "Douglas B. McKay" wrote:
>> I have found that Retrospect spends most of its time in my nightly
>> backups processing files and catalogs, not actually backing up data.
>> I have 15 clients which are backed up by Retrospect from a machine
>> with 4 OnStream ADR 50 drives. The full backups on the weekend take
>> about 26 hours (~120GB). Nightly backups take almost 12 hours
>> (usually less than 5GB).
>> Be careful about the half-million file limit. If you get around
>> 500,000 files, Retrospect has a problem with memory and dies. Dantz
>> knows about the issue and is working on ways to eliminate the memory
>> problem (at least that's what I was told several months ago).
>> Anyway, the bottom line is that what you are experiencing has been my
>> experience as well (long periods "building" catalogs and things).
>> It sure would be nice if my backup time could shrink and allow the
>> file copying to take place at full speed, but perhaps have the catalog
>> processing, etc. take place separately (perhaps on the client using
>> its CPU! - my backups happen after hours).
>>> Ben Eastwood wrote:
>>> When running a backup operation of a LOCAL disk (RAID 5 Array, actually)to a
>>> DLT7000 drive, I noticed that the Performance reported varied widely, from a
>>> high of 450MB/min to a low of 13 MB/min, and the "Time Remaining" would jump
>>> around pretty much based on this. Is that normal? The folder I backed up had
>>> over 36 GB in it, mostly little files. In fact there were over 300,000 files
>>> in about 10,000 folders, if that matters. Also the "Scanning" before the
>>> backup took a long time.
>>> The actual backup took about 6 hours, and then it went into "Updating
>>> catalog" for about 45 minutes and then on to "building snapshot," where it
>>> now. During these last two sections, it has said "time remaining 00:00:00,"
>>> but it's not really done... and the progress bar is only about halfway
>>> across. Is that any real indication of how much time I have left?
>>> More on this:
>>> Retrospect is still "building snapshot" and appears to be hogging about
>>> 95-99% CPU, but not stuck, really because it varies... I also notice that
>>> taskman reports memory usage of 86716K, which seems like a lot on a machine
>>> with 196 MB of RAM where the System takes up only 6364K... Any hints?
To subscribe: [EMAIL PROTECTED]
To unsubscribe: [EMAIL PROTECTED]
For urgent issues, please contact Dantz technical support directly at
[EMAIL PROTECTED] or 925.253.3050.