First, I want to say thanks to Eric Ullman for the details. It is
enlightnening indeed! After using Arcserve for a while I was happy to get
back to Retrospect because I feel it provides an easier way to find out
what's really going on. With Arcserve I was never really sure I had gotten
good backups, and setting it up was truly a pain. Restoring was VERY
counter intuitive. That seems to be part of what Retrospect does best.

But all of that scanning and matching does take time, and that is a
trade-off, I guess. It would be nice if the "Time remaining" could be made
a little more accureate, as it seems to just take what's left and compare
to the current MB/ minute, which changes dramatically. I wasn't sure
whether my 36 GB folder would take 2 hours or two days. It wound up taking
about 8.5 hours, which is acceptable given the extra checking that
Retrospect does. Of coure I won't be looking at it whe it's working in the
middle of the night (I hope), so maybe that doesn't matter....

I am still interested in imformation about the 500,000 file limit that Doug
McKay wrote about. What Eric U. says makes me think it may be "per
session," which would be OK, if not ideal. I definately have over 500,000
files total that need to be backed up in a full backup of my server. Any
info would be appreciated!

--Ben







Eric Ullman <[EMAIL PROTECTED]> on 12/05/2000 09:04:45 AM

Please respond to "retro-talk" <[EMAIL PROTECTED]>

To:   retro-talk <[EMAIL PROTECTED]>
cc:    (bcc: Ben Eastwood/HMG/Wilson Learning/US)
Subject:  Re: How Long does this take for you? (long)




Wow. There's a lot to respond to, here! I'll start with IncrementalPLUS,
Snapshot, and the scanning and matching process.

Retrospect's normal backup operation is something we call IncrementalPLUS.
Essentially, it's an incremental backup, but with one major difference from
the way that all other file-based backup software works: Retrospect's
IncrementalPLUS backups are incremental to the backup set currently in use,
not to the last full or incremental backup performed.

In other words, when you run a backup to a specific set of media,
Retrospect
will automatically complete that set, giving it one copy of every unique
file from the source volume(s).

To do this, Retrospect employs Snapshots. For each backup operation, it
scans the source volume and saves a listing of all the files, folders, and
(for Windows computers) registry information. That's the Snapshot. Then, to
determine what files need to be copied to the backup set, Retrospect
matches
that Snapshot against each session previously run to this set. Every time
Retrospect runs a backup of a volume, it generates a session. So, if you
have 30 drives to back up, and you've backed up each, four times to the
same
backup set, you'll have 120 sessions.

This matching process can take some time. It's dependent upon the total
number of files in those sessions and the speed of the backup computer. At
a
certain point, the scanning and matching process will take longer than
actually copying the files to the backup set. Scanning and matching
performance is dependent on the total number of files; copy performance is
dependent on the amount of data being moved.

It is because Retrospect functions this way that it is able to ensure each
backup set is complete. This is how Retrospect is able to restore a volume
exactly, with only a single pass over the backup media, from any backup
set.

Each "Normal" (IncrementalPLUS) backup performed to the same backup set
increases both the time needed for scanning and matching and the time it
will take to perform a restore. When the scanning and matching process is
preventing your backups from completing in the period allotted, then it's
time to perform a "Recycle" or "New Media" backup, resetting the session
count to zero for that set. Adjusting your media use in this fashion is not
an expensive ordeal. There are other options, but they cost more money
(getting a faster computer to run the backups or adding another backup
computer and splitting the clients between them).

So, for Ben Mihailescu, 15-20 days of Normal backups to the same set is too
many. At Dantz, we have to recycle or archive backup sets after five Normal
backups.

Regarding performance when you're dealing with a large number of relatively
small files, as Ben Eastwood commented on: fast backup devices will only
achieve their rated performance level when backing up large files. Small
files result in decreased performance, as shown in this table.

Drive model: IBM Ultrium LTO 3580 (Non-Compressible Data)

  Data Set      Backup    Copy    Compare
  1K files        7.2     10.3       5.5
  10K files     110.8    173.4      81.4
  100K files    285.0    362.4     234.8
  1MB files     716.4    857.1     618.5
  10MB files    800.6    870.2     741.3
  100MB Files   792.4    870.0     727.6
  500MB Files   840.7    882.8     802.5

  (With larger files, we've seen 1.2 GB/min.)

Finally, to answer Ben Eastwood's question about the time required to
update
the Catalog and build the Snapshot, it is during those processes that
Retrospect updates NTFS permissions and performs catalog compression, both
of which may take some time with large file sets.

I hope this helps everyone understand a little better how Retrospect works
and how to manage backup sets. Certainly respond to the list with
additional
questions or comments.

Best regards,

Eric Ullman
Dantz Development


Ben Mihailescu <[EMAIL PROTECTED]> wrote:

> I'm totally with you here. My full backup takes about 30 hours on a
> weekend. Fast and straight forward, no problem. After 15-20 days of
> incremental backup to the same set, it is totally impossible to collect
> all the clients in one night. I have posted at length about this before,
> yet no real solution - other than spending a ton of money - has been
> found. I gott'a say, Dantz should realize that disk space is so cheap and
> easy to add this days, that backup software and hardware should start
> matching the price and efficiency.
>
> "Douglas B. McKay" wrote:
>
>> I have found that Retrospect spends most of its time in my nightly
>> backups processing files and catalogs, not actually backing up data.
>> I have 15 clients which are backed up by Retrospect from a machine
>> with 4 OnStream ADR 50  drives.  The full backups on the weekend take
>> about 26 hours (~120GB).  Nightly backups take almost 12 hours
>> (usually less than 5GB).
>>
>> Be careful about the half-million file limit.  If you get around
>> 500,000 files, Retrospect has a problem with memory and dies.  Dantz
>> knows about the issue and is working on ways to eliminate the memory
>> problem (at least that's what I was told several months ago).
>>
>> Anyway, the bottom line is that what you are experiencing has been my
>> experience as well (long periods "building" catalogs and things).
>>
>> It sure would be nice if my backup time could shrink and allow the
>> file copying to take place at full speed, but perhaps have the catalog
>> processing, etc. take place separately (perhaps on the client using
>> its CPU! - my backups happen after hours).
>>
>>    ...Doug
>>
>>> Ben Eastwood wrote:
>>>
>>> When running a backup operation of a LOCAL disk (RAID 5 Array,
actually)to a
>>> DLT7000 drive, I noticed that the Performance reported varied widely,
from a
>>> high of 450MB/min to a low of 13 MB/min, and the "Time Remaining" would
jump
>>> around pretty much based on this. Is that normal? The folder I backed
up had
>>> over 36 GB in it, mostly little files. In fact there were over 300,000
files
>>> in about 10,000 folders, if that matters. Also the "Scanning" before
the
>>> backup took a long time.
>>>
>>> The actual backup took about 6 hours, and then it went into "Updating
>>> catalog" for about 45 minutes and then on to "building snapshot," where
it
>>> is
>>> now. During these last two sections, it has said "time remaining
00:00:00,"
>>> but it's not really done... and the progress bar is only about halfway
>>> across. Is that any real indication of how much time I have left?
>>>
>>> More on this:
>>> Retrospect is still "building snapshot" and appears to be hogging about
>>> 95-99% CPU, but not stuck, really because it varies... I also notice
that
>>> taskman reports memory usage of 86716K, which seems like a lot on a
machine
>>> with 196 MB of RAM where the System takes up only 6364K... Any hints?



--
----------------------------------------------------------
To subscribe:    [EMAIL PROTECTED]
To unsubscribe:  [EMAIL PROTECTED]
Archives:        <http://list.working-dogs.com/lists/retro-talk/>

For urgent issues, please contact Dantz technical support directly at
[EMAIL PROTECTED] or 925.253.3050.








--
----------------------------------------------------------
To subscribe:    [EMAIL PROTECTED]
To unsubscribe:  [EMAIL PROTECTED]
Archives:        <http://list.working-dogs.com/lists/retro-talk/>

For urgent issues, please contact Dantz technical support directly at
[EMAIL PROTECTED] or 925.253.3050.

Reply via email to