PFC wrote:

I once upon a time worked in a company doing backup software and I remember these problems, we had exactly the same !

  Prety neat. :)

The file tree was all into memory and everytime the user clicked on something it haaad to update everything. Being C++ it was very fast, but to backup a million files you needed a gig of RAM, which is... a problem let's say, when you think my linux laptop has about 400k files on it.

I want this to run on "average" systems (I'm developing it primarily on my modest P3 1GHz Thinkpad w/ 512MB RAM running Debian) so expecting that much free memory is not reasonable. As it is my test DB, with a realistic amount of data, is ~150MB.

So we rewrote the project entirely with the purpose of doing the million files thingy with the clunky Pentium 90 with 64 megabytes of RAM, and it worked.
    What I did was this :
    - use Berkeley DB
- the price of the licence to be able to embed it in your product and sell it is expensive, and if you want crash-proof, it's insanely expensive.

This is the kicker right there; my program is released under the GPL so it's fee-free. I can't eat anything costly like that. As it is there is hundreds and hundreds of hours in this program that I am already hoping to recoup one day through support contracts. Adding commercial software I am afraid is not an option.

bonus : if you check a directory as "include" and one of its subdirectory as "exclude", and the user adds files all over the place, the files added in the "included" directory will be automatically backed up and the ones in the 'ignored' directory will be automatically ignored, you have nothing to change.
    IMHO it's the only solution.

Now *this* is an idea worth looking into. How I will implement it with my system I don't know yet but it's a new line of thinking. Wonderful!

Now you'll ask me, but how do I calculate the total size of the backup without looking at all the files ? when I click on a directory I don't know what files are in it and which will inherit and which will not.

It's simple : you precompute it when you scan the disk for changed files. This is the only time you should do a complete tree exploration.

This is already what I do. When a user selects a partition they want to select files to backup or restore the partition is scanned. The scan looks at every file, directory and symlink and records it's size (on disk), it mtime, owner, group, etc. and records it to the database. I've got this scan/update running at ~1,500 files/second on my laptop. That was actually the first performance tuning I started with. :)

With all the data in the DB the backup script can calculate rather intelligently where it wants to copy each directory to.

On each directory we put a matrix [M]x[N], M and N being one of the three above state, containing the amount of stuff in the directory which would be in state M if the directory was in state N. This is very easy to compute when you scan for new files. Then when a directory changes state, you have to sum a few cells of that matrix to know how much more that adds to the backup. And you only look up 1 record.

In my case what I do is calculate the size of all the files selected for backup in each directory, sort the directories from all sources by the total size of all their selected files and then start assigning the directories, largest to smallest to each of my available destination medias. If it runs out of destination space it backs up what it can and then waits a user-definable amount of time and then checks to see if any new destination media has been made available. If so it again tries to assign the files/directories that didn't fit. It will loop a user-definable number of times before giving up and warning the user that more destination space is needed for that backup job.

    Is that helpful ?

The three states (inhertied, backup, ignore) has definately caught my attention. Thank you very much for your idea and lengthy reply!


---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
      choose an index scan if your joining column's datatypes do not

Reply via email to