Hi Vincent,If you back up only the gold files, and unpack "the most important" ones at restore time, you still need to be able to unpack on the fly. So why bother taking a guess at what are "the most important" uploads and slowing down the restore?
To me the possibilities are:1) backup and restore only gold (and license) files. This requires code changes to unpack on the fly. 2) reduce the size of the repository to make feasible backing up all the repository, and thus speeding up recovery and lowering backup and live disk space requirements.
But people should understand the tradeoffs. But first, here is some new data: - Total unpacked size will be less in a working repository due to file reuse. So to get a more realistic number by including file reuse, I measured the files and golds on grumpy (even though this repository is weighted towards isos):
gold 13 GB
files 99 GB
inflation 8x
The disk space requirements seem clear it takes roughly 8x the disk
space to save all the unpacked files over the size of the original
files. I think we also have a good handle on the time requirements
(at least for us at OSUOSL). On the system we use:
A 600GB (what we use today) repo backup would take roughly 3 hours
to copy to another disk (assuming similar transfer speed as we see
from their ftp server).
The 61 GB of gold files on our system would take 18 minutesBut keep in mind that disk backups (except for the first one) are incremental. If you add 6 GB of gold files a day, the incremental backup time for backing up the entire repo is around 18 minutes (for 60 GB of unpacked files), 2 minutes for just the gold files. I suspect very few people add that much to their repo in a day but it is for illustration.
On the restore side, restoring 600GB would still take roughly 3 hours with no unpack necessary. 61GB of gold take the 18 minutes to copy, but unpacking 61 GB of gold files would take over 3 days. Based on the 4.2 GB Fedora 11 iso which took 5.5 hrs to unpack, 61 GB could conceivably take 4,800 minutes = 80 hours.
Personally, I back up my laptop on a $200 1TB firewire raid that I can write to at over 60MB/s. So I'm speculating that some users would prefer to back up their entire repo because disk space can be pretty inexpensive and incremental disk backup time is not an issue (18 minutes / day for a large repo).
It also makes sense that people may want to clean up their repo by removing rarely needed files and just keep the gold.
So to me the backup solution is really two fold:1) With no changes to fossology, you can back up the system today. Just stop the scheduler, do a pg_dumpall and then rsync the entire repo to a backup drive.
2) Allow users to "archive" uploads. Meaning delete all the repo files except the gold. This will free up disk space for them at the cost of having to run unpack again should the files be needed. This will take some coding changes. This was actually in the initial fossology design, but we never implemented it. This will also allow someone to restrict their repo backup to just the gold (and license) files.
There is also a related issue that I mentioned briefly before - to not save all the files in the repo that we currently save. Here is some background:
There are three types of files that unpack saves: 1. leaf filesThe leaf files are simply files that can be unpacked no further. For example, file.c, myfile.spec, myfile.png, ...
2. containersContainers are archives and "artifacts". For example, containers are files like file.gz, file.rpm, file.jar, file.war, file.ott, ... Since these are unpacked to leaf nodes, we could delete the containers themselves.
3. artifactsArtifacts are files and directories created as a result of the unarchiving process. They have two names:
3.1 artifact.metaartifact.meta has data from the process used to unpack the data. These files can sometimes be as large as the archive itself. The purpose of saving these is to answer questions about the unpack. There is one artifact.meta for each artifact. We don't make use of these today. I've attached a snippet from one artifact.meta so you can see what they can look like.
artifact.meta
Description: Binary data
3.2 artifact.unpackedYou can't view artifact.unpacked files in the UI and they take up significant repository space. These are files, other than leaf files and embedded containers, that unpack needs to do it's work and represent a very significant disk space usage. By way of example, let's unpack BackupPC-3.1.0-5.fc11.src.rpm from Fedora 11.
Original file 484,679 bytes Unpacking the first level (rpms use rpm2cpio) creates two files:-rw-r--r-- 1 bobg bobg 486848 2009-08-05 14:33 BackupPC-3.1.0-5.fc11.src.rpm.unpacked drwxrws--- 3 bobg bobg 4096 2009-08-05 14:33 BackupPC-3.1.0-5.fc11.src.rpm.unpacked.dir
So just saving the unpacked file (called artifact.unpacked and not BackupPC-3.1.0-5.fc11.src.rpm.unpacked in the repository), gives us an inflation of 2x. Unarchiving the artifact.unpacked (the cpio file) give us a new directory BackupPC-3.1.0-5.fc11.src.rpm.unpacked.dir:
$ ls -l total 2484 -rw-rw-r-- 1 bobg bobg 474981 2009-08-05 14:33 BackupPC-3.1.0.tar.gz-rw-r--r-- 1 bobg bobg 2027520 2009-08-05 14:33 BackupPC-3.1.0.tar.gz.unpacked drwxrws--- 3 bobg bobg 4096 2009-08-05 14:33 BackupPC-3.1.0.tar.gz.unpacked.dir
-rw-r--r-- 1 bobg bobg 604 2009-08-05 14:33 BackupPC.htaccess -rw-r--r-- 1 bobg bobg 338 2009-08-05 14:33 BackupPC.logrotate -rw-r--r-- 1 bobg bobg 614 2009-08-05 14:33 BackupPC-README.fedora -rw-r--r-- 1 bobg bobg 8521 2009-08-05 14:33 BackupPC.spec-rw-r--r-- 1 bobg bobg 863 2009-08-05 14:33 BackupPC- TopDir_change.patch
and so on. Note the file BackupPC-3.1.0.tar.gz.unpacked, which is the gunzipped tar file, which is then untarred into the .dir directory. The final unpacked size for this 0.5 MB package is 5.2MB. The two unpacked artifacts account for 2.5 MB.
3.3 directoriesThe repository also contains files (usually small) with directory information
The point to all this is that the repository contains much more than leaf files. By being more selective on which files we save, we might be able to reduce the repository size by 2X.
Bob Gobeille
_______________________________________________ fossology mailing list [email protected] http://fossology.org/mailman/listinfo/fossology
