Hi Bob,
Thanks for your useful comments.
The consideration of unpack "the most important" gold files, as if we only
backup the gold files, I consider there will be some job needed some unpack
files at the backup point, when the restore time if we don't unpack these gold
files, some jobs may failed.
For example:
a. At the backup point, the unpack job is running, unpack a file
b. begin backup
c. when restore, I think the unpacked files will be needed
(these consideration will confirm with you, I am not sure about this, and I am
also not sure if other jobs depend on the unpack file)
With your test case data, you point
> So to me the backup solution is really two fold:
> 1) With no changes to fossology, you can back up the system today.
> Just stop the scheduler, do a pg_dumpall and then rsync the entire repo
> to a backup drive.
>
> 2) Allow users to "archive" uploads. Meaning delete all the repo files
> except the gold. This will free up disk space for them at the cost of
> having to run unpack again should the files be needed. This will take
> some coding changes. This was actually in the initial fossology design,
> but we never implemented it. This will also allow someone to restrict
> their repo backup to just the gold (and license) files.
I am fully agree with you, I think if we have enough disk spaces, your backup
solution is good to me.
Base on your solution, I think we can do follow things about this:
1. Discuss with team about the solution, talk with Dan if we have
enough disk space to backup all the repo files about our production system
2. Consider how to reduce the repo unpack files size, I think you give
a more detail information, I can work with you about this
3. Investigate how many code changes base on the solution(2), I think I
can initial to do the investigate
What about your thought?
Thanks,
Vincent
> -----Original Message-----
> From: Gobeille, Robert
> Sent: Thursday, August 06, 2009 5:55 AM
> To: Ma, Dong (vinc...@gdcc-bj-most)
> Cc: [email protected]
> Subject: Re: unpack size/time
>
> Hi Vincent,
> If you back up only the gold files, and unpack "the most important"
> ones at restore time, you still need to be able to unpack on the fly.
> So why bother taking a guess at what are "the most important" uploads
> and slowing down the restore?
>
> To me the possibilities are:
> 1) backup and restore only gold (and license) files. This requires
> code changes to unpack on the fly.
> 2) reduce the size of the repository to make feasible backing up all
> the repository, and thus speeding up recovery and lowering backup and
> live disk space requirements.
>
> But people should understand the tradeoffs. But first, here is some
> new data:
> - Total unpacked size will be less in a working repository due to file
> reuse. So to get a more realistic number by including file reuse, I
> measured the files and golds on grumpy (even though this repository is
> weighted towards isos):
> gold 13 GB
> files 99 GB
> inflation 8x
>
> The disk space requirements seem clear it takes roughly 8x the disk
> space to save all the unpacked files over the size of the original
> files. I think we also have a good handle on the time requirements
> (at least for us at OSUOSL). On the system we use:
> A 600GB (what we use today) repo backup would take roughly 3 hours
> to copy to another disk (assuming similar transfer speed as we see from
> their ftp server).
> The 61 GB of gold files on our system would take 18 minutes
>
> But keep in mind that disk backups (except for the first one) are
> incremental. If you add 6 GB of gold files a day, the incremental
> backup time for backing up the entire repo is around 18 minutes (for 60
> GB of unpacked files), 2 minutes for just the gold files. I suspect
> very few people add that much to their repo in a day but it is for
> illustration.
>
> On the restore side, restoring 600GB would still take roughly 3 hours
> with no unpack necessary. 61GB of gold take the 18 minutes to copy,
> but unpacking 61 GB of gold files would take over 3 days. Based on the
> 4.2 GB Fedora 11 iso which took 5.5 hrs to unpack, 61 GB could
> conceivably take 4,800 minutes = 80 hours.
>
> Personally, I back up my laptop on a $200 1TB firewire raid that I can
> write to at over 60MB/s. So I'm speculating that some users would
> prefer to back up their entire repo because disk space can be pretty
> inexpensive and incremental disk backup time is not an issue (18
> minutes / day for a large repo).
>
> It also makes sense that people may want to clean up their repo by
> removing rarely needed files and just keep the gold.
>
> So to me the backup solution is really two fold:
> 1) With no changes to fossology, you can back up the system today.
> Just stop the scheduler, do a pg_dumpall and then rsync the entire repo
> to a backup drive.
>
> 2) Allow users to "archive" uploads. Meaning delete all the repo files
> except the gold. This will free up disk space for them at the cost of
> having to run unpack again should the files be needed. This will take
> some coding changes. This was actually in the initial fossology design,
> but we never implemented it. This will also allow someone to restrict
> their repo backup to just the gold (and license) files.
>
> There is also a related issue that I mentioned briefly before - to not
> save all the files in the repo that we currently save. Here is some
> background:
>
> There are three types of files that unpack saves:
> 1. leaf files
> The leaf files are simply files that can be unpacked no further. For
> example, file.c, myfile.spec, myfile.png, ...
>
> 2. containers
> Containers are archives and "artifacts". For example, containers are
> files like file.gz, file.rpm, file.jar, file.war, file.ott, ... Since
> these are unpacked to leaf nodes, we could delete the containers
> themselves.
>
> 3. artifacts
> Artifacts are files and directories created as a result of the
> unarchiving process. They have two names:
> 3.1 artifact.meta
>
> artifact.meta has data from the process used to unpack the data.
> These files can sometimes be as large as the archive itself. The
> purpose of saving these is to answer questions about the unpack.
> There is one artifact.meta for each artifact. We don't make use of
> these today. I've attached a snippet from one artifact.meta so you can
> see what they can look like.
_______________________________________________
fossology mailing list
[email protected]
http://fossology.org/mailman/listinfo/fossology
