Hi Bob,
Also have 2 more questions talk with you:
1. Backup all repo files, the incremental disk backup time is not an issue, but
the restore process will also cost so much time when restore all the repo file
every time(one day or more). I just consider this situation:
a. only backup gold(and license) files
b. when restore, only unpack the files which have dependency with
backup point running jobs (if no dependency with running jobs, will not unpack
any gold files in restore)
c. unpack other files with user's demand after restore
I am not sure the percentage of 'unpack the files which have dependency with
backup point running jobs' with whole repo unpack files, if very little
unpacked files should be unpacked in restore process, the time cost will not
the issue. May be the time will less than restore all the repo file.
I hope this question not confuse you.
2. About ' Allow users to "archive" uploads.', I have a question is if users
have this requirement? If user want this feature of delete the unpacked files?
My thought is the users don't care about unpacked files occupied how many disk
space and want do delete them if disk spaces is enough.
What do you think about these?
Thanks,
Vincent
> -----Original Message-----
> From: Ma, Dong (vinc...@gdcc-bj-most)
> Sent: Thursday, August 06, 2009 8:44 PM
> To: Gobeille, Robert
> Cc: [email protected]
> Subject: RE: unpack size/time
>
> Hi Bob,
>
> Thanks for your useful comments.
>
> The consideration of unpack "the most important" gold files, as if we
> only backup the gold files, I consider there will be some job needed
> some unpack files at the backup point, when the restore time if we
> don't unpack these gold files, some jobs may failed.
> For example:
> a. At the backup point, the unpack job is running, unpack a file
> b. begin backup
> c. when restore, I think the unpacked files will be needed
> (these consideration will confirm with you, I am not sure about this,
> and I am also not sure if other jobs depend on the unpack file)
>
>
> With your test case data, you point
> > So to me the backup solution is really two fold:
> > 1) With no changes to fossology, you can back up the system today.
> > Just stop the scheduler, do a pg_dumpall and then rsync the entire
> repo
> > to a backup drive.
> >
> > 2) Allow users to "archive" uploads. Meaning delete all the repo
> files
> > except the gold. This will free up disk space for them at the cost
> of
> > having to run unpack again should the files be needed. This will
> take
> > some coding changes. This was actually in the initial fossology
> design,
> > but we never implemented it. This will also allow someone to
> restrict
> > their repo backup to just the gold (and license) files.
>
> I am fully agree with you, I think if we have enough disk spaces, your
> backup solution is good to me.
>
> Base on your solution, I think we can do follow things about this:
> 1. Discuss with team about the solution, talk with Dan if we have
> enough disk space to backup all the repo files about our production
> system
> 2. Consider how to reduce the repo unpack files size, I think you
> give a more detail information, I can work with you about this
> 3. Investigate how many code changes base on the solution(2), I
> think I can initial to do the investigate
>
> What about your thought?
>
> Thanks,
> Vincent
>
>
>
> > -----Original Message-----
> > From: Gobeille, Robert
> > Sent: Thursday, August 06, 2009 5:55 AM
> > To: Ma, Dong (vinc...@gdcc-bj-most)
> > Cc: [email protected]
> > Subject: Re: unpack size/time
> >
> > Hi Vincent,
> > If you back up only the gold files, and unpack "the most important"
> > ones at restore time, you still need to be able to unpack on the fly.
> > So why bother taking a guess at what are "the most important" uploads
> > and slowing down the restore?
> >
> > To me the possibilities are:
> > 1) backup and restore only gold (and license) files. This requires
> > code changes to unpack on the fly.
> > 2) reduce the size of the repository to make feasible backing up all
> > the repository, and thus speeding up recovery and lowering backup and
> > live disk space requirements.
> >
> > But people should understand the tradeoffs. But first, here is some
> > new data:
> > - Total unpacked size will be less in a working repository due to
> file
> > reuse. So to get a more realistic number by including file reuse,
> I
> > measured the files and golds on grumpy (even though this repository
> is
> > weighted towards isos):
> > gold 13 GB
> > files 99 GB
> > inflation 8x
> >
> > The disk space requirements seem clear it takes roughly 8x the disk
> > space to save all the unpacked files over the size of the original
> > files. I think we also have a good handle on the time requirements
> > (at least for us at OSUOSL). On the system we use:
> > A 600GB (what we use today) repo backup would take roughly 3 hours
> > to copy to another disk (assuming similar transfer speed as we see
> from
> > their ftp server).
> > The 61 GB of gold files on our system would take 18 minutes
> >
> > But keep in mind that disk backups (except for the first one) are
> > incremental. If you add 6 GB of gold files a day, the incremental
> > backup time for backing up the entire repo is around 18 minutes (for
> 60
> > GB of unpacked files), 2 minutes for just the gold files. I suspect
> > very few people add that much to their repo in a day but it is for
> > illustration.
> >
> > On the restore side, restoring 600GB would still take roughly 3 hours
> > with no unpack necessary. 61GB of gold take the 18 minutes to copy,
> > but unpacking 61 GB of gold files would take over 3 days. Based on
> the
> > 4.2 GB Fedora 11 iso which took 5.5 hrs to unpack, 61 GB could
> > conceivably take 4,800 minutes = 80 hours.
> >
> > Personally, I back up my laptop on a $200 1TB firewire raid that I
> can
> > write to at over 60MB/s. So I'm speculating that some users would
> > prefer to back up their entire repo because disk space can be pretty
> > inexpensive and incremental disk backup time is not an issue (18
> > minutes / day for a large repo).
> >
> > It also makes sense that people may want to clean up their repo by
> > removing rarely needed files and just keep the gold.
> >
> > So to me the backup solution is really two fold:
> > 1) With no changes to fossology, you can back up the system today.
> > Just stop the scheduler, do a pg_dumpall and then rsync the entire
> repo
> > to a backup drive.
> >
> > 2) Allow users to "archive" uploads. Meaning delete all the repo
> files
> > except the gold. This will free up disk space for them at the cost
> of
> > having to run unpack again should the files be needed. This will
> take
> > some coding changes. This was actually in the initial fossology
> design,
> > but we never implemented it. This will also allow someone to
> restrict
> > their repo backup to just the gold (and license) files.
> >
> > There is also a related issue that I mentioned briefly before - to
> not
> > save all the files in the repo that we currently save. Here is some
> > background:
> >
> > There are three types of files that unpack saves:
> > 1. leaf files
> > The leaf files are simply files that can be unpacked no further. For
> > example, file.c, myfile.spec, myfile.png, ...
> >
> > 2. containers
> > Containers are archives and "artifacts". For example, containers are
> > files like file.gz, file.rpm, file.jar, file.war, file.ott, ...
> Since
> > these are unpacked to leaf nodes, we could delete the containers
> > themselves.
> >
> > 3. artifacts
> > Artifacts are files and directories created as a result of the
> > unarchiving process. They have two names:
> > 3.1 artifact.meta
> >
> > artifact.meta has data from the process used to unpack the data.
> > These files can sometimes be as large as the archive itself. The
> > purpose of saving these is to answer questions about the unpack.
> > There is one artifact.meta for each artifact. We don't make use of
> > these today. I've attached a snippet from one artifact.meta so you
> can
> > see what they can look like.
_______________________________________________
fossology mailing list
[email protected]
http://fossology.org/mailman/listinfo/fossology
