Bob, Your analysis is correct.
One small clarification on the disk space: You said that we have additional space on the repo. We have unused space in the repository, but we do not have unallocated space in the SAN storage. In order to implement a full repo backup as you described, we would need to either shrink the allocated repo size by 1/2, or add twice as many disks as we currently have. All of the 5.7 TB of repository space is allocated currently. We would have to either shrink the repository to approximately 2.8 TB -- still quite a lot of space -- or else add an extra 5.7 TB of disks. Anyhow, I think the gold-file-only backup that you and Vincent have been describing is an OK alternative. Dan -----Original Message----- From: Gobeille, Robert Sent: Monday, August 10, 2009 10:45 PM To: Dan Stangel Cc: Ma, Dong (vinc...@gdcc-bj-most); [email protected] Subject: Re: [FOSSology] unpack size/time Dan, Perhaps my email wasn't clear. It says "also need to provide user instructions to backup the entire repo". By "also" I meant "in addition to" a gold backup/recovery. Thanks for clarifying a couple points I could not get an answer to: 1) Our only backup devices are the cluster disks. 2) RFO is driving the requirements I hate to continue this discussion since Vincent is only pursuing the gold option, but when you say that we do not have enough disk space on the rfo cluster, that is incorrect. The repo could grow ~4x it's current size before we would run out of disk space to duplicate the repo. We have roughly 5.7 TB on the cluster and the current repo is 671 GB. Also, no one ever commented on my previous email discussing how to make the repo significantly smaller. That's fine, because we should do the gold method anyway and can get more space efficient later. To wrap this up, 1) Vincent you are looking into all the code changes we need to implement the gold file backup. Is that correct? 2) In the mean time, we are not backing up the rfo repo. So if any one of the 6 repo sections (the repository is randomly distributed over 6 agent machines) crashes, then we will be left with random missing files. Our options then would be to either wait for the code changes (the changes for the gold recovery will recover from this situation) or we simply trash the entire database and repo and start over. Is there anything else? Bob Gobeille [email protected] On Aug 10, 2009, at 6:58 PM, Dan Stangel wrote: > Bob, > > I agree with you, the best user experience would come from backup and > restore of the entire repository. In our case, however, there's a big > problem: We do not have enough unallocated disk space on the RFO > cluster to > implement this. > > So I don't see a problem in recommending the full repo backup > method, but in > our case we will not be able to implement this, and will have to use > the > gold files-only method. > > Dan > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Gobeille, Robert > Sent: Monday, August 10, 2009 2:18 PM > To: Ma, Dong (vinc...@gdcc-bj-most) > Cc: [email protected] > Subject: Re: [FOSSology] unpack size/time > > Hi Vincent, > As per my previous email, this is probably not a great way to back up > a fossology database. But it's good functionality to have because we > could add a feature that allows users to remove all non-reused files > except gold to save disk space. I don't see any bad ramifications > except for the user experience (waiting for unpack). The code changes > needed in unpack are minimal. All the places in the UI that read > files will have to change. So you don't want to create this backup > solution until all the code changes are done. > > I think you also need to provide user instructions to backup the > entire repo. That is: > stop scheduler > pg_dumpall > myfile > backup myfile > backup repo gold, files, and license > start scheduler > > Bob > > > On Aug 10, 2009, at 1:31 AM, Ma, Dong (vinc...@gdcc-bj-most) wrote: > >> Hi Bob, >> >> If no dependencies with the unpacked files at the backup point, I >> think we should not do unpack on the fly at the restore time. We >> just restore the gold files and give the user's opinion to unpack >> the gold files as they needed. >> Is this opinion make sense? Or this will bring some bad >> ramifications? >> >> Thanks, >> Vincent >> >> >>> -----Original Message----- >>> From: Gobeille, Robert >>> Sent: Friday, August 07, 2009 10:56 PM >>> To: Ma, Dong (vinc...@gdcc-bj-most) >>> Cc: [email protected] >>> Subject: Re: unpack size/time >>> >>> >>> On Aug 7, 2009, at 2:20 AM, Ma, Dong (vinc...@gdcc-bj-most) wrote: >>> >>>> Hi Bob, >>>> >>>> Also have 2 more questions talk with you: >>>> 1. Backup all repo files, the incremental disk backup time is not >>>> an >>>> issue, but the restore process will also cost so much time when >>>> restore all the repo file every time(one day or more). I just >>>> consider this situation: >>>> a. only backup gold(and license) files >>>> b. when restore, only unpack the files which have dependency with >>>> backup point running jobs (if no dependency with running jobs, will >>>> not unpack any gold files in restore) >>> >>> See previous email. There are no dependencies from the interrupted >>> unpack. >>> >>>> c. unpack other files with user's demand after restore >>>> I am not sure the percentage of 'unpack the files which have >>>> dependency with backup point running jobs' with whole repo unpack >>>> files, if very little unpacked files should be unpacked in restore >>>> process, the time cost will not the issue. May be the time will >>>> less >>>> than restore all the repo file. >>>> I hope this question not confuse you. >>> >>> Restore only the gold (and license) files would typically be much >>> faster than restoring all the repo files. Se my previous email in >>> this thread for times. >>> >>> >>>> 2. About ' Allow users to "archive" uploads.', I have a question is >>>> if users have this requirement? If user want this feature of delete >>>> the unpacked files? My thought is the users don't care about >>>> unpacked files occupied how many disk space and want do delete them >>>> if disk spaces is enough. >>> >>> So far, this has not come up as a user requirement. The only thing >>> I've seen from users is the need to delete the entire upload from >>> the >>> database and repository. And we already do that. >>> >>> Bob Gobeille > > _______________________________________________ > fossology mailing list > [email protected] > http://fossology.org/mailman/listinfo/fossology > > > _______________________________________________ fossology mailing list [email protected] http://fossology.org/mailman/listinfo/fossology
