Hi Bob,
Thanks for continuing working on this discussion, I hope you are not tired:)
As discuss with you about a long mail loop and discuss with Dan, I summarize my
opinion of backup solution scope:
1. Provide user a instructions to backup the entire repo as you suggested:
stop scheduler
pg_dumpall > myfile
backup myfile
backup repo gold, files, and license
start scheduler
Suggest user if you have enough disk space, we recommend user to use this
backup solution. And I suggest we also list approximately backup and restore
time will cost in the instructions, in order to give user to tradeoff.
2. I see your comments about how to make the repo significantly smaller, I am
absolutely agree with your opinion. We should select which files we should save
and which files we should not save but now in the repository. I consider this
should also add to the backup scope, what do you think?
3. We should also provide a only backup gold files solution if user don't have
enough disk space or don't want to backup entire repo. This need more code
changes, so I am now working on investigate all the code changes we need to
implement the gold file backup.
Hope my email fully explain my thoughts.
Hi Dan, if I list something wrong or not fully clear please help give more
comments and clarity.
Thanks,
Vincent
> -----Original Message-----
> From: Gobeille, Robert
> Sent: Tuesday, August 11, 2009 12:45 PM
> To: Dan Stangel
> Cc: Ma, Dong (vinc...@gdcc-bj-most); [email protected]
> Subject: Re: [FOSSology] unpack size/time
>
> Dan,
> Perhaps my email wasn't clear. It says "also need to provide user
> instructions to backup the entire repo". By "also" I meant "in
> addition to" a gold backup/recovery.
>
> Thanks for clarifying a couple points I could not get an answer to:
> 1) Our only backup devices are the cluster disks.
> 2) RFO is driving the requirements
>
> I hate to continue this discussion since Vincent is only pursuing the
> gold option, but when you say that we do not have enough disk space on
> the rfo cluster, that is incorrect. The repo could grow ~4x it's
> current size before we would run out of disk space to duplicate the
> repo. We have roughly 5.7 TB on the cluster and the current repo is
> 671 GB. Also, no one ever commented on my previous email discussing
> how to make the repo significantly smaller. That's fine, because we
> should do the gold method anyway and can get more space efficient later.
>
> To wrap this up,
>
> 1) Vincent you are looking into all the code changes we need to
> implement the gold file backup. Is that correct?
>
> 2) In the mean time, we are not backing up the rfo repo. So if any
> one of the 6 repo sections (the repository is randomly distributed
> over 6 agent machines) crashes, then we will be left with random
> missing files. Our options then would be to either wait for the code
> changes (the changes for the gold recovery will recover from this
> situation) or we simply trash the entire database and repo and start
> over.
>
> Is there anything else?
>
> Bob Gobeille
> [email protected]
>
>
>
> On Aug 10, 2009, at 6:58 PM, Dan Stangel wrote:
>
> > Bob,
> >
> > I agree with you, the best user experience would come from backup and
> > restore of the entire repository. In our case, however, there's a
> big
> > problem: We do not have enough unallocated disk space on the RFO
> > cluster to
> > implement this.
> >
> > So I don't see a problem in recommending the full repo backup
> > method, but in
> > our case we will not be able to implement this, and will have to use
> > the
> > gold files-only method.
> >
> > Dan
> >
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of Gobeille,
> Robert
> > Sent: Monday, August 10, 2009 2:18 PM
> > To: Ma, Dong (vinc...@gdcc-bj-most)
> > Cc: [email protected]
> > Subject: Re: [FOSSology] unpack size/time
> >
> > Hi Vincent,
> > As per my previous email, this is probably not a great way to back up
> > a fossology database. But it's good functionality to have because we
> > could add a feature that allows users to remove all non-reused files
> > except gold to save disk space. I don't see any bad ramifications
> > except for the user experience (waiting for unpack). The code
> changes
> > needed in unpack are minimal. All the places in the UI that read
> > files will have to change. So you don't want to create this backup
> > solution until all the code changes are done.
> >
> > I think you also need to provide user instructions to backup the
> > entire repo. That is:
> > stop scheduler
> > pg_dumpall > myfile
> > backup myfile
> > backup repo gold, files, and license
> > start scheduler
> >
> > Bob
> >
> >
> > On Aug 10, 2009, at 1:31 AM, Ma, Dong (vinc...@gdcc-bj-most) wrote:
> >
> >> Hi Bob,
> >>
> >> If no dependencies with the unpacked files at the backup point, I
> >> think we should not do unpack on the fly at the restore time. We
> >> just restore the gold files and give the user's opinion to unpack
> >> the gold files as they needed.
> >> Is this opinion make sense? Or this will bring some bad
> >> ramifications?
> >>
> >> Thanks,
> >> Vincent
> >>
> >>
> >>> -----Original Message-----
> >>> From: Gobeille, Robert
> >>> Sent: Friday, August 07, 2009 10:56 PM
> >>> To: Ma, Dong (vinc...@gdcc-bj-most)
> >>> Cc: [email protected]
> >>> Subject: Re: unpack size/time
> >>>
> >>>
> >>> On Aug 7, 2009, at 2:20 AM, Ma, Dong (vinc...@gdcc-bj-most) wrote:
> >>>
> >>>> Hi Bob,
> >>>>
> >>>> Also have 2 more questions talk with you:
> >>>> 1. Backup all repo files, the incremental disk backup time is not
> >>>> an
> >>>> issue, but the restore process will also cost so much time when
> >>>> restore all the repo file every time(one day or more). I just
> >>>> consider this situation:
> >>>> a. only backup gold(and license) files
> >>>> b. when restore, only unpack the files which have dependency with
> >>>> backup point running jobs (if no dependency with running jobs,
> will
> >>>> not unpack any gold files in restore)
> >>>
> >>> See previous email. There are no dependencies from the interrupted
> >>> unpack.
> >>>
> >>>> c. unpack other files with user's demand after restore
> >>>> I am not sure the percentage of 'unpack the files which have
> >>>> dependency with backup point running jobs' with whole repo unpack
> >>>> files, if very little unpacked files should be unpacked in restore
> >>>> process, the time cost will not the issue. May be the time will
> >>>> less
> >>>> than restore all the repo file.
> >>>> I hope this question not confuse you.
> >>>
> >>> Restore only the gold (and license) files would typically be much
> >>> faster than restoring all the repo files. Se my previous email in
> >>> this thread for times.
> >>>
> >>>
> >>>> 2. About ' Allow users to "archive" uploads.', I have a question
> is
> >>>> if users have this requirement? If user want this feature of
> delete
> >>>> the unpacked files? My thought is the users don't care about
> >>>> unpacked files occupied how many disk space and want do delete
> them
> >>>> if disk spaces is enough.
> >>>
> >>> So far, this has not come up as a user requirement. The only thing
> >>> I've seen from users is the need to delete the entire upload from
> >>> the
> >>> database and repository. And we already do that.
> >>>
> >>> Bob Gobeille
> >
> > _______________________________________________
> > fossology mailing list
> > [email protected]
> > http://fossology.org/mailman/listinfo/fossology
> >
> >
> >
_______________________________________________
fossology mailing list
[email protected]
http://fossology.org/mailman/listinfo/fossology
