Hi,

> Now the problem comes when secondary tries to run the sweep phase.  It will
> first try to verify that a references file exists for each repository file
> in DS_P - and fail.  This fails because primary deleted its references file
> already.  Thus secondary will cancel GC and thus blob C never ends up
> getting deleted.  Note that secondary must delete C because it is the only
> repository that knows about C.
>
> This same situation exists also if secondary sweeps first.  If record D was
> created by primary after secondary was cloned, then D is deleted by
> primary, secondary never knows about blob D so it cannot delete it during
> the sweep phase - it can only be deleted by primary.
>

The solution for Shared DataStore currently is to require all repositories
to run a Mark phase then run the Sweep phase on one of them.


>
> The change I made to the garbage collector is that when a repository
> finishes the sweep phase, it doesn’t necessarily delete the references
> file.  Instead it marks the data store with a “sweepComplete” file
> indicating that this repository finished the sweep phase.  When there is a
> “sweepComplete” file for every repository (in other words, the last
> repository to sweep), then all the references files are deleted.
>
> Well currently the problem is that all repositories are not required to
run the sweep phase. The solution above would have been ok when the GC is
to be run manually at different times as in your case.
But in the real world applications typically there's a cron (e.g. AEM
maintenance task) which could be setup to execute weekly at a particular
time on all repositories. In this case in almost all cases the repository
which finished the Mark phase at the last would only be able to execute the
Sweep phase as it would be the only repository to see all the reference
files for other repos (others executing before it would fail). This is
still Ok for the Shared DataStore use cases we have.
But with the above solution since not all repositories would be able to run
the sweep phase the reference files won't be cleaned up.

Besides there's a problem of the Sweep phase on the primary encountering
blobs it does not know about (from the secondary) and which it cannot
delete creating an unpleasant experience. As I understand the Primary could
be a production system and having these sort of errors crop up would be
problematic.

So, generically the solution would be to use the shared DataStore GC
paradigm we currently have which requires Mark phase to be run on all
repositories before running a Sweep.

For this specific use case some observations and quick rough sketch of a
possible solution:
*  The DataStores for the 2 repositories - Primary & Secondary can be
thought of as Shared & Private
** Primary does not know about Secondary and could be an existing
repository and thus does not know about the DataStore of the Secondary as
well. In other words it could even function as a normal DataStore and need
not be a CompositeDataStore.
** Secondary does need to know about the Primary and thus registers itself
as sharing the Primary DataStore.
* Encode the blobs ids on the Secondary with the DataStore location/type
with which we can distinguish the blob ids belonging to the respective
DataStores.
* Secondary's Mark phase only redirects the Primary owned blobids to the
references file in the Primary's DataStore (Primary's DataStore operating
as Shared).
* Secondary executes GC for its DataStore independently and does not worry
about the Shared blobids (already taken care of above).

I presume some of the above steps are required to enable a generic or even
some restricted CompositeDataStore solutions.

Thanks
Amit

Reply via email to