[
https://issues.apache.org/jira/browse/GEODE-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fred Krone updated GEODE-2654:
------------------------------
Labels: storage_2 (was: )
> Backups can capture different members from different points in time
> -------------------------------------------------------------------
>
> Key: GEODE-2654
> URL: https://issues.apache.org/jira/browse/GEODE-2654
> Project: Geode
> Issue Type: Bug
> Components: persistence
> Reporter: Dan Smith
> Labels: storage_2
>
> Geode backups should behave the same as recovering from disk after killing
> all of the members.
> Unfortunately, the backups instead can backup data on different members at
> different points in time, resulting in application level inconsistency.
> Here's an example of what goes wrong:
> # Start a Backup
> # Do a put in region A
> # Do a put in region B
> # The backup finishes
> # Recover from the backup
> # You may see the put to region B, but not A, even if the data is colocated.
> We ran into this with with lucene indexes - see GEODE-2643. We've worked
> around GEODE-2643 by putting all data into the same region, but we're worried
> that we still have a problem with the async event queue. With an async event
> listener that writes to another geode region, because it's possible to
> recover different points in time for the async event queue and the region,
> resulting in missed events.
> The issue is that there is no locking or other mechanism to prevent different
> members from backing up their data at different points in time. Colocating
> data does not avoid this problem, because when we recover from disk we may
> recover region A's bucket from one member and region B's bucket from another
> member.
> The backup operation does have a mechanism for making sure that it gets a
> point in time snapshot of *metadata*. It sends a PrepareBackupRequest to all
> members which causes them to lock their init file. Then it sends a
> FinishBackupRequest which tells all members to backup their data and release
> the lock. This ensures that a backup doesn't completely miss a bucket or get
> corrupt metadata about what members host as bucket. See the comments in
> DiskStoreImpl.lockStoreBeforeBackup.
> We should extend this Prepare/Finish mechanism to make sure we get a point in
> time snapshot of region data as well. One way to do this would be to get a
> lock on the *oplog* in lockStoreBeforeBackup to prevent writes and hold it
> until releaseBackupLock is called.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
