RE: Adding a 'restore' method to the NodeStore apis
Hi, This assumption was challenged via OAK-1357 with the idea that the MongoDB backup can already produce consistent (non-blocking) backups so there's nothing to be done in this area. Also restoring means only replacing the old db with the backup one. the MongoDB management service (MMS) mentioned by Jukka is one option, but I think we should also have a backup mechanism that works without a third party dependency. the other methods discussed in the MongoDB documentation [0] do not guarantee consistent backups in a sharded cluster, but I wonder if we can make use of the oplog. we could first create an approximate point in time backup as described here [1]. the backed up shards may not provide a consistent view across the mongo cluster. to get the shards in a consistent state we would query the oplog on each of the shards and apply them up to a known point that is consistent. regards marcel [0] http://docs.mongodb.org/manual/core/backups/ [1] http://docs.mongodb.org/manual/tutorial/backup-sharded-cluster-with-filesystem-snapshots/
Re: Adding a 'restore' method to the NodeStore apis
Hi, On Wed, Jan 29, 2014 at 3:21 AM, Marcel Reutegger mreut...@adobe.com wrote: the MongoDB management service (MMS) mentioned by Jukka is one option, but I think we should also have a backup mechanism that works without a third party dependency. Right, as a cloud service MMS is probably not appropriate for all deployments. the other methods discussed in the MongoDB documentation [0] do not guarantee consistent backups in a sharded cluster, but I wonder if we can make use of the oplog. Instead of trying to solve that problem at MongoDB level, I suggest we simply use the Oak-level backup code that we already have. It guarantees a consistent snapshot of the running repository, is non-blocking, and can do incremental backups. BR, Jukka Zitting
RE: Adding a 'restore' method to the NodeStore apis
Hi, Instead of trying to solve that problem at MongoDB level, I suggest we simply use the Oak-level backup code that we already have. It guarantees a consistent snapshot of the running repository, is non-blocking, and can do incremental backups. this works fine for the backup part and partial restore. the only concern I have is disaster recovery. just replacing the MongoDB files would be much faster than copying over the nodes from the tar files to MongoDB. Regards Marcel
Re: Adding a 'restore' method to the NodeStore apis
Hi, I'm back trying to gather some feedback from the people involved in the MongoDB store impl. This is about creating a backup of the current repository using a native backup tool, a tool that is running at databasel level. It was my understanding that if you run a (non blocking) backup at any given time, you might catch a repository in an inconsistent state (large transactions half-way completed maybe?), so you might need a way to mark the latest stable head before basically copying everything. Next on restore you would simply need to reset the head to the last known stable state and you get the full circle. I've found that the checkpoint mechanism we use for the async indexing fits this model nicely, and I was planning on using it in this context as well: marking the last state with a checkpoint, then using the same checkpoint id as a reference for the restore. This would work both in the case of a MongoDB store (also the RDB one) but also in the cases where the repository is too big and out backup code cannot handle it efficiently (think huge repo + file system snapshots). This assumption was challenged via OAK-1357 with the idea that the MongoDB backup can already produce consistent (non-blocking) backups so there's nothing to be done in this area. Also restoring means only replacing the old db with the backup one. If this is true, I'm as happy as it gets, I can already close down a bunch of issues :) but I want clear confirmation that this is in fact the way it works and that everybody agrees with it and so there are no loose ends. thanks for your attention, alex On Mon, Jan 27, 2014 at 1:22 PM, Alex Parvulescu alex.parvule...@gmail.comwrote: Hi, I've created OAK-1357 asking for a new method on the NodeStore apis: 'restore', please add your thoughts to the issue. thanks, alex
Re: Adding a 'restore' method to the NodeStore apis
Hi, re This assumption was challenged via OAK-1357 with the idea that the MongoDB backup can already produce consistent (non-blocking) backups so there's nothing to be done in this area. Also restoring means only replacing the old db with the backup one. I am not sure if MongoDB is able to produce a consistent backup as it cannot be aware of Oak’s document model. For example, a consistent revision might involve changes to documents located in /content (in Oak’s POV) and also the corresponding index nodes. There is one caveat (and I actually might be wrong with the above): afaik Oak commits the new revision as the last document to MongoDB. If MongoMK can guarantee that and additionally guarantee that any changes that are committed without the corresponding revision-ID-commit we might be able to do without explicitly marking a checkpoint. Michael On 28 Jan 2014, at 15:30, Alex Parvulescu alex.parvule...@gmail.com wrote: Hi, I'm back trying to gather some feedback from the people involved in the MongoDB store impl. This is about creating a backup of the current repository using a native backup tool, a tool that is running at databasel level. It was my understanding that if you run a (non blocking) backup at any given time, you might catch a repository in an inconsistent state (large transactions half-way completed maybe?), so you might need a way to mark the latest stable head before basically copying everything. Next on restore you would simply need to reset the head to the last known stable state and you get the full circle. I've found that the checkpoint mechanism we use for the async indexing fits this model nicely, and I was planning on using it in this context as well: marking the last state with a checkpoint, then using the same checkpoint id as a reference for the restore. This would work both in the case of a MongoDB store (also the RDB one) but also in the cases where the repository is too big and out backup code cannot handle it efficiently (think huge repo + file system snapshots). This assumption was challenged via OAK-1357 with the idea that the MongoDB backup can already produce consistent (non-blocking) backups so there's nothing to be done in this area. Also restoring means only replacing the old db with the backup one. If this is true, I'm as happy as it gets, I can already close down a bunch of issues :) but I want clear confirmation that this is in fact the way it works and that everybody agrees with it and so there are no loose ends. thanks for your attention, alex On Mon, Jan 27, 2014 at 1:22 PM, Alex Parvulescu alex.parvule...@gmail.comwrote: Hi, I've created OAK-1357 asking for a new method on the NodeStore apis: 'restore', please add your thoughts to the issue. thanks, alex
Re: Adding a 'restore' method to the NodeStore apis
Hi, On Tue, Jan 28, 2014 at 9:30 AM, Alex Parvulescu alex.parvule...@gmail.com wrote: This assumption was challenged via OAK-1357 with the idea that the MongoDB backup can already produce consistent (non-blocking) backups so there's nothing to be done in this area. MMS Backup offers point in time recovery of MongoDB replica sets and a consistent snapshot of sharded systems. http://docs.mongodb.org/manual/core/backups/ Alternatively, if the external database (MongoDB, or other) doesn't support consistent backups, I'd argue that it's better *not* to use the external backup mechanism and instead do an Oak-level backup to a TarMK repository. BR, Jukka Zitting
