On Mon, Oct 27, 2014 at 9:26 PM, John Meinel <j...@arbash-meinel.com> wrote: > What are the tables we're wanting to filter? (charm storage because of its > size?)
The presence DB and a new one for backups. We actually do want to back up the blobstore DB (charms/tools). > Certainly 3 and 4 aren't reasonable if the point is to avoid having 4GB of > charm data in the backup. Yeah, I'd rather avoid that if possible. It would also make the restore situation messier. We should be able to avoid including unnecessary (even invalid) data from the backup archive. > > If we went with (1) for consistency sake, we should really go back to "stop > the db, dump to a backup". Yep. Alternatively we could lock the databases. Either way it would mean juju would be unresponsive during backup. I suppose when we have a replicaset we could take one member offline to make the backup, but I'd rather avoid that sort of complication. > It is also sensitive to us adding a collection and forgetting to include it > in the dump. (since it is a whitelist, right?) I've already implemented option 1 (http://reviews.vapour.ws/r/268/). The patch actually uses a blacklist, taking advantage of mgo.Session.DatabaseNames(). > > I'd really like (2) as a blacklist option, but I don't know how hard that is > to do, and how much we expend to dump GB of charm data just to filter them > out again. mongodump actually dumps each database to its own directory. If necessary, the trick would be to hack the dumped oplog to remove references to the ignored databases. I just need to verify that mongorestore --oplogReplay will be happy with that, or even if mongorestore offers us any other route. I'll be taking some time today to look into it. Thanks for the feedback. After hearing your thoughts and taking another look with fresh eyes, option 2 seems much more feasible and palatable. My only concern is that it relies on what amount to implementation details of mongodump (and mongorestore). -eric > > John > =:-> > > On Tue, Oct 28, 2014 at 3:07 AM, Eric Snow <eric.s...@canonical.com> wrote: >> >> For backup we currently call mongodump with the --oplog option. This >> dumps all databases at the same moment in time, which gives us >> assurances about the consistency of each database. However, we want >> to exclude some databases. Unfortunately --oplog is not compatible >> with the --db option. >> >> I'm looking for options here. I'm already considering, but am not >> satisfied with, the following: >> >> 1. (mongodump --db ...) Forgo the moment-in-time guarantee of --oplog >> (seems risky). >> 2. (mongodump --oplog ...) Manually remove the undesired databases >> from the dumped data (possible? risky!). >> 3. (mongodump --oplog ...) Skip the undesired databases when calling >> mongorestore (possible?). >> 4. (mongodump --oplog ...) Clear the undesired databases after calling >> mongorestore. >> >> The problem with 3 and 4 is that the backup archive will contain a >> bunch of erroneous data (the dumps of the databases we want to >> ignore). -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
