On Sun, Jun 19, 2016, at 08:16, Howard Chu wrote: > If by multi-level you mean LSM style DBs, you should be aware that these are > extremely fragile in the face of application and OS crashes. It's a > consequence of the fact that the DB state and metadata is scattered across > multiple files, and you cannot atomically update all of them at once. LevelDB > and RocksDB are both prone to self-corruption after a crash.
Great :( *sigh* > (Google "rocksdb corruption" ...) > > Not quite sure I understand your use case. It sounds like you'll maintain a > local store, with recent records, and a remote store with older records. Is > that correct? For the database I'd keep everything local, but I want to write everything back to the store such that a host crash doesn't lose any data - and I don't want to amplify writes out to absurdity doing it - though some level of amplification is obviously OK. > One of the features we'll be adding in LMDB 1.0 is incremental backup. This > will write out all the DB pages that are newer than a particular > transactionID. (A prototype of this feature is in the "incre" branch > https://github.com/LMDB/lmdb/tree/incre but it needs some rework before > releasing in 1.0) That's definitely a step in the direction I need. A good example of what I want is the way that the xapianactive file works in Cyrus search at FastMail: https://blog.fastmail.com/2014/12/01/email-search-system/ Because only the most recent database is writable (in this case on tmpfs, because we don't need 100% reliability for search, it only takes about 20 minutes to scan every mailbox and reindex the stuff that was on tmpfs after a crash) Every other database is read-only - and you can compact multiple of them together into a single database and then atomically switch the old ones out and the new one in with a single very quick xapianactive rewrite - so it's acceptable to stop the world while doing that. Bron. -- Bron Gondwana br...@fastmail.fm
