Bron Gondwana wrote:
On Sun, Jun 19, 2016, at 08:16, Howard Chu wrote:
Not quite sure I understand your use case. It sounds like you'll maintain a
local store, with recent records, and a remote store with older records. Is
that correct?
For the database I'd keep everything local, but I want to write everything back
to
the store such that a host crash doesn't lose any data - and I don't want to
amplify
writes out to absurdity doing it - though some level of amplification is
obviously
OK.
A good example of what I want is the way that the xapianactive file works in
Cyrus
search at FastMail:
https://blog.fastmail.com/2014/12/01/email-search-system/
Because only the most recent database is writable (in this case on tmpfs,
because
we don't need 100% reliability for search, it only takes about 20 minutes to
scan
every mailbox and reindex the stuff that was on tmpfs after a crash)
Every other database is read-only - and you can compact multiple of them
together
into a single database and then atomically switch the old ones out and the new
one
in with a single very quick xapianactive rewrite - so it's acceptable to stop
the world
while doing that.
This sounds like a lot of bother, particularly the bit about "checking if
tmpfs is full". It's also a bit confusing because you talk about "compacting"
which I interpret as "cleaning out empty/unused space inside a DB" but in
context it sounds like you really mean "merging" - combining multiple DBs into
a single DB.
If I were building this system with LMDB there would be no separate temp and
meta tiers. LMDB would just mmap the DB on the SSD and let the OS buffer cache
keep the hot pages in RAM. I'm not really sure I'd bother with multiple DBs
either, there's nothing to compact. The data tier would be no different from
the meta tier.
When you say you can reindex the stuff on tmpfs quickly, that means you're
only reindexing the most recent N emails?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
