On Sat, May 25, 2019, at 22:19, Dilyan Palauzov wrote:
Hello Bron,
For me it is still not absolutely clear how things work with the
Xapian seach backend.
Has search_batchsize any impact during compacting? Does this setting
say how many new messages have to arrive, before indexing them
together in Xapian?
No, search_batchsize just means that when you're indexing a brand
new mailbox with millions of emails, it will put that many emails at
a time into a single transactional write to the search index. During
compacting, this value is not used.
What are the use-cases to call "squatter -F" (In compact mode, filter
the resulting database to only include messages which are not expunged
in mailboxes with existing name/uidvalidity.) and "squatter -X"
(Reindex all messages before compacting. This mode reads all the
lists of messages indexed by the listed tiers, and re-indexes them
into a temporary database before compacting that into place)?
-F is useful to run occasionally so that your search indexes don't
grow forever. When emails are expunged, their matching terms aren't
removed from the xapian indexes, so the database will be bigger than
necessary and when you search for a term which is in deleted emails,
it will cause extra IO and conversations DB lookups on the document
id.
Why shall one keep index of deleted and expunged messages and how to
delete references from messages that are both expunged and expired
(after cyr_expire -X0, so removed from the hard disk), but keep the
index to messages that are still on the hard disk, but the user
expunged (double-deleted) them.
I'm not sure I understand your question here. Deleting from xapian
databases is slow, and particularly with the compacted form, it's
designed to be efficient if you don't write to it. Finally, since
we're de-duplicating by GUID, you would need to do a conversations
db lookup for every deleted email to check the refcount before
cleaning up the associated record.
How does re-compacting (as in
https://fastmail.blog/2014/12/01/email-search-system/) differ from
re-indexing (as in the manual page of master/squatter)?
"re-compacting" - just means combining multiple databases together
into a single compacted database - so the terms in all the source
databases are compacted together into a destination database. I used
"re-compacting" because the databases are already all compacted, so
it's just combining them rather than gaining the initial space
saving of the first compact.
"re-indexing" involves parsing the email again and creating terms
from the source document. When you "reindex" a set of xapian
directories, the squatter reads the cyrus.indexed.db for each of the
source directories to know which emails it claims to cover, and
reads each of those emails in order to index them again.
What gets indexed? For a mailbox receiving only reports (dkim, dmarc,
mta-sts, arf, mtatls), some of which are archived (zip, gzip) the
Xapian index increases very fast.
This would be because these emails often contain unique identifiers,
which do indeed take a lot of space. We have had lots of debates
over what exactly should be indexed - for example should you index
sha1 values (e.g. git commit identifiers)? They're completely
random, and hence all 40 characters need to be indexed each time!
But - it's very handy to be able to search your email for a known
identifier and see where it was referenced... so we decided to
include them.
We try not index GPG parts or other opaque blobs where nobody will
be interested in searching for the phrase. Likewise we don't index
MIME boundaries, because they're substructure, not something a user
would know to search for.
We have a work in progress on the master branch to index attachments
using an external tool to extract text from the attachment where
possible, which will increase index sizes even more if enabled!
How can I remove a tier, that contains no data, but is mentioned in
the .xapianactive files?
If you run a compact which includes that tier as a source and not as
a destination, then it should remove that tier from every
.xapianactive file, at which point you can remove it from your
imapd.conf.
How can I rename a tier?
The whole point of tier names not being paths on disk is so you can
change the disk path without having to rename the tier. Tier names
are IDs, so you're not supposed to rename them.
Having said that, you could add a new tier, compact everything
across to that tier, then remove the old tier.
How can I efficiently prepend a new tear in the .xapianactive file?
“squatter -t X -z Y -o” does add to the .xapianactive files the
defaultsearhtier, but first has to duplicate with rsync all existing
files. This is not efficient, as big files have to copied.
I'm afraid that's what we have right now. Again, tiers are supposed
to be set up at the start and not fiddled with afterwards, so the
system isn't designed to allow you to quickly add a new tier.
> What it does under the hood is creates a new database and copy all
> the documents over from the source databases, then compress the end
> result into the most compact and fastest xapian format which is
> designed to never write again. This compressed file is then stored
> into the target database name, and in an exclusively locked
> operation the new database is moved into place and the old tiers are
> removed from the xapianactive, such that all new searches look into
> the single destination database instead of the multiple source
> databases.
I do not get this. The amount of tiers to check does not reduce after
doing merging and with three tears the amount of databases is most of
the time three.
Not if you're compacting frequently. We do the following:
* hourly
- check if tmpfs is > 50% full - quit if not.
- run squatter -a -o -t temp -z data
* daily
- regardless of tmpfs size, compact everything on temp and meta down to data
- squatter -a -t temp,meta -z data
* weekly on Sunday - re-compact all data partitions together
- squatter -a temp,meta,data -z data
* And finally, once per week once the re-compact is done, check if
we need to filter and recompact the archive, if so:
- squatter -a data,archive -z archive -F
Since today is Monday, most users will have two, so the xapianactive
might be something like:
temp:66 data:52 data:51 archive:2
Later in the week, it might be:
temp:70 data:66 data:55 data:54 data:53 data:52 data:51 archive:2
And then maybe it will re-compact on Sunday and the user will have
temp:74 archive:3
What happens, if squatter is terminated during Xapian-compacting,
apart from leaving temporary files? Will rerunning it, just start
from beginning?
The source databases will still be in xapian.active, so yes - a new
compact run will take those same source databases and start again.
Is the idea to have three tiers like this:
At run time, new messages are indexed by Xapian in squatter-rolling
mode on tmpfs/RAM, say on tear T1.
That's certainly what we do, since indexing is too IO-intensive otherwise.
Regalarly, the RAM database is compacted to hard disk (tear T2), say
T1 and T2 are megred into T2. The database on the hard disk is
read-only and search in it is accelerated, as the database is “compact”.
As above - during the week we don't even merge T2 back together, we
compact from T1 to a single small database on T2 - leading to
multiple databases on T2 existing at once.
Only if two compactions happen in parallel of the same sources or
destination, the merge fails and is skipped for that user. The merge
is retried whenever merging T1 and T2 is rescheduled.
Yes - though that's pretty rare on our systems because we use a lock
around the cron task, so the only time this would happen is if you
ran a manual compaction at the same time as the cron job.
As the databases in T2 get bigger, merging T1 and T2 takes more and
more time. So one more Xapian tear is created, T3. Less regularly,
T2 and T3 are merged into T3. This process takes a while. But
afterwards, T2 is again small, so merging T1 and T2 into T2 is fast.
Yes, that's what we do. This is also the time that we filter the DB,
so the T3 database only contains emails which were still alive at
the time of compaction.
How many tears make sense, apart from having one more for power-off events?
Having another one for power off events doesn't make heaps of sense
unless you have a fast disk. That's kind of what our "meta"
partition is, it's an SSD RAID1 that's faster than the "data"
partition which is a SATA spinning RAID1 set.
When we power off a server, we run a task to compact all the temp
partitions down - it used to be to meta, but we found that
compacting straight to data was plenty fast, so we just do that now!
If you power off a server without copying the indexes off tmpfs,
they are of course lost. This means that you need to run squatter -i
on the server after reboot to index all the recent messages again!
So we always run a squatter -i after a crash or power outage before
bringing that server back into production.
Cheers,
Bron.
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
