I appreciate your help, but i have elected to switch back from Xapian to
squat search until the "longstanding issue" in Xapian is fixed. 🤷
This is a family email server with only five people's email on it, we
can definitely make do without Xapian.
On 8/15/25 2:52 PM, Nels Lindquist wrote:
On 2025-08-14 6:22 PM, Jonathan Kamens wrote:
On 8/14/25 2:23 PM, Nels Lindquist wrote:
What configuration directives do you have for the conversations
database (which is required for xapian) and for search?
conversations: 1
conversations_db: twoskip
search_engine: xapian
defaultsearchtier: default
defaultsearchpartition-default: /var/spool/imap-search
sync_log: on
sync_log_channels: squatter
You probably also want to add:
search_fuzzy_always: yes
That won't affect the behaviour you're seeing, but since Xapian can
only use fuzzy searches, it'll translate client requests for standard
searches for use with the Xapian engine which may increase client
performance.
There's a longstanding issue with xapian indexing which can be
worked around by setting search_batchsize to a very large number
(like 1000000), blowing away the contents of your search partition
and then regenerating with "squatter -v -i".
Yes, that sort of resolves the issue. I was able to rebuild my
indexes, but squatter -R still appears to be doing a ton of extra
work every time it starts up and causing excessive locking which is
blocking clients from functioning for noticeable delays while it is
considering reindexing every single message in every mailbox every
time it starts up.
It's disappointing that this longstanding issue hasn't been fixed and
doesn't seem to have been documented anywhere obvious that I could
find. Because of that I wasted many hours upgrading to 3.12.1 and
then digging into squatter in a debugger to try to figure out what's
going on.
Of note: when I said in my last message that the rolling squatter
process was doing the right thing I was mistaken. In fact the rolling
squatter was having the same problem, it was just rotating through
the mailboxes and reindexing the same messages in each one in turn,
rather than sticking with a single mailbox and indexing the same
messages in that mailbox over and over, so I didn't notice right away
that it was also malfunctioning.
Is your "squatter -R" setup in cyrus.conf still in the EVENTS section,
by chance?
If so, it should be moved to the DAEMON section. Once it's up and
running it should be continuously rolling search indexing from the
squatter replication channel you've defined, and shouldn't be doing
much of anything otherwise.
If you set up multiple search tiers for different types of storage,
you might still need some squatter EVENTS for periodically repacking
the search indexes between storage tiers. We haven't bothered with
that as the performance has been great with a single tier with our
typical loads. (See
http://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html
for some examples.)
We haven't noted mailbox locking due to squatter activity (even on
startup) in our environment, but we're still running Cyrus IMAPD 3.8.5
(self-built RPMs) on AlmaLinux 8, so no idea whether the behaviour's
changed in 3.10.x or 3.12.x
What kind of squatter logs are you seeing? This is pretty typical for us:
Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox
user/[redacted1]@maei.ca...
Aug 15 11:54:35 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 3
updates in 0.353203 sec
Aug 15 12:10:49 edm-cmbe01 cyrus/squatter[2716]: indexing mailbox
user/[redacted2]/[email protected]...
Aug 15 12:10:53 edm-cmbe01 cyrus/sync_client[2700]: Reprocessing sync
log file /var/lib/imap/sync/squatter/log-run
Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: Xapian committed 2
updates in 4.963152 sec
Aug 15 12:10:54 edm-cmbe01 cyrus/squatter[2716]: mappedfile: longlock
/var/lib/imap/user/uuid/5/c/5cf27ed7-050e-44dd-924d-eefb6835df76/xapianactive.db
for 5.0 seconds
------------------------------------------
Cyrus: Info
Permalink:
https://cyrus.topicbox.com/groups/info/T5a3c00cdf15ad8da-M2f4c2e7da4c04f1b926f9b20
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
