Hi, In our workload, cyrus replication latency is pretty critical as we serve most read requests from the replica. Having a single network channel between master & replica is a big issue for us.
Trying to improve our latency, we implemented the following approach : instead
of writing “channel/log” we write “channel/log.<shard_index>”.
We compute our shard key this way :
# cat log.0
APPEND devenv.blue!user.tom.Sent
MAILBOX devenv.blue!user.tom.Sent
# cat log.2
SEEN t...@devenv.blue 9f799278-a6cd-45b7-9546-0e861d5e15d6
root@bm1804:/var/lib/cyrus/sync/core# cat log.3
…
APPEND devenv.blue!user.sga
MAILBOX devenv.blue!user.sga
We compute an hashcode of the first argument. We normalize it so
devenv.blue!user.tom.Sent and devenv.blue!user.tom have the same hashcode then
we “hashcode % shard_count” to figure out which log file to use.
We patched sync_client to add a “-i <shard_index>”. sync_client -i 0 will
process log.0 and use log-run.0, etc.
We don’t spawn sync_client from cyrus.conf but we prefer systemd tricks :
/lib/systemd/system/bm-cyrus-syncclient@.service which is a template and we
then enable :
systemctl enable bm-cyrus-syncclient@{0..3} to spawn 4 sync_client.
Attached diff of what we changed.
As a side note, our usage forbids moving a mailbox folder into another mailbox
(ie. moving user.tom.titi into user.sga.stuff is forbidden in our setup). I
guess this approach would be problematic we moving a mailbox subfolder to
another mailbox as they might be sharded to separate log files.
Any feedback on this approach ? I read that you planned to turn sync_client
into a sync daemon. Any schedule estimate on that ?
Regards,
Thomas.
sync_client systemd configuration template :
/lib/systemd/system/bm-cyrus-syncclient@.service (%i is expanded to 42 by
systemd when you enable syncclient@42)
[Unit]
Description=BlueMind Cyrus sync_client service
After=bm-cyrus-imapd.service
PartOf=bm-cyrus-imapd.service
ConditionPathExists=!/etc/bm/bm-cyrus-imapd.disabled
[Service]
Type=forking
Environment=CONF=/etc/imapd.conf
ExecStartPre=/usr/bin/find /var/lib/cyrus/sync -name ‘log*.%i' -type f -exec rm
-f {} \;
ExecStart=/usr/sbin/sync_client -C $CONF -t 1800 -n core -i %i -l -r
SuccessExitStatus=75
RemainAfterExit=no
Restart=always
RestartSec=5s
TimeoutStopSec=20s
[Install]
WantedBy=bm-cyrus-imapd.service
Thomas Cataldo
Directeur Technique
(+33) 6 42 25 91 38
BlueMind
+33 (0)5 81 91 55 60
Hotel des Télécoms, 40 rue du village d'entreprises
31670 Labège, France
www.bluemind.net / https://blog.bluemind.net/fr/
replication_multiplexing.diff
Description: Binary data
