Hi,
We are working with a replication enabled cyrus 3.0.8.
Our replication endpoint is not another cyrus but I don’t think it matters
here. I am working on enabling object storage support with our setup.
Our replication settiings look like this :
sync_log: 1
sync_log_channels: core
core_sync_authname: admin0
core_sync_password: xxxxx
core_sync_realm: repl_realm
core_sync_host: 172.16.167.129
core_sync_port: 2501
core_sync_repeat_interval: 0
core_sync_try_imap: 0
We made a copy of objectstore_dummy.c and simplified it to hardcode the path to
/dummy-sds/<message guid> for testing purposes.
The object storage/archive setup looks like this :
object_storage_enabled: 1
archive_enabled: 1
archive_days: 0
archive_maxsize: 0
archive_keepflagged: 0
>From an imap point of view, everything works fine, emails are written through
>the object storage and are read from there.
# ls -l /dummy-sds/
...
-rw------- 1 cyrus mail 1491 Aug 28 11:35
ae3aaefa50d04042bc369e29e9c819ed600d3a03
-rw------- 1 cyrus mail 863 Aug 28 11:35
f996f4ad4e4c37c3c2cc852214fdeca778d9c43f
The problem occurs when sync_client triggers its replication code.
>From our logs we see:
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: creating sql_db
/var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/message.db
Aug 28 11:35:23 bm1804 cyrus/imap[116313]: USAGE ad...@devenv.blue user:
0.000000 sys: 0.010090
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES
devenv.blue!user.admin.Sent devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: Delivered:
<ffe06ca2d52c5d64a0d5401f5f23a...@devenv.blue> to mailbox: devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX
devenv.blue!user.admin.Sent
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: IOERROR: Failed to read file
/var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/a/user/admin/Sent/4.
Aug 28 11:35:23 bm1804 cyrus/lmtp[49143]: USAGE t...@devenv.blue user: 0.000000
sys: 0.011507
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOX devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: IOERROR: Failed to read file
/var/spool/cyrus/data/bm-master__devenv_blue/domain/d/devenv.blue/t/user/tom/1.
Aug 28 11:35:23 bm1804 cyrus/sync_client[60125]: MAILBOXES devenv.blue!user.tom
Aug 28 11:35:23 bm1804 cyrus/imap[33172]: session initialised for
ad...@devenv.blue
Aug 28 11:35:23 bm1804 cyrus/imap[33172]: login: bm1804.devenv.blue
[172.16.167.129] ad...@devenv.blue PLAIN User logged in
SESSIONID=<cyrus-33172-1566984923-1-5992269715888192037>
>From a protocol point of view we receive (REPL C is what sync_client sends us,
>REPL S is what we respond):
2019-08-28 09:35:23,310 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000001]: GET MAILBOXES
(devenv.blue!user.admin.Sent devenv.blue!user.tom)
2019-08-28 09:35:23,326 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000001]: * MAILBOX
%(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME
devenv.blue!user.admin.Sent SYNC_CRC 583232180 SYNC_CRC_ANNOT 0 LAST_UID 3
HIGHESTMODSEQ 6 RECENTUID 3 RECENTTIME 1566981904 LAST_APP... [truncated]
2019-08-28 09:35:23,329 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000002]: APPLY RESERVE
%(PARTITION bm-master__devenv_blue MBOXNAME (devenv.blue!user.admin.Sent
devenv.blue!user.tom) GUID (f996f4ad4e4c37c3c2cc852214fdeca778d9c43f
ae3aaefa50d04042bc369e29e9c819ed600d3a03))
2019-08-28 09:35:23,332 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000002]: * MISSING
(f996f4ad4e4c37c3c2cc852214fdeca778d9c43f
ae3aaefa50d04042bc369e29e9c819ed600d3a03)
OK success
2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000003]: APPLY MESSAGE
(NIL)
2019-08-28 09:35:23,334 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL S: [frame-00000003]: OK success
2019-08-28 09:35:23,337 [vert.x-eventloop-thread-1]
n.b.b.c.r.s.ReplicationSession INFO - REPL C: [frame-00000004]: APPLY MAILBOX
%(UNIQUEID 2971165a-f29e-46f1-8d97-ec2f62eb3e94 MBOXNAME
devenv.blue!user.admin.Sent SYNC_CRC 3650189183 SYNC_CRC_ANNOT 0 LAST_UID 4
HIGHESTMODSEQ 7 RECENTUID 3 RECENTTIME 1566981904 LAST_APPENDDATE 1566984923
POP3_LAST_LOGIN 0 POP3_SHOW_AFTER 0 UIDVALIDITY 1566980951 PARTITION
bm-master__devenv_blue ACL "admin0 lrswipkxtecdan
e8f1e28f-8778-442b-a581-c6b613ce9...@devenv.blue lrswipktecdan "
OPTIONS P ANNOTATIONS (%(ENTRY /specialuse USERID ad...@devenv.blue VALUE
{5+}{t28.bin})) RECORD (%(UID 4 MODSEQ 7 LAST_UPDATED 1566984923 FLAGS (\Seen)
INTERNALDATE 1566984923 SIZE 863 GUID
f996f4ad4e4c37c3c2cc852214fdeca778d9c43f)))
The problem seems pretty “obvious” : sync_client APPLY RESERVE for the 2 bodies
(from Sent folder + the copy for the recipient inbox), the it tries to read the
bodies but I imagine the sync_client code is not object-storage enabled
correctly and only tries the local filesystem path instead of asking to the
object storage.
It could also be an expected behaviour, as the replica(s) can share the object
storage with master and should reply that nothing is missing.
What is your point of view on that ? Could you give me a point to where in the
sync_client code I should look to “object storage”-enable it.
Quite un-related but we enjoyed reading your stuff about sync improvements and
having multiple sync streams between master & replica. If we can help on that
we would be happy to.
Regards,
Thomas.
Thomas Cataldo
Directeur Technique
(+33) 6 42 25 91 38
BlueMind
+33 (0)5 81 91 55 60
Hotel des Télécoms, 40 rue du village d'entreprises
31670 Labège, France
www.bluemind.net / https://blog.bluemind.net/fr/
