Re: minutes, cyrus call, 2020-10-12
On Tue, Oct 13, 2020, at 00:44, Ricardo Signes wrote: > * brong >* locking! conversationsdb has locking problems, annotations are "a whole > locking nightmare" >* there's a lock inversion between JMAP and other calls in the locking of > convdb vs. mailboxes >* we can change how convdb locking works internally so you can > take/release a lock without closing the entire db, add a user lock, then > we're done!(?) >* …but it's a bunch of work, and that will be Bron's next project for Cyrus Some more about this... So... I just reverted my attempt from https://github.com/cyrusimap/cyrus-imapd/pull/3232 to fix the deadlock problems. Here's the root problem: mailbox.c:mailbox_open_advanced() takes a *shared NAMELOCK on the mailbox name*, then opens the index and first locks the user.conversations file before locking the indexes. So far so good. BUT: JMAP (and other things which need a long lock) takes a *(shared/exclusive) lock on the user.conversations file BEFORE it opens mailboxes*. This leads to a lock inversion. My attempted fix was to release the namelock while locking the conversations - which reverted everything to the order that the JMAP code does. This caused errors because the mailbox could be deleted during that time that the lock is missing, and that broke invarients. There are a couple of options: *1) always lock conversations first* Refactor mailbox handling to always lock the conversations.db BEFORE it tries to lock the mailbox name. This means that we probably need to find a bunch of places in the code where we're doing copy and rename actions and lock one or both of the conversations databases early as well, so the lock is held across the entire action. *2) create a new user-level lock for JMAP* This would be useful for other things and would stop our current abuse of the conversations.db as a user-level lock. Instead we could open and close it along with mailboxes and commit more frequently, which would give better transaction safety. It does mean yet another lock, which would need to be managed in both mailbox_open_* and anywhere which is currently using the conversations database as a proxy lock. ... There are pros and cons of both. I'm leaning a bit towards the second one because it allows us to commit the conversations DB along with the mailbox in each case. ... Here's the log information that led to knowing about this, using https://github.com/cyrusimap/cyrus-imapd/pull/3179 in a build: 2020-10-07T07:30:37.415533-04:00 fastmail1 slotf1t01/sync_client[4156099]: LOCKORDER: 0=* 1= 2=*  2020-10-07T07:30:37.078907-04:00 fastmail1 slotf1t01/httpjmap[4156617]: LOCKORDER: *0= 1=* ... So the plan appears to be: *1) do a new user-level lock* that is held when any mailbox is opened as well as held for the entire command by JMAP. This will probably be a reuse of the existing user_namespacelock which is used for rename safety. *2) make conversations lock inside mailboxes* - for efficiency this will be done by making it not locked for the entire time it's open, so making it more like every other cyrusdb which can be held open for efficiency and only locked when used. Conversations will be lazyloaded inside mailbox if not already open, and it will just use refcounting like every other DB. *3) BONUS: move per-mailbox annotations to only be a mailbox internal thing too* - and just like cache, lazyload it when needed. I'm going to get started on that when I get a chance. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
A "Threeskip" database format (because there can never be too many skips)
I've been thinking about this one for a very long time. It's still not going to be as excellent as zeroskip, but it has the advantage that it's still a single file, and it fixes two of the three defects of twoskip: Those three defects are: 1) poor write locality - updates happen throughout the file 2) long repack freeze gives unpredictable performance 3) no MVCC reads - writers block all readers. Threeskip will fix two of those! Numbers 2 and 3. For comparison, here's the twoskip ondisk format: * HEADER: 64 bytes * magic: 20 bytes: "4 bytes same as skiplist" "twoskip file\0\0\0\0" * version: 4 bytes * generation: 8 bytes * num_records: 8 bytes * repack_size: 8 bytes * current_size: 8 bytes * flags: 4 bytes * crc32: 4 bytes * * RECORDS: * type 1 byte * level: 1 byte * keylen: 2 bytes * vallen: 4 bytes * * * ptrs: 8 bytes * (level+1) * crc32_head: 4 bytes * crc32_tail: 4 bytes * key: (keylen bytes) * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple Threeskip is almost exactly the same, with three types (and of course with 'threeskip file\0\0\0' in the header) RECORD NEWVALUE DELETE RECORD: * type 1 byte * level: 1 byte * keylen: 2 bytes * vallen: 4 bytes * * ** latestptr: 8 bytes (new)* * ptrs: 8 bytes * (level+1) * crc32_head: 4 bytes * crc32_tail: 4 bytes * key: (keylen bytes) * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple NEWVALUE: * type: 1 byte * padding: 3 bytes * vallen: 4 bytes * ** prevptr: 8 bytes* * crc32_head: 4 bytes * crc32_tail: 4 bytes * val: (vallen bytes) * padding: enough zeros to round up to an 8 byte multiple +1 DELETE: * type: 1 byte * ZEROS: 7 bytes * prevptr: 8 bytes * crc32_head: 4 bytes * crc32_tail: 4 bytes (ZERO) This gives us MVCC! When you start a read transaction, the transaction includes the length of the file at the time of transaction start. The reader holds the same filehandle open, meaning it can keep reading even after a repack. If it reads past the transaction length, it just moves on to the next record. If it follows the latestptr past the transaction length, it just follows the prevptr back until it's at the latest value within the transaction. (DELETE already had a prevptr, but it was less powerful) *DUAL LOCKING * With this comes a second MVCC help. We use two different writelocks, which can be achieved with ranged flock within the header. Lock number 1 is held for the entire transaction and avoids any other writer. Lock number 2 is ONLY held for the duration of writing a single record and updating the related pointers. Readers take a shared lock on number 2 only, which allows them to guarantee clean reads. *NON-BLOCKING REPACK* The repack process can take an MVCC read and stream those records to a new file $DB.new (on which it holds an exclusive lock to avoid racing repacks), and then it can replay the log from there. Even the log replay can be done with read locks if the log length is over a certain amount (though this can lead to starvation, so should be a limited number of times) . The final fsync and rename needs to be done with a writelock on the source file of course. Reading the log, if we hit a NEWVALUE or DELETE, we follow the prevptrs back to find out the key that it references. It might be necessary to have a "repackd" which does the actual repacking, or some other way of a process saying "I don't mind waiting on the repack" because otherwise the repack freeze will still be visible to the current client, even if other readers and writers can continue. *STORAGE COST COMPARISON** * This adds an extra 8 bytes to the overhead for every CREATE record. The DELETE record is the same size. The NEWVALUE record no longer has the cost of an extract copy of the key in it. So it will use slightly more storage on newly repacked databases. (The base-level average overhead of a twoskip record is 40 bytes, this makes it 48) *COMPUTATIONAL AND IO COST COMPARISON** * Calculating "repack_size" will be a little more costly, but not too bad. Replacing values will be much cheaper because we don't need to update pointers. In theory we could even "row lock" or "gap lock" if we wanted other concurrency levels. *CRC ALGORITHM** * Since we have a choice with a new format, clearly we'd go with something fast! CRC32c with hardware acceleration all the way. *CRASH SAFETY AND THE OTHER TWOSKIP BENEFITS* All still there just as much as ever. Of course, write locality is helped slightly (overwrite and delete only touch the current record) but it's still the issue that will make zeroskip a better DB format. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: expanding allowsetacl
On Thu, Jun 25, 2020, at 00:57, Ricardo Signes wrote: > Behold this commit: > > commit da8305164877735ba29b078151c70455f1aa6eea Author: Bron Gondwana Date: Wed Oct 30 14:13:38 2019 +1100 imapd: allow disabling the SETACL command > > I'm not sure what the intent here was, but I *assume* it was related to our > plan to kill of the ability of users to set their own ACLs. If so, I think we > need two small changes which I'd like to get out pretty soon. > > 1. relax the restriction so the admin user can still use SETACL > 2. tighten the restriction so it also restricts DELETEACL > > That's it! Then we'll move on to rolling that out and cleaning up users' > existing ACLs. I have tentatively made a task for this > <https://app.liquidplanner.com/space/14822/projects/show/58761531>. > > Bron: I just want to make sure we're not stepping on work intended for > something else! So... this does also handle DELETEACL, because they both call cmd_setacl, just with different paramenters. Maybe it should be called cmd_frobacl or something. As for allowing the admin - the way to do that is for the admin to connect to a different imapd which has no such restrictions. At least the way this is intended for use at Fastmail, it will only be on set on particular services entries rather than globally, so the `imapadmin` service won't have it set. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: Status of ftp server?
On Tue, Feb 25, 2020, at 10:16, Quanah Gibson-Mount wrote: > Hi, > > The documentation in numerous places states that release tarballs can be > obtained from "ftp.cyrusimap.org". However, this server appears to be > offline. > > Example reference: > <https://www.cyrusimap.org/imap/download/getcyrus.html#use-a-release-packaged-tarball> > > Do all the various documentation locations referencing FTP need to be > updated, or is it intended to bring the FTP server back online? Thanks for reminding us about the FTP server situation, we'll figure something out shortly one way or the other. I suspect we'll actually move to entirely HTTP, since FTP is a pretty awful protocol to firewall and manage compared to HTTP which can be easily hosted directly out of Github, which is where our code and issue tracker is stored. Cheers, Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: Cyrus IMAPd version 3.1.8
FYI: this is almost exactly what Fastmail is running in production right now - it has about 4 minor commits beyond current production, and is missing the handful of Fastmail specific magic! OldRev: cyrus-imapd-3.1.8 NewRev: fmstable-20191203v1 Removes the following commits: 44210c59 2019-12-02 brong: jmap: don't need to update the alive value, mailbox.c already did that on write 463f6291 2019-12-03 brong: caldav: fix bogus && to & in read_cb 41f6117e 2019-12-02 brong: caldav: scheduling enabled should always be checked on the shared annotation (aka: owner) 3d69f08c 2019-12-03 rsto: jmap_mail: report "xapian" perf filter for contact group searches 469cacc0 2019-12-04 rsto: jmap_mail: move Identity data to jmap:submission capability dc91d2d4 2019-12-04 rsto: jmap_mail: reject mutable search in queryChanges 0b757d88 2019-12-04 rsto: jmap_mail_query: don't crash for nested multipart alternatives 9b30ee3f 2019-12-04 ellie: release notes for 3.1.8 18d157e0 2019-12-04 ellie: fix cve link in 3.1.7 release notes 96d194de 2019-12-04 ellie: developer release 3.1.8 Adds the following commits: 1c6ed3ad 2015-03-30 brong: Fastmail Secrets (no rated) 5ba2dbee 2015-03-30 brong: Fastmail ONLY - make assertion failures and fatal errors into coredumps 26d563c1 2015-03-30 brong: Fastmail ONLY - Remove sieve action string 0e71d55c 2017-08-18 brong: Fastmail ONLY - don't fiddle timezone data in http_caldav_sched.c 61da4794 2018-06-26 brong: Fastmail ONLY - re-apply the VEVENTS ONLY patch for alarms c51f3989 2019-02-06 rsto: Fastmail ONLY - mailbox owners always have ACL_ADMIN in JMAP 2f3e9516 2015-08-07 brong: mkdebian: fastmail build script (v29) This is from the attached "GitBranchDiff" script. All the commits listed as only being on "master" will be merged into Fastmail production next week when we rebase our build on the 3.1.8 tag (and possibly more changes from master too) Cheers, Bron. On Wed, Dec 4, 2019, at 14:27, ellie timoney wrote: > The Cyrus team is pleased to announce the immediate availability of a new > version of Cyrus IMAP: 3.1.8 > > This is a snapshot of the master branch, and should be considered for testing > purposes and bleeding-edge features only. It is available as a git tag, which > can be found here: > > https://github.com/cyrusimap/cyrus-imapd/releases/tag/cyrus-imapd-3.1.8 > > Join us on Github at https://github.com/cyrusimap/cyrus-imapd to report > issues, join in the deliberations of new features for the next Cyrus IMAP > release, and to contribute to the documentation. > > On behalf of the Cyrus team, > > ellie -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com GitBranchDiff.pl Description: Perl program
Changing JMAP IDs for Calendar and Contacts to be server generated
Hi All, This was discussed into today's Cyrus call, but I figured I should put it here for a public note and to cover the discussion in more detail :) Fastmail has a "caldav_sync" tool, which replicates calendars from outside. Right now we rewrite the UID both ways in order to allow uniqueness of UIDs within our system, because we also constrain each UID to only exist once in all of a user's calendars (because of scheduling). This is variously buggy and annoying. Looking at various solutions for embedding mailboxid as well into the JMAP id, we came to the conclusion that the best move was actually to generate JMAP IDs synthetically on first receipt of a UID, and maintain that ID across changes. This has a couple of other good benefits: * doesn't use random junk off the wire as part of ids * can maintain the JMAP id even when moving between different calendars * fixed length IDs for JMAP, whereas UIDs can be quite long from some services * restricted character set means we don't have to escape parts of the UID (which is not ObjectId safe) All together, a big win. The same as not using the Message-Id header from emails, we won't use the UID from calendars or contacts. I'm looking at potential options for upgrade path for existing events - possibly even rewriting them on disk! It will definitely need a dav_db rewrite. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
CalendarEvents: should we patch the patch, or patch the model?
This is triggered by the fact that our current updates of calendar participants
on overridden events is broken, and the reason is this:
'recurrenceOverrides/2019-03-01T09:00:00/participants~1baz/participationStatus'
=> 'accepted',
This is a patch against an event with the following overrides.
"recurrenceOverrides": {
"2019-02-01T09:00:00": {
"duration": "PT2H"
},
"2019-03-01T09:00:00": {
"participants/baz": {
"@type": "Participant",
"sendTo": {
"imip": "mailto:baz@local;
},
"email": "baz@local",
"name": "Baz",
"kind": "individual",
"roles": {
"attendee": true
},
"locationId": "loc3",
"participationStatus": "needs-action",
"expectReply": true,
"scheduleSequence": 0
}
}
},
And the following participants at the top level:
"participants": {
"bar": {
"@type": "Participant",
"sendTo": {
"imip": "mailto:bar@local;
},
"email": "bar@local",
"name": "Bar",
"kind": "individual",
"roles": {
"attendee": true
},
"locationId": "loc2",
"participationStatus": "needs-action",
"expectReply": true,
"scheduleSequence": 0
},
"foo": {
"@type": "Participant",
"sendTo": {
"imip": "mailto:foo@local;
},
"email": "foo@local",
"name": "Foo",
"kind": "individual",
"roles": {
"owner": true,
"attendee": true,
"chair": true
},
"locationId": "loc1",
"participationStatus": "accepted",
"expectReply": false,
"scheduleSequence": 0,
"participationComment": "Sure; see you \\"soon\\"!"
}
},
Note that if 'baz' was also present on the top level, the patch with this
syntax would be:
'recurrenceOverrides/2019-03-01T09:00:00/participants~1baz~1participationStatus'
=> 'accepted',
And theoretically if there were no participants on the top level event (not
possible in this model) it would be:
'recurrenceOverrides/2019-03-01T09:00:00/participants/baz/participationStatus'
=> 'accepted',
Because there would be a rich participants object in the override.
*My contention is this:**
*
We should always patch the model, not the wire representation. The wire
representation is a compact representation of the event, in which each
recurrence override is just a patch of what needs to be changed, but critically
*an entire copy of the event is a valid patch as well*.
I would argue that we should always patch the overrides as if we were patching
a complete copy of the event, so that the last format is always correct.
Otherwise, we're assuming that the server always calculates the patch in the
same way, and there's no guarantee of that.
This means the client would need to be changed rather than Cyrus to fix our
production bug. I have vague memories of discussing this with Robert and
deciding that patching the entire model as if the datastructure was fully
resolved is the only sane approach, now that I come back an dig into this some
more. This means that it doesn't matter how the calendar event is stored on the
server, because it always has a way to expand every recurrence out to the full
object, and then calculate the minimised version for storage and /get again.
This also means that we never get weird '~1' items in the patch paths when we
update an event over the wire, which is kinda nice - and the same patch will
still apply cleanly even if 'baz' got invited to the master event in the
meantime.
Bron.
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Re: [RFC] multiplexing cyrus replication with log/log-run sharding & multiple sync_client
Oh I should just add the other thing that you might be interested in that I've
got some initial stabs at - synchronous replication. Embedding the sync_client
logic into mailbox commit such that any action that writes to a mailbox does a
pass through and creates a "SYNC APPLY MAILBOX" dlist stanza and shoves it down
the wire at a replica. There's a couple of bits missing so far - it needs a way
to upload the message content as well, which I'm probably just going to embed
directly the the RECORD compontent - it's kind of ugly but it makes it a single
DLIST - and obviously it needs to not fail entirely if the replica is down, so
it might be fire and forget or it might have a timeout after which it syslogs
but returns.
It builds on top of the existing SINCE_MODSEQ and SINCE_UIDNEXT logic that's
already in master, and will also want a "sync cache" - which will store the
remote MAILBOX line for each mailbox, so you can generate a SYNC APPLY without
first having to do a SYNC GET to find the current remote state. Assuming the
replica hasn't changed in the meanwhile, this will allow for single round trip
apply of changes rather than the current 4 round trips for an APPEND.
(truly, it's 4 round trips!)
S0 SYNCGET MAILBOX user.cassandane
S1 SYNCAPPLY RESERVE
S2 SYNCAPPLY MESSAGE
S3 SYNCAPPLY MAILBOX
In my plan, the "SYNCGET MAILBOX" would not be needed, because you'd already
know the remote state. The "RESERVE" would not be needed because you'd already
know from the local conversations.db that this message wasn't listed in any
other mailbox or with a UID less than the UIDNEXT of the remote mailbox from
that known remote state, so all you'd have left is the SYNCAPPLY MESSAGE and
the SYNCAPPLY MAILBOX. So it's just a matter of merging those into a single
round trip with some nice combined format, and you're done :)
Bron.
On Fri, Nov 22, 2019, at 02:25, Bron Gondwana wrote:
> Wow, interesting. That definitely works, though I'd probably normalise
> everything to the user ID so that the seen and mailbox events for the same
> user got the same channel.
>
> We're looking at similar things for our setup too, either shading or even per
> user logs with a daemon which farms users out to multiple channels.
>
> As for when we'd look at a sync daemon: probably next year. We're planning to
> land uuid based storage soon, which means that renaming users and mailboxes
> is really fast, then looking at replication channels on top of that would
> make more sense, because otherwise user renames become tricky.
>
> I'll have a look at the diff when it isn't 11:30pm for me.
>
> Cheers,
>
> Bron
>
> On Thu, Nov 21, 2019, at 18:50, Thomas Cataldo wrote:
>> Hi,
>>
>> In our workload, cyrus replication latency is pretty critical as we serve
>> most read requests from the replica.
>> Having a single network channel between master & replica is a big issue for
>> us.
>>
>> Trying to improve our latency, we implemented the following approach :
>> instead of writing “channel/log” we write “channel/log.”.
>> We compute our shard key this way :
>>
>> # cat log.0
>> APPEND devenv.blue!user.tom.Sent
>> MAILBOX devenv.blue!user.tom.Sent
>>
>> # cat log.2
>> SEEN t...@devenv.blue 9f799278-a6cd-45b7-9546-0e861d5e15d6
>> root@bm1804:/var/lib/cyrus/sync/core# cat log.3
>> …
>> APPEND devenv.blue!user.sga
>> MAILBOX devenv.blue!user.sga
>>
>> We compute an hashcode of the first argument. We normalize it so
>> devenv.blue!user.tom.Sent and devenv.blue!user.tom have the same hashcode
>> then we “hashcode % shard_count” to figure out which log file to use.
>> We patched sync_client to add a “-i ”. sync_client -i 0 will
>> process log.0 and use log-run.0, etc.
>>
>> We don’t spawn sync_client from cyrus.conf but we prefer systemd tricks :
>>
>> /lib/systemd/system/bm-cyrus-syncclient@.service which is a template and we
>> then enable :
>> systemctl enable bm-cyrus-syncclient@{0..3} to spawn 4 sync_client.
>>
>>
>> Attached diff of what we changed.
>>
>> As a side note, our usage forbids moving a mailbox folder into another
>> mailbox (ie. moving user.tom.titi into user.sga.stuff is forbidden in our
>> setup). I guess this approach would be problematic we moving a mailbox
>> subfolder to another mailbox as they might be sharded to separate log files.
>>
>> Any feedback on this approach ? I read that you planned to turn sync_client
>> into a sync daemon. Any schedule estimate on that ?
>>
>> Regards,
>> Thomas.
>>
>>
>> sync_client systemd configuration template :
>> /lib/systemd/system/bm-c
Re: [RFC] multiplexing cyrus replication with log/log-run sharding & multiple sync_client
Wow, interesting. That definitely works, though I'd probably normalise
everything to the user ID so that the seen and mailbox events for the same user
got the same channel.
We're looking at similar things for our setup too, either shading or even per
user logs with a daemon which farms users out to multiple channels.
As for when we'd look at a sync daemon: probably next year. We're planning to
land uuid based storage soon, which means that renaming users and mailboxes is
really fast, then looking at replication channels on top of that would make
more sense, because otherwise user renames become tricky.
I'll have a look at the diff when it isn't 11:30pm for me.
Cheers,
Bron
On Thu, Nov 21, 2019, at 18:50, Thomas Cataldo wrote:
> Hi,
>
> In our workload, cyrus replication latency is pretty critical as we serve
> most read requests from the replica.
> Having a single network channel between master & replica is a big issue for
> us.
>
> Trying to improve our latency, we implemented the following approach :
> instead of writing “channel/log” we write “channel/log.”.
> We compute our shard key this way :
>
> # cat log.0
> APPEND devenv.blue!user.tom.Sent
> MAILBOX devenv.blue!user.tom.Sent
>
> # cat log.2
> SEEN t...@devenv.blue 9f799278-a6cd-45b7-9546-0e861d5e15d6
> root@bm1804:/var/lib/cyrus/sync/core# cat log.3
> …
> APPEND devenv.blue!user.sga
> MAILBOX devenv.blue!user.sga
>
> We compute an hashcode of the first argument. We normalize it so
> devenv.blue!user.tom.Sent and devenv.blue!user.tom have the same hashcode
> then we “hashcode % shard_count” to figure out which log file to use.
> We patched sync_client to add a “-i ”. sync_client -i 0 will
> process log.0 and use log-run.0, etc.
>
> We don’t spawn sync_client from cyrus.conf but we prefer systemd tricks :
>
> /lib/systemd/system/bm-cyrus-syncclient@.service which is a template and we
> then enable :
> systemctl enable bm-cyrus-syncclient@{0..3} to spawn 4 sync_client.
>
>
> Attached diff of what we changed.
>
> As a side note, our usage forbids moving a mailbox folder into another
> mailbox (ie. moving user.tom.titi into user.sga.stuff is forbidden in our
> setup). I guess this approach would be problematic we moving a mailbox
> subfolder to another mailbox as they might be sharded to separate log files.
>
> Any feedback on this approach ? I read that you planned to turn sync_client
> into a sync daemon. Any schedule estimate on that ?
>
> Regards,
> Thomas.
>
>
> sync_client systemd configuration template :
> /lib/systemd/system/bm-cyrus-syncclient@.service (%i is expanded to 42 by
> systemd when you enable syncclient@42)
> [Unit]
> Description=BlueMind Cyrus sync_client service
> After=bm-cyrus-imapd.service
> PartOf=bm-cyrus-imapd.service
> ConditionPathExists=!/etc/bm/bm-cyrus-imapd.disabled
>
> [Service]
> Type=forking
> Environment=CONF=/etc/imapd.conf
> ExecStartPre=/usr/bin/find /var/lib/cyrus/sync -name ‘log*.%i' -type f -exec
> rm -f {} \;
> ExecStart=/usr/sbin/sync_client -C $CONF -t 1800 -n core -i %i -l -r
> SuccessExitStatus=75
> RemainAfterExit=no
> Restart=always
> RestartSec=5s
> TimeoutStopSec=20s
>
> [Install]
> WantedBy=bm-cyrus-imapd.service
>
>
>
>
>
> Thomas Cataldo
> Directeur Technique
>
> (+33) 6 42 25 91 38
>
> BlueMind
> +33 (0)5 81 91 55 60
> Hotel des Télécoms, 40 rue du village d'entreprises
> 31670 Labège, France
> www.bluemind.net / https://blog.bluemind.net/fr/
>
>
>
>
> *Attachments:*
> * replication_multiplexing.diff
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Re: time for cyrus-imap v3.2?
Given that Cyrus is unix processes, not threads, what mechanism are you
proposing for the atomic here?
I'd be keen to see something shaped like a pull request against Cyrus showing
how this would interact with the existing locking architecture.
Cheers,
Bron.
On Wed, Nov 13, 2019, at 16:12, Anatoli via Cyrus-devel wrote:
> > How do you know when the current write operations are finished?
>
> The pseudocode from my previous email:
>
> __start:
>
> atomic_inc(write_operations_in_execution_counter)
>
> atomic_load(write_barrier)
>
> if (write_barrier == ON) {
>
> atomic_dec(write_operations_in_execution_counter)
> sleep(100ms)
> goto __start
>
> } else {
>
> *perform_write_operation_with_its_own_locks()*
> atomic_dec(write_operations_in_execution_counter)
>
> }
>
>
> Here perform_write_operation_with_its_own_locks() is a function (one
> among many) that today performs a write operation (the algorithm implies
> that all occurrences of write operations should be wrapped with this
> pseudocode).
>
> Once a write function returns (which would mean that this particular
> write operation completed), the write_operations_in_execution_counter is
> decremented. When all operations complete, the counter would be == 0,
> which is detected by the barrier thread and it proceeds with the
> sync/fsync to have all files fully written to disk & notifies the
> external process waiting to take a snapshot. Then it turns off the
> barrier and the daemon continues normal operation.
>
> This is similar to a single global lock as you describe it, and it sort
> of behaves like it, but it doesn't need to be acquired as such every
> time and it has practically 0 overhead so it could be placed inside
> high-frequency loops. It's similar to how modern lock-free concurrency
> is implemented. The point it that it's an opportunistic locking which
> could be implemented with very low overhead.
>
> And if the code inside the perform_write_operation_with_its_own_locks()
> is guaranteed not to block on other
> perform_write_operation_with_its_own_locks() functions, then this
> implementation would be lock-free for when the barrier is OFF, i.e. no
> potential deadlocks. For when it's ON, there also would be no deadlocks,
> but the operation as such won't be fully lock-free.
>
> Also, the way I propose to implement the barrier thread, it won't block
> the server for more than a configurable amount of seconds no matter what
> (with this, the entire implementation would be lock-free (if we consider
> the snapshot as part of the process), i.e. it guarantees progress in a
> finite amount of time, though some threads could starve):
>
> atomic_store(write_barrier, ON)
>
> __check_write_counter:
>
> atomic_load(write_operations_in_execution_counter)
>
> if (write_operations_in_execution_counter == 0) {
>
> sync_data_to_disk()
> signal_data_ready()
> *wait_for_lock_release_with_timeout(5s)*
> atomic_store(write_barrier, OFF)
>
> } else {
>
> sleep(1ms)
> goto __check_write_counter
>
> }
>
>
> Here the wait_for_lock_release_with_timeout(5s) function will wait for
> the release-lock signal for 5 seconds and would turn off the barrier no
> matter if the external operation (snapshot-taking backup tool)
> completed, so the server would continue its normal operation once the 5s
> timeout expires.
>
> So while the barrier thread waits for the release-lock signal, the
> backup tool performs a snapshot and then sends the release-lock signal.
> The result of the signaling indicates whether the lock was released
> before or not. If the backup tool receives the code indicating that the
> lock was released before, it would mean that the snapshot that was taken
> could be inconsistent.
>
> In this case the backup tool could try to perform the operation again or
> proceed in another way (e.g. to notify the admin that the snapshot takes
> more than the preconfigured lock-wait time). Again this is the
> opportunistic locking, i.e. we try to perform an operation without a
> guarantee of success, so we don't need to wait indefinitely, again
> providing a lock-free guarantee. If we succeed, then all is good. If
> not, we try again or abandon the task with an error.
>
> And all this would be external to cyrus, it would be implemented in the
> backup utility.
>
> I guess the best way to start with this is to identify all places where
> data write operations occur (I suppose this is where the mail data and
> all sorts of databases are written). Once they are identified they could
> be tweaked a bit for better concurrency and lockability and then we
> could analyze how to wrap them with a global lock/barrier
Re: time for cyrus-imap v3.2?
On Tue, Nov 12, 2019, at 14:50, Anatoli wrote: > Bron, > > The proposed algo is a barrier before any single-lock. In itself it's a > single lock, but the same code (the pseudocode for the *worker thread* > in my previous mail) should be inserted at *every* single-lock/write > operation location. If there's no need to pause, the overhead is > non-existent. If a pause is requested, all worker threads would pause at > the entrance to any single-lock/write code. > > It would make the entire Cyrus daemon to complete all pending write > operations and pause new ones. At this stage, if I understand it > correctly, the data on disk would be in a consistent state, ready to > take a snapshot or to perform some other operation. "complete all pending write operations and pause new ones" How do you know when the current write operations are finished? > Without that, if we just take a snapshot of the fs, it could happen that > a) some files are not written entirely (i.e. caught in the middle of a > write operation) or b) the contents of some files are newer than the > other, i.e. the logical write operation was not atomic (e.g. mail data > is written but indexes are not updated yet or something similar). > > Maybe I didn't understand you correctly. Do you mean that finishing all > writes and pausing new ones is not enough to guarantee an integral state > of files on disk? If it's the case, what would have to be done to > guarantee it (i.e. to make it like Cyrus was shutdown normally)? I mean that to finish all writes and pause new ones, you need to know that the writes are finished. And not just writes, but sets of writes that are held under a lock together. The way I know to do this is a single global lock with the following properties: 1) every action which locks any file within Cyrus for writing takes a SHARED global lock before it takes the write lock on the file. 2) the SHARED lock is held for the duration of the writes, and released once the writes are finished. 3) the "backup utility" takes an EXCLUSIVE lock on the global lock, which will only be granted once each write is finished. It then takes a snapshot, and releases the EXCLUSIVE lock. This guarantees full consistency. The question that always exists for locks is "what granularity" - too wide, and you hold the lock for a long time. Too narrow, and you take and release it very frequently, adding overhead. My first and most dumbest theory is to go quite wide - add the lock in every runloop and command line utility such that it's held for the entire running of the loop or the utility! Mostly these are done within a fraction of a second. The one place that might be interesting is FETCH 1:* RFC822.PEEK or similar in imapd, where we already have some locking magic that holds a shared namelock on the mailbox to stop repacking while it releases the index lock to allow other actions on the mailbox in the meanwhile. So we could go down a layer and only lock when we lock mailboxes or cyrusdbs, and refcount the global lock. This seems more likely to be a good layer, and not too horrible. The other thing is that we'll need to assert that the lock isn't being held during each daemon's command loop, so that bugs don't leak out to deadlock entire servers. And I think that's nearly it :) Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: time for cyrus-imap v3.2?
On Thu, Nov 7, 2019, at 15:46, Anatoli via Cyrus-devel wrote: > Bron, > > Thanks for your detailed reply and the work the FM team is doing! > > > This is not easy unfortunately with all the different datastructures, > > because it means that everything else which takes a lock is going to > > need to first take a global shared lock before it does anything else, > > and that's going to have a performance and complexity impact on > > everything - because you have to find them ALL or you might wind up > > with lock inversions down the line. > > One solution I see is to have a separate single lock, not even a lock > per se, but a barrier. I.e. before every write operation How does that maintain consistency? I guess you don't get skew between files, but you still have to do crash recovery on every file. There's no single place to put this either. I think I still prefer the idea of a shared locks that wraps every single other lock, such that taking the snapshot pauses every attempt to start a new lock until it's done, so you always get a completely clean read equivalent to a clean shutdown. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: time for cyrus-imap v3.2?
On Wed, Nov 6, 2019, at 03:44, Anatoli via Cyrus-devel wrote: > Hi All! > > Bron, for deployments I manage these issues are also important: First of all - thanks for writing this up. It really helps! > * #1763 (Backups for SMB (lock entire server for a moment while taking a > snapshot)). Don't know if there was any progress on this. Basically, a > short (milliseconds to a few seconds) global write lock is needed on all > data structures. This is not easy unfortunately with all the different datastructures, because it means that everything else which takes a lock is going to need to first take a global shared lock before it does anything else, and that's going to have a performance and complexity impact on everything - because you have to find them ALL or you might wind up with lock inversions down the line. > * #1765 (Move SNMP out from master into a separate daemon) and related > pending PR #2100. Ellie had significant progress on this, don't know > what's blocking it, but this issue basically blocks any further work on > privilege separation like chroot as the main process should retain root > while running and the forked children should proceed with setuid & chroot. Good point - this is something the Greg was close to having done many years ago, but we're not using snmp so it hasn't caused us stress. Happy to put that on the consideration list for 3.2. The downside of making the list of tasks for 3.2 really long is that it could block releasing something which is otherwise still a good improvement over 3.0 and not a regression... *sigh*. But this one will be a good win, so let's do it! > * #2373 (Shared xDAV (CalDAV/CardDAV) resources are not discoverable). > Dilyan Palauzov sent a diff for this in the github repo and there was a > discussion with Ken on possible implementations (shared xDAV resources): > https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-May/004263.html. > I guess it had enough progress to try to close it. Labeled. I'm keen to have an answer to it somehow or other. > > * #2372 ([FR] ACL on autocreate folders). Basically, for automatic > "anyone p" ACL in plus+addressing folders. Yep - labeled. OK, the hard bit here isn't implementing (as ellie pointed out) - it's design. We want to make sure we create an interface that people can keep using reliably into the future. I'll have a chat with ellie about this one. > And there are 46 open PRs in the repo. Maybe they could be reviewed and > merged too? Yeah, maybe! Frustratingly the next couple of Cyrus call times aren't going to work for me, I've got a 7am Melbourne time meeting next Tuesday, then I'll be in Singapore for IETF where the Cyrus meeting time is 5am. One downside of pretty much everyone involved in direct Cyrus development being at Fastmail is that we discuss a lot of things in our private slack channel or internal mailing lists where we don't have to be quite so careful about stripping anything that could identify an internal customer... but it does create an impression that there's less happening than you'd otherwise see... and I haven't even posted the meeting minutes recently because they've been taken into a Dropbox paper doc and then langished there :( Sorry. Cheers, Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: time for cyrus-imap v3.2?
I've tagged those 4 issues for 3.2. We're going to try to work out what work is necessary for 3.2 to be done, so knowing that these are important is valuable. Cheers, Bron. On Tue, Nov 5, 2019, at 22:56, Michael Menge wrote: > Hi all, > > there are some bugs in cyrus 3.0/3.1 that i would like to see fixed > and I want to make sure that these changes will be able to be > included after 3.2 is released or will be fixed before 3.2 is released: > > #2659 allow rename back from deleted mailbox when conversations is enabled > #2599 bug renaming/deleting special use folders in murder setup > #2598 squat search_engine not used > > Also fixing "#2774 Murder does not work with TLS" would be > appreciate, if not possible the murder documentation should > at least been updated > > Quoting my mail > https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/004297.html > > > Quoting ellie timoney : > >> > >> Anyway, it looks to me like the STARTTLS support in mupdate is just > >> fundamentally broken at the moment, and my recommendation is to > >> not use it. If your IMAP servers need to connect to an mupdate > >> server that's not within their trusted network, I guess you'd need > >> to set up a VPN for it or something along those lines (but I'm no > >> network specialist). > >> > > could you add a warning in the relevant murder/installation guides > > and manuals? > > Quoting Bron Gondwana : > > > On Tue, Nov 5, 2019, at 12:04, Ricardo Signes wrote: > >> So, I think the plan was to cut a stable Cyrus 3.2 after we had > >> stable JMAP. Is that time now? We talked about this on the Zoom > >> call today. > > > > I think we're pretty close to it. The big question is: do we fork > > what will eventually become 3.2 and keep stabilising on it while we > > ship UUID mailboxes on master, or do we finish 3.2 before we merge > > uuid mailboxes. > > > >> Cyrus master has pretty stable for JMAP core and mail. I think we > >> need to do one more pass through to look for places where Cyrus > >> extensions might leak through without the correct `using` options, > >> but apart from that, I don't think we expect its mail API to change > >> apart from bugfixes. > > > > Yep, legit. The one big thing still missing there is > > PushSubscriptions. I'd be keen to finish writing that. I mean: > > > > https://github.com/cyrusimap/cyrus-imapd/issues?q=is%3Aopen+is%3Aissue+label%3A3.2 > > > > We should probably do a push and resolve all of those, then boom let's go. > > > > there are some bugs in cyrus 3.0/3.1 that i would like to see fixed > and I want to make sure that these changes will be able to be > included after 3.2 is released or will be fixed before 3.2 is released: > > #2659 allow rename back from deleted mailbox when conversations is enabled > #2599 bug renaming/deleting special use folders in murder setup > #2598 squat search_engine not used > > Also fixing "#2774 Murder does not work with TLS" would be > appreciate, if not possible the murder documentation should > at least been updated > > Quoting my mail > https://lists.andrew.cmu.edu/pipermail/cyrus-devel/2018-July/004297.html > > > Quoting ellie timoney : > >> > >> Anyway, it looks to me like the STARTTLS support in mupdate is just > >> fundamentally broken at the moment, and my recommendation is to > >> not use it. If your IMAP servers need to connect to an mupdate > >> server that's not within their trusted network, I guess you'd need > >> to set up a VPN for it or something along those lines (but I'm no > >> network specialist). > >> > > could you add a warning in the relevant murder/installation guides > > and manuals? > > > > > > > > M.Menge Tel.: (49) 7071/29-70316 > Universität Tübingen Fax.: (49) 7071/29-5912 > Zentrum für Datenverarbeitung mail: > michael.me...@zdv.uni-tuebingen.de > Wächterstraße 76 > 72074 Tübingen > > -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Re: Which imap command to rename a root mailbox while maintaining its partition
Wow - this looks like a bug in partition selection for user rename then :( We
should fix that.
https://github.com/cyrusimap/cyrus-imapd/issues/2907
Cheers,
Bron.
On Tue, Nov 5, 2019, at 17:30, Thomas Cataldo wrote:
>
>
> > On 29 Oct 2019, at 13:13, Ken Murchison wrote:
> >
> > x RENAME
> >
> >
> > should work
>
>
> Agree, but it does not :-)
>
> At least with version 3.0.8 :
>
> localhost> info user/r...@devenv.blue
> {user/r...@devenv.blue}:
> private:
> check: NIL
> checkperiod: NIL
> comment: NIL
> sort: NIL
> specialuse: NIL
> thread: NIL
> expire: NIL
> news2mail: NIL
> sieve: NIL
> squat: NIL
> shared:
> check: NIL
> checkperiod: NIL
> comment: NIL
> sort: NIL
> specialuse: NIL
> thread: NIL
> annotsize: 0
> duplicatedeliver: false
> expire: NIL
> lastpop: NIL
> lastupdate: 4-Nov-2019 15:32:13 +
> news2mail: NIL
> partition: bm-master__devenv_blue
> pop3newuidl: true
> pop3showafter: NIL
> sharedseen: false
> sieve: NIL
> size: 32310
> squat: NIL
> synccrcs: 2599665889 0
> uniqueid: ee8ede37-153a-4650-bf94-3da7d4f52043
>
>
> An IMAP session as admin :
>
> telnet localhost 1143
> Trying ::1...
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
> * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE AUTH=PLAIN SASL-IR] server ready
> . login admin0 admin
> . OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA
> MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN
> MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ
> SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS
> ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS
> LIST-MYRIGHTS LIST-METADATA WITHIN QRESYNC SCAN XLIST XMOVE MOVE SPECIAL-USE
> CREATE-SPECIAL-USE DIGEST=SHA1 X-REPLICATION URLAUTH URLAUTH=BINARY
> LOGINDISABLED COMPRESS=DEFLATE X-QUOTA=STORAGE X-QUOTA=MESSAGE
> X-QUOTA=X-ANNOTATION-STORAGE X-QUOTA=X-NUM-FOLDERS IDLE] User logged in
> SESSIONID=
> . RENAME user/r...@devenv.blue user/ren...@devenv.blue bm-master__devenv_blue
> . NO Cross-server or cross-partition move w/rename not supported
> . RENAME user/r...@devenv.blue user/ren...@devenv.blue
> * OK rename user/r...@devenv.blue user/ren...@devenv.blue
> * OK rename user/ren/dra...@devenv.blue user/rename/dra...@devenv.blue
> * OK rename user/ren/j...@devenv.blue user/rename/j...@devenv.blue
> * OK rename user/ren/out...@devenv.blue user/rename/out...@devenv.blue
> * OK rename user/ren/s...@devenv.blue user/rename/s...@devenv.blue
> * OK rename user/ren/tr...@devenv.blue user/rename/tr...@devenv.blue
> . OK Completed
>
> But if I use the version without an explicit partition, the new mailbox ends
> up in :
>
> > info user/ren...@devenv.blue
> {user/ren...@devenv.blue}:
> private:
> check: NIL
> checkperiod: NIL
> comment: NIL
> sort: NIL
> specialuse: NIL
> thread: NIL
> expire: NIL
> news2mail: NIL
> sieve: NIL
> squat: NIL
> shared:
> check: NIL
> checkperiod: NIL
> comment: NIL
> sort: NIL
> specialuse: NIL
> thread: NIL
> annotsize: 0
> duplicatedeliver: false
> expire: NIL
> lastpop: NIL
> lastupdate: 4-Nov-2019 16:43:36 +
> news2mail: NIL
> partition: default
> pop3newuidl: true
> pop3showafter: NIL
> sharedseen: false
> sieve: NIL
> size: 32310
> squat: NIL
> synccrcs: 2599665889 0
> uniqueid: ee8ede37-153a-4650-bf94-3da7d4f52043
>
>
> which forces me issue a second command in my imap session :
>
> . RENAME user/ren...@devenv.blue user/ren...@devenv.blue
> bm-master__devenv_blue
> * OK rename user/ren...@devenv.blue user/ren...@devenv.blue
> * OK rename user/rename/dra...@devenv.blue user/rename/dra...@devenv.blue
> * OK rename user/rename/j...@devenv.blue user/rename/j...@devenv.blue
> * OK rename user/rename/out...@devenv.blue user/rename/out...@devenv.blue
> * OK rename user/rename/s...@devenv.blue user/rename/s...@devenv.blue
> * OK rename user/rename/tr...@devenv.blue user/rename/tr...@devenv.blue
> . OK Completed
>
>
> Which moves the mailbox to the partition where I want it (its original one).
>
>
> The problem with the non-atomic rename is that our replication target
> receives data belonging to the default partition, which is not desired or
> expected.
>
>
>
>
> Thomas Cataldo
> Directeur Technique
>
> (+33) 6 42 25 91 38
>
> BlueMind
> +33 (0)5 81 91 55 60
> Hotel des Télécoms, 40 rue du village d'entreprises
> 31670 Labège, France
> www.bluemind.net / https://blog.bluemind.net/fr/
>
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Re: time for cyrus-imap v3.2?
On Tue, Nov 5, 2019, at 12:04, Ricardo Signes wrote: > So, I think the plan was to cut a stable Cyrus 3.2 after we had stable JMAP. > Is that time now? We talked about this on the Zoom call today. I think we're pretty close to it. The big question is: do we fork what will eventually become 3.2 and keep stabilising on it while we ship UUID mailboxes on master, or do we finish 3.2 before we merge uuid mailboxes. > Cyrus master has pretty stable for JMAP core and mail. I think we need to do > one more pass through to look for places where Cyrus extensions might leak > through without the correct `using` options, but apart from that, I don't > think we expect its mail API to change apart from bugfixes. Yep, legit. The one big thing still missing there is PushSubscriptions. I'd be keen to finish writing that. I mean: https://github.com/cyrusimap/cyrus-imapd/issues?q=is%3Aopen+is%3Aissue+label%3A3.2 We should probably do a push and resolve all of those, then boom let's go. > The other part of the conversation was declaring pre-3 releases EOL except > for security fixes. > > I don't have much of a horse in this race, but it felt like a bit of looming > question. Generally the pattern has been "current stable and oldstable are supported" - so that would be 3.2 and 3.0 once we release 3.2. I'm good with that. Bron. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Minutes of calls for the past couple of weeks
on master! * Discussed what to do with Snooze and how to handle IMAP and Sieve rules. ellie: * updated some virtdomains and sieve docs * fixed ipurge mboxevents bug * ye olde promstatsd update is back, 2nd time a charm * fixed a couple of bugs in my mboxlist_find* api change from a few weeks ago (doh) -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Cassandane now requires read access to syslog as the cyrus user
I've just pushed some changes to Cassandane to scan syslog for known "badness"
(basically anything with ERROR in it for now, but it's easy to add other
things).
Unfortunately, that does require that the cyrus user can access the syslog.
I've added the template to cassandane.ini.example to choose which file to look
at, default is /var/log/syslog.
On my Ubuntu machine I just ran "adduser cyrus adm" to grant it access - I'm
sure there's similar tricks on other machines, or syslog can be set up to send
just cyrus data to a special file and point cassandane there.
This allows Instance to watch syslog and die if there are any unhandled IOERROR
/ DBERROR / SYNCERROR lines, which picks up new sets of bugs automatically :)
There are a couple of cases already where I've put in tests which drain the
syslog ($self->{instance}->getsyslog) and check for the errors they expect.
There will need to be more, so some tests will be failing right now.
Cheers,
Bron.
--
Bron Gondwana, CEO, Fastmail Pty Ltd
br...@fastmailteam.com
Notes 29 July
Present: ellie, Ken, Bron ellie: * 3.0.11 release - hopefully tomorrow * 3.1 checkpoint release? - Let’s do one now-ish! - Then we can do another one after the delayed send changes land. Ken: * Mailbox by UUID status - Needs to be rebased. - Then - need to do some testing! Stand up a new one, replicate over. * Going to keep working on Delayed Send, including GUID from mailbox record. Bron: * Cyruslibs-v26+: - Need to fix tzdata issue and fixed libicu. * Calendar Sharing: - “Shared via JMAP" is clearly bad UI - Neil and Ken have discussed notifications Inbox and how to deal with these. - Have created a task to work on design for the various workflows. * Handling Calendar scheduling and X-Schedule-User-Address on updates - A user had issues with a complex mailflow causing participantId to be null because the schedule address didn't match. - For now we’ll fix this in the Fastmail middleware for our weird usecase. - Need to build a more general way to determine "IsYou" inside Cyrus. -- Bron Gondwana, CEO, Fastmail Pty Ltd br...@fastmailteam.com
Notes 24 June
Ken: * Spend last week pottering around with rebase of uuidmailbox work. * Also working on libical * Last week issue with broken libicu, now crashing on invalid data when before it ignored it. - might patch around it in Cyruslibs version of libicu. Robert: * Cyrus work: - some of the Xapian stuff would be good to test with older index db formats. - It would be good to run tests with two different binaries, one to create old-style data and the other one to see that things still work! - this would be a good general thing, both for replication and for our own testing! - The Xapian issues in the last little while would have been found if we had legacy version testing. * Query normalisation for contact groups - may need to cache. * JSCalendar and JSContact drafts! - JSCalendar unlikely to finish by the end of the week, so Bron to take on pushing it from here. - JSContact - will only be an interim draft, so will continue when back. * Will be bringing Cyrus up to date with JSCalendar - should we bring it up to date on master first, or wait until approved? Bron: * Wrote an email about SINCE_* keys in replication and replication cache. * Have shipped SINCE_* to FM production, working fine. * Next steps are exciting, but can't work on that just yet! ellie: * not sure if Cassandane is the right vehicle for general multiple version support * spent some time with imaptest this week and has opened an issue with their github - working on integrating the non-scripted benchmark/race condition searching tests. Times with Robert away - we'll stick with this time for now, it's still working for everyone! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
syncronous replication and replication cache
sync*** And now we get to the meat of this! We want to allow multiple parallel real-time sync calls, but probably not hundreds. I suggest that we use a sync_client-style tool which runs as a daemon within cyrus, listening probably on a UNIX socket and keeping a long running backend connection open, so there may be a handful of syncd / sync_server parings running at any one time, servicing random requests. The config would be something like sync_realtime_socket: [% confdir %]/socket/syncd and in cyrus.conf syncd cmd="syncd [% CONF %] -n [ %buddyslotchannel %]" listen="[% confdir %]/socket/syncd" (aka: use the named channel to choose the backend for realtime sync) The client code would keep EXACTLY the current sync_log support to all channels, but have the following very small modification added: at the end of mailbox_unlock_index or mailbox_close or wherever seems the most sane, we add a synchronous callout. It might even go up a layer, though there's many more places to think about. Anyway, this callout connects to the local syncd on the specified port and asks it to replicate just the one mailbox name to the backend, and waits for the reply. If the reply is a failure, then that gets syslogged, but otherwise the return is still success to the rest of the code. The end result of hooking at this point is that EVERY protocol which replies success after committing the changes will not reply back to the client until the data is replicated to the buddy machine (buddy naming is hearkening back to the short-lived career of Max at FastMail and his buddyslot idea!) The rolling sync_client will pick up the logged name still, but the good news is, it will either still be locked (local lock) and hence try later, or it will already be up to date, and the local state will match the cached state, so there's no network IO or heavy calculations at all! This lack of network IO at all for the followup case is why I think the cache and remote conversations-based GUID lookup (and local validation that it's worth trying) is worth doing first, because that means that synchronous replication stands a chance of being efficient enough to be sane! I _think_ this design is good. Please let me know if anything about how it's supposed to work is unclear, or if you have a better idea. In terms of the split if work: I envisage Ken or ellie writing the syncd stuff. I've already written the ifInState stuff, so I'll just ship that. The caching could be anybody, it's pretty easy, and the GUID lookup for master/replica could be me, or could be someone else if they're keen to have a look at it - I have a very clear idea of how that would look. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: squatter -F increases the index size
I saw your ticket about that - I'll check it out soon. Sorry, been busy at the calconnect conference in the UK this week. I also did this: https://github.com/cyrusimap/cyrus-imapd/commit/27513a9bc3f217f388bac163820f9879178071fb Which I believe will mean that if you change the defaultsearchtier, it will immediately start indexing to the new location. You'll definitely want to restart a server over changing that config option though, and not have it be different between different invocations of squatter, or you'll wind up creating a lot of xapainactive entries! Bron. On Tue, Jun 4, 2019, at 04:19, Dilyan Palauzov wrote: > Hello Bron, > > imap/squatter.c:do_compact() does call `if (sleepmicroseconds) > usleep(sleepmicroseconds);` so -S number is honoured with `squatter > -t… -z…`. > > Will `squatter -F -z… -t…` be fixed on the stable branch, or shall > calling `squatter -F -t… -z` be discouraged with 3.0? > > Providing that currently after `squatter -F -t… -z…` calling `squatter > -t… -z` does reindex all messages and therefore creates a new xapian > index, it must be possible to to create a compacted database directly, > without creating an bloated index first. > > My understaning to the rolling mode is that once a new message > appears/arrives/is APPENDed or deliver(ed), it is added to the sync > log and then indexed in rolling mode. Then arrives a message at a > different place, it is added to the log and then indexed. Whether the > first and second messages are in the same mailbox is completely > random. Why does squatter not sleep, if the two messages are in the > same mailbox and works non-stop otherwise, say why does it sleep > depending on random circumstances? > > https://wiki.dovecot.org/Plugins/FTS/Squat says for DoveCot that IMAP > requires that SEARCH is done also on substings, no IMAP server > implements this requirement, and dovecot does implement it only when > Squat indices are used. Is the same valid for Cyrus Imap (Squat index > offers substring search, Xapian index does not offer substring search)? > > Runnig squatter once printed “compressing X:0,X,Y:0 to Y:3 for … > (active Y:0,X:0,X,Y:0,Y:1,Y:2)” > (https://github.com/cyrusimap/cyrus-imapd/issues/2764) so I suspect a > tiername without a number was in the .xapianactive file. > > If I do any compact (-o, -F, -X, just -t -z), where the first tier is > not referenced, does squatter ensure that the default tier according > to imapd.conf is inserted in the xapianactive file. Or asking in > other ways, it I change imapd.conf and create a new tier T6 and > declare T5 to be the default tier, which of the following will insert > a reference to T5:0 in .xapianactive and which will not: > > squatter -t T2 -o -z T2 > squatter -t T5,T2 -z T2 > squatter -t T5 -o T4 > squatter -t T2 -F T3 > sqautter -t T2 -X T3 > or what else? (The name T5 is declared, and the root directory exist, > but neither there is data in the directory, nor is T5 yet in any > .xapianactive file). > > Regards > Дилян > - Message from Bron Gondwana - > Date: Tue, 04 Jun 2019 01:53:23 +1000 > From: Bron Gondwana > Subject: Re: squatter -F increases the index size > To: Cyrus Devel > > > > On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: > >> Hello, > >> > >> I gave squatter -F a try. > >> > >> Before I run it for a user tier T1 was not compacted and allocated 3,4 > >> MB (mega), T2 was compacted and contained 3.7GB (giga). After > >> removing the records of the deteled messages, say running squatter -F > >> T2 was 5.7GB and squatter printed “filtering” instead of “compacting”. > >> Then I run again “squatter -t T1,T2 -z T2” without -F, without -X > >> and squatter reindexed all messages, to create a 3.0 GB index. > >> > >> I expected, that using -F the resulting database will be compacted and > >> on the second call there will be no reindexing. > > > > I discovered some bad bugs in -F recently, so I suspect that's why. > > They should be fixed on master now. > > > >> When does squatter decide on its own to reindex? > > > > When the DB version is too old (which is one of the -F bugs - it > > wasn't setting the DB version, so it assumed the data was all > > version zero!) > > > >> What do G records in conversations.db contain? > > > > G records contain a mapping from GUID to folder number (offset into > > the $FOLDER_NAMES key) and UID and optionally IMAP part number as > > the key - mapping to values which contain some keywords and modseq > > from the original record as well. > > > >> My r
Re: squatter -F increases the index size
On Sat, Jun 1, 2019, at 04:34, Dilyan Palauzov wrote: > Hello, > > I gave squatter -F a try. > > Before I run it for a user tier T1 was not compacted and allocated 3,4 > MB (mega), T2 was compacted and contained 3.7GB (giga). After > removing the records of the deteled messages, say running squatter -F > T2 was 5.7GB and squatter printed “filtering” instead of “compacting”. > Then I run again “squatter -t T1,T2 -z T2” without -F, without -X > and squatter reindexed all messages, to create a 3.0 GB index. > > I expected, that using -F the resulting database will be compacted and > on the second call there will be no reindexing. I discovered some bad bugs in -F recently, so I suspect that's why. They should be fixed on master now. > When does squatter decide on its own to reindex? When the DB version is too old (which is one of the -F bugs - it wasn't setting the DB version, so it assumed the data was all version zero!) > What do G records in conversations.db contain? G records contain a mapping from GUID to folder number (offset into the $FOLDER_NAMES key) and UID and optionally IMAP part number as the key - mapping to values which contain some keywords and modseq from the original record as well. > My reading is that the way to create a Xapian index of an indexed > mailbox, is that first squatter has to be run in INDEX mode and then > in COMPACT mode. In particular it is not possible to create in one > step a compacted database. No, it's not - due to the way to compact API works. At least, I haven't figured out how. > Does squatter -R -S sleep after each mailbox or after each message indexed? It sleeps after each mailbox. > When compacting, squatter deals just with messages and on search or > reindex the conversations.db is used to map the messages to mailboxes. > How does squatter -S sleep after each mailbox during compacting, if > it knows nothing about mailboxes? -S is not used when compacting. > What does mean a tier name in a xapianactive file without a number? that shouldn't happen. It will be parsed as the same as tier:0 I believe. > What are XAPIAN_DBW_CONVINDEXED and _XAPINDEXED? Two different ways to know if a document is indexed. CONVINDEXED uses the conversations DB to look up mailbox and uid and then the cyrus.indexed.db databases to see if the message has already been seen. XAPINDEXED uses the metadata inside the Xapian databases to know if a particular message has been indexed based on the cyrusid.*G* metadata values which are identical to the GUIDs themselves. > What does the file sync/squatter? It's a sync/$channel directory which squatter watches on. This is a method for providing a queue of mailboxes to look at based on the APPEND sync_log statements. > squatter can print “Xapian: truncating text from message mailbox > user uid 7309”. When are messages truncated for the purposes of > indexing? When they are too long! The comment in the source code says this: /* Maximum size of a query, determined empirically, is a little bit * under 8MB. That seems like more than enough, so let's limit the * total amount of parts text to 4 MB. */ #define MAX_PARTS_SIZE (4*1024*1024) This is a holdover from when Greg was working on it. We could switch this to be a configurable option. > Do I understand correctly, that for a Xapianactive file with "A B C D > E", to remove C one has to call "squatter -t C,D -z D". But A cannot > be removed, if it the defaultsearchtier. Is the defaultsearchtier > always included in the xapianactive file, if the tier is missing, > whenever the file is modified (and the only way to modify it is to > call squatter in COMPACT mode)? When you do any compact, if it includes the first item (the writable database) then a new writable database will be created on the default tier. So if you try to compact the default tier away, a new default tier item will be created. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Prepending Xapian Tiers
On Wed, May 29, 2019, at 06:39, Dilyan Palauzov wrote: > Hello, > > so the .conversations database does, apart of the descriptions > at > https://www.cyrusimap.org/imap/concepts/deployment/databases.html#conversations-userid-conversations, > also store per user a G record for each message, mapping the mailboxes where > the message is located and the results from Xapian search return G > records. > > Are a G record, GUID and a conversation ID the same thing? G records are identical to GUIDs. There are also G records (in latest master at least) for sub parts of message, which map to a blobId in JMAP and allow direct addressing of every part by a content-based ID. conversation ID is something different, it's based on a permutation of the GUID of the first message that arrived within that thread - and was the original point of the conversations database. Sadly this has all evolved over time. I would like to migrate Cyrus towards using the terminology in JMAP, which has EmailId (which is a prefix on the GUID in JMAP) and ThreadId (which is the conversation ID from Cyrus with 'T' as a prefix). As well as MailboxId which was previously known in Cyrus as UniqueId on mailboxes. > When a message is expunged, are its records from > .conversations removed? They are removed when it is UNLINKED, which may be at the same time depending on your expunge_mode setting. > When a message is unexpunged, is it again inserted in > .conversations and referenced in the sync_log_channels: > squatter? Yes, unexpunge is treated as a new APPEND, and since the bytes are the same, the GUID will be the same. > squatter has the modes: indexer, search, rolling, synclog, compact, > indexfrom (deprecated) and audit. Is search_batchsize used only in the > indexer mode, in particular it is not used when squatter -t … -z -X is > called (compact and reindex simultaneously)? Hmm let me check! Nope, when you run with -X it reindexes all the messages in an entire mailbox in a single batch, ignoring search_batchsize. > What is the application for squatter -X (Reindex all messages before > compacting. This mode reads all the lists of messages indexed by the > listed tiers, and re-indexes them into a temporary database before > compacting that into place)? It is very useful when index formats have changed over time and you want to reindex all emails with the latest format, or when you believe a search database might be corrupted and want to rebuild it from source. > Does it index messages, that were not indexed yet for any reason, or > it deletes the database, scans each message again and creates a > compact Xapian database? It uses the cyrus.indexed.db of each of the source databases (selected by -t) to know which range of UIDs in each mailbox were claimed to be indexed by those databases, and then scans over those same ranges of UIDs again and indexes the contents of those messages if they are not yet expunged. > In the case I described, mailbox receiving reports, having an index > grow very fast, the cause was a mail loop - a lot of emails arriving > in short time. Once the loop stopped, the index does not exand faster > than other mailboxes. That makes sense. > So by default for now, unless some extra setup is performed, only > words in text/plain and text/html get indexed, possibly with headers, > and attachments are ignored? Yes, that's is correct. In fact, it's all text types. text/calendar and text/vcard are processed specially. Other text/* types are treated the same as text/plain for indexing purposes. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Prepending Xapian Tiers
ig files have to copied. I'm afraid that's what we have right now. Again, tiers are supposed to be set up at the start and not fiddled with afterwards, so the system isn't designed to allow you to quickly add a new tier. > > What it does under the hood is creates a new database and copy all > > the documents over from the source databases, then compress the end > > result into the most compact and fastest xapian format which is > > designed to never write again. This compressed file is then stored > > into the target database name, and in an exclusively locked > > operation the new database is moved into place and the old tiers are > > removed from the xapianactive, such that all new searches look into > > the single destination database instead of the multiple source > > databases. > > I do not get this. The amount of tiers to check does not reduce after > doing merging and with three tears the amount of databases is most of > the time three. Not if you're compacting frequently. We do the following: * hourly - check if tmpfs is > 50% full - quit if not. - run squatter -a -o -t temp -z data * daily - regardless of tmpfs size, compact everything on temp and meta down to data - squatter -a -t temp,meta -z data * weekly on Sunday - re-compact all data partitions together - squatter -a temp,meta,data -z data * And finally, once per week once the re-compact is done, check if we need to filter and recompact the archive, if so: - squatter -a data,archive -z archive -F Since today is Monday, most users will have two, so the xapianactive might be something like: temp:66 data:52 data:51 archive:2 Later in the week, it might be: temp:70 data:66 data:55 data:54 data:53 data:52 data:51 archive:2 And then maybe it will re-compact on Sunday and the user will have temp:74 archive:3 > What happens, if squatter is terminated during Xapian-compacting, > apart from leaving temporary files? Will rerunning it, just start > from beginning? The source databases will still be in xapian.active, so yes - a new compact run will take those same source databases and start again. > Is the idea to have three tiers like this: > > At run time, new messages are indexed by Xapian in squatter-rolling > mode on tmpfs/RAM, say on tear T1. That's certainly what we do, since indexing is too IO-intensive otherwise. > Regalarly, the RAM database is compacted to hard disk (tear T2), say > T1 and T2 are megred into T2. The database on the hard disk is > read-only and search in it is accelerated, as the database is “compact”. As above - during the week we don't even merge T2 back together, we compact from T1 to a single small database on T2 - leading to multiple databases on T2 existing at once. > Only if two compactions happen in parallel of the same sources or > destination, the merge fails and is skipped for that user. The merge > is retried whenever merging T1 and T2 is rescheduled. Yes - though that's pretty rare on our systems because we use a lock around the cron task, so the only time this would happen is if you ran a manual compaction at the same time as the cron job. > As the databases in T2 get bigger, merging T1 and T2 takes more and > more time. So one more Xapian tear is created, T3. Less regularly, > T2 and T3 are merged into T3. This process takes a while. But > afterwards, T2 is again small, so merging T1 and T2 into T2 is fast. Yes, that's what we do. This is also the time that we filter the DB, so the T3 database only contains emails which were still alive at the time of compaction. > How many tears make sense, apart from having one more for power-off events? Having another one for power off events doesn't make heaps of sense unless you have a fast disk. That's kind of what our "meta" partition is, it's an SSD RAID1 that's faster than the "data" partition which is a SATA spinning RAID1 set. When we power off a server, we run a task to compact all the temp partitions down - it used to be to meta, but we found that compacting straight to data was plenty fast, so we just do that now! If you power off a server without copying the indexes off tmpfs, they are of course lost. This means that you need to run squatter -i on the server after reboot to index all the recent messages again! So we always run a squatter -i after a crash or power outage before bringing that server back into production. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Does squatter retry on transient errors (xapian)?
On Tue, May 21, 2019, at 18:56, Dilyan Palauzov wrote: > Hello, > > https://fastmail.blog/2014/12/01/email-search-system/ says: > > The process is as follows: > 5. if the xapianactive file has changed, discard all our work (we lock > against this, but it's a sanity check) and exit > > Does squatter exit with an error message in this case, and has to be > started again, or does it retry automatically, until it successes? No, it just keeps going. It will syslog something about the failure, but a compact failure is generally considered to be not a bad thing. > What means “indexing”? -T can be passed to squatter only in compact > mode, and is a directory for temporary files. But the documentation > says, this directory is used when “indexing”. How big is the data > stored there? Lets say if tier A allocates K MB for its data, and > tier B allocates L MB for the data, is during compacting of A and B > approximately K+L MBs used in the temporary directory? It doesn't work like that - you don't allocate a particular amount of space for each tier. The space used in -T during a single compact is roughly the sum of the database sizes of the databases which are being compacted together. I expect the documentation could be made more clear here. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Prepending Xapian Tiers
On Tue, May 21, 2019, at 18:41, Dilyan Palauzov wrote: > Hello, > > thanks, Bron, for your answer. > > I gave it a try. > > squatter does not remove .NEW directories when aborted (SIGINT), the > directories have to be removed manually https://github.com/cyrusimap/cyrus-imapd/issues/2765 > > squatter -t X -z X -o recognizes, when the directory structure behind > tier X exists, that nothing has to be done, prints “Skipping X for > user.ABC, only one” and quits, without updating the .xapianactive files. yeah right, that won't work. Glad to know :) > squatter -t Y -z Y -o, when the directory structructure behind tier Y > does not exist, prints “compressing Y:1,Y:0 to Y:2 for user... (active > Y:1,Y:0)”. As far as I remember this has not updated the xapianactive > files. Yeah right, it won't add a new target unless you are compressing the current first item in xapianactive. > squatter -t X -z Y -o does add to the .xapianactive files the > defaultsearhtier, but first has to duplicate with rsync all existing > files. This takes a while… But at the end did what I wanted. > Afterwards the directory structure for the new tier was not created. > The directory structure was created once I started all the cyrus > processes again. That makes sense. We don't create a directory structure until a document gets created in there. > squatter -t X -z Y -o emits the message “undefined search partition > X,Ysearchpartition-default” and then “compressing X:0,X,Y:0 to Y:2 for > ... (active Y:0,X:0,X,Y:0,Y:1)”. That sounds like a sanity checking failure! Good catch: https://github.com/cyrusimap/cyrus-imapd/issues/2764 > Does squatter -t X -z Y append X to Y, or it deletes Y and copies X to > Y? In the latter case, is there any (performance) difference between > "squatter -t X,Y -z Y" and “squatter -t Y,X -z Y”? There's no difference in what order you add items to -t. -t is a comma separated list of selectors for source items. You can even explicitly say: squatter -t X:0,X:2,Y:45 -z Y and it will compact just those three sources into a new target in Y. What it does under the hood is creates a new database and copy all the documents over from the source databases, then compress the end result into the most compact and fastest xapian format which is designed to never write again. This compressed file is then stored into the target database name, and in an exclusively locked operation the new database is moved into place and the old tiers are removed from the xapianactive, such that all new searches look into the single destination database instead of the multiple source databases. > Can one xapian tier store a document, and another tier store the > information, that the address of the document has changed? It doesn't work like that. The addresses of the documents never change (they are the sha1 of the document contents, and Cyrus documents are all immutable). The xapian engine searches across the full set of databases listed in xapianactive in order to find document ids, then maps them through the conversations.db file to find the actual emails. A copy/move of an email updates the conversations.db lookups, so the next search will find the new location without anything changing in xapian. the cyrus.indexed.db file is just a convenience to allow rolling squatter to avoid having to re-scan records that it knows are already indexed. Bron. > Regards > Дилян > > - Message from Bron Gondwana - > Date: Mon, 20 May 2019 18:52:07 +1000 > From: Bron Gondwana > Subject: Re: Prepending Xapian Tiers > To: Cyrus Devel > > > > On Fri, May 17, 2019, at 23:52, Дилян Палаузов wrote: > >> Hello, > >> > >> I set up a Cyrus system with one tier. I think it works. The > >> .xapianactive files contain 'tiername: 0'. > >> > >> How can I insert a second tier? > > > > I have never tried this on a live server! Clearly the right thing to > > do is to build a cassandane search which implements doing this so > > that we can make sure it works. > > > >> Adding a XYZsearchpartition-default to imapd.conf, together with > >> defaultsearchtier: XYZ does not utilize the new directory: it stays > >> empty and the .xapianactive files do not get updated to mention the > >> new tier. > > > > That looks like it should work. I assume you have restarted your > > cyrus since making the change? I'm not certain that a rolling > > squatter will discover a new config in the way that imapd does. > > > > Also - you'll need to run squatter in compact mode in order to add a > > new xapianactive entry. The simplest could be: > > > > squatter -z tiername -t tiername -o > > > > I belie
Notes 20 May
Present: Bron, Robert, Ken, ellie Bron in Vienna with Robert! ellie: * now has proper internet after many months of unreliable provider issues! * not much else to report Ken: * libical and multi-value parameters/properties discussion (there's a FastMail issue about it) - what's the expected behaviour on the client side? - not a blocker right now, but has been allocated over from Robert to Ken * Mailbox/get handling of ACL changes - this is long and complex because it requires keeping modseq per ACL entry, so it's a mailboxes.db format change - will work on that this week * did some bugfixing last week * nearly finished with vacation support - might have to go back to forcing user to include the vacation script, because otherwise STOP rules will mean that vacation doesn't happen. - plan is to parse active script and check if our custom jmapvacation.script is included - if not, reject the vacation object. - we're using :fcc at FastMail, may need to add a custom extension to allow setting that via JMAP. - current spec doesn't have a way to specify how many days to suppress responses for -> will need to fall back to server default. Could also add this in FM extension, but it's odd that it's not supported in upstream. - addresses: how to know it was specifically addressed to you rather than a bulk email. Maybe need to add Identity support to Cyrus too! Right now it only returns authenticated ID. Maybe add a property to INBOX with a list of email addresses! - https://github.com/cyrusimap/cyrus-imapd/issues/2760 One per identity. Probably uuid as the key. * promised to write a draft for discovery of shared calendars before next week. Robert: * Finally landed Xapian language indexing on upstream master * Spent a lot of time on JSCalendar RFC based on feedback from the mailing list * Now started working on annotations API changes. Will be working on this for the next week. Bron: * working on caching jmap objects in the dav database. Bron won't be available at the standard time next week, but will be around working. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Prepending Xapian Tiers
On Fri, May 17, 2019, at 23:52, Дилян Палаузов wrote: > Hello, > > I set up a Cyrus system with one tier. I think it works. The .xapianactive > files contain 'tiername: 0'. > > How can I insert a second tier? I have never tried this on a live server! Clearly the right thing to do is to build a cassandane search which implements doing this so that we can make sure it works. > Adding a XYZsearchpartition-default to imapd.conf, together with > defaultsearchtier: XYZ does not utilize the new directory: it stays empty and > the .xapianactive files do not get updated to mention the new tier. That looks like it should work. I assume you have restarted your cyrus since making the change? I'm not certain that a rolling squatter will discover a new config in the way that imapd does. Also - you'll need to run squatter in compact mode in order to add a new xapianactive entry. The simplest could be: squatter -z tiername -t tiername -o I believe that given your current setup, this will just copy the entry from tiername:0 to tirename:1 and also create XYZ:0 in the xapianactive file at the same time. > Besides, if a message is MOVEd over IMAP, is any optimization utilized, to > avoid reindexing the message, but just change the address of the document? Yes, both XAPINDEXED mode where the GUID is read from xapian, and CONVINDEXED mode where the GUID is looked up via user.conversations and then mapped into the cyrus.indexed.db files in each xapian tier allow Xapian to skip reindexing when a message is already indexed. This works for both MOVE and for re-uploading of an identical message file via IMAP. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: https://imapwiki.org/ImapTest/ServerStatus update for Cyrus Imap 3.0
On Fri, May 3, 2019, at 12:50, ellie timoney wrote:
> This ("ImapTest") looks like the thing that we can run from Cassandane as
> Cassandane::Cyrus::ImapTest, I haven't had it set up for a while (haven't
> gotten around to it on new laptop yet...) but I think last time I looked we
> passed a lot of the tests on 3.0, and some of the ones we failed looked like
> bad tests
We fail two tests in vanilla upstream ImapTest last I checked, and there are
pull requests that I made for them years ago, but maybe they fell through the
cracks. They were cases where Cyrus is technically correct in rejecting
something that you could also argue is OK to accept.
> Bron, are you still running this test suite regularly? How are we looking on
> master? I think I remember you were planning to get in touch with Timo to get
> some of the bogus tests fixed, but I don't remember if this ever happened
Yeah, I still run it regularly. It has run clean since 2.4.something.
> It would be good to get this page updated, even if we're still a
> "non-compliant server", at least to have something more recent than 2.3 on
> there(!)
There are some things that ImapTest can test which we don't implement, but
we're fully compliant on all the things we do implement. (at least, if you
don't turn on deliberately standards-breaking stuff like autoexpuge or
search_fuzzy_always).
> It's a wiki so I'll try and create an account and see if I can just update it
> myself, but I'd like to refresh my understanding of where we're at before I
> embarrass myself!
Excellent. I don't remember having difficulty creating an account when I last
did anything on that wiki.
Bron.
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Notes Apr 29
JMAP slow query performance: * lots of discussion of why NOT in:Mailbox is really slow * Bron will build out a test user using the enron dataset to see if we can easily debug in dev * will also get a callgrind dump from a production user who has issues with both AJAX and JMAP * might try to build an array of folder numbers in conversations.db to "yes, include", "no, exclude" or "not mentioned" at each tree level in the search expression for quick filtering Ken: * have migration tool almost done * need to add support for tiered Xapian indexes * added namelocks to ensure nobody reads a middle state * can give a mailbox or an entire user * start work on the Vacation JMAP support. We need to figure out what to do with custom scripts! - probably add an "include" to the end of the current active script - create an empty active script if there isn't one * for testing the by-UUID code, we'll try replicating to a new machine with the new code and see if that works. Robert: * closed out Cyrus 3.2 issues from github last week * default values - being returned on get * should we return empty string for properties of type string that don't have defaults? * refactored JMAP language indexing, would like to get it on the branch with attachment indexing. * with CalConnect coming close, running out of time. Need to work on JSCalendar first. * for JSContact, there's somebody interested in implementing it already! They'll have to expect lots of rewrites before it's standard. Bron: * Been working on index upgrade/downgrade for 3.2 * readonly conversations support to mailbox_open_iwl inversion - changed to abort immediately rather than later when conversations DB write failed - needed to fix a bunch of places which were doing that. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting Notes 8 Apr 2019
Present: Bron, Ken, Robert Ken: * Wrote tool to do mailbox upgrades * Today working on tool to migrate people from old structure to new structure using a reconstruct * Will add back in all the code to read/write the old structure until they get migrated * Took a lot of time working on the annotations.db work. Moved work into annotations.c * CalConnect - going to wait to see if prices drop before committing * libical pull request for millisecond support for recurrences! Some discussion about finding out the use cases it's for. Robert: * Fixed some JMAP stuff last week - hasAttachment flag is now preserved on copy. * Copy code in append.c doesn't trigger the annotator - wonder if it should trigger the annotator regardless of JMAP. No, because the annotator should have always run. * Issue from Neil: made Cyrus more lenient about message-ids being invalid in emails. * Going to integrate attachment searching code tomorrow and work with Rob N on making it work for FM. * TC-API work. Got feedback for JSContact and * For JSCalendar: properties conversion. - split out all the ICALENDAR conversion work into a separate doc, and put it there. * Separated out the qsort_r code * implemented jmap sortAsTree and filterAsTree code for Mailbox/query. Bron: * Made it so the annotation layer doesn't throw an error if you try to write back the existing value - helps with annotator - do need to check that if you can't READ, then you don't get different errors when you try to write Bron and Ken meeting up in Philly in a few weeks - will prepare a planning doc for the visit: * caching contact/calendarEvent renderings in DAV DB. * lots of calendar stuff * finishing mailbox-by-id? NEXT WEEK: * read through issues for anything that should be a 3.2 release blocker and we'll discuss the tasks at the next meeting. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 11 March
Present: Ken, Robert, Bron Ken: * wrote a CGI module for HTTP in Cyrus * got all the drafts updated for IETF * facility with eventsource to give a state token at the end of streaming events. Do we want similar mechanism for push over websockets? - might have to be optional, some servers might not have a way to do a token for the user. - open issue to discuss at Prague * did some http refactoring in the code to make it easier to maintain. * found at least one thing in JMAP spec we haven't done - making settings response non-cacheable. * shareWith will be this week Robert: * partially on sick leave last week. * worked on JSContact and JSCalendar drafts for IETF. - haven't received substantial feedback yet, but may get more at IETF. * landed multi-language support on fastmailtest.com last week. * next step - maybe use additional timing information to collect data for current setup, then start collecting performance data for new setup. * implemented Blob/get * now focusing on attachment search indexing code - planning to support http based backend for text extraction with blocking requests. * could imaging defining a configurable parallel implementation - ship multiple to backend at the same time. - not sure if it's worth it, but might be! * sidetracked by issue from github - QP encoding -> we are splitting illegally within UTF-8 words. Can't reproduce that, but we do overrun the hard limit of 76 characters! Making that more RFC compliant. Current code never breaks within UTF8, but keeps going on that line rather than wrapping early. * Next: annotations topics -> will look through existing github issues. Bron: * Email from Michael Menge - making conversations DB easier for people to use / understand / manage. - should document how the tools work better - and document when it's needed! - split into two bits: conversations part and "reverse lookup" part. - make the core part non-optional. * maybe split all "per-user" databases into two files: one being master data and one being cache. * issue with sqlite and modseqs seen in the wild - still not yet debugged. - debugged LIVE on the call! We need to use sqlite3_column_int64 for the modseq fields! Next week, same time. Week after will clash with IETF. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 5 March
Present: Ken, Bron, ellie Ken: * going to work on drafts pre-IETF-deadline * shareWith should be done in next week or two * going to get back to following up on copyright and license discussion ellie: * it's been a while since a stable release, going to do a 3.0.9 later this week Bron: * would be good to do a 3.2 series release soon - once we're at a good stable point * planning for IETF. Next week the meeting is moving back to Monday 11am UTC (7am US East, midday Europe, 10pm Melbourne) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Feb 12
Present: Bron, Ken, Robert, ellie Ken: * last week at CalConnect * fixed issue with Windows TZ names - used existing code in ICU * fixed bug with Outlook truncated timezones - made our code not truncate existing RRULE by sliding start-date forward * did some reorg towards implementing shareWith for /set * new: - maybe 40-50% done with shareWith work - then back to mailbox path by ID on-the-fly upgrade work - confident that what's there now will work on a new system! - but first: drafts for calext! (at least managed attachments before IETF) Bron: * fixed issue with G keys not being created on replicas if messages added without CID, breaking search. * went to Fosdem - key notes are: - spoke about JMAP. Plenty of interest, including some people from Dovecot who are keen to implement JMAP but busy, so can't promise exactly when - went to a session on fsync in Postgresql. We should make our code just die on failed fsync as well, that way we'll clean up properly. - dinner with Perl people - went to Dovecot's Chat Over IMAP presentation - it's actually pretty cool. Just a couple of email headers and then layers on top to make the chat experience good. I like it. Suggested the should bring it to IETF EXTRA. * Then CalConnect: - I hacked on a new module, Text::JSCalendar, which is just the conversion parts of JSCalendar. I'll change Net::CalDAVTalk to use it. - new member Alex does an Outlook connector for CalDAV and is based in Vienna, will catch up next week. - timezones - ongoing problems with naming. ISO interested in being clearinghouse for country-owned names. IANA already do central DB. Need to find a home for a tzdist service now it's standard. - IETF work - I did a presentation about why to keep bringing work to IETF. - enumerated existing half-done drafts and proposed to bring most of them through - I suggested just submitting them all at IETF then we can choose to progress them as interest applies, but at least the process is started. - JSCalendar - a few points were raised, Robert is working on finishing them into the doc. - I'm keen to find another person or two from non-FastMail who can join me at IETF and maybe take the mantle at some point in the future. Succession planning! - next CalConnect in Bedford in June, then FM hosting in Philly in October. Hopefully Apple in the bay area next Jan/Feb. Robert: * fixed a couple of bugs in the JMAP ical layer last week * updated parts of JSCalendar doc and will discuss the rest during the week * this week, continue with Xapian languages update - would like to get it to the point where we can experiment with it * bug Michael reported with recurrence overrides - going to put a test together! - we'll try to recreate it ellie: * the chat stuff sounded glastly at first, but seeing more, it seems pretty neat actually! - how will we implement it in JMAP? - could use EmailSubmission delivery status to show that it's been delivered and maybe even that it's been seen if permissions allow! * been lots of pull requests recently, good quality, been merging them! - nice to see the community chipping in! Planning to hold the meeting again next week. Bron and Robert stayed on the call and confirmed the bug that Michael found. We'll work on it tomorrow. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 29/1/2019
Present: Ken, Bron, Robert, Partha, ellie Ken: * was in Philly last week, so did very little Cyrus programming * found some bugs in scheduling code Robert: * Had a short week last week * Working on rewriting cyrus.indexed.db for Xapian to use mailbox uuids rather than names - Bron to review * Tested locally and on fastmailtest. * When cyrus.indexed.db is read, is migrated on the fly. - will test to see if the indexed.db is read for search, or only in squatter. * Had to update a cassandane test built on internal format of indexed.db * tested revert on fastmailtest.com * to finish, needs UUID changes to conversations.db to land. * will land it along with Ken's storage by UUID patches. * also: updated JMAP Mailbox/set to update subscriptions DB. * working on Xapian multilingual feature - hope to have a demo this week. - do we want to support i;unicode-casemap ? - suspect we'll discover that no clients use this * either way, may need to change uni2searchform to support multilingual correctly. * also the snippet generator needs to know which language was matched to correctly identify snippets. * maybe persist detected languages as an annotation? Rik: * will have a look at the i;unicode-casemap algorithm. * hit calendar-address-set bugs Bron: * fixed ACL creation bug in children of INTERMEDIATE folders * fixed readonly conversations DB issue with XCONV commands because expunge wants to write ellie: * ongoing hosting and relicensing discussion * been doing mailing list discussions * updated build environment to use cyruslibs and now testing against zeroskip Cyrus needs a way to translate from username to identities. * a lookup that does userid -> email address or email address -> "who does this belong to". * is this user local on this server? - for free-busy queries, need a different lookup that returns "is this user in the same customer as me". * ptloader and ldap support for group RACLs already works * maybe implement a redis protocol cyrusdb backend and redis protocol server in postfixlookup at FastMail or whatever Next week Robert, Bron and Ken will be in Zurich, so we won't be having an online meeting next week. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Generating MIME boundaries within cyrus
Yes, a general "create mine boundary" function could be written. I guess nobody figured it was worth the effort yet. Bron. On Sat, Jan 26, 2019, at 00:33, Дилян Палаузов wrote: > Hello, > > sending a MIME message implies generating multipart boundary (random) string. > Cyrus does this in: > > imap/cyrdump.c:g > enerate_boundary() > imap/httpd.c:write_multipart_body() > imap/http_caldav_sched.c:imip_send_sendmail() > imap/jmap_mail.c:_mime > _make_boundary() > imap/lmtp_siele.c:send_rejection() > > Isn’t it feasible to avoid on this occassion code duplication? > > Regards > > Дилян > > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: MMAP performance and using mmap writes
On Sat, 1 Dec 2018, at 01:32, Дилян Палаузов wrote: > Hello, > > > > 3: if your DB is larger than RAM, writing thru mmap is slower than using > > > write() syscalls. Whenever you > > > access a page for the first time, the OS will page it in. This is a > > > wasted I/O if all you're doing is > > > overwriting the page with new data. > > > > I doubt it... especially now we're running on servers with 256Gb of data. > > These databases are usually under a gigabyte in size. I also don't think we > > ever overwrite a page without reading from it first - we're usually > > updating pointers which we've just had to read. > > > > Are there recommendations on RAM for running cyrus imap, that reflect > this aspect? Yes ram is cheap :P Seriously though, we budget for 120 bytes per message in open mailboxes, so if you probably want at least 1 gigabyte ram per 100 users on a machine. If you turn on conversations support (needed for jmap) I would multiply by 4 to allow for the databases there, though jmap doesn't hold as much in ram between connections. Most of the benefit really comes from caching, particularly if you have data on non SSD storage, so the more the better. Bron. > Greetings > Dilyan > > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: MMAP performance and using mmap writes
On Fri, Nov 30, 2018, at 22:09, Howard Chu wrote: > Bron Gondwana wrote: > > Hi All, > > > > We were debugging the CPU usage in a ctl_conversationsdb rebuild yesterday, > > and noticed an interesting thing. 70% of the CPU utilisation for this one > > process > > was inside the kernel! Mostly with dirty pages. > > > > ctl_conversationsdb -R is particularly heavy on the twoskip database - it's > > rewriting a lot of random keys. This leads to writes all over the place, as > > it > > stitches records into the skiplists. > > > > Of course the "real answer"[tm] is zeroskip, which doesn't do random writes > > - but until then, we suspect that the cost is largely due to the face that > > we use > > mmap to read, and fwrite to write! We know that might be less efficient > > already from Linus' comments about 10 years ago! And I guess here's the > > proof. > > > > An option would be to switch to using mmap to write as well. We could > > easily modify lib/mappedfile to memcpy to do the writes. > > > > Does anybody see any strong reason not to? > > I've covered the reasons for/against writing thru mmap in my LMDB design > papers. I > don't know how relevant all of these are for your use case: > > 1: writing thru mmap loses any control over write ordering - the OS will page > dirty pages out in arbitrary order. > If you're using a filesystem that supports ordered writes, it will preserve > the ordering of data from write() calls. This is not a concern at all - twoskip is deliberately designed such that it does a single write and then flush to "dirty" the file, all changes made while dirty are fully revertable if it crashes, and then it does a fsync (msync now I guess!) before a single write which clears the dirty flag. So long as a single 256 byte write is consistent, it's safe. > 2: making the mmap writable opens the possibility of undetectable data > structure corruption if any other code > is doing stray writes through arbitrary pointers. You need to be very sure > your code is bug-free. Yes, this is a significant concern. > 3: if your DB is larger than RAM, writing thru mmap is slower than using > write() syscalls. Whenever you > access a page for the first time, the OS will page it in. This is a wasted > I/O if all you're doing is > overwriting the page with new data. I doubt it... especially now we're running on servers with 256Gb of data. These databases are usually under a gigabyte in size. I also don't think we ever overwrite a page without reading from it first - we're usually updating pointers which we've just had to read. > 4: you can't use mmap exclusively, if you need to grow the output file. You > can only write thru the mapping > to pages that already exist. If you need to grow the file, you must > preallocate the space, otherwise you > get a SEGV when referencing unallocated pages. We always know what we're planning to write, so I'm fine with using an ftruncate call on the file descriptor to extend it. > And a side note, multiple studies have shown that skiplists are not > cache-friendly, and thus have > inferior performance to B+tree organizations. A skiplist is a very poor > choice for a read/write data structure. Yeah, hence zeroskip - it's coming. > Obviously I would recommend you use something carefully designed and heavily > tested, like LMDB, instead > of whatever you're using. We tried and had a bad experience last time - it didn't fit in well with the expectations how our code uses database. I'm not super keen to try again right now. I do appreciate your persistence and passion for your project though :) It's good to see this level of engagement. > There's one point in favor of writing thru mmap - if you take care of all the > other potential gotchas, > it will work on every OS that implements mmap. Using mmap for reads, and > syscalls for writes, is only > valid on OSs with a unified buffer cache. While this isn't a problem on most > modern OSs, OpenBSD is a > notable example of an OS that lacks this, and so that approach always results > in file corruption there. Yeah - that's an interesting point to me as well. At the moment we use a wrapper which is called map_stupidshared (don't blame me, was named before my time) which unmaps and remaps every time if the file has been changed. Insanity. It gets tested for during the configure stage. We have something even more awful called map_nommmap which just reads the entire file into a buffer every time. As you can imagine, performance is awful - but it does work! Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
MMAP performance and using mmap writes
Hi All, We were debugging the CPU usage in a ctl_conversationsdb rebuild yesterday, and noticed an interesting thing. 70% of the CPU utilisation for this one process was inside the kernel! Mostly with dirty pages. ctl_conversationsdb -R is particularly heavy on the twoskip database - it's rewriting a lot of random keys. This leads to writes all over the place, as it stitches records into the skiplists. Of course the "real answer"[tm] is zeroskip, which doesn't do random writes - but until then, we suspect that the cost is largely due to the face that we use mmap to read, and fwrite to write! We know that might be less efficient already from Linus' comments about 10 years ago! And I guess here's the proof. An option would be to switch to using mmap to write as well. We could easily modify lib/mappedfile to memcpy to do the writes. Does anybody see any strong reason not to? Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Nov 12
Present: ellie, Bron, Robert, Ken, Partha Bron: * was at IETF last week * JMAP is close to last call and will probably recharter for calendars/contacts early next year * EXTRA is also close to rechartering, last major work is IMAP4rev2. * have built a package with the rename intermediate patch, but not yet been able to test it. Robert: * JSCalendar - in last call for RFC now. - New version last week. - One update pending: updating RRULEs to better define non-Gregorian RSCALEs. - also a feature branch for Cyrus with latest spec version for testing * pushed keyword updates to mboxset update * main current project is how best to proceed with conversationsdb. - two current issues open due to not including expunged state of messages. - can't search for multiple ANDed "inMailboxId" filter, or showing deleted messages in query results. - have a fix which adds these flags to conversationsdb. - prototyping a workaround to handle legacy records, but it's really a pain. - as long as not rebuilt, might still return deleted messages in query results. - go ahead and switch to binary format as it's more efficient. * Working on smaller JMAP bugs. - there was an issue that we closed with accepting message IDs not according to spec. Now issue where they actually exist, so need to make it more lenient so we can handle real world messages. * haven't started working on Quota/get - that's next. ellie: * imap labels over fetch is pretty close, hopefully can just bang it out tomorrow! * it'll definitely benefit from robert's convdb additions, but the merge conflicts will be trivial to resolve Ken: * working on mailbox history and tombstones * will probably rebase path-by-id onto rename intermediate * spent time working on drafts - need to do more on FCC draft this week * with mailbox-paths-by-id, need to keep mailbox name in the cyrus.header in case we lose mailboxes.db. * is it cheaper to use symlinks on disk? - Bron isn't in favour, couple of issues - disk path lengths and consistency/cleanup. - also - update mailboxes.db twice - once at start saying what will happen, then afterwards once the on disk files have been updated. - that way, we always know which mailboxes might be in an unknown state and can be repaired. * one last release candidate for SASL - build issues and Coverity issues. - planning to release this week. Partha: * no cyrus work last week or coming up in the next week. >From next week, switching to 8am Tuesday Melbourne time (4pm US East, 10pm >Vienna) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 22 Oct
Present: Partha, ellie, Bron We forgot that both Ken and Robert weren't coming, so we didn't need to stay up late! Oops. Partha: *merged zeroskip a couple of weeks back, but it has issues with unused variables which he will fix. ellie: * trying to find tags for cyrus-sasl - it looks like they never got exported from Phabricator, so they might need to be recreated by hand for older releases. * planning for X-GM-LABELS equivalent (probably called MAILBOXIDS to match objectid and JMAP naming) via IMAP. Bron has been sick all last week and didn't get much of anything done, let alone any Cyrus work! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Notes Oct 1st
That's excellent Matt. Do let us (Cyrus upstream team) know if there's anything we can do to help you with taking on packaging. Cheers, Bron. On Tue, Oct 2, 2018, at 12:02, Matt via Cyrus-devel wrote: > On 02/10/18 14:16, Ondřej Surý wrote: > > Hi, > > > > On Mon, Oct 1, 2018, at 16:18, Bron Gondwana wrote: > >> ellie: > >> * Cyrus is no longer being packaged directly by Debian > >> - we should look at how we want to distribute Cyrus for Debian users - > >> nightly builds? Releases? > > I am sorry for my lack of time after I changed jobs. Today, I finally moved > > the Debian packaging git repositories to Debian GitLab instance and I can > > help other people to get involved in packaging Cyrus IMAP and Cyrus SASL > > in Debian. > > > > The repositories are here: > > > > https://salsa.debian.org/debian/cyrus-imapd > > https://salsa.debian.org/debian/cyrus-sasl2 > > > > Max Kosmach has already volunteer to fix the cyrus-imapd in Debian stable > > and I can mentor and sponsor other people to move the packaging forward. > > > > Unfortunately, I am busy elsewhere to continue with the main bulk of > > packaging > > Cyrus IMAP and SASL in Debian. Also it's not just "take whatever upstream > > has" > > due tu past legacy and upgrade paths between Debian stable releases. > Hi Ondřej, > > I have been using Cyrus for almost a decade, and Debian since 1994. I > run a few Cyrus servers, albeit not using all the features. I have done > Debian packaging before (for work projects not for Debian itself though) > so I know my way around a bit but I'm not completely up to speed on some > of the newer packaging features. > > I'm willing to help with packaging, coding, testing, documenting, etc. > > Matt > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting notes 24/9 - no meeting this week
As expected, we're not having a meeting this week due to most of the devs being at CalConnect in Karlsruhe and otherwise occupied. Almost all the dev work in the past week has been JMAP related, as FastMail finishes off the beta of the FastMail-on-JMAP conversion project, and because JMAP is currently at Working Group Last Call within the IETF, so we're keen to get real world testing experience on it before it becomes a published standard. Other interesting little bits: fixed (and then properly fixed, oops) an issue where utf-8 sequences were being split inside header encoding. We do expect to be getting back to work on non-JMAP things soonish. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Sep 17
Present: Bron, Robert, ellie, Partha Ken sick. Next week - Bron, Robert and Ken will be in Karlsruhe for CalConnect. Robert: * have been working on JMAP bugs all last week * will continue this week! Picking whatever issues tagged with "jmap" label on github * updating JSCalendar draft - still some questions open, so CalConnect next week is a good time to close them. - will send an updated draft this week. Partha: * nothing to report on Cyrus this week ellie: * fixed a crash in 3.0 today * working on moving website to github pages * looking at bringing back annotatemore for 3.0 and 3.2 - it's still in 3.0, so just master to update - not yet sure if config time option or compile time option - Bron: make it config, people can use upstream packages and code gets better tested if compiled. Bron: * mailing list: talked about annotatemore end-of-life and supporting existing users for longer while they migrate installed client base. * lots of JMAP stuff too! Been working on JMAP calendars, testing ACL edge cases, builds. * calendar shareWith - read from Cyrus, write in Perl (for now) * will be traveling for next three weeks (Karlsruhe, Philly, New York), so finishing up stuff in Melbourne. Next week: not sure if it will run as usual - depends on CalConnect schedule since 3 of us will be there. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Bring back ANNOTATEMORE support
On Sat, Sep 15, 2018, at 00:20, Samir Aguiar wrote: > Hi Bron, > > On 09/12/2018 03:49 AM, Bron Gondwana wrote: > > Samir: what's your plan for moving those users to something which > > supports standard METADATA? Because the downside of keeping on > > supporting a legacy ANNOTATEMORE in Cyrus is that somebody has to > > maintain that forever if you're never planning to get off it, either you > > or everyone else touching that code! > > Yes, that's a fair point. > > The plan is to update our client to use METADATA instead of > ANNOTATEMORE, while keeping support for both in the server. After this > we deprecate ANNOTATEMORE to signal customers that they should upgrade > their clients. This deprecation should happen mostly likely within two > years, and in the next years we will be removing it completely from the > server. Right! OK, we'll add it back in 3.0 and 3.2 then (with a config option to turn it on), and deprecate it after that :) I have CC'd ellie who is making the releases and created https://github.com/cyrusimap/cyrus-imapd/issues/2525 Cheers, Bron. -- Bron Gondwana br...@fastmail.fm
Re: Bring back ANNOTATEMORE support
If it's an easy revert, I think we'd be happy to leave it available but marked deprecated with a flag that means you need to explicitly turn it on: deprecated_annotatemore_support: yes And include that in a 3.2 release. Longer term though, the main reason for dropping it is that the annotations code is a weird shape in order to support both old and new style ANNOTATEMORE and METADATA. The plan was to clean up that mess once it no longer had to support METADATA. Samir: what's your plan for moving those users to something which supports standard METADATA? Because the downside of keeping on supporting a legacy ANNOTATEMORE in Cyrus is that somebody has to maintain that forever if you're never planning to get off it, either you or everyone else touching that code! Regards, Bron. On Wed, Sep 12, 2018, at 11:22, Dilyan Palauzov wrote: > Hello Samir, > > if the opt-in ANNOTATEMORE feature is something that only you will use, as it > currently looks like, I see no reason to bother others with it. > > Greetings > Дилян > > On September 11, 2018 2:40:48 PM PDT, Samir Aguiar > wrote: > >Hi Dilyan, > > > >On 09/11/2018 06:17 PM, Dilyan Palauzov wrote: > >> it does not make sense to keep the ANNOTATEMORE code just for your > >specific case. > >> > >> You are entitled to issue an updated client software dealing with > >METADATA and ask users to update, or for your server to revert the > >respective change. > > > >Yes, absolutely. We will need to revert that change and restore > >ANNOTATEMORE support anyway since it's not possible at the moment for > >all of our clients to upgrade. > > > >But I believe my question was ill-phrased. I actually meant to ask if > >such revert, after done by us, would be accepted upstream as an opt-in > >feature, or if the team has already decided to drop that implementation > >for good. > > > >Kind regards, > >Samir Aguiar > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Bring back ANNOTATEMORE support
Hi Samir, The ANNOTATEMORE was a draft which got replaced with METADATA. If there a reason why you are still using the draft rather than the spec which was published in 2009? Bron. On Tue, Sep 11, 2018, at 09:17, Samir Aguiar wrote: > Hi all, > > We are currently upgrading to 3.0.8, but 3.1.x will probably be stable > by the time we are finished. From what I could see ANNOTATEMORE support > was dropped in 3.1. > > Would it possible to bring it back and maybe make it opt-in in > imapd.conf? Our groupware clients depend heavily on it and it would take > a while until we can fully deprecate it. > > I ran some tests and apparently reverting #26fbf999312 seems to be > enough, but please correct me if I'm wrong. > > Thank you in advance. > > Kind regards, > Samir Aguiar > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Fwd: Build failed in Jenkins: ci #8244
The issues in TesterJMAP are related to spec updates that JMAPTester doesn't have yet. I have 50% fixes done on my work desktop and will finish them today. - Original message - From: jenkins@cyrus.works To: jenkins@cyrus.works Subject: Build failed in Jenkins: ci #8244 Date: Wed, 5 Sep 2018 03:23:55 + (UTC) See <https://cyrus.works/job/ci/8244/display/redirect> -- Started by remote host 192.30.252.39 Started by remote host 192.30.252.44 Building in workspace <https://cyrus.works/job/ci/ws/> [ci] $ /bin/sh -xe /tmp/jenkins1765257438755413091.sh + git -C /cyrusworks/cyrus-docker/ pull Already up-to-date. Waiting for the completion of master-jessie master-jessie #8327 completed. Result was FAILURE Build step 'Trigger/call builds on other projects' marked build as failure == This automated email was sent via FMQA.com. Please contact the FastMail QA department for more information. == -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Sep 3
Present: Partha, Robert, ellie, Bron Robert: * This week - fixing github issues - patching Calendar events will now work - Mailbox query changes - every property is immutable, so there's no immutable sort, so have to issue a deleted for everything which has been touched, and then an added at the place where it is, if present * Reading the subscription code in IMAP, so can use the user.subs data for isSubscribed JMAP facility. - by default, mailbox created over JMAP is being created unsubscribed. * Last week: - did lazy loading of Email/get data Partha: * Made a pull request for zeroskip, not yet merged, but fails with -Wall because libzeroskip APIs expect everything unsigned char *. * Working a patch to convert all of cyrusdb to use unsigned char *! - making sure all unit and cassandane tests are continuing to pass. * Microsoft's new kv database has benchmarks, trying to integrate them with libcyrusdb. ellie: * have set cyrusimap.github.io up, which will eventually get the real cyrusimap.org website moved to it. * have been working on a new 2.5 release and getting Cassandane working better with multiple versions. * converted Cassandane from tabs to spaces (like cyrus-imapd repo was a while back) * potentially use perltidy on the repo? Will check with perl experts. Bron: * added spamScore fetch item to avoid header parsing overhead and make fetching it faster. * cyr_expire: wasn't cleaning up tombstones, fixed that. * have been stracing and valgrinding common queries to speed things up. * working on making conversations parsing faster * at some point will get back to master-to-master sync code - have an in-progress branch -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Updating Cyrus Bylaws
aff because I'll stop paying them if they don't fix their mistakes! It's harder when we take something large and not well architected (the mboxevents module is probably the most notorious case in the past decade) and it causes ongoing maintenance headaches. So we do have to evaluate committers. I'd be happy to accept additional contributors who are showing up to the public meetings we hold once per week and collaborating with the team. I'm loathe to add people with commit rights who only communicate by dropping large chunks of code that haven't been had the design socialised with the rest of the group first, because that leads to a fragmented codebase. > I understand that for some of you JMAP support is very interesting, > for others having IMAP/WebDAV/CalDAV/CardDAV where known problems are > fixed on master, but are not on the stable branch is suboptimal. I agree. Please do tell us when you hit such issues. We're not doing much on those subsystems on master, and we do try to backport them where they seem to be independent issues and not side effects of the JMAP subsystem changes. But I'm sure we miss them sometimes. Since JMAP is a key project for FastMail, and the people doing the bulk of the work right now are being paid by FastMail, clearly JMAP is getting the bulk of development. Once JMAP is final, we expect to cut another stable release, and then focus on cleanups and optimisations. > My preference is to have soon stable release with any fixes for > IMAP/WebDAV/Sieve that were not backported and have later a release > with JMAP/multi-master/better backup when these are ready. And from > that moment on take care that any fixes relevant for > IMAP/WebDAV/Sieve/something else are backported to the stable branch, > while development for JMAP or (new RFCs) goes on master. We always go through phases of this. We were backporting everything to 3.0 for a while, but then master diverged sufficiently. There are now new features in 3.1 (e.g. SAVEDATE and STATUS=SIZE) which depend on index format changes which will never be backported to 3.0. It was also unfortunate that there was an upgrade bug that hurt the 3.0 series for a while. But I expect that once 3.2 comes out, again there will be a flurry of backported fixes to the 3.2 stable series, followed by a while where there's high churn on the master branch, and only key bugfixes backported. That's a normal open-source project lifecycle. Right now we're in the unstable, high churn section where there's no much that's both important enough and unentangled enough to justify backporting. > Are the concerns raised recently by Quanah the only blockers for cyrus > sasl 2.1.27 and what reasons prevent releasing cyrus sasl 2.1.27 > within two months? I will leave this for Ken to answer, as SASL is more his department. I believe the blockers were waiting on testing to ensure there wasn't any regression - the cyrus-sasl code doesn't have a comprehensive test suite. Regards, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Internal error: assertion failed imap/message.c: 4246: !message_need(m, M_RECORD)
Hi Nic, Damn! That interface is really frustrating in terms of inability to debug. One possible culprit in all this is b56e391c9579b3c4c9fd6e64fa29e627ec358339. I've just copied that back to 3.0 asc665af571dc1f9d870d53f2aa4ea29bb5cd52738 . Are you able to try with that added? Cheers, Bron. On Fri, Jul 27, 2018, at 06:02, Nic Bernstein wrote: > Bron, > Answering my own question, I now see that the fix you refer to is in cfb3054, imap/mailbox.c on 1/2/2017, which is in both master and 3.0... However, I've still got the problem, so same failed assertion, but different bug? > -nic > > > On 07/24/2018 12:10 PM, Nic Bernstein wrote: >> Bron, et al., >> Was this change ever cherry-picked to 3.0? I am seeing the same issue with recent 3.0 HEAD, but slightly different location: >> >>> user.masked: updating sync_crc 521983118 => 503807715 >>> fatal error: Internal error: assertion failed: imap/message.c: 4286: !message_need(m, M_RECORD) >> A git log of imap/message.c doesn't show a commit from 1/2/2017, and nothing affecting imap/message.c around that time seems to line up with this. >> >> Please advise, >> -nic >> >> >> On 01/02/2017 07:13 AM, Bron Gondwana via Cyrus-devel wrote: >>> Thanks for the data. It was 8 bytes of zeros across a UID and INTERNALDATE in the cyrus.index file. >>> >>> I now have a fixed reconstruct which can detect and repair this rather than aborting, pushed to master. >>> I also have a Cassandane testcase for this and a couple of other things that reconstruct does :) >>> >>> Bron. >>> >>> On Thu, 29 Dec 2016, at 09:45, Bron Gondwana via Cyrus-devel wrote: >>>> Wow, interesting. Are you willing to send me a tarball containing the spool folder including cyrus.index and cyrus.cache files as well as the email files themselves? I'll need your imapd.conf file as well :) >>>> >>>> Cheers, >>>> >>>> Bron. >>>> >>>> >>>> On Thu, 29 Dec 2016, at 00:28, Thomas Cataldo via Cyrus-devel wrote: >>>>> Hi, >>>>> >>>>> Running a build of 3.0.0-beta6 I hit the following assertion on one of my test mailboxes after playing a bit with the replication stuff : >>>>> >>>>> root@bm1604:~# /usr/lib/cyrus/sbin/sync_client -n eclipse -o -u t...@ex2016.vmw >>>>> Fatal error: Internal error: assertion failed: imap/message.c: 4246: !message_need(m, M_RECORD) >>>>> root@bm1604:~# cyradm -u admin0 localhost >>>>> Password: >>>>> localhost> version >>>>> name : Cyrus IMAPD >>>>> version : 3.0.0-beta6-3-gf721e5b >>>>> vendor : Project Cyrus >>>>> support-url: http://www.cyrusimap.org >>>>> os : Linux >>>>> os-version : 4.4.0-57-generic >>>>> environment: Built w/Cyrus SASL 2.1.26 >>>>> Running w/Cyrus SASL 2.1.26 >>>>> Built w/OpenSSL 1.0.2g 1 Mar 2016 >>>>> Running w/OpenSSL 1.0.2g 1 Mar 2016 >>>>> Built w/zlib 1.2.8 >>>>> Running w/zlib 1.2.8 >>>>> CMU Sieve 2.4 >>>>> mmap = shared >>>>> lock = fcntl >>>>> nonblock = ioctl >>>>> idle = idled >>>>> >>>>> root@bm1604:~# telnet localhost 1143 >>>>> Connected to localhost. >>>>> Escape character is '^]'. >>>>> * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE STARTTLS AUTH=PLAIN SASL-IR] server ready >>>>> . login t...@ex2016.vmw xx >>>>> . OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE ACL RIGHTS=kxten QUOTA MAILBOX-REFERRALS NAMESPACE UIDPLUS NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY CATENATE CONDSTORE ESEARCH SEARCH=FUZZY SORT SORT=MODSEQ SORT=DISPLAY SORT=UID THREAD=ORDEREDSUBJECT THREAD=REFERENCES THREAD=REFS ANNOTATEMORE ANNOTATE-EXPERIMENT-1 METADATA LIST-EXTENDED LIST-STATUS LIST-MYRIGHTS
Meeting notes Jul 16
Present: Ken, Bron, ellie, Partha, Robert Ken: * finished Intermediate Mailboxes last week * played around with JMAP - consolidation of JSON output last week. * yesterday - attempted to fix a github issue, but broke other stuff (returning IDs when not asked for / Email/parse) Robert: * two weak points with spec compliance - currently hard-coded what we return on object creation - by spec, we should error on any unknown properties, but currently don't abort on unknown (breaks JMAP Tester) ~ probably need a registry of JMAP property per capability - JMAP Mail code is the most up-to-date code, will generalise to other modules, so best to improve JMAP mail* reworking JMAP code for intermediate mailboxes (may have broken things)* code is trying to be a bit clever by assuming most things are valid, only doing expensive parts if it runs into an issue* looking at the JMAP backlog - quite a backlog! Labeled on github as JMAP. - multiple calendar events issues, there have been some spec changes - #2437 - do we want a more generic approach to extension properties, or hard code them in Cyrus? - SMTP client protocol error - can make it smarter and return SMTP error. Couple of other requests around making EmailSubmission smarter. - Ken will take the performance stats task (#2400) Partha: * have been busy studying during meeting time for the last while * mostly working on FastMail operations * zeroskip: cunit tests all passing - haven't run Cassandane with it yet, but will get it onto next sprint.* won't have much time for big issues, but could take on some smaller issues ellie: * haven't been feeling well for the last week and didn't work much * mostly keeping on top of mailing list stuff Bron: * released 3.1.5 so others at Hackathon had a snapshot to work from * been working on JMAP Proxy this weekend at IETF102 Hackathon * Changed FM Cyrus operations to always build new code to new binary locations, such that we never have old and new binaries running at the same time! (is a problem when file formats are upgraded and long- running procs load files created by new binaries) Meeting same time next week -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Cyrus IMAPd version 3.1.5
The Cyrus team is pleased to announce the immediate availability of a new version of Cyrus IMAP: 3.1.5 This is a snapshot of the master branch, and should be considered for testing purposes and bleeding-edge features only. It is available as a git tag, which can be found here: https://github.com/cyrusimap/cyrus-imapd/releases/tag/cyrus-imapd-3.1.5 Join us on Github at https://github.com/cyrusimap/cyrus-imapd to report issues, join in the deliberations of new features for the next Cyrus IMAP release, and to contribute to the documentation. On behalf of the Cyrus team, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting report Jun 2
Present: Robert, Bron Since there were just the two of us, we chatted about a few things: Japan: Robert had a great time, it was lovely. JS Calendar: Robert to ship an update with the changes from the mailing list discussion today (in time for IETF Montreal) Mailbox/set: Robert will keep working on this * we have a conversations lock for the entire time each MethodCall runs, so it's always going to appear atomic to the outside * will do a first pass check that things will probably succeed (and order them), then actually run the Mailbox actions. * if there's a failure during, then users might see intermediate states, but otherwise they won't show up in /changes any more with the new "added" code. the double base64 incident: Robert felt bad about it because Email is so key, and once it's corrupted it's hard to uncorrupt. * We didn't have tests that found it. We now do :) * would be great to have more complete testing, not just bug testing as Cassandane has * will work on getting more full protocol coverage into JMAPTester * shit happens, it was found and fixed. Don't feel too bad! CREATEDMODSEQ: Bron has shipped that code to master and FastMail's future/staging branches * now have "added" on all /changes responses. * and queryState on all /query and /queryChanges. Intermediate Mailboxes: Bron will chat with Ken about them. Close to ready to deploy. Xapian: New version 1.4.6 is available with snippet support (even better than what we have) * Robert will try it out. * still need to apply our patches for full CJK support. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
state -> queryState change just landed on master
Just pushed a change to master (and cassandane) changing the "state" to "queryState" for all query and queryChanges JMAP methods, as per latest spec draft. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting minutes Jun 18, 2018
Present: Ken, ellie, Bron, Dave Robert is still away on holiday, so we don't have a report from CalConnect yet! Dave (visiting!): * CTO for ClearCentre (ClearOS) Linux distro for SMB. * customer who needs to disable duplicate elimination * Running 2.5.x? * pointed out that there's a per mailbox annotation, and how to set it.* also found some missing slashes on the docs! Murch: * Have been working on IETF drafts * Planning to get to intermediate messages * nobody working with gssapi stuff, might need to talk to Alexey at IETF* waiting to hear back from Ned about sieve-fcc wording * should be arriving Sunday afternoon at IETF102. ellie: * dillian has some questions about changing content disposition from download to inline.* planning to have a crack at group racl's since i've got that section of code mentally swapped in at the moment* have implemented cassandane infrastructure for testing with an ldap backend* questions about ptscache and auth_krb - our goal is Bron: * fixed takefv bug in sieve where a change to strarray meant zero-item lists returned NULL rather than [], and this broke things.* somehow failed to create the Extra session for IETF102. Working on getting that fixed! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting notes from Feb 27th, 2018 (oops)
(look what I found in my drafts folder! I checked the list, and it was never posted) Ken: - last week away first half - second half - updating calendar drafts for IETF drafts. (Bron will be in the session)- recurrence bugs in libical - thought close to having it fixed, but regression tests hanging.- SASL - still not released, now on hold because person reporting kerberos bug is getting a kerberos expert to look at it. If not sorted by IETF, get Alexey to meet people in London for it.- libical - support for jumping into the middle of an rrule. * issues with rule starting on a leap day - calconnect started TC-CALSPAM. Robert: - unsure what to do with jscalendar - not much discussion on calext mailing list, because already discussed in calconnect. Ken will ping calconnect people to make comments on list.- 99% of people involved in calendaring have already given feedback! - tasks: there's a separate draft on that, and race / ordering issue between jscalendar and icalendar tasks.- backported xapian fix from master to 3.0 for tibbs' request from IRC. * related to cyruslibs issues we see with different compilers too. - we will have to run two different cyrus.works instances to test both the older compiler version and the newer compiler versions. - last week: working on Xapian fork. * Finally successfully building in Travis system at Xapian again. * Making progress at glacier speed at getting upstreamed - depending on Olly's time. * Hope snippet generator will land very soon. - JMAP * in process of fixing bugs, restarted work on rewriting JMAP core. * initially started on separate branch, but now writing inside out. * more iterative style. * might introduce more bugs, but catch quicker. - expect JMAP work to fill up the next 2 weeks. - JMAP - object create, spec says you should return any server-set properties and any properties that the client didn't explicitly set. * Bron: should be possible to request what you actually want via backrefs - don't tell the client stuff it doesn't care about! Partha: - spent last week on build issues and FastMail customer issues. - cyrus.works updated to more recent version of Debian and more recent version of compiler.- easier to work on zeroskip with Bron around - will make cyrus.works send us warnings - tried having stretch and jessie builds on cyrus.works, but it can't work in parallel, so might have to do sequentially or something. * Bron: cyrus.works is really complex to be safe from FastMail infrastructure, but maybe we're better off just paying some money and spinning it up somewhere else! ellie: - been helping tibbs with Fedora builds and the odd issue and mailing list- cassandane now spits out rlimits in verbose mode * will be useful later. Nic: - should get discussion on 2265 - documentation of tombstone records. - working on dilyanpalauzov's reports Bron: - email about sync_client changes * response showed "it looks scary" is a bit bad even on dev list! Bron to reply.- should get a strategy which can be backported all the way to 2.5. - how's core rewrite going? Ken: distracted by other things, but back to it very soon.- Robert: good time to implement same thing in search layers. - JMAP-proxy work New internals: - Lots of API changes for all sorts of things - Conversations DB doesn't have uniqueid right now, so can't change xapian. So do core changes first. Matt (via Rik): - been working on Cyrus test suite against Cyrus and reporting bugs. - patch against Cassandane to run JMAP test suite. - mostly passing! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Cyrus Jmap support question
Hi,
I'm really sorry you've had such difficulty! We're using Cyrus with
http/1.1 at FastMail, so I don't have any direct experience with http/2.
We're also not using TLS directly in Cyrus, because we use an nginx
proxy in front which terminates the TLS connections.
You can enable regular telemetry logging to get http logs:
https://cyrusimap.org/imap/reference/faqs/o-telemetry.html
Ken (CC'd) is our expert on the http/2 code, so hopefully he can help
you.
Bron.
On Tue, May 22, 2018, at 03:47, Zhivko Vasilev wrote:
> Hello Everyone,
>
> I hope you can help me.
>
> I'm implementing a JMAP client and try to use Cyrus as
> reference server.> Send and Receive via IMAP/SMTP work fine.
> But when I try to connect via HTTP/2 , connection is reset by HTTP/2
> reset stream command.>
> imapd.log file show only "starttls: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-
> SHA384 (256/256 bits new) no authentication">
> My http2 call is POST https://[1]cyrus.mailtemi.com[2]:443/jmap[3]
> HEADERS:
> "accept", "application/json"
> "content-type", "application/json"
> "Authorization", "base64 encoded user name:pass"
> BODY:
> {"methodCalls":[["Mailbox/get",{},"#1"]],"using":["jmap-
> core","jmap-mail"]}>
> Is there a way to turn on HTTP log to track what I do incorrectly.
>
> I tried to track with curl the server allways reponded with HTTP
> error 301.> The command I've tried was
> "curl --verbose --http2 -X POST \
> -H "Content-Type: application/json" \
> -H "Accept: application/json" \
> --user u...@cyrus.mailtemi.com:q \
> -d '[["Mailbox/get", {}, "#1"],"using":["jmap-core","jmap-
> mail"]]' \> http://[4]35.192.121.94:443[5]/jmap[6];
> returned
> * Connected to 35.192.121.94 (35.192.121.94) port 80 (#0)
> * Server auth using Basic with user 'u...@cyrus.mailtemi.com'
> > POST /jmap HTTP/1.1
> > Host: 35.192.121.94
> > Authorization: Basic dTFAY3lydXMubWFpbHRlbWkuY29tOnE=
> > User-Agent: curl/7.58.0
> > Connection: Upgrade, HTTP2-Settings
> > Upgrade: h2c
> > HTTP2-Settings: AAMAAABkAARAAAIA
> > Content-Type: application/json
> > Accept: application/json
> > Content-Length: 61
> >
> * upload completely sent off: 61 out of 61 bytes
> < HTTP/1.1 101 Switching Protocols
> < Connection: Upgrade
> < Upgrade: h2c
> * Received 101
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after
> upgrade: len=0> * Connection state changed (MAX_CONCURRENT_STREAMS updated)!
> < HTTP/2 301
> < date: Mon, 21 May 2018 17:36:58 GMT
> < location: /jmap/
> < vary: Accept-Encoding
> < content-length: 0
> <
> * Connection #0 to host 35.192.121.94 left intact
>
> Please give me some advice or clue how to track what is the problem.
> I'm stuck at this point for two days :(
>
> Thanks ,
> Zhivko Vasilev
>
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Links:
1. https://my.ip.address:443/jmap
2. mailto:u...@cyrus.mailtemi.com
3. https://my.ip.address:443/jmap
4. http://35.192.121.94/jmap
5. https://my.ip.address:443/jmap
6. http://35.192.121.94/jmap
Notes May 7
Ken: * worked on some Cassandane stuff * spent far too much time working out why smtpserver was playing up, seems maybe just a local issue* kept working on mailboxes.db stuff. * looked at SASL a bit - waiting for feedback from Alexey, will probably release this week. - RC today - final release Friday * waiting for rest of feedback on LIST-MYRIGHTS before doing fixes * expect something back on fcc too! Bron will ask list for feedback - had already * implemented UNAUTHENTICATE in master, seems to work fine. Would be good for murder. - ellie: should it be only for admins? - text suggests: should only be used for admins. - security risk if non-admins, intermediate proxy may not be aware, and then connection is still available to user, who can authenticate to backend as another user without proxy checking for validity. Bron: * Discussion about semi-synchronous replication. * Github issue from FM - issue with mailbox with no parent. Assigned to Ken. - could simplify the LIST code - Ken and Robert to look into - needed soon, so on 3.1.x * added support for sentdate, which includes a version change for both cache and index - created github issues around version upgrades, replication and costs. - problem is: if replicating new field that replica doesn't support, creates CRC mismatches - fix for now: make savedate optional and turned off by default, so can upgrade everything first before turning on - better fix is to support storing in annotation for older versions, and seamlessly upgrade/downgrade without breaking CRCs. Robert: * been implementing latest JSCalendar in Cyrus for the last week - more or less done - minor differences between Cyrus and spec still - with experience, can update JSCalendar RFC draft, will send out tomorrow - will discuss with Daniel (IETF calext) when to schedule last call - make it overlap CalConnect and overlap IETF102 deadline to get more feedback.* JMAP Mail work - have implemented Email/parse - so can now look at attached emails. - believe we are now feature complete for parts which are implemented in Cyrus - missing feature: interim states (e.g. A => B, B => A mailbox renames) - Ken: do we need it for reference implementation status? Yes, we should! - question: priority vs other things (calendar / contacts code cleanup) - not implemented: + vacation responses (hook up sieve to JMAP) + JMAP notifications (push in-process) + Making EmailSubmission more sophisticated (means big change in how we work in Cyrus) - discussion of future-scheduled events (e.g. snooze, undo send, etc) ellie: * been picking at cassandane and the imapparse stuff but haven't made concrete progress* spent some time investigating moving the parse_ functions from imapd into imapparse.c so they can be tested - would be a pretty big faff and the whole reason is to backport the eatline changes to 3.0 - amount of surgery before cunit can see them would be too much for a 3.0.x release* spent a bunch of time reading about github pages for www.cyrusimap.org possible rehome Partha: * just started working on splitting system flags in struct index_field.* other work was FM specific (mostly perl) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Cyrus Meeting Minutes April 30, 2018
Present: Bron, Ken, Robert, ellie, Partha, Nicola Ken: * short week last week, was traveling. * continuing to work on mailboxdb keys * need to do SASL release, we have a solution to GSSAPI thing. * sieve-fcc: what needs to be done. - Do we need to force people to do it for notify? - Tricky to implement. - do we need it? - mostly for people to either prove what happened for audit trail, or for debugging, so maybe it doesn't need to be exact. - Ken didn't push latest revision, so will have to redo and push. Robert: * been working on JMAP-mail mostly * except for Email/parse and maybe some other gaps in the specification.* now working on JSCalendar spec updates in calendar code. * will come back to Email/parse afterwards. * CalConnect Tokyo - goal is to take updated JSCalendar spec in tandem with Last Call, and get any last feedback. - plan to submit updated specification this week (implementation will be done then!) - other idea: get contacts happening more (work with Ribose people). Need to decide best strategy.* Index record updates: - Partha can have a look at it. - Will try to fix current bug (2329/2330) in JMAP separately * JSCalendar - for participant, there are two properties - schedule sequence (SEQ of the last update from the participant), schedule- updated (DTSTAMP of the last update from the participant). - in FastMail this is done with Perl, so we will need to add it there. - right now there's no inbound iMIP handling in Cyrus, so it's up to the client. - should put it in lmtp, sieve extension, etc. - will have to store as an X- parameter on the Attendee. - good discussion point to look at JMAP X-* and see if they should be standardised. - might dovetail with Mike's Participant work. ellie: * have released 3.0.6 * been working more on the imap parsing stuff * got a maybe-fix for this evening's fetch bug (2329) but it's untested * have implemented prot_peek and prot_lookahead, which might be useful for http2 Partha: * mostly been doing FastMail stuff * largely GDPR work Bron: * did some JMAP work with allInThreadHaveKeyword last week. * does IMAP need a "can't be returned for legal reasons" error. - one-way flag - once set, can't be unset by user. ACL to allow override? - per-message ACL? Use-case outside this? - Ken: need per-message ACLs for manage attachments in CalDAV anyway. Nicola: * cyrus probably doesn't have much for GDPR. * maybe Cyrus website needs a privacy policy which says that we keep web logs. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: shared xDAV resources
Ken knows this code best. I bet there's something which is requiring that there's a user on the mboxname because we implement the same behaviour at FastMail by having a separate user on which shared resources are kept. The DAV resources are stored per-user, and without a place to keep them for "shared calendars" that code might just not be accessible. I'm sure it would be possible to create a shared DAV database as well for this case, but it just needs some programming effort. Bron. On Sun, 8 Apr 2018, at 07:30, Anatoli wrote: > Hi All, > > I'm trying to understand the code responsible for enumerating user > calendars (and xDAV resources in general) to try to make the > discovery work for shared resources too (currently there's no way to > access shared resources with Apple xDAV client implementation, yes > with Thunderbird as it doesn't use the discovery mechanism, but > instead should be pointed to the exact URL for each calendar). If I > understand it correctly, the functionality is in imap/http_caldav.c.> > Could you please point me to the place where the enumeration occurs > and briefly mention how the general workflow looks like?> > The client asks for: > > PROPFIND /dav/calendars/user/<user@domain>/ > > ...> > The server responds with: > > HTTP/1.1 207 Multi-Status > > > > /dav/calendars/user/<user@domain>/ > ... > > /dav/calendars/user/<user@domain>/Default/ > > ...> > The idea is to include in the returned lists the shared calendars too > with the discovery logic based on the IMAP shared folders.> > Below goes the initial exchange between the calendar app on iOS > 10.2.6 and Cyrus 3.0.5 when the exact URL (/dav/calendars/shared/) > for the shared calendar is provided in the advanced settings of the > app (the URL finally resets to the user principals folder > (/dav/principals/user/t...@domain.com/) as iOS is pointed to it by > Cyrus). In the attached file goes the telemetry for the rest of the > communication.> > Thanks, > Anatoli > > -- t...@domain.com Sun Mar 25 06:05:36 2018 > > <1521968736<*PROPFIND* */dav/calendars/shared/* HTTP/1.1 Accept: */* > Content-type: text/xml Connection: keep-alive Content-length: 181 > Host: mail.domain.com User-agent: iOS/11.2.6 (15D100) accountsd/1.0 > Prefer: return=minimal Depth: 0 Brief: t Accept-language: en-us > Authorization: Basic ... Accept-encoding: br, gzip, deflate > > <1521968736< xmlns:A="DAV:"> > > > > > > > >1521968736>HTTP/1.1 207 Multi-Status > Date: Sun, 25 Mar 2018 09:05:36 GMT Strict-Transport-Security: max- > age=600 Vary: Accept-Encoding, Brief, Prefer Preference-Applied: > return=minimal Content-Type: application/xml; charset=utf-8 Content- > Length: 546 > > xmlns:C="urn:ietf:params:xml:ns:caldav"> > > */dav/calendars/shared/* > > > >*/dav/principals/user/t...@domain.com/* >principal> > > > >HTTP/1.1 200 OK > > > <1521968736 Host: mail.domain.com Connection: keep-alive Accept: */* User-Agent: > iOS/11.2.6 (15D100) accountsd/1.0 Accept-Language: en-us Content- > Length: 0 Accept-Encoding: br, gzip, deflate > > >1521968736>HTTP/1.1 200 OK > Date: Sun, 25 Mar 2018 09:05:36 GMT Strict-Transport-Security: max- > age=600 Cache-Control: no-cache Link: ; > rel="server-info"; token="80769c2c66d340ecd178710db26d56b9c4699e3e" > DAV: 1, 2, 3, access-control, extended-mkcol, resource-sharing DAV: > calendar-access, calendar-auto-schedule DAV: calendar-query-extended, > calendar-availability, calendar-managed-attachments DAV: calendarserver- > sharing, inbox-availability DAV: addressbook Allow: OPTIONS, GET, > HEAD Allow: PROPFIND, REPORT, COPY Content-Length: 0> Email had 1 attachment: > * telemetry.log > 36k (text/x-log) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
The CyrusDB interface
turn code is CYRUSDB_DONE to know if the foreach found a key. If foreach is called with a transaction pointer, then it is your responsibility as the caller to also pass that pointer (and a pointer to the database) in that rock, so that callees can make further operations within the same transaction. A foreach with a transaction does NOT unlock before calling its callback. Writing: === There are three write operations: cyrusdb_create cyrusdb_store cyrusdb_delete cyrusdb_store will either create or overwrite an existing key. cyrusdb_create will abort if the key already exists. cyrusdb_delete takes a flag 'force' which just makes it return CYRUSDB_OK (0) rather than CYRUSDB_NOTFOUND if they key doesn't exist. Strangely, 'force' is after , making it the only cyrusdb API that does that, but hey - keeps you on your toes. behaves exactly the same for the write APIs. If not passed, then the database engine will behave as it if creates a writable transaction, does the operation, then commits all within the cyrusdb_* call. Gotchas! === * NULL is permitted in both keys and values, though 'flat' and 'quotalegacy' have 8-bit cleanliness issues. * zero-length keys are not supported * zero-length values are theoretically supported, but a little interesting. Certainly, pass "" rather than NULL as the value when writing or things will get weird. I'm pretty sure at least the *skip databases assert on these kinds of weirdness. * unlocked foreach: this is the land of the gotcha! They key and data pointers (const char *) passed to your foreach_cb are only valid *UNTIL YOU TOUCH THE DATABASE*. A common cause of rare and hard to diagnose bugs is writing something to the same database in the same process (*OR EVEN READING FROM IT AGAIN*). I cannot emphasise this enough. If you want to zero-copy access that data, you need to access it first, before touching that DB again. Otherwise the map in which the data was a pointer may have been replaced as the next read found a new file and mapped it in! also: if you're implementing a backend. Unlocked foreach must find future records created by the current callback. Consider a database containing 4 keys: A B C D if you are at key B and insert a key BB, then it must be iterated over. If you insert AA while at B, it must NOT be iterated over. * Opening the same database multiple times. In the bad old days, opening the same database multiple times in the same process led to locking bugs (fcntl is braindead). Each database engine is responsible for making sure this doesn't happen. Most engines keep a linked lists of open databases. If you try to open the same database again, they will just return the existing opened copy and bump a refcount. Beware. If a database is locked and you try to lock again - thinking you were opening it brand new, it will assertion fail and/or error. I think that covers about everything! Cyrusdb is used just about everywhere that sorted key-value databases give what's needed, including mailboxes.db, annotations.db (global and per mailbox databases), seen state (non-owner), subscriptions, cyrus.indexed.db for Xapian, and the rather massive (and increasingly inaccurately named) user.conversations. Future plans are to increase the usage of cyrusdb databases, possibly by building an indexing layer on top and using that instead of the sqldb interface used for sqlite databases by DAV code, and possibly also moving other custom file formats into a cyrusdb to allow easier stateless server builds on a distributed backend. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: sync_client behaviour improvement planning
Followup on this - I pushed a commit to master which included a few comments highlighting places in the code where changes would help. ellie is going to work on the changes, to be reviewed by Partha. None of these changes make any alterations to the replication protocol, all they change is: 1) which mailboxes are batched together when multiple mailboxes are being checked at once (SYNC GET MAILBOXES) 2) what gets done in interesting cases (e.g. when a folder named in the sync log is present on the replica but not on the master) Cheers, Bron. On Mon, 26 Feb 2018, at 18:28, Bron Gondwana wrote: > Hey - here's me posting something to the public list instead of > internal FastMail slack. We've been really bad at making our random > ruminations public, sorry.> > Tomorrow (Tues 27th) 2pm Melbourne time, I'm going to be meeting with > ellie and maybe Partha in the Melbourne office with a whiteboard and a > screen to flesh out some ideas for things we can do to fix some of the > issues that came up after a recent machine failure event at FastMail.> > In particular, sync sheer is a very real problem. The core issue is > something like this, either:> > a) > > sync_log MAILBOX A > sync_log MAILBOX A > sync_log MAILBOX B > > Underlying cause - something happened on mailbox A, then mailbox A was > renamed to B.> > Result - if there is a log split between those two lines, the > sync_client first sees just MAILBOX A, and so it just processes that > one mailbox. It sees:> > local: MAILBOX A IMAP_MAILBOX_NONEXISTENT > remote: MAILBOX A exists > > so it issues an UNMAILBOX A, then processes the second file. > > In the second file, it gets: > > local: MAILBOX A IMAP_MAILBOX_NONEXISTENT > local: MAILBOX B exists > > remote: MAILBOX A IMAP_MAILBOX_NONEXISTENT > remote: MAILBOX B IMAP_MAILBOX_NONEXISTENT > > So it creates B and copies all the messages again. This is correct, > but it's both inefficient and creates a gap where the replica doesn't > have the messages at all.> > > b) there are over 1000 mailboxes, and the log file got deduplicated >and then run in sets, and we had this:> > sync_log MAILBOX Z > sync_log MAILBOX B > > (for a rename of Z to B) > > local: MAILBOX B exists > remote: MAILBOX B IMAP_MAILBOX_NONEXISTENT > > We upload the entire mailbox. Later we see both mailbox Z and mailbox > B, and due to uniqueid duplication and the existence of mailbox B, we > forget about mailbox Z entirely - leaving a duplicate on the server. > Until a recent change, this led to real mess when running reconstruct > caused mailbox Z to get a new uniqueid, just on the end where the > reconstruct was run. Run it on both ends later, you could wind up > with different uniqueids, and replication bails on that because it's > confused!> > > The long term solution to all this is to replicate by uniqueid, and > replicate the name history entirely for each folder such that you can > calculate the delta and converge on the latest name for the folder in > split brain. But for now, maybe we can make this safer.> > > My initial thought is something like: if the folder exists at one end, > but not at the other (either way) do a full user sync.> > Also, if splitting > 1000 folders in sync_client, make sure we keep > all the folder for a user in a single batch, so don't split batches > inside a user.> > We may be able to use the tombstone records we've been storing for a > while to decide whether the lack of a folder is "it used to exist, but > it doesn't any more" or "it never existed here" - handy for figuring > out split brain recovery.> > Added complications: what about cross-user renames? What about > renaming users entirely?> > I know some of what Ken has been working on will also possibly > interact with this, so we're looking for some simple heuristic changes > that can make everyday situations safer while we wait for the real > solution.> > Bron. > > -- > Bron Gondwana, CEO, FastMail Pty Ltd > br...@fastmailteam.com > > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: sync_client behaviour improvement planning
On Mon, 26 Feb 2018, at 10:37, Sebastian Hagedorn wrote: > Hi, > > we're only getting started with replication, so I can't contribute any> > useful comments, but your post raised two questions for me: > > This only affects the "new", post-3.0 replication protocol, > right? We're> planning a migration from 2.4 to 3.0 using replication, and I'd > like to be> aware of all potential issues we might encounter. A migration will be totally fine regardless. I realise now the danger of posting about a really rare edge case to the dev list in a way that implies that it's anything anyone is likely to hit in regular usage. It surfaced something like 3 times over tens of thousands of mailboxes and a full disk event causing some weirdness around retries in sync. Cyrus handles full disk quite well these days (after an event in 2014 where we fixed a bunch of bugs!) but it does make some things odd. In particular, it can cause replication to fail for a mailbox due to disk full, but then later mailboxes in the same list get processed OK because there was space for that operation. In summary, don't let your disks get full! >> Hey - here's me posting something to the public list instead of >> internal>> FastMail slack. We've been really bad at making our random >> ruminations>> public, sorry. >> Tomorrow (Tues 27th) 2pm Melbourne time, I'm going to be meeting with>> >> ellie and maybe Partha in the Melbourne office with a >> whiteboard and a>> screen to flesh out some ideas for things we can do to fix >> some of the>> issues that came up after a recent machine failure event at >> FastMail.> > > >> b) there are over 1000 mailboxes, and the log file got >>deduplicated and>> then run in sets, and we had this: >> sync_log MAILBOX Z >> sync_log MAILBOX B >> >> (for a rename of Z to B) > > Where and when does deduplication happen? Deduplication in this case is sync_action_list_add(mailbox_list, args[1]) in sync_client.c:do_sync(). If the same item is logged twice in the same log file, then it's only added once to the list of mailboxes later passed to do_mailboxes. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
sync_client behaviour improvement planning
Hey - here's me posting something to the public list instead of internal FastMail slack. We've been really bad at making our random ruminations public, sorry. Tomorrow (Tues 27th) 2pm Melbourne time, I'm going to be meeting with ellie and maybe Partha in the Melbourne office with a whiteboard and a screen to flesh out some ideas for things we can do to fix some of the issues that came up after a recent machine failure event at FastMail. In particular, sync sheer is a very real problem. The core issue is something like this, either: a) sync_log MAILBOX A sync_log MAILBOX A sync_log MAILBOX B Underlying cause - something happened on mailbox A, then mailbox A was renamed to B. Result - if there is a log split between those two lines, the sync_client first sees just MAILBOX A, and so it just processes that one mailbox. It sees: local: MAILBOX A IMAP_MAILBOX_NONEXISTENT remote: MAILBOX A exists so it issues an UNMAILBOX A, then processes the second file. In the second file, it gets: local: MAILBOX A IMAP_MAILBOX_NONEXISTENT local: MAILBOX B exists remote: MAILBOX A IMAP_MAILBOX_NONEXISTENT remote: MAILBOX B IMAP_MAILBOX_NONEXISTENT So it creates B and copies all the messages again. This is correct, but it's both inefficient and creates a gap where the replica doesn't have the messages at all. b) there are over 1000 mailboxes, and the log file got deduplicated and then run in sets, and we had this: sync_log MAILBOX Z sync_log MAILBOX B (for a rename of Z to B) local: MAILBOX B exists remote: MAILBOX B IMAP_MAILBOX_NONEXISTENT We upload the entire mailbox. Later we see both mailbox Z and mailbox B, and due to uniqueid duplication and the existence of mailbox B, we forget about mailbox Z entirely - leaving a duplicate on the server. Until a recent change, this led to real mess when running reconstruct caused mailbox Z to get a new uniqueid, just on the end where the reconstruct was run. Run it on both ends later, you could wind up with different uniqueids, and replication bails on that because it's confused! The long term solution to all this is to replicate by uniqueid, and replicate the name history entirely for each folder such that you can calculate the delta and converge on the latest name for the folder in split brain. But for now, maybe we can make this safer. My initial thought is something like: if the folder exists at one end, but not at the other (either way) do a full user sync. Also, if splitting > 1000 folders in sync_client, make sure we keep all the folder for a user in a single batch, so don't split batches inside a user. We may be able to use the tombstone records we've been storing for a while to decide whether the lack of a folder is "it used to exist, but it doesn't any more" or "it never existed here" - handy for figuring out split brain recovery. Added complications: what about cross-user renames? What about renaming users entirely? I know some of what Ken has been working on will also possibly interact with this, so we're looking for some simple heuristic changes that can make everyday situations safer while we wait for the real solution. Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Cyrus IMAP 3.1.4 development version available
Those following along will be pleased to note that this new version contains very recent JMAP changes, including the new method naming, backreference support, and top level request/response objects. Bron. On Wed, 14 Feb 2018, at 14:50, Partha Susarla wrote: > The Cyrus team is pleased to announce the immediate availability of a > new version of Cyrus IMAP: 3.1.4> > This is a snapshot of the master branch, and should be considered for > testing purposes and bleeding-edge features only. It is available as a > git tag only, which can be found here:> > https://github.com/cyrusimap/cyrus-imapd/releases/tag/cyrus-imapd-3.1.4> > Join us on Github at https://github.com/cyrusimap/cyrus-imapd to > report issues, join in the deliberations of new features for the next > Cyrus IMAP release, and to contribute to the documentation.> > On behalf of the Cyrus team, > > Kind regards, > ..Partha > > -- > Partha Susarla > par...@fastmailteam.com -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 22 Jan 2018
ellie: * will be in office tomorrow - setting up build environment on new laptop* need to discuss JMAP in 3.0 Partha: * most of last week was spent on zeroskip - have iterators for both in-memory and multiple types of on- disk format - locking in place - code is on cyrusimap github repo * last couple of weeks didn't have Bron around, but back now * this is a short week (Australia Day on Friday) and off for 3 days next week Robert: * jscalendar - RFC draft is v5 now. - confident that it's ready for acceptance - off the list discussion about tasks details - waiting for more feedback on tasks: when tasks start - at the moment we can track when a task is completed, but not when work started - next step - official call on Calsify list * Ken - giving it a slow careful read - will be done by next week (CalConnect)* Robert will update slides for Ken to use for presentation * Ken and Mike will discuss parameters on properties for tasks * JMAP in 3.0 - why remove it? - best thing to do - not only about format of JMAP request envelopes, but also data inside it. - will make people unhappy if they build applications on top of an API which has been outdated for months. - Bron approves * Have two feature branches in limbo. * smtp API rewrite - was asking about SMTP authentication on TCP-based backends. - suggest just pushing to master as is - for SMTP authentication - just put in config file - can include a file with more sensitive data* index attachments - people at FastMail have been on leave, hence lack of movement * JMAP code refactor - currently concentrating efforts on - started last week - generally looks fine - would like to show a prototype next week for at least one type of JMAP method call to decide how to go forward - would like to generalise validation code - thinking about implementing poor-man's JSON-schema validation Bron: * just got back from being away for 2 weeks. Testing append cost on large mailboxes: * test appending on mailboxes with different scale of messages (10, 100, 1000, etc...) and see how performance scales.* Robert will look at. Ken: * finally got laptop where it needs to be after replacing hard drive (Windows partition booting)* had to reinstall Linux - back to where Cyrus build works. * wasted more of the week than wanted to * working on some FastMail tickets as well * Bron - bug Apple and Google people about bugs in their products at CalConnect!* This week - preparing slides on sharing and proxying/delegation to get a discussion going - make sure VPOLL works so Mike and Ken can do a demonstration - next week? Change time of call? - make Australian early morning? - tibbs, nic maybe. NEXT WEEK: Tuesday 8am Melbourne == Monday 1pm for Ken == Monday 10pm for Robert. (UTC 2100) Nicola: * wanted to know if JMAP 3.0 was out, tibbs wanted to know. Hadn't clearly communicated to list. - Robert will mail list to explain what's happening. - or ellie, as release manager. * want to make sure west/central US aren't left out all the time, so maybe do alternating meeting times. - don't have regular contributors outside FM right now - might be more if we do more outreach - invite people to the meetings - Ken will invite people - figure out if we want to alternate times. - we also know who we've been dealing with, so individual invites can be made too - ellie from bugs/list. - be nice to get 3.0 / 3.2 into next major releases. * coverage on IRC channel - Ken needs a better client (popups, etc). Ditto Robert. - irccloud is good for phone - also has bouncer - why not use slack for Cyrus? Hard for drive-by people, non-open- source, etc. - on the flip side, lots of idle people on IRC who might not really be around. - people who want to administer Cyrus - maybe happier staying with IRC. Calconnect: * Robert - be good to talk to Peter from Ribose * Nicola - would like to write up ideas about what would be good to see in calendars* Per-user vs shared calendars, how delegation is supposed to work SASL release: * the guy who's been testing GSSAPI issue will try to get back to it this week - just moving slowly.* Ken isn't GSSAPI expert and doesn't have infrastructure any more -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Dicerolls
This is done from a script called pick5.pl: https://github.com/cyrusimap/BugzillaImport/blob/master/pick5.pl The script is run with a single argument: % perl pick5.pl doit And it outputs the dump below, as well as doing the updates to the tickets. Please note, you'll need your own token! Bron. --- Ken (ksmurchison) = https://github.com/cyrusimap/cyrus-imapd/issues/509 give ipurge a test mode = https://github.com/cyrusimap/cyrus-imapd/issues/878 Fails building on FreeBSD kernel with GNU userspace (PS_STRING defined but not usable)= https://github.com/cyrusimap/cyrus-imapd/issues/1708 Replication breaks if a mailbox has expire annotations on folders= https://github.com/cyrusimap/cyrus-imapd/issues/1728 EXAMINE reset \recent flag in statuscache = https://github.com/cyrusimap/cyrus-imapd/issues/1848 cyrus-2.5.10: segmentation faults caused by sieve/sievec on Gentoo Partha (ajaysusarla) = https://github.com/cyrusimap/cyrus-imapd/pull/71 Preserve the existing unique ID of a mailbox renamed / transferred= https://github.com/cyrusimap/cyrus-imapd/issues/352 remove need for imapd_userisproxyadmin = https://github.com/cyrusimap/cyrus-imapd/issues/1467 ptloader - ptclient ldap failure = https://github.com/cyrusimap/cyrus-imapd/issues/1618 interruption in communication between master and frontend = https://github.com/cyrusimap/cyrus-imapd/issues/1808 3.0.0-rc1 / Solaris = https://github.com/cyrusimap/cyrus-imapd/issues/1981 lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags Robert (rsto) = https://github.com/cyrusimap/cyrus-imapd/issues/40 [2.5.9] SIGSEGV on murder frontend if backend disappears = https://github.com/cyrusimap/cyrus-imapd/issues/457 managesieve should handle AUTHENTICATE referrals = https://github.com/cyrusimap/cyrus-imapd/issues/1561 virtdomains: off should return 'invalid mailbox name' when '@' included in mailbox name= https://github.com/cyrusimap/cyrus-imapd/issues/1637 sendmail output shall be isolated from current protocol + https://github.com/cyrusimap/cyrus-imapd/issues/1033 Improve unix socket permissions ellie (elliefm) = https://github.com/cyrusimap/cyrus-imapd/issues/50 Integrate doc/internal docs with sphinx repo = https://github.com/cyrusimap/cyrus-imapd/pull/73 Several changes.master = https://github.com/cyrusimap/cyrus-imapd/issues/1765 Move SNMP out from master into a separate daemon = https://github.com/cyrusimap/cyrus-imapd/issues/1767 2.5 imap/: close some resource leaks + https://github.com/cyrusimap/cyrus-imapd/issues/743 glob.c: glob_test() function returns wrong matches in some cases -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Accidental push to master branch
I accidentally pushed a few FastMail internal patches to master and then turned off branch protection for a few moments and force pushed an earlier commit. Sorry to anyone who happened to pull in those few seconds! Branch protection is back on again now. There's nothing secret in there, it's just a pain to push a bunch of reverts. Regards, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
The issue with calendar alarms
Calendar alarms and replication are a real problem for the following reason: 1) calalarmd needs to only run on the master, not the replica - otherwise the alarm will fire on both, and the alarmd on the replica will write annotations, causing split brain issues. 2) the caldav_alarms.sqlite3 database on the replica needs to have all the nextalarm values in sync with the master, so if there's a failover event, the calalarmd on the ex-replica knows which records are interesting without having to parse every single mailbox! 3) replication and annotations is a disaster - the record gets written first, and the annotations later. So we solve this as follows: a) if record.silent is set, we don't write the caldav_alarms.sqlite3 event line for the record immediately. b) AFTER we have applied both the record and any annotations for the record in sync_support.c, we call caldav_alarm_touch_record(), which can then read the annotation to see when the nextalarm should be. The problem is this: if there's already a nextalarm value, and we change a record to have an earlier alarm for whatever reason, the alarm annotation just gets copied by dav_store_resource to the new record, and hence there's already a nextalarm in the future. This OVERWRITES the earlier alarm value created by caldav_alarm_new_record(). The fix is this: instead of blindly overwriting, caldav_alarm_touch_record needs to have two modes: sync mode and non- sync mode. In sync mode, it's a replica - so it just does what it does now - blindly writes the value from the annotation. In non-sync mode, it instead always reads the record (plus any per-user overrides) and finds the time that it next needs to be checked. Easy-peasy :) I might even just call it two different things to make it extra clear what's going on! Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Xapian partition definition when using archiving?
Totally! The name "archive" is overused. It could be called something else easily enough. Bron. On Thu, 21 Dec 2017, at 01:05, Nic Bernstein wrote: > Bron, > Thanks for the swift reply. So if I understand this correctly, the > "archivesearchpartition-default" is named such because it's for the > archive location of Xapian search data, not because it's Xapian > search data from an Archive partition. Is that correct? In other > words, this is just another circumstance where seemingly obvious > partition names (like default-default) get us into documentation > trouble. Right?> > Thanks again, > -nic > > > On 12/20/2017 05:39 AM, Bron Gondwana wrote: >> Hi Nic, >> >> The Xapian partitions are entirely separate from archive partitions. >> The indexing code will find the file in the correct location if it >> needs to read it.>> >> Here's an example from one of our servers: >> >> defaultpartition: default >> defaultsearchtier: temp >> partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool >> tempsearchpartition-default: /tmpfs-search/sloti21d2t40 >> metasearchpartition-default: >> /mnt/ssd21d2/sloti21d2t40/store254/search>> datasearchpartition-default: >> /mnt/i21d2search/sloti21d2t40/store254/search>> >> archivesearchpartition-default: >> /mnt/i21d2search/sloti21d2t40/store254/search- >> archive>> archivepartition-default: >> /mnt/i21d2t40/sloti21d2t40/store254/spool- >> archive>> >> (clearly auto-generated!) >> >> Bron. >> >> >> On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: >>> Bron, et al., >>> We're about to set up a whole new bunch of partitions in support of >>> Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be >>> introducing archive functionality, too. How do archive partitions >>> and Xapian partitions interact?>>> >>> For example, the server currently has the following in imapd.conf: >>> >>>> defaultpartition: default partition-default: >>>> /var/mailstores/default metapartition-default: >>>> /var/imapmeta/default partition-1: /var/mailstores/1 partition-2: >>>> /var/mailstores/2 partition-3: /var/mailstores/3 partition-4: >>>> /var/mailstores/4 ... partition-29: /var/mailstores/29 partition- >>>> 30: /var/mailstores/30 partition-100: /var/mailstores/100 partition- >>>> temp: /var/mailstores/temp ... # non-default metapartitions metapartition- >>>> 1: /var/imapmeta/1 metapartition-2: /var/imapmeta/2 metapartition- >>>> 3: /var/imapmeta/3 metapartition-4: /var/imapmeta/4 ... metapartition- >>>> 29: /var/imapmeta/29 metapartition-30: /var/imapmeta/30 metapartition- >>>> 100: /var/imapmeta/100 metapartition-temp: /var/imapmeta/temp >>>>>>> Going by the documentation, which I wrote with help from you good >>> folk at Fastmail, the Archive partition scheme might look something >>> like this:>>> >>> https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html>>> >>>> archivepartition-default: /var/mailarchives/default archivepartition- >>>> 1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 >>>> archivepartition- >>>> 3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... >>>> archivepartition-29: /var/mailarchives/29 archivepartition-30: >>>> /var/mailarchives/30 archivepartition-100: /var/mailarchives/100 >>>> archivepartition-temp: /var/mailarchives/temp >>>>>>> And the Xapian partition structure to mate with this would look >>> something like this (again, from the docs):>>> >>> https://www.cyrusimap.org/imap/developer/install-xapian.html >>> >>>> defaultpartition: default partition-default: >>>> /var/mailstores/default search_engine: xapian search_index_headers: >>>> no search_batchsize: 8192 defaultsearchtier: t1 1searchtier: t1 >>>> 2searchtier: t1 3searchtier: t1 4searchtier: t1 ... 29searchtier: >>>> t1 30searchtier: t1 100searchtier: t1 tempsearchtier: t1 ... >>>> t1searchpartition- >>>> default: /var/search/default t1searchpartition-1: /var/search/1 >>>> t1searchpartition- >>>> 2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition- >>>> 4: /var/search/4 ... t1searchpartition-29: /var/search/29 >>>> t1searchpartition- >>>> 30: /var/search/30 t1searchpartition-100: /var/search/100 >>&g
Re: Xapian partition definition when using archiving?
Hi Nic, The Xapian partitions are entirely separate from archive partitions. The indexing code will find the file in the correct location if it needs to read it. Here's an example from one of our servers: defaultpartition: default defaultsearchtier: temp partition-default: /mnt/ssd21d2/sloti21d2t40/store254/spool tempsearchpartition-default: /tmpfs-search/sloti21d2t40 metasearchpartition-default: /mnt/ssd21d2/sloti21d2t40/store254/search datasearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/searcharchivesearchpartition-default: /mnt/i21d2search/sloti21d2t40/store254/search-archivearchivepartition-default: /mnt/i21d2t40/sloti21d2t40/store254/spool-archive (clearly auto-generated!) Bron. On Wed, 20 Dec 2017, at 07:32, Nic Bernstein wrote: > Bron, et al., > We're about to set up a whole new bunch of partitions in support of > Xapian indexing, for a 2.5.10-to-3.0.4 upgrade, and then will be > introducing archive functionality, too. How do archive partitions > and Xapian partitions interact?> > For example, the server currently has the following in imapd.conf: > >> defaultpartition: default partition-default: /var/mailstores/default >> metapartition-default: /var/imapmeta/default partition-1: >> /var/mailstores/1 partition-2: /var/mailstores/2 partition-3: >> /var/mailstores/3 partition-4: /var/mailstores/4 ... partition-29: >> /var/mailstores/29 partition-30: /var/mailstores/30 partition-100: >> /var/mailstores/100 partition-temp: /var/mailstores/temp ... # non- >> default metapartitions metapartition-1: /var/imapmeta/1 metapartition- >> 2: /var/imapmeta/2 metapartition-3: /var/imapmeta/3 metapartition-4: >> /var/imapmeta/4 ... metapartition-29: /var/imapmeta/29 metapartition- >> 30: /var/imapmeta/30 metapartition-100: /var/imapmeta/100 metapartition- >> temp: /var/imapmeta/temp >>> Going by the documentation, which I wrote with help from you good > folk at Fastmail, the Archive partition scheme might look something > like this:> > https://www.cyrusimap.org/imap/reference/admin/locations/archive-partitions.html> >> archivepartition-default: /var/mailarchives/default archivepartition- >> 1: /var/mailarchives/1 archivepartition-2: /var/mailarchives/2 >> archivepartition- >> 3: /var/mailarchives/3 archiveartition-4: /var/mailarchives/4 ... >> archivepartition- >> 29: /var/mailarchives/29 archivepartition-30: /var/mailarchives/30 >> archivepartition-100: /var/mailarchives/100 archivepartition-temp: >> /var/mailarchives/temp >>> And the Xapian partition structure to mate with this would look > something like this (again, from the docs):> > https://www.cyrusimap.org/imap/developer/install-xapian.html > >> defaultpartition: default partition-default: /var/mailstores/default >> search_engine: xapian search_index_headers: no search_batchsize: 8192 >> defaultsearchtier: t1 1searchtier: t1 2searchtier: t1 3searchtier: t1 >> 4searchtier: t1 ... 29searchtier: t1 30searchtier: t1 100searchtier: >> t1 tempsearchtier: t1 ... t1searchpartition-default: >> /var/search/default t1searchpartition-1: /var/search/1 t1searchpartition- >> 2: /var/search/2 t1searchpartition-3: /var/search/3 t1searchpartition- >> 4: /var/search/4 ... t1searchpartition-29: /var/search/29 t1searchpartition- >> 30: /var/search/30 t1searchpartition-100: /var/search/100 t1searchpartition- >> temp: /var/search/temp> First question, since there's no examples to work >> from; Is this Xapian > layout correct?> > Do I need to define & create Xapian partitions for the metadata > partitions, as is indirectly implied in Bron's original email on > this topic:> > https://lists.tartarus.org/pipermail/xapian-discuss/2014-October/009112.html> > Also, how do these Xapian and Archive, interact? Do I need to add a > separate Xapian partition for each Archive partition, or will the > Archive partition be treated like a child of the non-Archive > partition? (again, implied but not directly addressed in that > email).> > Any guidance gladly accepted, and whatever I learn will be repackaged > into more complete documentation on same.> > Cheers, > -nic > > -- Nic Bernstein n...@onlight.com > Onlight, Inc. www.onlight.com 6525 W > Bluemound Road, Suite 24 v. 414.272.4477 Milwaukee, > Wisconsin 53213-4073 >> Email had 1 attachment: > * nic.vcf > 1k (text/x-vcard) -- Bron Gondwana br...@fastmail.fm
Notes 27/11
Present: Bron, Ken, Nicola, ellie Bron: * Lots of Xapian issues last week * including seqset bug * incorrect UID issue for per-user-data only updates with calendar events.* index_snippets is keeping the mailbox locked while reading from disk and generating snippets, which can be slow and block out other actions. We should find a solution to this. Ken: * working on docs for internals changes - sync, rename safety etc * SASL - looks like Alexey is correct, and people were relying on buggy behaviour* found a logic error with Bearer Token authentication. * short week this week Nicola: * went to write-the-docs on Friday, single day conference * didn't fix any doc bugs * had people with mail server knowledge - used them as guinea-pigs to look at our existing docs - have notes which will write up and send to list - if you don't already know what Cyrus is and how it works, it's a wall of acronyms - FastMail webpage was the kind of thing he was expecting - but at least Cyrus docs look better than Dovecot!* FastMail caldav sync doesn't handle alarms particularly nicely. * Also, FastMail caldav sync should be copy through calendar-user- address from the source and passing that through for clients to use for scheduling. ellie: * just been looking into the jmap getmessagelist stuff -- that email thread today is v useful sasl is just a point release, so should be safe to upgrade independently Partha is back working on zeroskip after being involved in the xapian fixes most of last week. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Nov 20
Present: ellie, Bron, Partha, Ken, Nicola Robert is away for a few weeks. Bron: * standards work * xapian locking fiasco * cyrus.index v14 soon (SAVEDATE, ANNOTUSED) * multiple failed renames in the past week - plan to remove asserts from conversations db* there's a bunch of JMAP changes on master that are an intermediate state, no point moving Topicbox to that when a whole lot more changes are landing soon.* spoke to Alexey about Cyrus SASL changes - he will have a call with Ken. Ken: * spent most of last week on drafts - documenting binary timezone format * fixed bugs found by BusyCal * been writing up thoughts on user change * crash during Calendar auto-creation on new user, looks like a race when two sessions both create the calendar at the same time. - could be a lock inversion between create code and use code. ellie: * merged some patches from the list/github, and working on some additional prometheus metrics* planning to keep working on metrics Nicola: * not sure if done any cyrus work in the past week! * this Friday - write the docs conference. - will do a doc sprint - doc areas good for first timers? Partha: * last week mostly calendaring - customer tickets * almost done with coding - translating mailing list post to C! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Cyrus notes Nov 6th
Present: Bron, Robert, Ken, Nicola Robert: * finished JMAP-mail updates (unless missed something) * cyrus.works issues - timezone problems - should do a cyrus-timezones package using Ken's vzic * Setting up Ken's vzic on Jenkins - what do we need to do? - not much - just pick a place to put the vtimezone files (e.g. in $confdir/timezones) - option to autoconfig? - ship a copy of Olson DB with Cyrus and build that, or have configure point to - Robert will set up: ship a copy of Olson DB in Cyrus, compile with vzic during build process, document how to update. - will benefit Cyrus.works and FM environment * Wanted to update JS Calendar draft at IETF - but it's locked. Can be updated while in Singapore.* Working on updating SMTP client API - will have something by end of this week.* What next? - indexing attachments - indexing in xapian with mailbox uniqueid - xapian caching changes will go live on FastMail this week. Ken: * have been kind of obsessed with per-user-data stuff. - committed code to do most annotations in a transaction - looking at extending API so that annotations get set with the append - need to make sure private annotations get set properly - msgrecord API? Rather than using storeall, maybe store individually. - Bron: want to look at sync protocol when writing this. - also: some history about why there's an inversion around per-message- annotations and index records. Bron: * prepare for URL changes - blog post and plan * email to customers in-flight, and list of customers to email (non- tested or failing clients)* away in Singapore 10th-19th for IETF. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting notes Oct 23
to reload its configuration in a more rigorous way -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Cyrus Meeting Notes [9th October]
We've been keeping it at 10pm Melbourne, which pushed it back to 7am USA East, and that will go back to 6am when they switch in a few weeks! Bron. On Mon, 9 Oct 2017, at 08:14, ellie timoney wrote: > What's the conf call time for next week, now that Melbourne is in DST?> > > On Mon, Oct 9, 2017, at 10:20 PM, Partha Susarla wrote: > > Present: Bron, Ken, RobS, Nicola, Partha > > > > Bron: > > * Cyrus Board Meeting [With Ken] - decision yet to be taken on > > foundation > > association. Next meeting on Friday 13th. > > * JMAP Proxy now supports all of the new JMAP spec. Thread keyword > > search needs to be implemented. > > * getMessageListUpdates is implemented in Cyrus. > > > > Ken: > > * Cyrus Board Meeting [With Bron] - decision yet to be taken on > > foundation > > association. > > * Working through CalDav tests. > > * CardDav tests cleaned-up. > > > > RobS: > > * Few fixes on Cyrus JMAP. > > * Reviewed getMessageListUpdates patch from Bron. > > * Working on Xapian search improvements: > >- Migration of mailboxnames to unique ids > >- Caching search tiers in faster tiers > >- To discuss these changes with Bron > > * Revisit JMAP spec to make sure it is updated. > > > > Nicola: > > * Not much Cyrus work, was on vacation. > > > > Partha: > > * No Cyrus work done last week. > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting notes Oct 2
Present: Bron, Ken, Robert, Nicola, Partha (and Ricard Signes for a moment at the end) CalConnect was the main thing this week! *Robert:*** * this CalConnect was really good - bigger group than Seattle, and more momentum* did an internal FM report - will also do reports to the cyrus- devel list.* contacts - work with Ribose on data model together, to extend/replace VCARD - design based on GraphQL - map to JMAP data model - will take quite a while (3-4 CalConnects) to be stable - may a year or so - workable and protocol independent data format * Xapian caching work - can cache lower order indexes in the fast top tier cyrus.indexed.db - need to change compact code to not copy cache records * Xapian snippet fix - will submit upstream - still needs to be used at FastMail, so will wait for that first * Will be off for vacation in 7 weeks - be good to work our what's still to do with JMAP first and finish. - define JMAP roadmap. *Ken:*** * VPOLL: Mike and Ken tried to do a demo - ran into issues with Access Control and Organizer tracking - 1&1 people are really interested in getting something going * Marten and Ken did a demo with subscriptions. - FastMail sharing works differently, so we need to convert existing clients to new sharing style - plan now that CaldavTester is working is to make sure Cyrus Daboo's tests work as expected - in a couple of weeks, formulate plan for changing over * Per-user calendar alarms - ready to go - need to check that server-side stuff based on who is acting is ready to go - will roll it out while Ken in Melbourne in mid-late October * Next: - work with CalDAVTester - SASL release - will schedule a call with Bron/Rik to talk about JMAP Calendar planning *Bron:*** * Cyrus Board meeting this week to discuss licensing * cyrus builds: will probably build timezone data directly from upstream rather than from cyruslibs.* 3.1.2 build changed JMAP behaviour - interesting for upgrades. * cyr_expire wasn't reading conversations config, and hence not expiring conversations - changed to assert if we try to read a config value without having loaded config! - refactor improved code *Partha:*** * 3.1.2 release made last week * FastMail specific is just a few patches now * cyruslibs - not building timezone data yet * Started working on zeroskip - as a separate project (standalone library) - chatted with Neil - and grabbed spec file - next couple of weeks - hope to have most ideas fleshed out - also moving twoskip into standalone project - tests and comparison between the two versions - benchmarking tests - rsto: is this based on cyrus-imapd cunit tests? probably have separate unit tests in the library as well* apart from this, nothing Cyrus specific. *Nicola:*** * have been herding all Dilyan's bugs! * it's school holidays, so not doing that much * at end of November - will be representing FastMail and Cyrus at Write the Docs conference - talking about Sphinx work - doing a doc-sprint, so collecting bugs that are good for first- timers to work with.* Prometheus is being used at FastMail now, will be using Cyrus on. Next meeting: same time next week! -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Dicerolls
Ken (ksmurchison) = https://github.com/cyrusimap/cyrus-imapd/issues/509 give ipurge a test mode = https://github.com/cyrusimap/cyrus-imapd/issues/558 per-service shutdown method? = https://github.com/cyrusimap/cyrus-imapd/issues/1708 Replication breaks if a mailbox has expire annotations on folders= https://github.com/cyrusimap/cyrus-imapd/issues/1728 EXAMINE reset \recent flag in statuscache = https://github.com/cyrusimap/cyrus-imapd/issues/1848 cyrus-2.5.10: segmentation faults caused by sieve/sievec on Gentoo Partha (ajaysusarla) = https://github.com/cyrusimap/cyrus-imapd/pull/71 Preserve the existing unique ID of a mailbox renamed / transferred= https://github.com/cyrusimap/cyrus-imapd/issues/352 remove need for imapd_userisproxyadmin = https://github.com/cyrusimap/cyrus-imapd/issues/1467 ptloader - ptclient ldap failure = https://github.com/cyrusimap/cyrus-imapd/issues/1618 interruption in communication between master and frontend + https://github.com/cyrusimap/cyrus-imapd/issues/1808 3.0.0-rc1 / Solaris Robert (rsto) = https://github.com/cyrusimap/cyrus-imapd/issues/40 [2.5.9] SIGSEGV on murder frontend if backend disappears = https://github.com/cyrusimap/cyrus-imapd/issues/457 managesieve should handle AUTHENTICATE referrals = https://github.com/cyrusimap/cyrus-imapd/issues/1152 Recursive reconstruct and @ in mailbox name = https://github.com/cyrusimap/cyrus-imapd/issues/1981 lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags+ https://github.com/cyrusimap/cyrus-imapd/issues/1561 virtdomains: off should return 'invalid mailbox name' when '@' included in mailbox name -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes 18 Sep
Present: Jeroen, Ken, Nicola, Partha, Robert, Bron Ken: * past week mostly SASL stuff * Added support for Channel Binding to Cyrus * Hopefully fixed client-side ordering of mechanisms for the last time * Autoconfig junk * Planning to 5th release candidate of SASL midweek Robert: * been working on JSCalendars RFC draft. Going to send it out today.* pushed JMAP message submission on master branch - supports real SMTP backend - debugging backend (to a file for testing - is default) - sendmail executable (considering removing, but could be useful for some installations)* started working on Xapian indexdb improvements. Will have a simplistic variant by today.* Contacts RFC: haven't worked on a document yet. Could do before CalConnect, but might be a bit much in a rush. - might be best to talk to Ron and Ribose people at CalConnect. Partha: * didn't do much Cyrus work in the last week * working on fixing build issues with latest Debian (autotools/makemaker and Perl modules)* Rik (FastMail perl dev) suggested a few things, but still not working.* Did a couple of dicerolls from last week. Bron: * been pairing with Partha on FM backup system * remove subscription on source of rename Jeroen: * preparing to do some performance testing on Power 8 - not sure exactly when they'll be testing it yet - certainly won't be happening for the next couple of weeks - maybe October some time.* interesting to compare x86 against Power 8. * will be testing a whole environment. * would be interesting to create some benchmark tests. - parallelisation could be a problem with a testing suite. - might need multiple client systems so you can truly load a system with 20-30k sessions in parallel.* User was able to rename over own INBOX the other day which broke things! - will try to create a test to recreate it, then we can test a solution Nicola: * Small misc doc fixes from me, just going through github items. And done in 15 minutes! Record :) Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
dicerolls
Ken (ksmurchison) = https://github.com/cyrusimap/cyrus-imapd/issues/1728 EXAMINE reset \recent flag in statuscache = https://github.com/cyrusimap/cyrus-imapd/issues/1848 cyrus-2.5.10: segmentation faults caused by sieve/sievec on Gentoo+ https://github.com/cyrusimap/cyrus-imapd/issues/558 per-service shutdown method? + https://github.com/cyrusimap/cyrus-imapd/issues/509 give ipurge a test mode + https://github.com/cyrusimap/cyrus-imapd/issues/1708 Replication breaks if a mailbox has expire annotations on folders Partha (ajaysusarla) = https://github.com/cyrusimap/cyrus-imapd/pull/71 Preserve the existing unique ID of a mailbox renamed / transferred= https://github.com/cyrusimap/cyrus-imapd/issues/352 remove need for imapd_userisproxyadmin = https://github.com/cyrusimap/cyrus-imapd/issues/1467 ptloader - ptclient ldap failure = https://github.com/cyrusimap/cyrus-imapd/issues/1618 interruption in communication between master and frontend = https://github.com/cyrusimap/cyrus-imapd/issues/1710 SSL private key cannot be read if chmod is 640 and path contains a symlink Robert (rsto) = https://github.com/cyrusimap/cyrus-imapd/issues/457 managesieve should handle AUTHENTICATE referrals = https://github.com/cyrusimap/cyrus-imapd/issues/1152 Recursive reconstruct and @ in mailbox name = https://github.com/cyrusimap/cyrus-imapd/issues/1981 lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags+ https://github.com/cyrusimap/cyrus-imapd/issues/561 /etc/krb.equiv bad choice of filename + https://github.com/cyrusimap/cyrus-imapd/issues/40 [2.5.9] SIGSEGV on murder frontend if backend disappears -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Notes Sep 4th
Present: Bron, Ken, Robert, Nicola Bron: * wants changes to reduce IO for Xapian reads when indexing (requires using conversations DB and cyrus.indexed.db more cleverly)* Finally replied to Benn about license choices. Haven't heard back yet. Robert: * have been working on JMAP Message Submission - have a simplistic implementation ready.* put this on hold for now, design in Perl for now and then backport it later.* working on improving HTML body processing - backporting the body processing from Perl back to C.* TC-API - still updating draft, not yet sent out. Aiming for 1 week before calconnect.* Planning to work on this plus contact format in Cologne. * Will complete current JMAP tasks and then look at Xapian Ken: * spent most of last week with OpenSSL 1.1 and SASL. * fixed plugins to use API. * downloaded augmented API. * Added SCRAM-SHA256. * Jan Parcel from Oracle issues in git. They build outside of the source tree.* Nicola: about docs - there's a list in a rst, and will need a man page for the plugin.* Promised Alexey an update to Managed Attachments draft, working with HTTP folks.* Plan for CalConnect: nothing on radar at this point - waiting to see what Mike does with VPOLL. Might try to get demo going.* would like to run through sharing stuff if there's a client there. * working on Subscription upgrade (enhanced subscriptions) - sync-report with simple GETs. Wants to add time-range query. - nice thing is, if you treat the etag as a sync-token, you get updates. Nicola: * spent the week on a beach. Partha is unwell, so didn't show up - but has released 3.0.4 with a fix to an off-by-one error in mboxlist_findall and some other little fixes. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Minutes Aug 21
Present: Bron, Ken, Nicola, Robert, ellie, Partha Ken: * believes he has all the non-scheduling sharing stuff done * just need to do some testing with real clients to make sure what they need to see advertised does get advertised so they know they can write alarms for read-only stuff.* there are tasks in LP - do they have associated issues? Nicola - imagine there are placeholders for tracking time.* Nicola: if something has its own email address, i.e. meeting room - there's no oversight if somebody declines.* do I need to look at calalarmd? Probably yes. How will we test all this? Run up a fastmail instance on a testbed machine. * can cassandane run with valgrind? Yes - but not default. * SRS sieve patch from one of the dicerolls, will make sure it applies. Config options. Robert: * mainly working on JMAP spec updates last week. They are ready to land on master.* Need to update Cassandane repository because tests are breaking. * Will split 3.0 tests out against 3.1 tests. * Also - the Cassandane tests - there are 3 sources of timezone data. Only third option (vzic) is the good one.* libical database doesn't include GMT timezones. * 4th option - bundle our timezone database with source distro. * hard on sysadmins - because you have to keep multiple timezone databases updated. Already a mess, so??? how do we untangle?* Ken: could create a tool which monitors IANA and pulls down updates and rebuilds? Maybe, but some people might not want to run it.* Two decisions: what to do at FastMail vs what to do for upstream * from FastMail - it's not a big deal to rebuild cyrus 2-3 times per year.* Deliberately omitting them in libical right now. Probably because strictly not an Olson name.* For now - look at cyruslibs libical and add support for Etc timezones. * Not totally sure what to work on next. Will update tomorrow. * One of the issues is mailbox selects for unknown files pile up locks in the lock directory. Partha: * mentioning about vagrant earlier. Couldn't get it working, will talk to robn/marc later this week.* Will get it running on a cloud instance somewhere. * Got it working on desktop at work - it's pretty cool - has a full stack running.* Last week - a lot of time on something trivial - #2086 * Trying to debug calendaring issue on Android - customer issue - pairing with Matt (FM support) to understand more of calendars to reproduce.* Have more information than had last week, so hopefully progress. * Have a few more bugs on github working on - will be pairing for Rob later this week for IOERROR bug.* Spent time with Chris Davies fixing cyrus.works issues. * Multi-level docker instance directly exposed to the internet, so lots of workarounds to restrict access. Nicola: * nothing exciting this week - small doc fixes * email about changing release structure - does it make sense to everybody? Yes, everybody fine with it.* regarding CI - have working setup with cyrus.works, but it has issues with manual work needed for libs update. - Relies heavily on Chris right now. - many of the scripts are known only to Chris. - idea is to see if we can get Jenkins working better with existing system, and each have own branches working against Jenkins. - some tickets for features - both Chris and Partha will work on them. - wants to know if can completely get rid of Jenkins and use Travis completely? - One of the goals is to have builds for multiple plaforms, multiple distros, etc. Bron: * nothing exciting this week. * dicerolls sent ellie: * going away for a while - last conf call for a bit. * should remove from dicerolls for a bit! * sync protocol - closest thing to documentation * loving that travis builds and tests all the pull requests. * Partha: still need to get cassandane into Travis. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
dicerolls
Ken (ksmurchison) = https://github.com/cyrusimap/cyrus-imapd/issues/950 Unified murder - if one node is down during mailbox deletion, deleted remote mailboxes end up local mailboxes of that node with an invalid partition upon startup= https://github.com/cyrusimap/cyrus-imapd/issues/1156 SRS implementation upon sieve script redirect action = https://github.com/cyrusimap/cyrus-imapd/issues/1650 Support the i;ip-mask comparator for sieve = https://github.com/cyrusimap/cyrus-imapd/issues/1848 cyrus-2.5.10: segmentation faults caused by sieve/sievec on Gentoo+ https://github.com/cyrusimap/cyrus-imapd/issues/1728 EXAMINE reset \recent flag in statuscache Partha (ajaysusarla) = https://github.com/cyrusimap/cyrus-imapd/issues/1467 ptloader - ptclient ldap failure = https://github.com/cyrusimap/cyrus-imapd/issues/1618 interruption in communication between master and frontend = https://github.com/cyrusimap/cyrus-imapd/issues/1710 SSL private key cannot be read if chmod is 640 and path contains a symlink+ https://github.com/cyrusimap/cyrus-imapd/pull/71 Preserve the existing unique ID of a mailbox renamed / transferred+ https://github.com/cyrusimap/cyrus-imapd/issues/352 remove need for imapd_userisproxyadmin Robert (rsto) = https://github.com/cyrusimap/cyrus-imapd/issues/457 managesieve should handle AUTHENTICATE referrals = https://github.com/cyrusimap/cyrus-imapd/issues/1134 cyr_expire silently quits if it hits a non-existant account = https://github.com/cyrusimap/cyrus-imapd/issues/1152 Recursive reconstruct and @ in mailbox name = https://github.com/cyrusimap/cyrus-imapd/issues/1692 SELECT writes .lock file for unknown/inexistent mailboxes = https://github.com/cyrusimap/cyrus-imapd/issues/1981 lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags ellie (elliefm) = https://github.com/cyrusimap/cyrus-imapd/issues/1053 SETANNOTATION fails when using front-end in unified murder = https://github.com/cyrusimap/cyrus-imapd/issues/1765 Move SNMP out from master into a separate daemon + https://github.com/cyrusimap/cyrus-imapd/issues/730 Patch to add -a (preauth) switch to lmtpproxyd + https://github.com/cyrusimap/cyrus-imapd/issues/1096 faster tonum in sieve/sieve-lex.l + https://github.com/cyrusimap/cyrus-imapd/issues/2104 master process miscounts/conflates "ready workers" and "active" processes? -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: issue #46 not fixed in 2.5.11 and 3.0/master
Ooh, that's nasty. We only use the default position at FastMail, so I can see how we would miss this in our own usage. We don't test partitions much in Cassandane either. On Tue, 15 Aug 2017, at 19:30, Stephan Lauffer wrote: > Hello! > > A rename of a mailbox went wrong in our 3.0.x test environment. During> the > rename the partition information got lost and the default > partition was used instead. > > See my comment in issue #46: > > https://github.com/cyrusimap/cyrus-imapd/issues/46 > > -- > Liebe Gruesse, with best regards > Stephan Lauffer > > Pedagogical University Freiburg - Germany > http://www.ph-freiburg.de/zik/ > Fon/ Fax: +49 761 682 -559/ -486 > Email had 1 attachment: > * smime.p7s 8k (application/pkcs7-signature) -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Cyrus meeting notes 31 Jul 2017
Present: Nicola, Ken, Jeroen, ellie, Bron, Partha, Robert ellie: * has been mostly moving this week, so hasn't done much. * picking nits to fix on sasl and imapd Jeroen: * has updated personal copy of old Cyrus docs to point to current docs.* feels like a guest, which we don't want - we want to be the Cyrus project, which just has some people who are FastMail people.* has many patches on top of 2.5 - they need to be cleaned up and merged back.* sasl? what changed - mostly bug fixes and build fixes. Might need build changes to be coordinated since packagers are often different.* Murder improvements would be interesting to Kolab. Bron: * nothing really - been busy with FastMail management tasks Ken: * didn't touch dierolls. * Been working on per-user alarms which are coming along nicely. * Still struggling with how to handle etags and make sure they only change for the user who added their information.* decision is between storing changes as annotations, or storing patches in the file along with everything else. Will keep looking at it.* sasl v2.x vs v3 - no binary changes, so don't need to bump major version. Will do more invasive pull requests later.* will have to put together release notes for sasl release. Robert: * JMAP multi-account support 99% done. * Xapian crasher has kept busy - waiting for opinion from Xapian upstream. Cherry-pick commit and test in prod. -> bron and partha will be testing it tomorrow * Two different issues - non-deterministic and hard to debug stuff. Any way to improve process? Pair program with someone with FM prod access.* Second issue - user-flags dump. * this week - update TC-API document for Calendars. Partha: * first couple of days - trying to fix autotools issue with sasl * also massive patch in a branch that does code compliance with cyrus-imapd style (whitespace, etc)* working on functional testing framework for sasl using shell scripts - hoping to send initial pull request later this week - started by using cassandane for sasl, but too tightly knit with cyrus-imapd for now* didn't get to dicerolls last week * got prod access to FastMail, so need to spend some time trying to understand how Cyrus is used in FM production Nicola: * not much this week, just some docs for the sasl release -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: dicerolls
As noted by ellie, this didn't properly deal with the 'shelved' status. I have now fixed the script for next time: https://github.com/cyrusimap/BugzillaImport/commit/5a8db5bdf1f8611db102f590fe4a2970dad175c0 Bron. On Mon, 24 Jul 2017, at 22:21, Bron Gondwana wrote: > have some more bugs :) (for those of us who did anything) > > Bron (brong) > = https://github.com/cyrusimap/cyrus-imapd/issues/594 >Patch for cyrus-imapd to support OpenSLP > = https://github.com/cyrusimap/cyrus-imapd/issues/1005 >Only return IMAP_QUOTA_EXCEEDED if needed > = https://github.com/cyrusimap/cyrus-imapd/issues/1515 >sync_client general protection fault in libcrypto.so.0.9.8 > = https://github.com/cyrusimap/cyrus-imapd/issues/1911 >Two definitions of fillin_interactions() > = https://github.com/cyrusimap/cyrus-imapd/issues/2001 >Incorrect paths for autocreated calendars and addressbook in cyrus > 2.5.11> > Ken (ksmurchison) > = https://github.com/cyrusimap/cyrus-imapd/issues/497 >mupdate slave thread should commit less frequently > = https://github.com/cyrusimap/cyrus-imapd/issues/1598 >Make mupdate auto-resync after a master outage > + https://github.com/cyrusimap/cyrus-imapd/issues/1981 >lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags> > + https://github.com/cyrusimap/cyrus-imapd/issues/1692 >SELECT writes .lock file for unknown/inexistent mailboxes > + https://github.com/cyrusimap/cyrus-imapd/issues/1760 >implement blobs for calendar and contacts > > Partha (ajaysusarla) > = https://github.com/cyrusimap/cyrus-imapd/issues/1155 >PATCH TLS doesn't check CRLs > = https://github.com/cyrusimap/cyrus-imapd/issues/1467 >ptloader - ptclient ldap failure > = https://github.com/cyrusimap/cyrus-imapd/issues/1618 >interruption in communication between master and frontend > = https://github.com/cyrusimap/cyrus-imapd/issues/1710 >SSL private key cannot be read if chmod is 640 and path contains a > symlink> = https://github.com/cyrusimap/cyrus-imapd/issues/1907 >sieveshell: upload files, whose names end with ".script" without > truncating the names> > Robert (rsto) > = https://github.com/cyrusimap/cyrus-imapd/issues/457 >managesieve should handle AUTHENTICATE referrals > + https://github.com/cyrusimap/cyrus-imapd/issues/2069 >get rid of SMakefile > + https://github.com/cyrusimap/cyrus-imapd/issues/950 >Unified murder - if one node is down during mailbox deletion, deleted > remote mailboxes end up local mailboxes of that node with an invalid > partition upon startup> + https://github.com/cyrusimap/cyrus-imapd/issues/1440 >CID 385: STRING_NULL in ptload > + https://github.com/cyrusimap/cyrus-imapd/issues/1152 >Recursive reconstruct and @ in mailbox name > > ellie (elliefm) > = https://github.com/cyrusimap/cyrus-imapd/issues/1053 >SETANNOTATION fails when using front-end in unified murder > = https://github.com/cyrusimap/cyrus-imapd/issues/1650 >Support the i;ip-mask comparator for sieve > = https://github.com/cyrusimap/cyrus-imapd/issues/1765 >Move SNMP out from master into a separate daemon > = https://github.com/cyrusimap/cyrus-imapd/issues/2012 >Cyrus 2.5.10 - centos 7.3. Cyrus process hangs causing a lock that > spawns lots of other hanging processes come back.> + > https://github.com/cyrusimap/cyrus-imapd/issues/423 >configure should detect need for -ldoor with staticsasl > > > -- > Bron Gondwana, CEO, FastMail Pty Ltd > br...@fastmailteam.com > > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
dicerolls
have some more bugs :) (for those of us who did anything) Bron (brong) = https://github.com/cyrusimap/cyrus-imapd/issues/594 Patch for cyrus-imapd to support OpenSLP = https://github.com/cyrusimap/cyrus-imapd/issues/1005 Only return IMAP_QUOTA_EXCEEDED if needed = https://github.com/cyrusimap/cyrus-imapd/issues/1515 sync_client general protection fault in libcrypto.so.0.9.8 = https://github.com/cyrusimap/cyrus-imapd/issues/1911 Two definitions of fillin_interactions() = https://github.com/cyrusimap/cyrus-imapd/issues/2001 Incorrect paths for autocreated calendars and addressbook in cyrus 2.5.11 Ken (ksmurchison) = https://github.com/cyrusimap/cyrus-imapd/issues/497 mupdate slave thread should commit less frequently = https://github.com/cyrusimap/cyrus-imapd/issues/1598 Make mupdate auto-resync after a master outage + https://github.com/cyrusimap/cyrus-imapd/issues/1981 lmtpd aborts when compiled with optimisation and FORTIFY_SOURCE flags+ https://github.com/cyrusimap/cyrus-imapd/issues/1692 SELECT writes .lock file for unknown/inexistent mailboxes + https://github.com/cyrusimap/cyrus-imapd/issues/1760 implement blobs for calendar and contacts Partha (ajaysusarla) = https://github.com/cyrusimap/cyrus-imapd/issues/1155 PATCH TLS doesn't check CRLs = https://github.com/cyrusimap/cyrus-imapd/issues/1467 ptloader - ptclient ldap failure = https://github.com/cyrusimap/cyrus-imapd/issues/1618 interruption in communication between master and frontend = https://github.com/cyrusimap/cyrus-imapd/issues/1710 SSL private key cannot be read if chmod is 640 and path contains a symlink= https://github.com/cyrusimap/cyrus-imapd/issues/1907 sieveshell: upload files, whose names end with ".script" without truncating the names Robert (rsto) = https://github.com/cyrusimap/cyrus-imapd/issues/457 managesieve should handle AUTHENTICATE referrals + https://github.com/cyrusimap/cyrus-imapd/issues/2069 get rid of SMakefile + https://github.com/cyrusimap/cyrus-imapd/issues/950 Unified murder - if one node is down during mailbox deletion, deleted remote mailboxes end up local mailboxes of that node with an invalid partition upon startup+ https://github.com/cyrusimap/cyrus-imapd/issues/1440 CID 385: STRING_NULL in ptload + https://github.com/cyrusimap/cyrus-imapd/issues/1152 Recursive reconstruct and @ in mailbox name ellie (elliefm) = https://github.com/cyrusimap/cyrus-imapd/issues/1053 SETANNOTATION fails when using front-end in unified murder = https://github.com/cyrusimap/cyrus-imapd/issues/1650 Support the i;ip-mask comparator for sieve = https://github.com/cyrusimap/cyrus-imapd/issues/1765 Move SNMP out from master into a separate daemon = https://github.com/cyrusimap/cyrus-imapd/issues/2012 Cyrus 2.5.10 - centos 7.3. Cyrus process hangs causing a lock that spawns lots of other hanging processes come back.+ https://github.com/cyrusimap/cyrus-imapd/issues/423 configure should detect need for -ldoor with staticsasl -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Meeting not happening this week
The Cyrus meeting time runs pretty much into the start of the JMAP session at IETF, so you should all show up there instead (in person or virtually) https://datatracker.ietf.org/wg/jmap/meetings/ We're also considering whether to run a second meeting every week at a time more suitable to US West Coast people - it would be Australian morning and US afternoon/evening, and probably not suitable for Europeans. Let me know if this is something you're interested in. Cheers, Bron. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Future of Cyrus-SASL
Thanks Jan, I wonder if we want to set up a second meeting time, either fortnightly or weekly, that does the other cross over and allows us to more easily include middle and west America. Our current meeting time is 10pm for Australia, so something that was during the Australian day and afternoon/evening across the USA would work fine for everybody except Europeans. As Ken has already noted - we still want to support SASL. Now that we have him full time, he's going to release a new version, and at our scheduling meeting on Monday, Nicola bumped up the priority of updating the SASL documentation and converting it to the sphinx-based build system. Regardless of whether Cyrus-* leaves CMU (which is still being discussed with both CMU and potential new homes), the project intends to keep supporting SASL. Bron. On Thu, 13 Jul 2017, at 09:40, jan parcel wrote: > On problem I have is that people in imap development seem to say (and, on the > web, too) that cyrus-sasl means imap, leaving cyrus-? to mean libsasl?> > Other than that > > I cannot volunteer as a regular contributor because my regular job uses up > all my time. However, company policy requires us to send "upstream" all > relevant bug fixes we create, and to file bugs we find to upstream regardless > of whether or not we need to fix them -- for instance, I might never fix the > bug with failing to exclude ldapdb when so configured, unless a customer > demands that plugin, which I suspect will not happen.We are also required > to SECRETLY and SECURELY report security bugs upstream. > > Company policy definitely encourages us to have a relationship with upstream > developers. > > But I'm also in the U.S. west coast timezone (Pacific Time) and 4am is not > doable for me except in extreme emergencies (such as a high-scoring CVE that > needs immediate attention, for instance.) > > So, I don't quite know what you will find acceptable in the way of > participation from me, but I want to be cooperative to a libsasl community > and contribute where I can, and participate if there are meetings that I can > reasonably attend. I'd say times between 6:00 AM - 1AM would be doable. > > And if the libsasl project were to lose all upstream support, I would be > required to report that to my management, and abide by any decisions they > make wrt whether to look for a supported alternative. > > sendmail can also profitably use libsasl2. > > So, thanks for including me in this thread. > > Jan Parcel > Software Developer > Oracle Systems Server & Cloud Engineering > > > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
New dicerolls this week
hangs causing a lock that spawns lots of other hanging processes come back. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
This week's dice roll
Now not including Nicola, and with extra magic in the script to set the
label and alter the owner:
Bron (brong)
* https://github.com/cyrusimap/cyrus-imapd/issues/1911
* https://github.com/cyrusimap/cyrus-imapd/issues/1621
* https://github.com/cyrusimap/cyrus-imapd/issues/1073
* https://github.com/cyrusimap/cyrus-imapd/issues/1697
* https://github.com/cyrusimap/cyrus-imapd/issues/1841
Ken (ksmurchison)
* https://github.com/cyrusimap/cyrus-imapd/issues/1893
* https://github.com/cyrusimap/cyrus-imapd/issues/519
* https://github.com/cyrusimap/cyrus-imapd/issues/690
* https://github.com/cyrusimap/cyrus-imapd/issues/951
* https://github.com/cyrusimap/cyrus-imapd/issues/1746
Partha (ajaysusarla)
* https://github.com/cyrusimap/cyrus-imapd/issues/608
* https://github.com/cyrusimap/cyrus-imapd/issues/1155
* https://github.com/cyrusimap/cyrus-imapd/issues/1710
* https://github.com/cyrusimap/cyrus-imapd/issues/1618
* https://github.com/cyrusimap/cyrus-imapd/issues/1907
Robert (rsto)
* https://github.com/cyrusimap/cyrus-imapd/issues/60
* https://github.com/cyrusimap/cyrus-imapd/issues/1284
* https://github.com/cyrusimap/cyrus-imapd/issues/735
* https://github.com/cyrusimap/cyrus-imapd/issues/1569
* https://github.com/cyrusimap/cyrus-imapd/issues/1498
ellie (elliefm)
* https://github.com/cyrusimap/cyrus-imapd/issues/1053
* https://github.com/cyrusimap/cyrus-imapd/issues/1793
* https://github.com/cyrusimap/cyrus-imapd/issues/1650
* https://github.com/cyrusimap/cyrus-imapd/issues/1726
* https://github.com/cyrusimap/cyrus-imapd/issues/1938
Here's the icky ghetto GitHub API code... seriously, all the docs
say "id", but the mean "number", and you get back label and assignee
objects, but write just names. It's really an odd API. Anyway,
this works :)
brong@bat:~/src/BugzillaImport$ cat pick5.pl
use strict;
use warnings;
use Net::GitHub;
use Data::Dumper;
use XML::Fast;
use HTML::Entities;
use DateTime::Format::DateParse;
use Encode qw(decode_utf8 encode_utf8);
use JSON;
use List::Util qw(shuffle);
BEGIN {
Net::GitHub::V3::Issues->__build_methods(
import_issue => { url => "/repos/%s/%s/import/issues", method =>
'POST', args => 1 },);
};
open(FH, "<oauth_token.txt");
my $token = ;
chomp($token);
close(FH);
my $gh = Net::GitHub::V3->new(
access_token => $token,
);
$gh->ua->default_header(Accept => 'application/vnd.github.golden-comet-
preview+json');
my $issue = $gh->issue;
$issue->set_default_user_repo('cyrusimap', 'cyrus-imapd');
use HTTP::Message;
$issue->ua->default_header('Accept-Encoding' => scalar
HTTP::Message::decodable());# for debugging
#$issue->ua->add_handler("request_send", sub { shift->dump; return
});#$issue->ua->add_handler("response_done", sub { shift->dump; return });
my @labels = $issue->labels();
my %labels = map { $_->{name} => $_ } @labels;
my %have;
my @issues = $issue->repos_issues({filter => 'all', state => 'open'});
push @issues, $issue->next_page() while $issue->has_next_page();
my @list = shuffle grep { not has_diceroll($_) } @issues;
my $n = 0;
my %map = qw(
Ken ksmurchison
Robert rsto
Partha ajaysusarla
Bron brong
ellie elliefm
);
foreach my $name (sort keys %map) {
print "$name ($map{$name})\n";
for (1..5) {
my $item = $list[$n++];
my @labels = map { $_->{name} } @{$item->{labels}};
push @labels, 'diceroll';
print " * $item->{html_url}\n";
$issue->update_issue($item->{number}, {
assignee => $map{$name},
labels => \@labels,
});
}
print "\n";
}
sub has_diceroll {
my $issue = shift;
foreach my $label (@{$issue->{labels}}) {
next unless $label->{name} eq 'diceroll';
return 1;
}
return 0;
}
--
Bron Gondwana, CEO, FastMail Pty Ltd
br...@fastmailteam.com
Re: Cyrus meeting minutes 26 June 2017
Hey, nice. I'll add that to the script so it automatically dicerolls them and assigns them to the person when I run it :) (and I'll have it skip issues with diceroll already set) Bron. On Mon, 26 Jun 2017, at 23:17, Robert Stepanek wrote: > FYI - I've just created a new Github issue label 'diceroll' for the > cyrus-imapd repository, so I can tag these random issues in my open > issue list. Maybe that's also useful to you.> > > On Mon, Jun 26, 2017, at 14:48, Bron Gondwana wrote: >> Happy birthday: >> >> Bron >> * https://github.com/cyrusimap/cyrus-imapd/issues/1680 >> * https://github.com/cyrusimap/cyrus-imapd/issues/689 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1515 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1090 >> * https://github.com/cyrusimap/cyrus-imapd/issues/398 >> >> ellie >> * https://github.com/cyrusimap/cyrus-imapd/issues/2012 >> * https://github.com/cyrusimap/cyrus-imapd/issues/742 >> * https://github.com/cyrusimap/cyrus-imapd/issues/560 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1434 >> * https://github.com/cyrusimap/cyrus-imapd/issues/744 >> >> Ken >> * https://github.com/cyrusimap/cyrus-imapd/issues/1177 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1598 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1895 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1955 >> * https://github.com/cyrusimap/cyrus-imapd/issues/947 >> >> Nicola >> * https://github.com/cyrusimap/cyrus-imapd/issues/1130 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1775 >> * https://github.com/cyrusimap/cyrus-imapd/issues/466 >> * https://github.com/cyrusimap/cyrus-imapd/issues/967 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1136 >> >> Partha >> * https://github.com/cyrusimap/cyrus-imapd/issues/1995 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1467 >> * https://github.com/cyrusimap/cyrus-imapd/issues/488 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1976 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1159 >> >> Robert >> * https://github.com/cyrusimap/cyrus-imapd/issues/1074 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1359 >> * https://github.com/cyrusimap/cyrus-imapd/issues/497 >> * https://github.com/cyrusimap/cyrus-imapd/issues/1556 >> * https://github.com/cyrusimap/cyrus-imapd/issues/575 >> >> Chosen by fair dice roll (ok - by perl List::Util::shuffle) >> >> Bron. >> >> On Mon, 26 Jun 2017, at 22:28, Nicola Nye wrote: >>> Cyrus team met once again, same bat time, same bat channel. >>> >>> Here's what we've been working on! >>> >>> Ken- >>> Shared alarms: splitting them out on the fly rather than doing >>> them in bulk>>> What do we do if something gets unshared. >>> - Check ACL at alarm time and not re-send, not re-schedule. >>> - Owner keeps the alarm state in the event themselves, alarm >>>state for everyone else kept as a separate annotation>>> - But >>> if you share with someone else, need to strip owner's >>>alarm and patch in the annotated state.>>> - (Do not fiddle >>> owner) >>> - For calendar with no owner, no alarm set on event: all >>>people's alarm state stored as annotations>>> Last week at CMU, >>> from next week he's a FastMailer! >>> >>> Partha >>> - Inducted in Bron's Zeroskip plan for a new, improved skiplist >>> implementation to replace twoskip>>> - Starting to design API for >>> Zeroskip >>> - Improve behaviour on reconstruct (GitHub #2041) >>> - Released 3.0.2 >>> >>> Nicola >>> - Working on adding a git datestamp to the docs so readers know >>> when the last time a piece of documentation was updated (and how >>> out of date/unreliable it might be)>>> >>> Robert >>> - Going to remove lmdb support - too much effort required to make >>> it stable and supported.>>> - JMAP message record events >>> - Xapian fixes sent upstream via Pull Request >>> - TC API RFC document. IETF RFCs frozen from July 7 for two weeks, >>> looking to finish TC API doc before them.>>> >>> Ellie >>> - Released 3.0.2 >>> - Endian fixes (Thanks to Tibbs for lots of testing and excellent >>> bug reports)>>> >>> Bron >>> - Pushing JMAP author to get next round of JMAP spec changes out, >>> so we can implement in Cyrus and proxy.>>> - Wants to start clearing >>> out some of the tech debt bug backlog in >>> GitHub.>>> - We'll each take 5 bugs per week and look at them to >>> close or >>>reassign or fix.>> >> -- >> Bron Gondwana, CEO, FastMail Pty Ltd >> br...@fastmailteam.com >> >> > -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Re: Cyrus meeting minutes 26 June 2017
Happy birthday: Bron * https://github.com/cyrusimap/cyrus-imapd/issues/1680 * https://github.com/cyrusimap/cyrus-imapd/issues/689 * https://github.com/cyrusimap/cyrus-imapd/issues/1515 * https://github.com/cyrusimap/cyrus-imapd/issues/1090 * https://github.com/cyrusimap/cyrus-imapd/issues/398 ellie * https://github.com/cyrusimap/cyrus-imapd/issues/2012 * https://github.com/cyrusimap/cyrus-imapd/issues/742 * https://github.com/cyrusimap/cyrus-imapd/issues/560 * https://github.com/cyrusimap/cyrus-imapd/issues/1434 * https://github.com/cyrusimap/cyrus-imapd/issues/744 Ken * https://github.com/cyrusimap/cyrus-imapd/issues/1177 * https://github.com/cyrusimap/cyrus-imapd/issues/1598 * https://github.com/cyrusimap/cyrus-imapd/issues/1895 * https://github.com/cyrusimap/cyrus-imapd/issues/1955 * https://github.com/cyrusimap/cyrus-imapd/issues/947 Nicola * https://github.com/cyrusimap/cyrus-imapd/issues/1130 * https://github.com/cyrusimap/cyrus-imapd/issues/1775 * https://github.com/cyrusimap/cyrus-imapd/issues/466 * https://github.com/cyrusimap/cyrus-imapd/issues/967 * https://github.com/cyrusimap/cyrus-imapd/issues/1136 Partha * https://github.com/cyrusimap/cyrus-imapd/issues/1995 * https://github.com/cyrusimap/cyrus-imapd/issues/1467 * https://github.com/cyrusimap/cyrus-imapd/issues/488 * https://github.com/cyrusimap/cyrus-imapd/issues/1976 * https://github.com/cyrusimap/cyrus-imapd/issues/1159 Robert * https://github.com/cyrusimap/cyrus-imapd/issues/1074 * https://github.com/cyrusimap/cyrus-imapd/issues/1359 * https://github.com/cyrusimap/cyrus-imapd/issues/497 * https://github.com/cyrusimap/cyrus-imapd/issues/1556 * https://github.com/cyrusimap/cyrus-imapd/issues/575 Chosen by fair dice roll (ok - by perl List::Util::shuffle) Bron. On Mon, 26 Jun 2017, at 22:28, Nicola Nye wrote: > Cyrus team met once again, same bat time, same bat channel. > > Here's what we've been working on! > > Ken- > Shared alarms: splitting them out on the fly rather than doing them > in bulk> What do we do if something gets unshared. > - Check ACL at alarm time and not re-send, not re-schedule. > - Owner keeps the alarm state in the event themselves, alarm >state for everyone else kept as a separate annotation> - But if > you share with someone else, need to strip owner's alarm >and patch in the annotated state.> - (Do not fiddle owner) > - For calendar with no owner, no alarm set on event: all people's >alarm state stored as annotations> Last week at CMU, from next week > he's a FastMailer! > > Partha > - Inducted in Bron's Zeroskip plan for a new, improved skiplist > implementation to replace twoskip> - Starting to design API for Zeroskip > - Improve behaviour on reconstruct (GitHub #2041) > - Released 3.0.2 > > Nicola > - Working on adding a git datestamp to the docs so readers know when > the last time a piece of documentation was updated (and how out of > date/unreliable it might be)> > Robert > - Going to remove lmdb support - too much effort required to make it > stable and supported.> - JMAP message record events > - Xapian fixes sent upstream via Pull Request > - TC API RFC document. IETF RFCs frozen from July 7 for two weeks, > looking to finish TC API doc before them.> > Ellie > - Released 3.0.2 > - Endian fixes (Thanks to Tibbs for lots of testing and excellent > bug reports)> > Bron > - Pushing JMAP author to get next round of JMAP spec changes out, so > we can implement in Cyrus and proxy.> - Wants to start clearing out > some of the tech debt bug backlog in > GitHub.> - We'll each take 5 bugs per week and look at them to close > or >reassign or fix. -- Bron Gondwana, CEO, FastMail Pty Ltd br...@fastmailteam.com
Crash in ... jansson??
Core was generated by `httpd -C /etc/cyrus/imapd-sloti21d2t19.conf'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0049d71c in getcontacts_cb (rock=0x7fffed02f100,
cdata=0x7f193ea66880 ) at imap/jmap_contact.c:1652
1652json_t *obj = jmap_contact_from_vcard(vcard->objects, cdata,
,
(gdb) bt
#0 0x0049d71c in getcontacts_cb (rock=0x7fffed02f100,
cdata=0x7f193ea66880 ) at imap/jmap_contact.c:1652
#1 0x7f193e811a09 in read_cb (stmt=0xcc74c8, rock=0x7fffed02efb0) at
imap/carddav_db.c:228
#2 0x7f193e4c3358 in sqldb_exec (open=0xcc2e20,
cmd=0xc9aa60 "SELECT rowid, creationdate, mailbox, resource, imap_uid,
lock_token, lock_owner, lock_ownerid, lock_expire, version, vcard_uid, kind,
fullname, name, nickname, alive, modseq FROM vcard_objs WHERE a"...,
bval=0x7fffed02f028, cb=0x7f193e81178c ,
rock=0x7fffed02efb0) at lib/sqldb.c:335
#3 0x7f193e8130b6 in carddav_get_cards (carddavdb=0xc9a900,
abookname=0xcb3b50 "fastmail.fm!user.stefaniegray.#addressbooks.Default",
vcard_uid=0x0, kind=0, cb=0x49d538 , rock=0x7fffed02f100)
at imap/carddav_db.c:722
#4 0x00498f0b in jmap_contacts_get (req=0x7fffed02f290, cb=0x49d538
, kind=0, resname=0x4de1d3 "contacts")
at imap/jmap_contact.c:528
#5 0x0049d7a4 in getContacts (req=0x7fffed02f290) at
imap/jmap_contact.c:1663
#6 0x0048a530 in jmap_post (txn=0x7fffed02f490, params=0x0) at
imap/http_jmap.c:338
#7 0x00474387 in cmdloop (conn=0x7fffed032670) at imap/httpd.c:2071
#8 0x00471973 in service_main (argc=1, argv=0xc64040,
envp=0x7fffed0341b8) at imap/httpd.c:1095
#9 0x00489a16 in main (argc=3, argv=0x7fffed034198,
envp=0x7fffed0341b8) at master/service.c:632
...
1651/* Convert the VCARD to a JMAP contact. */
1652json_t *obj = jmap_contact_from_vcard(vcard->objects, cdata,
,
1653 crock->props,
crock->mailbox->name);
The crash is there - but what could be segfaulting?
(gdb) p vcard->objects
$1 = (struct vparse_card *) 0xe0bdf0
(gdb) p *vcard->objects
$2 = {type = 0xe0be20 "VCARD", properties = 0xe0bd40, objects = 0x0, next = 0x0}
(gdb) p cdata
$3 = (struct carddav_data *) 0x7f193ea66880
(gdb) p *cdata
$4 = {dav = {rowid = 250, creationdate = 1435863920, mailbox = 0xcc28b8
"fastmail.fm!user.stefaniegray.#addressbooks.Default",
resource = 0xcc8a68 "a6a43115-1c03-4f45-a38e-273520b3db9f.vcf", imap_uid =
558, modseq = 336446, lock_token = 0x0, lock_owner = 0x0,
lock_ownerid = 0x0, lock_expire = 0, alive = 1}, version = 0, vcard_uid =
0xc9b6c8 "a6a43115-1c03-4f45-a38e-273520b3db9f", kind = 0,
fullname = 0xcb3ad8 "Phaedra Burke", name = 0xcb43b8 "\004", nickname =
0xcad338 "\001", emails = 0x0, member_uids = 0x0}
(gdb) p record
$5 = {uid = 558, internaldate = 1435863920, sentdate = 1435852800, size = 535,
header_size = 314, gmtime = 1435863920, cache_offset = 213828,
last_updated = 1497457449, system_flags = 0, user_flags = {1, 0, 0, 0},
content_lines = 11, cache_version = 6, guid = {status = GUID_UNKNOWN,
value = "\300\300\a\260\242}&\023,\311\034^/\217\027\352\350UX:"}, modseq =
336446, cid = 93421607515573021, cache_crc = 458859899,
recno = 250, silent = 0, basecid = 0, crec = {buf = 0x0, offset = 0, len = 0,
item = {{offset = 0, len = 0}, {offset = 0, len = 0}, {
offset = 0, len = 0}, {offset = 0, len = 0}, {offset = 0, len = 0},
{offset = 0, len = 0}, {offset = 0, len = 0}, {offset = 0, len = 0}, {
offset = 0, len = 0}, {offset = 0, len = 0
(gdb) p crock->props
$6 = (struct hash_table *) 0x0
(gdb) p crock->mailbox->name
$7 = 0xcad370 "fastmail.fm!user.[CENSORED].#addressbooks.Default"
There's nothing here which should be causing a crash! The only thing I can see
is that jansson was corrupting somehow :( Boo. I'll keep an eye on it and see
if we get any more.
Bron.
--
Bron Gondwana
br...@fastmail.fm
FastMail feature planning discussion
While this isn't a full "roadmap for the project" this is what some FastMail staff thought was important to us to focus on in the next little while. Here's what we came to. If there's something you think is important which is NOT in this list, let us know :) No promises that we'll schedule our staff time on it, but we'll help you with code review and tests if you want to work on anything! Cheers, Bron. *Short term goals: June** **-** August 2017* Full JMAP support. What’s left? * getMessageListUpdates - brong * spec updates * sending support (per new spec) * push channel for non-FM people * efficiency improvements maybe * test suite coverage Global lock / per domain lock / per-user lock options? * requested on the list - a different way to do backups. freeze / snapshot / unfreezeNew backup tools running in production at FastMail and declared stable for the world. * Meeting with @el...@fastmailteam.com[1] today! Attachment search * Robert S JMAP calendar and contacts objects * Robert S Good init scripts and packaging for Cyrus so we can point people to it. * Debian / Ubuntu (andre) * Redhat / Fedora / CentOS (tibbs) * Arch Linux * FreeBSD / NetBSD / OpenBSD (?) * Joyent / Solaris (?) * MacOS ? - need to make it build! Monitoring : Prom statistics * ellie WIP / testing things Migrating off CMU and SPI/Apereo * Bron to do *Medium term goals: End of 2017* Improve shared calendaring support. * Ken to work on Per-user notifications. * Ken to work on Event creator vs calendar owner. * Ken to work on Cyrus SASL needs some love * Ken to work on Zeroskip! * Partha to work on with Bron Cyrus 3.0 in distro packages. Debian, RedHat, Fedora, Ubuntu, … ? Usable “clone existing server” tooling - import/export helpers All Cyrus projects with sphinx docs - sasl, cassandane, … - nicola *Long term goals: 2018* Murder improvements. (Why doesn’t FastMail run it and how do we get there/do we want to?)Calendar availability checking. *Wishlist/Unscheduled* Calendaring resource support. Room bookings. Sieve extensions? RFC additions? Cyrus.works on multiple versions and distros Simplify/tidy the conversations database Usable backup tooling shipping with Cyrus Not having 410 “open issues” in GitHub - DONE! -- Bron Gondwana br...@fastmail.fm Links: 1. https://paper.dropbox.com/ep/profile/iX86Va0dCNNJDjPG9iZ958P1v6NA7mQYxIijKgJMdedctBc0IX
Meeting minutes May 22nd
Present: Ken, Partha, ellie, Nicola, Bron, Robert, Leena We really should make ourselves an agenda for these things! Here's what we talked about in rough order... Ken: * while in Pittsburgh chatted with Cyrus Daboo about per-user alarms on calendar events. * Cyrus suggested storing it all in the event and stripping it on FETCH * downside is per-user exceptions if a user changes just a single alarm * Ken thinks VPATCH as ANNOTATIONS makes sense for us * foreach Robert * has been doing work on Annotations branch, lots to push to master. * deprecated ANNOTATEMORE - including cyradm fixes * tiny JMAP fix today * working on calalarmdb and rescheduling instances of VEVENTs - there's clearly a bug in there somewhere, FastMail users not moving due to events that haven't run properly, and a user report too. Working on tests to find the cause. * working on docs for TC-API. Currently working on Calendar Event wrapper - same as JMAP calendar - for publishing webcal, etc. * doing for Calendar Wrapper first, then JMAP wrapper for VTODO if time before calconnect * Attachment search - not sure what to do... * Bron suggested - index attachments in a separate channel so that message search data becomes available fast, attachments later if the engine is down. * Will see if we can do two snippet fetches over JMAP, one for snippets that don't include attachment data, and a separate one for attachment snippets which might be very slow. ellie * been working on some misc bugs from list/github * sketching out a prototype of prometheus metrics (which will make robn happy) Partha: * Mostly been doing trivial patches in Cyrus and spending time with ellie and Nicola trying to soak up as much info as possible! * Still struggling with Cassandane tests - haven't finished on a bug because of that. * Meanwhile sent patches which are being reviewed and merged! * Fortify plus O1 is killing Cyrus, slow progress on looking at compiling with it. * Aborting in multiple places with O1, will send report tomorrow. Leena: * been ages since I worked on Cyrus, last time was the Perl modules years ago, just trying to work out what's going on * waiting for RHEL7 builds to arrive * is the 3.0 administration still scriptable with perl? - yes * wondering what's happened with the Kolab builds * have already been using / for separator, so defaults in 3.0 will be good :) Nicola says update docs cover the things that have changed. (aside on cyrus packages) * Partha will contact Jeroen and tibbs to ask what's happening with packages. * Bron - would like to get continuous integration builds also building SRPM/deb-src packages for common distros or operating systems that we can get a build environment for and make sure packages can be built from our tree at any time. * Leena - the depencies on 3.0 seemed quite daunting, as rhel7 is quite old now. Current is rhel 7.3. * it might shake out build system assumptions since we're all running pretty recent stuff * with could build on Centos 7 and it should just work. centos7+epel. * Ken has compiled on RHEL7 at CMU. Nicola * partway through the big doc restructure (again). * Hoping for a review from NicB before I issue a big PR. But may just go ahead this week anyway. Bron * no news on finding a new home for Cyrus - haven't done anything sorry. * haven't had much time for Cyrus this week Next meeting same time next week. -- Bron Gondwana br...@fastmail.fm
Re: Discussion on Hacker News about CMU discontinuing Cyrus
On Tue, 9 May 2017, at 03:04, jan parcel wrote: > What about libsas2? We seem to have "Cyrus" and "Cyrus-sasl", it is the > latter that I'm concerned with. It makes sense for the SASL project to come to the same home. It's already using the same Github project. Bron. -- Bron Gondwana br...@fastmail.fm
Discussion on Hacker News about CMU discontinuing Cyrus
https://news.ycombinator.com/item?id=14282678 Lots of interesting discussion in there. Luckily I got in on time to post the highest rated response! --- Cyrus development will not be affected by this. While CMU has been running Cyrus, and employing one of the key developers, FastMail has a team dedicated to supporting the biggest open source project that we use. We have a new developer starting on Wednesday next week as well as Ken from CMU who has agreed to keep working on Cyrus as a FastMail employee and representing the project at conferences. We are committed to improving the project and keeping it open. As a member of the Cyrus IMAP board, I'm very proud of the 3.0 release that we recently made, and we're currently planning for the 3.1 release which will include further significant improvements. --- There's plenty more interesting comments in there. I'll be working on finding a good legal home for the project this week. Cheers, Bron. -- Bron Gondwana br...@fastmail.fm
