Re: MESSAGE quota resource implemention
On 05/09/11 20:16, Greg Banks wrote: On 02/09/11 20:03, Bron Gondwana wrote: On Fri, Sep 02, 2011 at 07:36:20PM +1000, Greg Banks wrote: How's about this for a strategy? When a quota resource is first enabled, (i.e. the limit is changed from UNLIMITED to some finite value), the usage is stored as some special value which I'll call INDETERMINATE. What about 'getquota'? I don't support any solution which leaves getquota returning bogus values or failing to respond. That's just icky and confusing. I don't think you can avoid two passes, and I don't even think you can avoid two values during if you really want to be good about it. [...] There are pure ways to do this, that guarantee consistency. [...] That's a real, robust solution. But it's pretty heavy engineering. After some thought, I agreed, decided it's quite doable, and decided to start. I have an implementation coded, so far the hardest bit is testing it. Implemented, tested, and I'd appreciate a review. https://github.com/gnb/cyrus-imapd/commit/af8d5bd3b40c4cb49fb7c943c85cb0aeab209215 https://github.com/gnb/cassandane/commit/c529f745b51aa0b566020ecbffca774e783124ee https://github.com/gnb/cassandane/commit/569eea2b9a2ad56b98c3c5cea60509689fa49bad -- Greg.
Re: Subject : git best practices
Olivier ROLAND wrote: 2011/8/19 Henrique de Moraes Holschuh h...@debian.org: IMHO, if you're going to come up with rules for that, better do as it is done in the Linux kernel or in git itself. The rules are: * detailed changelogs on every commit * only cleaned up commits are sent up for merging in the mainline. The cleaned-up commits are not large squashes (those are impossible to understand), instead they should look like you did everything perfectly at the first try, but step-by-step. * Bissectability is very important and should be preserved, so the tree should build _and run_ at every commit. * Don't rebase topic branches on top of the latest mainline, unless you actually need to do it for it to run. Base topic branches on stable points of the mainline. Sounds good to me. Bron, you ask us to just point you to a git branch with our patches. The problem is that sometimes we discuss things and the branch become a development branch with (useless for master) history. Maybe it would be more convenient if we can work with pull request like on github : http://help.github.com/send-pull-requests/ If genuinely a topic-branch, try the following: $ git checkout --track -b dev-something origin/master $ git config branch.dev-something.autosetuprebase always ^ I have this in my ~/.gitconfig to apply globally, BTW $ make edit #1 $ git commit $ send patch to mailing list / attach to ticket $ receive feedback, further edits $ git commit (possibly git pull rebasing your working copy changes) $ git diff [options] origin/master [-- files and paths] or $ git diff dev-something...master For what [options] can give you (such as -c and --cc). Is that acceptable? Should we document such (with comments and feedback appreciated!)? Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Cyrus CalDAV design decision
Dave McMurtrie wrote: Good day, Ken, Bron and I have had various disjointed conversations about where CalDAV data should be stored. We're getting to a point where we really need to finalize that design decision, so I'm soliciting feedback here. The current Cyrus CalDAV code stores DAV resources as subfolders of a user's INBOX. For example, user.dave64.#calendars and user.dave64.#addressbooks. Under each resource, of course, may be additional sub-resources like user.dave64.#calendars.Work. This was fairly easy for Ken to implement, so he did so as a proof of concept without much thought beyond that. One issue, however, is that IMAP clients have access to these mailboxes and it would likely be a system administrator's nightmare to roll something like this into production. First: Does the Cyrus CalDAV daemon / interface require to use the same configuration / runtime data (imapd.conf, config and partition directories)? Second: The IMAP clients to a Kolab Groupware solution, heavily dependent on Cyrus IMAP, uses annotations to indicate the folder type, for example; INBOX.Calendar - /vendor/kolab/folder-type value.shared event In these folders we store email messages that have .xml attachments. The .xml attachments are the actual 'object'; an event, a contact, etc. A 'dumb' IMAP client will see a message with a .xml attachment. The message body in our case contains a small note on the object being a Kolab object, and a link to a page listing 'Kolab Smart clients'. We find that we have to make many potential Kolab clients 'smart', however. I suppose, with CalDAV integrating into Cyrus IMAP in the foreseeable future, Kolab Groupware will want to make use if these additional capabilities as well. We have previously found iCalendar to have a number of serious restrictions and/or flaws, however, so I'm not certain what/who/where/how on our side. Anyway, perhaps the way we have successfully used for the past ~7 years inspires someone to come up with a way that works for all of us. Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: Cyrus CalDAV design decision
Ken Murchison wrote: Bron Gondwana wrote: On Tue, Aug 23, 2011 at 02:44:46PM -0400, Dave McMurtrie wrote: 3) Store DAV resources in a separate hierarchy like the DELETED hierarchy. I think Ken and I initially liked this idea, but the more we talk about it, the more it seems like this is the hardest to implement and we can't remember why we thought it was a good idea. Also, I think Bron suggested that he'd like to move away from having the DELETED hierarchy at some point. I'm pretty sure we were at a bar when we discussed this, which may explain why my memory is so foggy on the details. I actually like this best - put it in a separate namespace at the top level, like: addressbook.brong addressbook.brong.Work calendar.brong calendar.brong.Work This could also be hooked in with altnamespace more sensibly, and even advertised as separate namespaces or suppressed to IMAP clients completely. Where would shared mailboxes reside? I don't know if there is a viable use case for shared mailboxes, (...snip...) There's definitely a use-case for shared calendars (think resources such as conference rooms, beamers and cars). Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeuwen at kolabsys.com t: +44 144 340 9500 m: +44 74 2516 3817 w: http://www.kolabsys.com pgp: 9342 BF08
Re: MESSAGE quota resource implemention
Le 06/09/2011 10:23, Greg Banks a écrit : So I think we'll need another round, sorry :( Given that the annotations quota is broken and I'll be reimplementing it anyway, you may as well ignore QUOTA_ANNOTSTORAGE in all commits, and leave out the function annotatemore_computequota() for now. We'll use something very much like it for reconstruct later. I'm hoping to be able to pull your next round of changes into my annotate branch before I reimplement the annotation quota in the next few days. If the new code is capable of returning actual resource usages upon getquota (I think Bron wanted it that way), then I guess I can drop some more of the functions I added (annotatemore_computequota, but also mboxlist_updatequota and associated code including the quota.sets[]; then maybe I could also change the code as proposed earlier and prevent writing of resource limit in quota entry if it is QUOTA_UNLIMITED). Will look at it tomorrow. Worked on DUMP/UNDUMP today: https://github.com/worldline-messaging/cyrus-imapd/commit/45148b20a4f2343d72aa7436e7255a92508d7bf8 I used an IMAP list to send data, as for annotations. For future usages, maybe the UNDUMP code could try to skip IMAP lists if it finds any instead of the expected file content LITERAL ? (to prevent breaking compatibility too many times, in case other things would need to be transmitted this way through DUMP/UNDUMP). By the way, I noticed that UNDUMPing annotations fails for now (function annotate_state_write uses 'int_mboxname' in the annotation state struct, but it is NULL in this context). I'm still not convinced we'll need quota.sets[], but I'll play along. Thanks again for your work, and sorry that my annotate branch wasn't quite as stable a base as you first thought :) No problem, I was somehow prepared to that kind of scenario :) Regards Julien
Re: MESSAGE quota resource implemention
Le 06/09/2011 19:48, Julien Coloos a écrit : Le 06/09/2011 10:23, Greg Banks a écrit : So I think we'll need another round, sorry :( Given that the annotations quota is broken and I'll be reimplementing it anyway, you may as well ignore QUOTA_ANNOTSTORAGE in all commits, and leave out the function annotatemore_computequota() for now. We'll use something very much like it for reconstruct later. I'm hoping to be able to pull your next round of changes into my annotate branch before I reimplement the annotation quota in the next few days. If the new code is capable of returning actual resource usages upon getquota (I think Bron wanted it that way), then I guess I can drop some more of the functions I added (annotatemore_computequota, but also mboxlist_updatequota and associated code including the quota.sets[]; then maybe I could also change the code as proposed earlier and prevent writing of resource limit in quota entry if it is QUOTA_UNLIMITED). Will look at it tomorrow. When I said 'actual', I meant based on the upcoming mailbox index field. But wait, I got a bit confused with the last point about playing along with quota.sets[]. If what you discussed with Bron is that, in the end, usage (re)computing will only be done with the quota utility (no more automatic computing upon setquota, neither for getquota), then after upgrading people shall call 'quota -f' once and for all and quota.sets[] must disappear - and all resources must be written in the quota entry -, otherwise users would need to call 'quota -f' on a given quotaroot each time they set a new quota resource limit to a mailbox. Regards Julien
Re: Cyrus CalDAV design decision
(Resending to include cyrus-devel) At the moment, the storage format in use is iCalendar, being stored as RFC5322 messages. That sounds very much like what Kolab did in version 1. After trying to make this interoperate for several years it was given up in favor of the Kolab XML format because iCalendar vCard looked good on paper but suffered from severe interoperability issues between implementations. While I like the idea of the Kolab format, I'd like to point out that it's not all quite as rosy as you suggest. Most the criticisms you have of iCalendar and vCard, the Kolab format has equal amount of or more. Having had a look at the spec and putting together comments from several people: 1. The format never seems to have ever made it to the finished state. It's been stuck in RC states for several years. I don't think that gives strong confidence to implementers that it's actually stable. http://kolab.org/doc/kolabformat-2.0rc7-html/index.html 2. The description is that's it's in XML, but there's no DTD or ABNF, just a pseudo english format description. So there's no way to clearly test that you can create/read conforming XML blobs in a client implementation. 3. The document is very incomplete in places, for example this node: http://kolab.org/doc/kolabformat-2.0rc7-html/x180.html 4. It's also inconsistent in places. Some attributes being described as both MAY be present and mandatory: http://kolab.org/doc/kolabformat-2.0rc7-html/c188.html 5. Some parts are underspecified. The sensitivity field is inadequately described: http://kolab.org/doc/kolabformat-2.0rc7-html/c188.html. 6. For that matter, there's no complete set of examples (an mbox full of example Events, Tasks, Contacts, etc) or a test suite that clients should be able to read and interoperate with. 7. The Datetime format seems arbitrarily made up, but is actually the ISO8601 combined extended UTC format with second precision, but you don't reference ISO8601 or RFC3339 which also defines Internet time the same way. Likewise Date. 8. Recurrence seems primitive. Recurrence across DST boundaries seems to be as broken as every other format. There doesn't seem to be a way to specify public holiday exclusions. 9. The docs talk about ANNOTATEMORE more everywhere, which really needs updating to include METADATA and the final RFC 10. The /vendor/kolab/folder-type annotation should be updated now that SPECIALUSE has been made an RFC When you add these things together, I don't think it makes it easy to create a client or server for the Kolab format. And I think that goes part of the way to explaining why the quality of alternate clients out there hasn't exactly been great. I've tried Bynari (Outlook) and SyncKolab (Thunderbird), and their implementations are pretty bad. Having said a bunch of negative things, let me finish on how I think we can actually fix these and take these forward: 1. The Kolab Format spec needs to be completely finished. Edge cases cleaned up, todo parts finished, and unspecified bits specified completely. 2. It needs to be marked as a true 2.0 version, not an RC. You can add 2.1 later if you want, but I think the eternal RC status just leaves it feeling abandoned. 3. We need some sort of test suite/code. That should consist of at least two main things: a) A large bunch of emails that test all the different edge cases of the spec that clients should be able to read provided in a simple mbox format (separate one for each folder type). That way at least client implementers can quickly test their reading code against a bunch of cases b) Some standalone C/C++ executable that should be able to read these emails, check they conform to the right format, and dump a memory structure of the parsed result. That will allow implementers to quickly check that they're generating correctly formatted data structures. Yes, that would require a bit of work from someone, but I totally think it's worth it, and I think it would go a long way to helping more implementers work on the Kolab format in the future. What do you think the chances are of dedicating a developer or two for a few months to actually getting the spec up to scratch, and building the above 3a and 3b testing sets/tools? Cheers Rob Rob Mueller r...@fastmail.fm
