Re: competition
Adam Tauno Williams wrote: On Wed, 2010-09-22 at 18:18 +0200, Jeroen van Meeuwen (Kolab Systems) wrote: The scenario is integration, not extension of Cyrus -which in and of itself works perfecly fine and reliable for us. We're not seeking to improve Cyrus' performance with *SQL db backends. But, this assumes the data that Cyrus stores in that DB would be useful outside the context of the Cyrus' processes. I've lightly played with the idea, and not gone any further - the data available isn't really very useful. Well, for one, our ActiveSync implementation wants the following information; - List of (subscribed) IMAP folders, - Annotations, per IMAP folder, - Current status of the contents of such IMAP folder, such as new messages or deleted messages, in comparison to what the client currently holds, - Message contents. While connecting through the IMAP server and have Cyrus hand over the answers, and correlate such information on the side of the 3rd party application is perfectly feasible, I think it may be more efficient to correlate the requested information from a database directly, as opposed to attempting to cache the results handed over by Cyrus by each 3rd party application. Imagine the following scenario; a client polls 3rd party application for a list of mailboxes and the content's status very regularly -basically what it's interested in knowing is whether anything changed. Doesn't condstore solve this issue inexpensively? [maybe I misunderstand condstore]. I thought it was equivalent to WebDAV/CalDAV ctags (which are mightily nice). I'm not sure whether the IMAP server's capabilities with regards to modification sequences has anything to do with this thread, but maybe I'm misunderstanding the IMAP CONDSTORE extension. Each 3rd party app will seek to cache the results somehow, for improved performance interacting with its clients and as to not continuously put load on the Cyrus server. Which is what WebDAV/CalDAV ctags are for. The WebDAV/CalDAV scenario doesn't really fly with mailboxes. For one, mailboxes tend to have plenty more folders and plenty more messages. The question is not how the 3rd party app *can* get the needed information, the question is how many 3rd party apps can be integrated *most efficiently* (both in terms of performance/lower overhead, as well as architecture and 3rd party app's design -DYI cache for each 3rd party app?). Kind regards, -- Jeroen van Meeuwen Senior Engineer, Kolab Systems AG e: vanmeeu...@kolabsys.com t: +316 42 801 403 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Mon, Sep 27, 2010 at 09:38:08AM -0400, Jeff Eaton wrote: Better to just use an internal DB codebase (like skiplists) that has nothing to do with Sleepycat. But then someone has to write and maintain this code. I think the best compromise I've heard yet is to use something like skiplists by default and make the use of libdb an optional feature like the use of mysql. I'd be in favor of deprecating the use of BerkeleyDB in Cyrus. Making at least the default for new installations to be all-skiplist would be a good step in that direction, as well as encouraging people to move away from BDB where possible. Just committed this change to dev/2.4.0-alpha1. Along with the code that automagically converts databases on ctl_cyrusdb -r, this should make most sites that are running the default layout transparently start running all-skiplist without even realising. Ideally, I'd suggest making the default behavior of ./configure; make; make install to _not_ link in BDB at all unless --with-bdb is explicitly given. As I recall, you have to jump through a few hoops to get a build of Cyrus to not link in the BDB libraries, even if you're not using them. I haven't tried this. Maybe I should. All of that said, it's probably worth directing future Cyrus DB backend work away from BDB and more to improving SQLite or other integration. What about Hamster, or any of the other similar things? What about memcached for tls, pts and statuscache. Worth looking at. By the way - I'm travelling for the next few weeks, including meeting Ken and Dave in Buffalo mid next month. We have some new programmers starting with FastMail/Opera in Melbourne while I'm away, and I'm hoping to dedicate a large part of one of them's time to improving Cyrus. My shortlist of tasks to look at while I'm away is: 1) figure out what's up with BDB 2) seen db cleaner. Remove stale entries from seen databases (including entries for user's own mailboxes now that the owner's seen is absorbed back into the cyrus.index) Apart from these two things, I suspect most of the work we'll do on Cyrus over the next few months will be quite tactical in nature. Targetted towards making cross-datacentre replication better, and working towards supporting conversations (cross folder threads) in an efficient way. And there'll be a fair bit of work on the supporting code that manages replication from the outside too. Possibly pulling some back into a C daemon within Cyrus that can track replication status and monitor how the replication is running. Particularly of interest here is safe failover, and knowing when a replica is complete - so if you bring up a new replica, knowing when all the users have been replicated and rolling replication is close enough that you can shut down and switch. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Mon, Sep 27, 2010 at 09:38:08AM -0400, Jeff Eaton wrote: Better to just use an internal DB codebase (like skiplists) that has nothing to do with Sleepycat. But then someone has to write and maintain this code. I think the best compromise I've heard yet is to use something like skiplists by default and make the use of libdb an optional feature like the use of mysql. I'd be in favor of deprecating the use of BerkeleyDB in Cyrus. Making at least the default for new installations to be all-skiplist would be a good step in that direction, as well as encouraging people to move away from BDB where possible. Just committed this change to dev/2.4.0-alpha1. Along with the code that automagically converts databases on ctl_cyrusdb -r, this should make most sites that are running the default layout transparently start running all-skiplist without even realising. Ideally, I'd suggest making the default behavior of ./configure; make; make install to _not_ link in BDB at all unless --with-bdb is explicitly given. As I recall, you have to jump through a few hoops to get a build of Cyrus to not link in the BDB libraries, even if you're not using them. I haven't tried this. Maybe I should. Oh yes please, would be great being able to build completely without BDB. Even better if it would be possible to still build cvt_cyrusdb with BDB support so one can convert things. Or, if database backends could be compiled as loadable plugins, just imagine :) Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 18:18 +0200, Jeroen van Meeuwen (Kolab Systems) wrote: Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. What makes you think the query parsing and other overheads of SQL will be faster than IMAP? I'd be interested in any numbers that you might have to support the effort. The scenario is integration, not extension of Cyrus -which in and of itself works perfecly fine and reliable for us. We're not seeking to improve Cyrus' performance with *SQL db backends. But, this assumes the data that Cyrus stores in that DB would be useful outside the context of the Cyrus' processes. I've lightly played with the idea, and not gone any further - the data available isn't really very useful. Imagine the following scenario; a client polls 3rd party application for a list of mailboxes and the content's status very regularly -basically what it's interested in knowing is whether anything changed. Doesn't condstore solve this issue inexpensively? [maybe I misunderstand condstore]. I thought it was equivalent to WebDAV/CalDAV ctags (which are mightily nice). Each 3rd party app will seek to cache the results somehow, for improved performance interacting with its clients and as to not continuously put load on the Cyrus server. Which is what WebDAV/CalDAV ctags are for. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Better to just use an internal DB codebase (like skiplists) that has nothing to do with Sleepycat. But then someone has to write and maintain this code. I think the best compromise I've heard yet is to use something like skiplists by default and make the use of libdb an optional feature like the use of mysql. I'd be in favor of deprecating the use of BerkeleyDB in Cyrus. Making at least the default for new installations to be all-skiplist would be a good step in that direction, as well as encouraging people to move away from BDB where possible. Ideally, I'd suggest making the default behavior of ./configure; make; make install to _not_ link in BDB at all unless --with-bdb is explicitly given. As I recall, you have to jump through a few hoops to get a build of Cyrus to not link in the BDB libraries, even if you're not using them. All of that said, it's probably worth directing future Cyrus DB backend work away from BDB and more to improving SQLite or other integration. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Andy Bennett andy...@ashurst.eu.org wrote: Hi, Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. What makes you think the query parsing and other overheads of SQL will be faster than IMAP? I'd be interested in any numbers that you might have to support the effort. The scenario is integration, not extension of Cyrus -which in and of itself works perfecly fine and reliable for us. We're not seeking to improve Cyrus' performance with *SQL db backends. Imagine the following scenario; a client polls 3rd party application for a list of mailboxes and the content's status very regularly -basically what it's interested in knowing is whether anything changed. Each 3rd party app will seek to cache the results somehow, for improved performance interacting with its clients and as to not continuously put load on the Cyrus server. Our idea is to prevent that caching from needing to happen, and needlessly be duplicated technology across 3rd party apps, by using what Cyrus would consider it's live data in SQL as the 3rd party app's cache. Now, I don't have any numbers to compare while there is no Cyrus SQL db backend for the relevant databases. I'm just thinking that a single database query -if it could provide accurate status info- can be more efficient -to the Cyrus server(s) itself as well as the 3rd party app- then folderlist, iterate, request status info, parse, only to ultimately throw back at the client there's no changes -which is the result most of the time. It'd also eliminate duplicating attempts to cache folderlist and status results somehow, and would ultimately improve the scalability of such 3rd party apps as part of the infra they require to be shared..., since its cache is in SQL, etc. etc. The big downside to using an SQL database is the enormous temptation to point all the Cyrus servers at the same Database server and lose the redundancy and scalability inherent in a multi node or Murder setup. One would set up the database backend server infrastructure just as much conform to H/A and L/B requirements as the rest of the Cyrus/groupware infrastructure, not unlike how you would set up a sustainable database backend server infrastructure in any other type of critical environment. -- Jeroen Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
For situations where we need just random access, not sequential, can we use GDBM? Is that library better than Berkeley DB? ^ G = GNU = GPL. Licencing issues I suspect. We're BSD licence, not GPL. Yes, you're quite right, I just checked. Till your comment, I had assumed that GDBM would of course, obviously be offered under the LGPL. But it seems that the free software activism has brought even these small libraries under GPL, much like MySQL client libraries. No further questions. :) Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 9/22/2010 10:20 PM, Shuvam Misra wrote: I was a strong advocate of bundling DB libraries, etc, with Cyrus. The points you've made here are very interesting. I didn't know many of these things. I'm re-thinking whether bundling is such a good idea now. Thanks. There's a lot to be said for the Mac OS X approach. However, as someone already pointed out, most distributions won't allow this (for many reasons), so it's pointless to even talk about going there. Better to just use an internal DB codebase (like skiplists) that has nothing to do with Sleepycat. But then someone has to write and maintain this code. I think the best compromise I've heard yet is to use something like skiplists by default and make the use of libdb an optional feature like the use of mysql. For situations where we need just random access, not sequential, can we use GDBM? Is that library better than Berkeley DB? Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sat, Sep 25, 2010 at 01:46:10PM +0530, Shuvam Misra wrote: For situations where we need just random access, not sequential, can we use GDBM? Is that library better than Berkeley DB? ^ G = GNU = GPL. Licencing issues I suspect. We're BSD licence, not GPL. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 9/22/2010 10:20 PM, Shuvam Misra wrote: I was a strong advocate of bundling DB libraries, etc, with Cyrus. The points you've made here are very interesting. I didn't know many of these things. I'm re-thinking whether bundling is such a good idea now. Thanks. There's a lot to be said for the Mac OS X approach. However, as someone already pointed out, most distributions won't allow this (for many reasons), so it's pointless to even talk about going there. Better to just use an internal DB codebase (like skiplists) that has nothing to do with Sleepycat. But then someone has to write and maintain this code. I think the best compromise I've heard yet is to use something like skiplists by default and make the use of libdb an optional feature like the use of mysql. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 9/20/2010 8:59 AM, Marc Patermann wrote: And still, if someone asks a mailing list (not here certainly) how to start with IMAPd, many people shout, to go with dovecot and not using Cyrus. Hi - A little late to this thread, but here are a couple of modest observations: 1. I have cyrus and dovecot installations (with postfix smtp). dovecot doesn't support an lmtp transport, which I finally decided was a deal killer. In our (~1000 user) dovecot system, we're using procmail to ferry messages from the smtp server to dovecot deliver. One very disturbing recent finding is that with some regularity messages are not being delivered with no notification to the sender -- they just get dropped. We think it's an ldap authentication problem, but it doesn't really matter what the cause is: mail should either be delivered or someone should find out that it wasn't withough having to snoop around in log files. Procmail is *supposed* to send this stuff back up the pipe when something goes wrong, but it's not happening. Using lmtp is clean and simple and affords the administrator a huge amount of flexibility when using postfix: mailbox_transport = lmtp:unix:/var/run/cyrus/socket/lmtp This doesn't work with dovecot; you have to use the mailbox_command. I think dovecot now has LMTP support in the latest (dev?) version. That said, after spending the summer on the cyrus-devel list, I have a lot of confidence in the work that particularly Bron Gondwana has been doing and think that with -- with an infusion of some clear documentation, a project I'm more than happy to contribute to -- cyrus can probably become the default open source imap server. Documentation is one thing, and dependencies like BDB another. But there is something else I guess, for servers which are not dedicated mail server, it would be really nice if one could install Cyrus and it just works for every user having an account on the system. That can be done with the autocreate patches but they are also not included in Cyrus. In fact, Cyrus could have two main modes, one for dedicated, sealed mail servers and one for allround servers with user accounts. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 07:50:25AM +0200, Simon Matter wrote: Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) What migration path? Cyrus 2.3 supports all of the same database backends that Cyrus 2.2 did. I don't know the Debian packages well but from a quick look I did long time ago it seems like those packages leave quite some work to do by the user of the packages. That also means there are more ways for a user to break things which may stop them making the 2.3 packages go in easily. To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and with the same config files as 2.2 used, everything will just work. You can't easily go back, because I believe that 2.3 will update cyrus.index files to a format which 2.2 doesn't recognize, but that shouldn't prevent anyone from upgrading. IIRC it's a little bit more complicated. Beside BDB, there are also some steps to do at upgrade, like compiling sieve scripts, possibly converting its encoding. With BDB backends you have to make sure everything fits correctly. Binaries need to be linked against the correct version of BDB and also the on disk BDB files need to be of the same version. Now, think you want to do a distribution upgrade which comes with the latest greatest BDB release and new Cyrus binaries which are using them, but your spool is still on the old release... We all know how this ends. The only solution - you can call it dirty workaround - I found for our RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus shutdown and convert them back on startup. That way the spool is always in a state which can be migrated once Cyrus is shutdown. Of course getting rid of any BDB in the default configuration of the RPM binaries has helped much to prevent BDB mess. Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. I've already changed ctl_cyrusdb -r (run on startup) to automatically convert between skiplist, bdb and bdb-hash formats based on reading the magic at the start of the file on disk. So the only thing really left to do is making sure we can upgrade BDB databases automatically. Sure downgrades will suck, but that's not quite so scary. given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I know it's generally considered bad taste, but it sure makes keeping in sync easier! Bron ( the downside being security issue tracking, of course ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 21.09.2010 23:15, schrieb Jeffrey T Eaton: Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) What migration path? Cyrus 2.3 supports all of the same database backends that Cyrus 2.2 did. To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and with the same config files as 2.2 used, everything will just work. You can't easily go back, because I believe that 2.3 will update cyrus.index files to a format which 2.2 doesn't recognize, but that shouldn't prevent anyone from upgrading. From a cyrus point of view this might be true, but in real upgrades most of the time also the DB version changes and here is one of the pitfalls (Not only for cyrus) Here the current statement from the debian guys. http://lists.alioth.debian.org/pipermail/pkg-cyrus-imapd-debian-devel/2010-August/003427.html André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: Documentation is one thing, and dependencies like BDB another. But there is something else I guess, for servers which are not dedicated mail server, it would be really nice if one could install Cyrus and it just works for every user having an account on the system. That can be done with the autocreate patches but they are also not included in Cyrus. In fact, Cyrus could have two main modes, one for dedicated, sealed mail servers and one for allround servers with user accounts. We're planning to include the autocreate patches in 2.4 as soon as we make sure they work nicely with virtdomains and murder. Making it work nicely with user accounts would be very handy. Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site friendly. I also want auto-recompilation of sieve scripts, and auto-fixing of broken .cache files ) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
--On 22. September 2010 16:10:15 +1000 Bron Gondwana br...@fastmail.fm wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. I wouldn't exactly call myself an expert, but I think I understand the basics of BDB operation. The biggest problem is that there may be multiple version of the BDB utilities installed. So you need to know which BDB version Cyrus was linked against and somehow figure out what the corresponding tools are on the box you're on. The tool are named db_stat, db_verify, db_upgrade etc., but it's not unusual to have db41_stat, db42_stat etc . The file utility is pretty useful, because it tells you the version of a BDB file. Unfortunately that version has nothing to to with the version number of the BDB libraries themselves. Instead they look like this: Berkeley DB (Hash, version 8, native byte-order) I suppose one could do this: - at startup, create a new BDB hash file using a Cyrus binary - check its version number using file - compare that to the existing BDB files - if they're different you need to perform an upgrade The upgrade itself *should* be doable using db_upgrade (using the version that goes with the libraries Cyrus is linked against), although I think there are cases where you have to use db_dump followed by db_load. The problem with the latter is that you need the dump from the *old* version of the libraries! given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I know it's generally considered bad taste, but it sure makes keeping in sync easier! That would certainly help with the issues I described above, but only if you either also include the utilities or have your own binaries that fulfill their function. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7sOFjStOGfiA.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 22 Sep 2010 09:12 +0200, Sebastian Hagedorn haged...@uni-koeln.de wrote: --On 22. September 2010 16:10:15 +1000 Bron Gondwana br...@fastmail.fm wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. I wouldn't exactly call myself an expert, but I think I understand the basics of BDB operation. The biggest problem is that there may be multiple version of the BDB utilities installed. So you need to know which BDB version Cyrus was linked against and somehow figure out what the corresponding tools are on the box you're on. The tool are named db_stat, db_verify, db_upgrade etc., but it's not unusual to have db41_stat, db42_stat etc . The file utility is pretty useful, because it tells you the version of a BDB file. Unfortunately that version has nothing to to with the version number of the BDB libraries themselves. Instead they look like this: Berkeley DB (Hash, version 8, native byte-order) Yeah, I can get that just by reading the value as byte 12 in the file. I pulled a copy of the BDB source and grabbed their magic numbers already. The thing is - I don't want to be calling out to the tools - I want to be doing it in_process with a Cyrus tool that's already compiled against the correct version of BDB. I suppose one could do this: - at startup, create a new BDB hash file using a Cyrus binary - check its version number using file - compare that to the existing BDB files - if they're different you need to perform an upgrade Oooh. Actually, that's a point. Create a temporary file of each type and pull the version field from it, then check each BDB file's version field. Hmm. That's quite viable actually. I like it :) The upgrade itself *should* be doable using db_upgrade (using the version that goes with the libraries Cyrus is linked against), although I think there are cases where you have to use db_dump followed by db_load. The problem with the latter is that you need the dump from the *old* version of the libraries! http://download.oracle.com/docs/cd/E17076_02/html/upgrading/index.html seems to have good information on what's changed in each version. On the topic of upgrades: http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/am_upgrade.html We do guarantee that any archived database may be upgraded using a current Berkeley DB software release and the DB-upgrade() method, and there is no need to step-wise upgrade the database using intermediate releases of Berkeley DB. Sites should consider archiving appropriate copies of their application or application sources if they may need to access archived databases without first upgrading them. given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I know it's generally considered bad taste, but it sure makes keeping in sync easier! That would certainly help with the issues I described above, but only if you either also include the utilities or have your own binaries that fulfill their function. No need I don't think if we do the little trick above to figure out BDB version and upgrade. Ideally, I'd like to not be using BDB because the incompatibilities are a killer. If we could just say use a backwards compatible format to BDB and have it readable by any 3.x + version, I'd be a very happy camper. Bron. -- Bron Gondwana br...@fastmail.fm Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote: On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: snipped Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site friendly. I also want auto-recompilation of sieve scripts, and auto-fixing of broken .cache files ) Now that you mention it, what would be needed to do to auto-recompile sieve- scripts? If that would work, I could edit the file directly in the respective location, rather then having to play with the sieve management tool or a MUA with sieve- support. :) -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:13:20AM +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 09:01:33 Bron Gondwana wrote: On Wed, Sep 22, 2010 at 08:02:35AM +0200, Simon Matter wrote: snipped Bron ( really trying to make Cyrus newbie-friendly as well as advanced-site friendly. I also want auto-recompilation of sieve scripts, and auto-fixing of broken .cache files ) Now that you mention it, what would be needed to do to auto-recompile sieve- scripts? If that would work, I could edit the file directly in the respective location, rather then having to play with the sieve management tool or a MUA with sieve- support. :) Hmm... I take it all back. We did this a while back, and it's the path to great sadness. Directly editing the file in place means we can't auto-replicate it, can't add CRC checks on the contents, etc. The files on disk are internal format files, not to be fiddled with directly. Use an API. Now the lack of a good command to copy a file into place, enable it, etc. I'm sold on that. A command line tool to install a new sieve file for a user would be handy. But no direct editing of files on disk please. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sep 22, 2010, at 1:50 AM, Simon Matter wrote: Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) What migration path? Cyrus 2.3 supports all of the same database backends that Cyrus 2.2 did. I don't know the Debian packages well but from a quick look I did long time ago it seems like those packages leave quite some work to do by the user of the packages. That also means there are more ways for a user to break things which may stop them making the 2.3 packages go in easily. To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and with the same config files as 2.2 used, everything will just work. You can't easily go back, because I believe that 2.3 will update cyrus.index files to a format which 2.2 doesn't recognize, but that shouldn't prevent anyone from upgrading. IIRC it's a little bit more complicated. Beside BDB, there are also some steps to do at upgrade, like compiling sieve scripts, possibly converting its encoding. To the best of my knowledge, the complied Sieve format has not changed since 2.2. With BDB backends you have to make sure everything fits correctly. Binaries need to be linked against the correct version of BDB and also the on disk BDB files need to be of the same version. Now, think you want to do a distribution upgrade which comes with the latest greatest BDB release and new Cyrus binaries which are using them, but your spool is still on the old release... We all know how this ends. This would be a problem anytime you upgrade BDB, independent of the version of Cyrus. If you are rolling out a new distribution with a new version of BDB, you will need to address this problem even if you don't upgrade Cyrus. (Unless of course, you continue to ship a Cyrus built against the _old_ BDB, in which case the argument is irrelevant). The only solution - you can call it dirty workaround - I found for our RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus shutdown and convert them back on startup. That way the spool is always in a state which can be migrated once Cyrus is shutdown. Of course getting rid of any BDB in the default configuration of the RPM binaries has helped much to prevent BDB mess. All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
--On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton jea...@cmu.edu wrote: All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. I'm pretty sure that's not true. *If* an application uses BDB's environment features (not all do!), you *cannot* simply upgrade, unless the application is programmed to perform the upgrade automatically. Otherwise you have to either use db_upgrade to convert the environment to the current version, or you use db_dump and db_load. There's a db_upgrade() call in the library that could be used, but it might not work in all cases: Errors The DB-upgrade() method may fail and return one of the following non-zero errors: DB_OLD_VERSION - The database cannot be upgraded by this version of the Berkeley DB software. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7s98D0p7NYL6.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote: --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton jea...@cmu.edu wrote: All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. I'm pretty sure that's not true. *If* an application uses BDB's environment features (not all do!), you *cannot* simply upgrade, unless the application is programmed to perform the upgrade automatically. Otherwise you have to either use db_upgrade to convert the environment to the current version, or you use db_dump and db_load. There's a db_upgrade() call in the library that could be used, but it might not work in all cases: Errors The DB-upgrade() method may fail and return one of the following non-zero errors: DB_OLD_VERSION - The database cannot be upgraded by this version of the Berkeley DB software. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. Hmm. I thought that I had updated BDB in the past. I probably just nuked the databases between updates, since we only used them for transient data (tls_sessions.db, deliver.db, ptscache,db). The rest of my point stands; changing the BDB version is a problem in and of itself, not changing the Cyrus version. I know of no reason why one couldn't update a Cyrus 2.2 installation to 2.3. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Sep 22, 2010, at 7:57 AM, Sebastian Hagedorn wrote: --On 22. September 2010 07:47:26 -0400 Jeffrey T Eaton jea...@cmu.edu wrote: All of that said, I believe that, in general, you can safely upgrade BDB. If you have a Cyrus installation using BDB X, you can drop in a new Cyrus using BDB Y, as long as everything is shut down in between. You can't go back without effort, but upgrades should work. I'm pretty sure that's not true. *If* an application uses BDB's environment features (not all do!), you *cannot* simply upgrade, unless the application is programmed to perform the upgrade automatically. Otherwise you have to either use db_upgrade to convert the environment to the current version, or you use db_dump and db_load. There's a db_upgrade() call in the library that could be used, but it might not work in all cases: Errors The DB-upgrade() method may fail and return one of the following non-zero errors: DB_OLD_VERSION - The database cannot be upgraded by this version of the Berkeley DB software. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - â +49-221-478-5587.:. Hmm. I thought that I had updated BDB in the past. I probably just nuked the databases between updates, since we only used them for transient data (tls_sessions.db, deliver.db, ptscache,db). You can upgrade between some versions but not all. The rest of my point stands; changing the BDB version is a problem in and of itself, not changing the Cyrus version. I know of no reason why one couldn't update a Cyrus 2.2 installation to 2.3. Well, I never said it's a problem of Cyrus, it's a problem of BDB. Just ask the OpenLDAP folks - and those doing distribution packages for it. The only way I see for Cyrus is to stay away from it. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22/09/2010, at 22:17, Adam Tauno Williams wrote: On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too. On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. Fine, that may all be true. But I never ever heard someone going from BDB to skiplist only and coming back because he had any performance issues. Maybe it doesn't matter so much with Cyrus and we should try to find out what is Poetry and Truth? If BDB is not really required for Cyrus it could be made optional for those who really want it. BTW, it's even worse than Bron said. You not only get errors in the logs even without using any BDB, you can also end up with a broken BDB environment which prevents Cyrus from starting up :) Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:27:11PM +0930, Daniel O'Connor wrote: On 22/09/2010, at 22:17, Adam Tauno Williams wrote: On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. I'm not sure it's Cyrus, I see issues with OpenLDAP and BDB too. On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Cheers, Ken Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22/09/2010, at 22:33, Kenneth Marshall wrote: On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au The nice thing about standards is that there are so many of them to choose from. -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: On 22/09/2010, at 22:33, Kenneth Marshall wrote: On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) Sadly, OpenLDAP does need the performance of BDB. Cheers, Ken Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 2010-09-22 at 15:04 +0200, Simon Matter wrote: On Wed, 2010-09-22 at 14:44 +0200, J. Roeleveld wrote: On Wednesday 22 September 2010 13:47:26 Jeffrey T Eaton wrote: snipped I am probably missing some info here, but And, as Bron has said, there's something wrong with the way Cyrus uses BDB. I've never been able to understand BDB well enough to figure it out myself, nor have I ever found anyone who can help. For what its worth, I solved the problem by not using BDB at all on the Cyrus systems I used. If it is possible to not use BDB, and BDB causes problems with upgrades, why is BDB still used then? BDB is wicked fast and scales well. At least that is the typical argument in defense of BDB. And given the stellar performance one sees from OpenLDAP I'm prone to believing it. Fine, that may all be true. But I never ever heard someone going from BDB to skiplist only and coming back because he had any performance issues. True; and we moved to Skiplist, and no BDB, a long time ago. I was just answering the question why is BDB still used then?. It may be some degree faster, but IMO isn't worth the agony of figuring out BDB issues. Maybe it doesn't matter so much with Cyrus and we should try to find out what is Poetry and Truth? If BDB is not really required for Cyrus it could be made optional for those who really want it. BTW, it's even worse than Bron said. You not only get errors in the logs even without using any BDB, you can also end up with a broken BDB environment which prevents Cyrus from starting up :) Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: On 22/09/2010, at 22:33, Kenneth Marshall wrote: On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) Sadly, OpenLDAP does need the performance of BDB. With current LDIF exports of OpenLDAP, the downtime caused by a corrupt BDB database is acceptable. Is a similar method possible with cyrus? Eg. a simple command that exports all BDB-database-files to a simple format that can easily be re-imported into Cyrus when the DBD is corrupted? -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: On 22/09/2010, at 22:33, Kenneth Marshall wrote: On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) Sadly, OpenLDAP does need the performance of BDB. With current LDIF exports of OpenLDAP, the downtime caused by a corrupt BDB database is acceptable. Is a similar method possible with cyrus? Eg. a simple command that exports all BDB-database-files to a simple format that can easily be re-imported into Cyrus when the DBD is corrupted? That's exactly what our RPM does. It converts all BDB databases to skiplist on shutdown. On startup, it detects the format of all database files and converts it back to what is configured if needed. As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My question remains, why can't we get rid of BDB completely? Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) Zinato On Wed, Sep 22, 2010 at 11:06 AM, Simon Matter simon.mat...@invoca.ch wrote: On Wednesday 22 September 2010 15:29:20 Kenneth Marshall wrote: On Wed, Sep 22, 2010 at 10:48:49PM +0930, Daniel O'Connor wrote: On 22/09/2010, at 22:33, Kenneth Marshall wrote: On a bad shutdown it requires admin intervention very frequently which is pretty tedious. And yes, upgrading it is also a PITA. That is why we moved to skiplist. The server would require manual intervention to even restart after certain types of system events. It is hard to explain why mail was unavailable for 30+mins. when the server rebooted in 2mins. BDB would need to be much, much easier to manage and be more robust for us to use. Yep, I plan to migrate to skiplist some day :) Now if only I could use skiplist with openldap ;) Sadly, OpenLDAP does need the performance of BDB. With current LDIF exports of OpenLDAP, the downtime caused by a corrupt BDB database is acceptable. Is a similar method possible with cyrus? Eg. a simple command that exports all BDB-database-files to a simple format that can easily be re-imported into Cyrus when the DBD is corrupted? That's exactly what our RPM does. It converts all BDB databases to skiplist on shutdown. On startup, it detects the format of all database files and converts it back to what is configured if needed. As Bron told us, with 2.4 the magic on startup is done inside Cyrus. My question remains, why can't we get rid of BDB completely? Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/22/2010 10:17 AM, Lucas Zinato Carraro wrote: For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) In 2.3.13 (I think) and newer, there is the option of using an SQL backend. It hasn't been widely used and tested yet, though. Along the lines of Simon's question, other than the fact that some of the Cyrus database defaults happen to be BDB, is anyone using BDB with Cyrus because they really want to? Considering the state of Cyrus' interoperability with BDB and all the recent fixes to skiplist, would it make sense to at least not make BDB a default backend from now on? Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) Beside interesting what would you get for a real benefit from this ? They are ver verly likely to be slower. André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/22/2010 10:52 AM, André Schild wrote: Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) Beside interesting what would you get for a real benefit from this ? They are ver verly likely to be slower. We wanted to use it for the user_deny database so we could insert a row into one database table that every host has access to. This way we didn't need to come up with a way to update the local user_deny across each frontend server. Also, we were interested in testing SQLite for things like tls_sessions.db and deliver.db because we were tired of BDB breaking our service. Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE : competition
We wanted to use it for the user_deny database so we could insert a row into one database table that every host has access to. This way we didn't need to come up with a way to update the local user_deny across each frontend server. Such database provides the same benefit to the tlscache DB. Our frontend servers share a Mysql DB (HEAP) to cache TLS Also, we were interested in testing SQLite for things like tls_sessions.db and deliver.db because we were tired of BDB breaking our service. Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, 22 Sep 2010, Bron Gondwana wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. All I know is that there used to be an API call to upgrade the db environment, which basically did exactly what the db_upgrade utility does. It was easy to find before Oracle bought SleepyCat and screwed over with the documentation, nowadays I don't even know where the API docs are. If it is still exists, you could probably upgrade all DBs the first time they're accessed (if the db-upgrade() call is cheap enough when there is no work to be done, something I don't know). So the only thing really left to do is making sure we can upgrade BDB databases automatically. Sure downgrades will suck, but that's not quite so scary. Downgrades can only be done manually anyway, by dump+restore. However, the file formats don't change as much as the API/ABI anyway, often two different BDB releases use the same file format. given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I That way lies madness. BDB is one of those things where arcane blackmagic skills are needed to keep it working on all arches. It uses scary crap to be fast and archive high performance with large concurrency, and I've seen it break OpenLDAP in very nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to mmap() ;-) ). And it will break the world if you get symbol versioning wrong. Lots of libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link BDB with your application, and all sort of non-funny crap can happen if the wrong linker magic is applied. You *really* don't want to go there. -- One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie. -- The Silicon Valley Tarot Henrique Holschuh Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. Kind regards, Jeroen van Meeuwen -From his Android André Schild an...@schild.ws wrote: Am 22.09.2010 16:17, schrieb Lucas Zinato Carraro: For me it would be very interesting a option to save cyrus tables in a traditional database. ( mysql, postgresql, etc... ) Beside interesting what would you get for a real benefit from this ? They are ver verly likely to be slower. André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Hi, Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. What makes you think the query parsing and other overheads of SQL will be faster than IMAP? I'd be interested in any numbers that you might have to support the effort. The big downside to using an SQL database is the enormous temptation to point all the Cyrus servers at the same Database server and lose the redundancy and scalability inherent in a multi node or Murder setup. Regards, @ndy -- andy...@ashurst.eu.org http://www.ashurst.eu.org/ 0x7EBA75FF Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
The big downside to using an SQL database is the enormous temptation to point all the Cyrus servers at the same Database server and lose the redundancy and scalability inherent in a multi node or Murder setup. But the SQL world has this figured out, at least for reads. For situations where DB *writes* need to be scaled, which the SQL world doesn't really have figured out, there is always the large list of up-and-coming noSQL solutions and they seem to fit Cyrus' use case pretty well. Cyrus' single-murder-master situation is a write bottleneck that could be alleviated with such a solution. John -- John Madden Sr UNIX Systems Engineer Ivy Tech Community College of Indiana jmad...@ivytech.edu Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 22 Sep 2010, at 10:46, Dave McMurtrie wrote: Considering the state of Cyrus' interoperability with BDB and all the recent fixes to skiplist, would it make sense to at least not make BDB a default backend from now on? Yes, and sane defaults was to be one of the themes of the 2.4 release. Very pertinent if you're interested in making Cyrus IMAPd welcoming to new people... :wes Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. It was quite some time ago I last upgraded a Cyrus instance, but my upgrade script did the following: - Run the _old_ version of db_recover. Just shutting down Cyrus is not enough, without this step db_upgrade can fail. - Run the _new_ version of db_upgrade. - Run db_checkpoint -1 to force the log to be rotated. I think it won't be hard to implement steps 2 3 inside Cyrus, and I don't remember those steps taking a significant amount of time, but I don't know how it would be possible to implement step 1. However if the BDB version change does not bring a change to the on-disk DB log format, then it may be possible to skip steps 1 3. Gabor Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 11:24:04PM +0200, Gabor Gombas wrote: On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote: Now - BDB database SHOULD be upgradable. I want to find a BDB expert to help me with that - (a) detecting that an upgrade is necessary, and (b) doing the upgrade. It was quite some time ago I last upgraded a Cyrus instance, but my upgrade script did the following: - Run the _old_ version of db_recover. Just shutting down Cyrus is not enough, without this step db_upgrade can fail. - Run the _new_ version of db_upgrade. - Run db_checkpoint -1 to force the log to be rotated. I think it won't be hard to implement steps 2 3 inside Cyrus, and I don't remember those steps taking a significant amount of time, but I don't know how it would be possible to implement step 1. However if the BDB version change does not bring a change to the on-disk DB log format, then it may be possible to skip steps 1 3. Ok - so the upshot is that we need to implement a shutdown path, either in master.c or in ctl_cyrusdb - and we need to make sure it gets run during the usual shutdown path. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
given the issues with BDB. Is it worth embedding a copy of BDB into the Cyrus distribution rather than using the OS one? I That way lies madness. BDB is one of those things where arcane blackmagic skills are needed to keep it working on all arches. It uses scary crap to be fast and archive high performance with large concurrency, and I've seen it break OpenLDAP in very nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to mmap() ;-) ). And it will break the world if you get symbol versioning wrong. Lots of libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link BDB with your application, and all sort of non-funny crap can happen if the wrong linker magic is applied. You *really* don't want to go there. I was a strong advocate of bundling DB libraries, etc, with Cyrus. The points you've made here are very interesting. I didn't know many of these things. I'm re-thinking whether bundling is such a good idea now. Thanks. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Thu, 23 Sep 2010, Shuvam Misra wrote:
given the issues with BDB. Is it worth embedding a copy of
BDB into the Cyrus distribution rather than using the OS one? I
That way lies madness.
BDB is one of those things where arcane blackmagic skills are needed to keep
it working on all arches. It uses scary crap to be fast and archive high
performance with large concurrency, and I've seen it break OpenLDAP in very
nasty arch-specific ways in the past (OpenLDAP is to BDB what Cyrus is to
mmap() ;-) ).
And it will break the world if you get symbol versioning wrong. Lots of
libraries (SASL *and* glibc included!) might decide to shadow-dynamic-link
BDB with your application, and all sort of non-funny crap can happen if the
wrong linker magic is applied. You *really* don't want to go there.
I was a strong advocate of bundling DB libraries, etc, with Cyrus. The
points you've made here are very interesting. I didn't know many of these
things. I'm re-thinking whether bundling is such a good idea now. Thanks.
Bundling is always a pain to get exactly right.
Bundled libs are also a security hazard. Bundled libs *will* be disabled by
any Linux distro worth being called that, but even disabled they are not
harmless. They will waste resources of the security teams while acessing if
an embedded copy of whatever lib needs to be updated with a security fix.
Bundling also complicate the build system. Complicated build systems are a
VERY BAD idea on most projects, where nobody is really paying attention to
them.
--
One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie. -- The Silicon Valley Tarot
Henrique Holschuh
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Kolab Systems is thinking of such SQL databases for integration purposes, where the performance penalty now lies within having to use the IMAP protocol to gain access to the underlying metadata (seen status, message indexes) in distributed groupware environments where Cyrus itself is not the only component that needs access to such information (but would still remain authoritative, of course, and would be the only component with write access to most tables). While not necessarily the best approach, it seems worth exploring. Yes, this is one of the big reasons why we use RDB backends in many of our products. The downside is performance, of course. But the upside is very big when you need it -- ability to keep one instance of important metadata and allow multiple pieces of software on multiple servers to access it. Most of the time, this metadata is read-often-update-rarely, so we don't need huge write performance. But if we have to synchronise the data across multiple servers and use local files like SQLite/BerkeleyDB/gdbm, we land up building entire synchronisation layers just to ship updates around from server to server. This is not Cyrus-specific -- any metadata which needs sharing across multiple servers seems to be easier to build with a network-accessible database, and an RDB is the commonest of them (SQLite being an exception). Basically, memcached is better than BerkeleyDB hashmaps in these situations. Shuvam Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Wed, Sep 22, 2010 at 04:10:15PM +1000, Bron Gondwana wrote:
Now - BDB database SHOULD be upgradable. I want to find a BDB expert
to help me with that - (a) detecting that an upgrade is necessary, and
(b) doing the upgrade.
It was quite some time ago I last upgraded a Cyrus instance, but my
upgrade script did the following:
- Run the _old_ version of db_recover. Just shutting down Cyrus is not
enough, without this step db_upgrade can fail.
- Run the _new_ version of db_upgrade.
- Run db_checkpoint -1 to force the log to be rotated.
I think it won't be hard to implement steps 2 3 inside Cyrus, and I
don't remember those steps taking a significant amount of time, but I
don't know how it would be possible to implement step 1. However if the
BDB version change does not bring a change to the on-disk DB log format,
then it may be possible to skip steps 1 3.
Unfortunately that's not enough in many situations. I did uncountable
tests with different BDB versions and came up with the following method in
our RPMs:
...
# make sure our Berkeley databases are in a sane state
# wait for db_checkpoint to end successfully or kill it after a timeout
db_checkpoint -v -1 -h ${imap_prefix}/db
DB_CHECK_PID=$!
CNT=0
while [ $CNT -lt 60 ]; do
if ! kill -0 $DB_CHECK_PID /dev/null 21; then
break
fi
sleep 1
let CNT+=1
done
if kill -0 $DB_CHECK_PID /dev/null 21; then
kill -USR1 $DB_CHECK_PID /dev/null 21
sleep 1
kill -KILL $DB_CHECK_PID /dev/null 21
wait $DB_CHECK_PID /dev/null 21
fi
# do a normal recovery
db_recover -v -h ${imap_prefix}/db
RETVAL=$?
if [ $RETVAL -ne 0 ]; then
# try a catastrophic recovery instead of normal recovery
db_recover -v -c -h ${imap_prefix}/db
RETVAL=$?
ERRVAL=$(( $ERRVAL + $RETVAL ))
if [ $RETVAL -ne 0 ]; then
echo ERROR: catastrophic recovery of Berkeley databases failed
fi
fi
if [ $EXPORT = export ]; then
# convert all db files to portable format for migration
# TODO: quota_db, we don't touch it for now
cvt_file ${imap_prefix}/deliver.db skiplist
cvt_file ${imap_prefix}/mailboxes.db skiplist
cvt_file ${imap_prefix}/tls_sessions.db skiplist
cvt_file ${imap_prefix}/annotations.db skiplist
cvt_file ${imap_prefix}/ptclient/ptscache.db skiplist
cvt_file ${imap_prefix}/statuscache.db skiplist
cvt_file ${imap_prefix}/user_deny.db flat
rm -vf ${imap_prefix}/db/log.*
rm -vf ${imap_prefix}/db/__db.*
else
...
This all is done not for upgrades of BDB but for simple cases where BDB is
in a bad state for some reasons.
Now, the db_upgrade may be good for those situations where it works. But
I'm quite sure it doesn't work between some major versions of BDB. And
that's exactly what we had to care in the RPMs, so you can run RHEL2.1 on
i386 and upgrade it to RHEL5 on x86_64 and it will just work.
It works, but I'd be very happy to get rid of this stuff, really!
Simon
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Le 20 sept. 2010 à 15:59, Marc Patermann a écrit : Hi, where does Cyrus IMAPd stand today? When I was starting to think about moving to a open source mail system (migrating away from Lotus Domino btw.), there ware Cyrus IMAPd, Courier and UW-IMAP I think. Cyrus was the only full flavored IMAP server with active development. We were going the 2.2. path, while 2.3 seemed to fresh. So there was development. On the other side there were still many people complaining about Cyrus being too complex and too unstable with all the BDB fiddlings. Then dovecot emerged and quickly evolved. I don't know why, I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? For me, this is a typical open source crowd behaviour. If one product tends to have better results, change immediately on the faster running train (which is believed to run faster) and to dump the reliable solution immediately, beginning to do advocacy in all forums, newsgroups and discussion groups to tell people that the new product is even better than sliced bread. Today it is dovecot, tomorrow it can be insert new name of a superb new blinkenlights imap server project here. This kind of advocacy kills capacities, kills work time (because you have to tell to your boss why it is not a good idea to dump the existing installation only because another linux guy plotted out some coloured powerpoint slides stating that dovecot is the way to go) and focusses on the wrong side of the story. Cyrus has been proven to be reliable for over ten years. The security record for Cyrus is quite high. It runs perfectly on our infrastructure. It is still actively developed. Until now, it does not hurt any performance barrier and staff is trained with it. Just to change because numbers seem to be better or an IMHO arrogant founder offers 1000 euros if you can hack my server (which is like being in a kindergarden from my point of view) is not a viable option. Image the cost of training staff to a new system. And even a easy migration path often turns out to get downtime. Downtime - for nothing you get in plus - no service will be better - it is far worse: as your staff isn't experienced with dovecot you likely will do more errors in administering ist. As for a new installation, that can be another case. We are very satisfied with the performance and flexibility Cyrus IMAP gives to us. There's no need to change apart from being in head with the open source advocacy croud. -- Pascal Gienger Jabber/XMPP/Mail: pascal.gien...@uni-konstanz.de University of Konstanz, IT Services Department (Rechenzentrum) Electronic Communications and Web Services Building V, Room V404, Phone +49 7531 88 5048, Fax +49 7531 88 3739 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 21.09.2010 09:31, schrieb Pascal Gienger: I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? I think we have two different cases to consider: 1. Experienced users, running a cyrus installation for several months/years. Here I see no danger that they switch over to another imap project, unless that project provides better overall experience for the them. Running cyrus since ~8 years now, we don't intend to switch to another project. 2. Newbies, which do not know cyrus, they just ask for advice and a simple starter experience with the imap server. In the last month there has been much posts of type Dovecot is the greates/simples/best/etc. So the new users are very likely to go to dovecot instead of cyrus, because of advices from other internet users. So I think we should take care, that cyrus is not loosing new users, because the userbase will shrink over the years, compared to dovecot. One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) Also the SASL stuff is not first-user friendly (But very powerfull when understood) So helping new users in this area would probably also help for getting cyrus recommendations for new users. André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: competition
where does Cyrus IMAPd stand today? When I was starting to think about moving to a open source mail system (migrating away from Lotus Domino btw.), there ware Cyrus IMAPd, Courier and UW-IMAP I think. Cyrus was the only full flavored IMAP server with active development. We were going the 2.2. path, while 2.3 seemed to fresh. So there was development. On the other side there were still many people complaining about Cyrus being too complex and too unstable with all the BDB fiddlings. Then dovecot emerged and quickly evolved. I don't know why, I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? For me, this is a typical open source crowd behaviour. If one product tends to have better results, change immediately on the faster running train (which is believed to run faster) and to dump the reliable solution immediately, beginning to do advocacy in all forums, newsgroups and discussion groups to tell people that the new product is even better than sliced bread. Personally I like the competition, it encourage the continuous improvement of open source software. Dovecot has some good ideas or design solutions (see this rackspace document about Dovecot. It indicates that Apple did the switch from Cyrus to Dovecot http://www.dovecot.org/talks/macsysadmin-2009-09-16.pdf ) We run Cyrus for years and are happy with it. We host nearly 30 million mailboxes on our biggest platform. But begin to have mailboxes with hundreds of thousands of emails and it becomes difficult to balance the storage. I don't know what is really sharding in dovecot, but I would like to have a virtual storage in Cyrus (emails localized in an IMAP folder stored on several storage nodes). Sébastien Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité du groupe Atos Origin ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
I don't know, where this bad karma is coming from - I'm still happy with I guess it's simply because for many years there were no clean packages for the most used operating systems. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Hi, André Schild schrieb am 21.09.2010 10:40 Uhr: Am 21.09.2010 09:31, schrieb Pascal Gienger: I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? Yes, I'm too. But this is what you see in forums and mailing list. And my post is intended to see what the Cyrus community can do to /explain/ the product any better. 2. Newbies, which do not know cyrus, they just ask for advice and a simple starter experience with the imap server. In the last month there has been much posts of type Dovecot is the greates/simples/best/etc. So the new users are very likely to go to dovecot instead of cyrus, because of advices from other internet users. This is my experience, too. One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? - Everyone is talking about Push-Mail - how to do it with Cyrus? - What if I want Groupware later, can I still use Cyrus? (link to projects supporting Cyrus, OX etc) - I want to have virtual users in LDAP/RDBMS - how to do it? There may be much more basic task which may be well know to us but would be interesting to newbies to know. IMHO the What is Cyrus? page should be worked over. Why do we not talk about who is using Cyrus IMAPd and how - Success Stories (fastmail.fm?)? Just my 2 Cent Marc Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Am 21.09.2010 11:35, schrieb Simon Matter: I don't know, where this bad karma is coming from - I'm still happy with I guess it's simply because for many years there were no clean packages for the most used operating systems. Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) André Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote: Hi, André Schild schrieb am 21.09.2010 10:40 Uhr: Am 21.09.2010 09:31, schrieb Pascal Gienger: I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? Yes, I'm too. But this is what you see in forums and mailing list. And my post is intended to see what the Cyrus community can do to /explain/ the product any better. Be there is the only thing I can think of. The only way to counter advocates is to be one. 2. Newbies, which do not know cyrus, they just ask for advice and a simple starter experience with the imap server. In the last month there has been much posts of type Dovecot is the greates/simples/best/etc. So the new users are very likely to go to dovecot instead of cyrus, because of advices from other internet users. This is my experience, too. Yep. Also, for some reason, many people have the opinion that Cyrus is 'arcane' in some way. I honestly have no idea where that comes from except perhaps from the dreadful state of the documentation. But over the years I've been able to fairly easily figure out what I need to know. One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? Ugh, please no. Distro specific directions are lame, pointless, and unmaintainable [beyond possibly packages-are-here]. Once the packages are installed setting up is essentially the same everywhere. - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? - Everyone is talking about Push-Mail - how to do it with Cyrus? It just happens if you your client supports IMAP idle. K9 on the Android seems to work very well with Cyrus in the 'push' mode. Although I've noticed users usually turn it off within a couple of weeks - really, nobody wants push-mail once they have it. - What if I want Groupware later, can I still use Cyrus? (link to projects supporting Cyrus, OX etc) Shameless self promotion -I have a chapter on Cyrus in WMOGAG (OpenGroupware Administrator's Guide) http://docs.opengroupware.org/Members/whitemice/wmogag/file_view I've gotten quite a few compliments on it. - I want to have virtual users in LDAP/RDBMS - how to do it? There may be much more basic task which may be well know to us but would be interesting to newbies to know. IMHO the What is Cyrus? page should be worked over. Why do we not talk about who is using Cyrus IMAPd and how - Success Stories (fastmail.fm?)? -- Adam Tauno Williams awill...@whitemice.org LPIC-1, Novell CLA http://www.whitemiceconsulting.com OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Re: competition
Am Dienstag, den 21.09.2010, 11:48 +0200 schrieb André Schild: Am 21.09.2010 11:35, schrieb Simon Matter: I don't know, where this bad karma is coming from - I'm still happy with I guess it's simply because for many years there were no clean packages for the most used operating systems. Debian is still stuck on 2.2 and there seems to be no progress in that area. Most people like to use versions from repositories. For obvious reasons. Might make sense for cyrus to host their own repository. If it's just a simple entry in sources.list(.d) more people would use the recent version. Greetings, Syren Baran +++ Niederlassung Deutschland: BIT-HOUSE Ltd. internet professionals Erthalstr. 17 63739 Aschaffenburg Deutschland Amtsgericht:Aschaffenburg HRB-Nr.:9136 St/Nr.: 204/104/60603 USt-Id-Nr.: DE814408164 Gechäftsführer: Thomas Witzel +++ Telefon:+49.6021.8622-680 VoIP: +49.6021.8622-680 Fax:+49.6021.8622-676 E-Mail: m...@bit-house.com Homepage: www.bit-house.com +++ Hauptsitz: BIT-HOUSE Ltd. 69 Great Hampton Street Birmingham West Midlands United Kingdom Registergericht: Companies House of Cardiff Registernummer: 05325636 Registereintrag: United Kingdom Director: Thomas Witzel +++ attachment: winmail.dat Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Tuesday 21 September 2010 11:44:45 Marc Patermann wrote: Hi, André Schild schrieb am 21.09.2010 10:40 Uhr: Am 21.09.2010 09:31, schrieb Pascal Gienger: snipped One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? - Everyone is talking about Push-Mail - how to do it with Cyrus? - What if I want Groupware later, can I still use Cyrus? (link to projects supporting Cyrus, OX etc) - I want to have virtual users in LDAP/RDBMS - how to do it? There may be much more basic task which may be well know to us but would be interesting to newbies to know. Don't forget one of the most important details (which I think should be listed right after how to install / configure): - How do I backup and restore my email? (eg. what needs to be backed up and what can be rebuild) Just my 2 Cent And mine, makes 4 cents :) -- Joost Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Hi, Adam Tauno Williams schrieb am 21.09.2010 12:04 Uhr: On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote: André Schild schrieb am 21.09.2010 10:40 Uhr: Am 21.09.2010 09:31, schrieb Pascal Gienger: I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? Yes, I'm too. But this is what you see in forums and mailing list. And my post is intended to see what the Cyrus community can do to /explain/ the product any better. Be there is the only thing I can think of. The only way to counter advocates is to be one. Yes, Be there - but Be prepared and have something to point to: See it is easy and all there, dude! One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? Ugh, please no. Distro specific directions are lame, pointless, and unmaintainable [beyond possibly packages-are-here]. Yes, you're right. But in many cases it just point to somewhere else, which might be better than having nothing. For Ubuntu there is a howto: http://wiki.ubuntuusers.de/Cyrus_IMAPD and what do you get? 2.2.13 - uargh: http://packages.ubuntu.com/lucid/cyrus-imapd-2.2 That may not be, what we want. Once the packages are installed setting up is essentially the same everywhere. Yes. But install is always first. If you do not get the user to get it installed, everything else is useless. If you were a newbie and most people were trying to talk you away from Cyrus and there is no deb or rpm, would you try to compile the software yourself? - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? - Everyone is talking about Push-Mail - how to do it with Cyrus? It just happens if you your client supports IMAP idle. _I_ know that. :) It is just one example what I think user could search for. And Push mail IMHO was one of the email buzz words of the last few years. K9 on the Android seems to work very well with Cyrus in the 'push' mode. Although I've noticed users usually turn it off within a couple of weeks - really, nobody wants push-mail once they have it. I disagree. - What if I want Groupware later, can I still use Cyrus? (link to projects supporting Cyrus, OX etc) Shameless self promotion -I have a chapter on Cyrus in WMOGAG (OpenGroupware Administrator's Guide) http://docs.opengroupware.org/Members/whitemice/wmogag/file_view I've gotten quite a few compliments on it. You could put your Shameless self promotion in the wiki then. :) Marc Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Hi, [...] The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? I think this may also be a problem: Where to get up to date packages? In Debian they are still on Version 2.2.13. And Ubuntu probably also. And as Squeeze will be finished sooner or later, they will still have the 2.2.13 packages in their repositories. Having a repository with the latest 2.3 and then the 2.4 cyrus versions would be a good thing. This helps in getting a working cyrus server for not only newbie users. But I know that keeping a seperate repository is quite a lot of work. Best regards, Claus Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Tue, 2010-09-21 at 14:14 +0200, Marc Patermann wrote: Hi, Adam Tauno Williams schrieb am 21.09.2010 12:04 Uhr: On Tue, 2010-09-21 at 11:44 +0200, Marc Patermann wrote: André Schild schrieb am 21.09.2010 10:40 Uhr: Am 21.09.2010 09:31, schrieb Pascal Gienger: I begin to be tired from this dovecot is much more besser, you HAVE TO USE IT, why don't you migrate, ... ...? Yes, I'm too. But this is what you see in forums and mailing list. And my post is intended to see what the Cyrus community can do to /explain/ the product any better. Be there is the only thing I can think of. The only way to counter advocates is to be one. Yes, Be there - but Be prepared and have something to point to: See it is easy and all there, dude! One important thing is the documentation of the imap server, and there cyrus could offer more (Just my opinion) The new web site is a good start. We should start a best practice section in the wiki. - How do I install Cyrus on Debian/Ubuntu/...? Ugh, please no. Distro specific directions are lame, pointless, and unmaintainable [beyond possibly packages-are-here]. Yes, you're right. But in many cases it just point to somewhere else, which might be better than having nothing. For Ubuntu there is a howto: http://wiki.ubuntuusers.de/Cyrus_IMAPD and what do you get? 2.2.13 - uargh: http://packages.ubuntu.com/lucid/cyrus-imapd-2.2 That may not be, what we want. It is an Ubuntu wiki - *of course* it is out of date. And probably wrong/bad in the first place. Once the packages are installed setting up is essentially the same everywhere. Yes. But install is always first. If you do not get the user to get it installed, everything else is useless. If you were a newbie and most people were trying to talk you away from Cyrus and there is no deb or rpm, would you try to compile the software yourself? There are RPMs. The first Bing result for cyrus-imapd rpm. http://www.invoca.ch/pub/packages/cyrus-imapd/ So, while I understand your point, I don't think the situation is as dire as building-from-source. - From single server to multi server? - How to start with partitions and why? - Where to get latest releases (rpm/deb) when is not in my distriburtion (simon matter etc.)? - Everyone is talking about Push-Mail - how to do it with Cyrus? It just happens if you your client supports IMAP idle. _I_ know that. :) It is just one example what I think user could search for. And Push mail IMHO was one of the email buzz words of the last few years. K9 on the Android seems to work very well with Cyrus in the 'push' mode. Although I've noticed users usually turn it off within a couple of weeks - really, nobody wants push-mail once they have it. I disagree. That IMAP idle works, or that people don't actually want push? - What if I want Groupware later, can I still use Cyrus? (link to projects supporting Cyrus, OX etc) Shameless self promotion -I have a chapter on Cyrus in WMOGAG (OpenGroupware Administrator's Guide) http://docs.opengroupware.org/Members/whitemice/wmogag/file_view I've gotten quite a few compliments on it. You could put your Shameless self promotion in the wiki then. :) It certainly isn't an 'official' Cyrus project document, so I'm hesitant to do so. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) What migration path? Cyrus 2.3 supports all of the same database backends that Cyrus 2.2 did. To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and with the same config files as 2.2 used, everything will just work. You can't easily go back, because I believe that 2.3 will update cyrus.index files to a format which 2.2 doesn't recognize, but that shouldn't prevent anyone from upgrading. -jeaton Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
Debian is still stuck on 2.2 and there seems to be no progress in that area. The main problem they apparently have, is the migration path for the various DB files from 2.2 to 2.3. (The 2.3 version itself works fine as .deb packages) What migration path? Cyrus 2.3 supports all of the same database backends that Cyrus 2.2 did. I don't know the Debian packages well but from a quick look I did long time ago it seems like those packages leave quite some work to do by the user of the packages. That also means there are more ways for a user to break things which may stop them making the 2.3 packages go in easily. To the best of my knowledge, you can drop in Cyrus 2.3 binaries, and with the same config files as 2.2 used, everything will just work. You can't easily go back, because I believe that 2.3 will update cyrus.index files to a format which 2.2 doesn't recognize, but that shouldn't prevent anyone from upgrading. IIRC it's a little bit more complicated. Beside BDB, there are also some steps to do at upgrade, like compiling sieve scripts, possibly converting its encoding. With BDB backends you have to make sure everything fits correctly. Binaries need to be linked against the correct version of BDB and also the on disk BDB files need to be of the same version. Now, think you want to do a distribution upgrade which comes with the latest greatest BDB release and new Cyrus binaries which are using them, but your spool is still on the old release... We all know how this ends. The only solution - you can call it dirty workaround - I found for our RedHat EL RPMs was to convert all BDB databases to skiplist on Cyrus shutdown and convert them back on startup. That way the spool is always in a state which can be migrated once Cyrus is shutdown. Of course getting rid of any BDB in the default configuration of the RPM binaries has helped much to prevent BDB mess. Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/20/2010 06:59 AM, Marc Patermann wrote: But where does Cyrus IMAPd stand today? It may be Murder/Aggregator - but how to get the people, when on first contact, where they just need a simple IMAP server, they are pointed to other product, which they then stay with? Umm, what? We run Cyrus IMAP server with no Murder for 20K+ people. Murder may be a feature but it's not a deployment requirement. We used Perdition, originally just thrown up to provide a transparent bridge as we migrated from Uwash to Cyrus. But as things moved along, we ended up with Perdition and multiple Cyrus backends and no good reason to switch over to a Murder. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Mon, 20 Sep 2010, Vincent Fox wrote: Umm, what? We run Cyrus IMAP server with no Murder for 20K+ people. Murder may be a feature but it's not a deployment requirement. We used Perdition, originally just thrown up to provide a transparent bridge as we migrated from Uwash to Cyrus. But as things moved along, we ended up with Perdition and multiple Cyrus backends and no good reason to switch over to a Murder. I end up granting myself rights to various users' mailboxes to investigate when we see one of our users sending out spam. It usually turns out that they have been phished recently. Once I grant myself rights to their mailbox, I see the mailbox in my regular email client (Alpine) in the Other Users hierarchy. Does this work without Murder? How do you investigate users' mailboxes in cases like this? Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On 09/20/2010 04:23 PM, Andrew Morgan wrote: I end up granting myself rights to various users' mailboxes to investigate when we see one of our users sending out spam. It usually turns out that they have been phished recently. Once I grant myself rights to their mailbox, I see the mailbox in my regular email client (Alpine) in the Other Users hierarchy. Does this work without Murder? How do you investigate users' mailboxes in cases like this? I just login to the backend and look at their mailbox directly. Admittedly clunkier but the number of times where I've had to poke directly is small. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: competition
On Mon, 20 Sep 2010, Andrew Morgan wrote: On Mon, 20 Sep 2010, Vincent Fox wrote: Umm, what? We run Cyrus IMAP server with no Murder for 20K+ people. Murder may be a feature but it's not a deployment requirement. We used Perdition, originally just thrown up to provide a transparent bridge as we migrated from Uwash to Cyrus. But as things moved along, we ended up with Perdition and multiple Cyrus backends and no good reason to switch over to a Murder. I end up granting myself rights to various users' mailboxes to investigate when we see one of our users sending out spam. It usually turns out that they have been phished recently. Once I grant myself rights to their mailbox, I see the mailbox in my regular email client (Alpine) in the Other Users hierarchy. Does this work without Murder? it definantly works with a single back-end. In the case of a multi-server setup, it depends on if your murder substatute will show you the folders across back-ends or not in a listing. you can always connect directly to the back-end (bypassing murder or it's replacement) to see all other users for that back-end. David Lang How do you investigate users' mailboxes in cases like this? Andy Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
