Re: Problem with Cyrus 2.4.0 when subscribe to folders in other backends
Bron Gondwana wrote: On Wed, Oct 13, 2010 at 05:27:32PM -0300, Lucas Zinato Carraro wrote: With Thunderbird 3.1.3 i can subscribe only in folders located in Backend1 where my account lucas.carraro exist. The other folder in backend2 are not avaiable for me: Actually, you're subscribing fine - we're just not showing them. Oops. I know exactly which change fixed this - it's an obviously over-eager patch that I applied for FastMail because users were complaining about non-existant mailboxes appearing in their list view - things that were subscribed ages ago, and appearing despite not being real. I don't have a one-line patch for you I'm afraid - I'll have to talk to Ken about this one and come up with a real fix. Definitely a bug though, in the way murdered imapds handle lsub. Lucas, could you submit this to bugzilla so that I can set a blocker bug for 2.4- next? That will make absolutely sure it is resolved before we release 2.4.1. If you put me in CC: (vanmeeu...@kolabsys.com), that'd make it pop up on my radar the sooner. Thanks in advance, Kind regards, -- Jeroen van Meeuwen Senior Engineer, Kolab Systems AG e: vanmeeu...@kolabsys.com t: +316 42 801 403 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Problem with Cyrus 2.4.0 when subscribe to folders in other backends
--On 13. Oktober 2010 17:50:55 -0400 Bron Gondwana br...@fastmail.fm wrote: On Wed, Oct 13, 2010 at 05:27:32PM -0300, Lucas Zinato Carraro wrote: With Thunderbird 3.1.3 i can subscribe only in folders located in Backend1 where my account lucas.carraro exist. The other folder in backend2 are not avaiable for me: Actually, you're subscribing fine - we're just not showing them. Oops. I know exactly which change fixed this - it's an obviously over-eager patch that I applied for FastMail because users were complaining about non-existant mailboxes appearing in their list view - things that were subscribed ages ago, and appearing despite not being real. just so I get that straight: does that affect all instances of subscriptions to non-existent mailboxes? If so that should definitely be made an option only. I agree that we have more users complain about deleted folders still showing up in the GUI of their client than there are people who actually use that as a feature, but as long as section 6.3.6 of RFC 3501 reads like the following, we just have to live with that: A server MAY validate the mailbox argument to SUBSCRIBE to verify that it exists. However, it MUST NOT unilaterally remove an existing mailbox name from the subscription list even if a mailbox by that name no longer exists. Note: This requirement is because a server site can choose to routinely remove a mailbox with a well-known name (e.g., system-alerts) after its contents expire, with the intention of recreating it when new contents are appropriate. -- .:.Sebastian Hagedorn - RZKR-R1 (Gebäude 52), Zimmer 18.:. .:.Regionales Rechenzentrum (RRZK).:. .:.Universität zu Köln / Cologne University - ✆ +49-221-478-5587.:. p7sCl9Z7XbtLk.p7s Description: S/MIME cryptographic signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: user_deny.db, very high load and Apple-Spotlight
Hi, Mark Heisterkamp schrieb am 12.04.2010 09:03 Uhr: we graded up to cyrus 2.3.16 a few weeks ago and since then the load average showed values from 200 to 300 a few times per day. The server has 16 cores, 64GB RAM an is attached to a SAN. This machine is quite powerfull. It serves about 5000 mailboxes. First we touched user_deny.db to get rid of these annoying IOERROR-messages. These messages where replaced by (annoying) 'fetching user_deny.db'-entries. A normal IMAP-user causes about 500 to 1500 of such messages in eight hours. But we found two users who 'generated' 500 (!) and 25 of such messages in this period. After phoning them we found out, that they where using Mac OS X and Thunderbird 3 (the one with 5 Mio messages) and Mail.app (the other one). Turning off the spotlight-search on IMAP-folders immediately turned the load average down to a normal value (about 0.2). I think we shouldn't advise 5000 users not to use Spotlight, we should deactivate user_deny.db. By the way, what is this database really good for? If we want someone not to use cyrus-service we deny this person by ldap for example. Kenneth Murchison stated in some mail on this list that user_deny.db is used once per login, that's definitely not true, it is used every time the client 'uses' an IMAP-folder and that can be pretty often! Maybe we can change this behaviour by some config? Is it possible to deactivate fetching user_deny.db-entries by some config-option or do we have to patch the sources? I have this issue, too. But there is no Mac involved. It is just Thunderbird on the client (Win*/Linux) side. I created user_deny.db with touch to make the one error message go away. Now I have lots of fetching ... messages in the log (2.3.16). Is there anything to do about this? Marc Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Problem with Cyrus 2.4.0 when subscribe to folders in other backends
On Thu, Oct 14, 2010 at 12:43:39PM +0200, Sebastian Hagedorn wrote: --On 13. Oktober 2010 17:50:55 -0400 Bron Gondwana br...@fastmail.fm wrote: On Wed, Oct 13, 2010 at 05:27:32PM -0300, Lucas Zinato Carraro wrote: With Thunderbird 3.1.3 i can subscribe only in folders located in Backend1 where my account lucas.carraro exist. The other folder in backend2 are not avaiable for me: Actually, you're subscribing fine - we're just not showing them. Oops. I know exactly which change fixed this - it's an obviously over-eager patch that I applied for FastMail because users were complaining about non-existant mailboxes appearing in their list view - things that were subscribed ages ago, and appearing despite not being real. just so I get that straight: does that affect all instances of subscriptions to non-existent mailboxes? If so that should definitely be made an option only. I agree that we have more users complain about deleted folders still showing up in the GUI of their client than there are people who actually use that as a feature, but as long as section 6.3.6 of RFC 3501 reads like the following, we just have to live with that: It doesn't remove them, it just doesn't show them in LSUB. And that's the issue here - the backend isn't showing them in the LSUB response any more. A server MAY validate the mailbox argument to SUBSCRIBE to verify that it exists. However, it MUST NOT unilaterally remove an existing mailbox name from the subscription list even if a mailbox by that name no longer exists. This is an option. Note: This requirement is because a server site can choose to routinely remove a mailbox with a well-known name (e.g., system-alerts) after its contents expire, with the intention of recreating it when new contents are appropriate. And we never do this :) So we're complient - we just aren't displaying subscriptions for non-existent folders. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
Greetings all, We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. We will be deploying new servers for 2.3.16, not upgrading the existing servers. In the past, we have simply rsync'd /var/spool/imap and /var/lib/imap to the new servers, but this doesn't seem like a good idea when dealing with different versions and packagings. Which files, directories, and databases do I need to copy to the new servers? Am I correct in assuming that we will need to recover the databases and mailboxes to make them compatible with the newer version? Thank you in advance. I apologize if this is a basic question -- I don't have a lot of operational experience with Cyrus, and have not been able to find specific instructions in the documentation, wiki, or mailing list. Links to anything I missed would also be appreciated. Best regards, John John Simpson Senior Software Engineer, I. T. Engineering and Operations Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
Greetings all, Hi John, We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. We will be You are welcome! The RHEL package is based on ours but it's a bit old now. deploying new servers for 2.3.16, not upgrading the existing servers. In the past, we have simply rsync'd /var/spool/imap and /var/lib/imap to the new servers, but this doesn't seem like a good idea when dealing with different versions and packagings. The good news is you should not have to worry about anything, the RPM takes care of it automagically :) If you are able to test things and quickly shutdown the old production servers I tried it the following way: 1) Check /etc/imapd.conf and /etc/cyrus.conf on the old and the new server. I think they should be identical but better look at them. And of course also check your auth config, for example if you are running saslauthd and such. 2) rsync -aH --delete (you may really want to sync hardlinks as well!) /var/spool/imap and /var/lib/imap from old to new while old is running. 3) service cyrus-imapd stop on the old box so databases are in a consistent state (and are converted to skiplist for migration if they are not already skiplist) 4) Rerun the rsync, should go quite fast now 5) Start Cyrus on the new box and check how it works. Please let me know if you have any problems. Regards, Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
On 10/14/2010 09:36 AM, Simpson, John R wrote: We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. Speaking of replication, is there any documentation at all on how to set this up and what it does? What about for creating a murder server pool? -- Patrick Goetz Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
On 10/14/2010 11:20 AM, Patrick Goetz wrote: On 10/14/2010 09:36 AM, Simpson, John R wrote: We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. Speaking of replication, is there any documentation at all on how to set this up and what it does? http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/install-replication.php What about for creating a murder server pool? http://www.cyrusimap.org/docs/cyrus-imapd/2.3.16/install-murder.php Thanks, Dave -- Dave McMurtrie, SPE Email Systems Team Leader Carnegie Mellon University, Computing Services Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
Thank you Simon! To clarify one point, when you say the RPM takes care of it automagically, does that mean that I should rsync the old data to the new server BEFORE installing the cyrus-imapd 2.3.16-8 RPMS? The new servers will be bare-bones -- no pre-existing data or older version of Cyrus. John John Simpson Senior Software Engineer, I. T. Engineering and Operations -Original Message- From: Simon Matter [mailto:simon.mat...@invoca.ch] Sent: Thursday, October 14, 2010 10:56 AM To: Simpson, John R Cc: info-cyrus@lists.andrew.cmu.edu Subject: Re: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca) Greetings all, Hi John, We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. We will be You are welcome! The RHEL package is based on ours but it's a bit old now. deploying new servers for 2.3.16, not upgrading the existing servers. In the past, we have simply rsync'd /var/spool/imap and /var/lib/imap to the new servers, but this doesn't seem like a good idea when dealing with different versions and packagings. The good news is you should not have to worry about anything, the RPM takes care of it automagically :) If you are able to test things and quickly shutdown the old production servers I tried it the following way: 1) Check /etc/imapd.conf and /etc/cyrus.conf on the old and the new server. I think they should be identical but better look at them. And of course also check your auth config, for example if you are running saslauthd and such. 2) rsync -aH --delete (you may really want to sync hardlinks as well!) /var/spool/imap and /var/lib/imap from old to new while old is running. 3) service cyrus-imapd stop on the old box so databases are in a consistent state (and are converted to skiplist for migration if they are not already skiplist) 4) Rerun the rsync, should go quite fast now 5) Start Cyrus on the new box and check how it works. Please let me know if you have any problems. Regards, Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
Thank you Simon! To clarify one point, when you say the RPM takes care of it automagically, does that mean that I should rsync the old data to the new server BEFORE installing the cyrus-imapd 2.3.16-8 RPMS? The new No, you have to install the cyrus-imapd RPMs first. You can even start it to see if it works. But then before syncing, shut it down. What the RPM takes care of it automagically is the conversion of databases according to the defaults in the package and/or your settings in /etc/imapd.conf. Simon servers will be bare-bones -- no pre-existing data or older version of Cyrus. John John Simpson Senior Software Engineer, I. T. Engineering and Operations -Original Message- From: Simon Matter [mailto:simon.mat...@invoca.ch] Sent: Thursday, October 14, 2010 10:56 AM To: Simpson, John R Cc: info-cyrus@lists.andrew.cmu.edu Subject: Re: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca) Greetings all, Hi John, We have been running Cyrus 2.3.7, as packaged by Red Hat (cyrus-imapd-2.3.7-2.el5) for some time, and we're upgrading to 2.3.16 (cyrus-imapd-2.3.16-8, thanks Simon) to support replication. We will be You are welcome! The RHEL package is based on ours but it's a bit old now. deploying new servers for 2.3.16, not upgrading the existing servers. In the past, we have simply rsync'd /var/spool/imap and /var/lib/imap to the new servers, but this doesn't seem like a good idea when dealing with different versions and packagings. The good news is you should not have to worry about anything, the RPM takes care of it automagically :) If you are able to test things and quickly shutdown the old production servers I tried it the following way: 1) Check /etc/imapd.conf and /etc/cyrus.conf on the old and the new server. I think they should be identical but better look at them. And of course also check your auth config, for example if you are running saslauthd and such. 2) rsync -aH --delete (you may really want to sync hardlinks as well!) /var/spool/imap and /var/lib/imap from old to new while old is running. 3) service cyrus-imapd stop on the old box so databases are in a consistent state (and are converted to skiplist for migration if they are not already skiplist) 4) Rerun the rsync, should go quite fast now 5) Start Cyrus on the new box and check how it works. Please let me know if you have any problems. Regards, Simon Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Problem with Cyrus 2.4.0 when move message to Mailbox Trash
Hi, Bron This patch works fine. Regards Zinato On Wed, Oct 13, 2010 at 6:30 PM, Bron Gondwana br...@fastmail.fm wrote: On Tue, Oct 12, 2010 at 06:21:36PM -0300, Lucas Zinato Carraro wrote: When i delete a message ( technically, move to trash ) one copy of message is created in mailbox Trash but the message remains on INBOX. Ok - here's the patch that fixes this issue. It will be in the next release, and you can either pull it immediately from the git repository at git://git.cyrusimap.org/cyrus-imapd or you can apply from the attached file. Regards, Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Problem with Cyrus 2.4.0 when move message to Mailbox Trash
On Thu, Oct 14, 2010 at 04:26:30PM -0300, Lucas Zinato Carraro wrote: Hi, Bron This patch works fine. Great - so now I just have to fix your subscription issue :) Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
RE: Moving data from Cyrus 2.3.7 (RHEL) to 2.3.16 (Invoca)
John Simpson Senior Software Engineer, I. T. Engineering and Operations All the basics seem to be working, thank you again for your help. One last question. With rolling replication, should the replica automatically catch up to the master or do I need to seed the replica with rsync (or a batched sync_client)? I can replicate individual mailboxes with sync_client -m mailbox and mailboxes are replicated when they receive a new message, but if neither of those actions takes place the replica is not populated. John Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: messages in mailbox aren't\H\H\H visible
On 10/13/2010 08:43 AM, Bron Gondwana wrote:
Or you could do:
stop cyrus
ctl_mboxlist -d mailbox_list.txt
mv $confdir/mailboxes.db $confdir/mailboxes.db.unsorted
add improved_mboxlist_sort: 1 to /etc/imapd.conf
ctl_mboxlist -u mailbox_list.txt
start cyrus
Thanks! This worked perfectly, and the messages (in mailboxes for which
they were inexplicably hidden) are now visible. It's still unclear how
the out-of-order sorting in mailboxes.db would trigger this.
This brings back the issue of a safe, canonical procedure for moving a
(single server) cyrus mail system to a new server. Since particularly
new users frequently have problems with this (including me, when I first
started using cyrus).
I'm thinking something like this. Suppose there were a command, say
called ctl_cyrusdb2, which offered the same features as ctl_mboxlist,
but could be used on any cyrus db file, e.g.
ctl_cyrusdb2 -d cyrus_db
dumps the cyrus_db database.. (The following also assumes ssh
authorized_keys have been set.)
On the old_cyrus_server:
---
stop cyrus
for i in `cat /var/lib/cyrus/cyrus_db_list`
do
ctl_cyrusdb2 -d $i | ssh new_server cd /var/lib/cyrus
ctl_cyrusdb2 -u -
done
(cd /var/spool/cyrus; tar cf - .) | ssh new_server cd
/var/spool/cyrus tar xpf -
---
start cyrus on new_server and you're done -- a total of 4
command-line commands (viewing the for as a single command).
There are a few problems with this in addition to the absence of a
ctl_cyrusdb2, namely it's still unclear what cyrus db files are created
and why. On my system, I have:
annotations.db deliver.db mailboxes.db tls_sessions.db
but also have
db:
__db.001 __db.002 __db.003 __db.004 __db.005 __db.006
log.01 skipstamp
and
user/[a-z]/{user_name.seen, user_name.sub}
Are the contents of db important for migration? And why isn't the stuff
in /var/lib/cyrus/user stored in the appropriate location in
/var/spool/cyrus? From the file names, this seems like important
metadata that users would want to see preserved. I suppose all this
could be handled with no extra knowledge on the admin user's part by
populating the /var/lib/cyrus/cyrus_db_list file only with the database
names necessary for successful replication of metadata.
Finally, I see many instances of the following error message in
/var/log/syslog:
Oct 14 16:33:01 www cyrus/imap[32001]: IOERROR: opening
/var/lib/cyrus/user_deny.db: No such file or directory
Apparently
touch /var/lib/cyrus/user_deny.db
is not a solution, either:
http://www.mail-archive.com/info-cyrus@lists.andrew.cmu.edu/msg39304.html
2 questions about this:
1.
I have rsyslog configured to log all mail stuff to /var/log/mail.*; e.g.
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail.none,news.none -/var/log/messages
So why is anything from cyrus being written to syslog at all? This
message should be going to /var/log/mail.err, as far as I can tell. Is
this a compile-time error?
2.
Under any circumstance, shouldn't this error message be written to the
log files only when cyrus is started?
--
Patrick Goetz
Cyrus Home Page: http://www.cyrusimap.org/
List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: Cyrus IMAPd 2.4.0 Released
On 10/13/10 18:44, Simon Matter wrote: Simon Matter wrote: On Tue, Oct 12, 2010 at 02:23:18PM -0700, David Lang wrote: (...) IIRC autocreate is on the official todo list, isn't it? That would solve it forever. Simon Excuse my ignorance, but where can I find the Official ToDo list? Thanks, -- per Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
