Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
On 11/2/10 5:42 PM, Bron Gondwana wrote: Ok - your IMAP client was holding the lock open so the cleanup didn't finish. Basically 2.4.x delays deletion of a mailbox until all the current users have closed it - which is why the files were still around. This may be pretty broken with subfolders when deleting an entire user though - I'll have to think about it for a bit... This sounds correct. lsof showed that imapd had files open within the folders that I was trying to delete. As soon as I closed my email client, lsof no longer showed imapd with those open files, and the files and directories were gone. Basically, it was correct to refuse to recreate the mailbox because the old mailbox was still held open by another connection. I'm leaning towards adding kill currently open connections for this user logic inside Cyrus actually, because replication failover and murder XFER have similar issues with a still-open connection existing as they clean up. Bron. Bron. attachment: rspell.vcf smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
On Wed, Nov 03, 2010 at 09:49:12AM -0400, Robert Spellman wrote: On 11/2/10 5:42 PM, Bron Gondwana wrote: Ok - your IMAP client was holding the lock open so the cleanup didn't finish. Basically 2.4.x delays deletion of a mailbox until all the current users have closed it - which is why the files were still around. This may be pretty broken with subfolders when deleting an entire user though - I'll have to think about it for a bit... This sounds correct. lsof showed that imapd had files open within the folders that I was trying to delete. As soon as I closed my email client, lsof no longer showed imapd with those open files, and the files and directories were gone. Yep. So we need a kill all currently open connections API within Cyrus, and we need to use it when deleting a folder. This will probably involve trolling the proc directory looking for processes. It's almost certainly going to be a pain to implement single-threadedly! I'll have to keep thinking about it. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
cyrus-imapd v2.4.2 on CentOS issue with deleting users
I'm looking into upgrading our cyrus configuration to 2.4.2, and have been playing with a test configuration that mimics our production environment. The test configuration consists of three servers: 1 backend server (mailstore04) 1 murder server (murderdev) 1 front end server (postoffice05) All servers are running CentOS from a clean installation, along with all available patches. I login via cyradm to the backend server mailstore04. I create a user, along with a Trash, Sent and Junk folder I modify the permissions to allow me full access to the user's folders I send an email to the new user, and view it from my account. I then try to delete the account. Here's what I'm seeing in the syslog on the backend: Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Junk Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Sent Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Trash Nov 2 12:32:52 mailstore04 imap[6444]: Remove of supposedly empty directory /home/imap/f/user/frodo failed: Directory not empty and now, when I try to recreate the mailbox: mailstore04.bates.edu cm user.frodo default createmailbox: Mailbox is locked mailstore04.bates.edu If I login to the backend and delete /home/imap/f/user/frodo along with lock/f/frodo* and user/f/frodo*, I can create the folder, however, I start seeing ioerrors until I restart master on the backend: Nov 2 12:07:31 mailstore04 imap[6303]: IOERROR: header CRC mismatch user.frodo: EA10BDA4 E4C3FF07 Nov 2 12:07:31 mailstore04 imap[6303]: IOERROR: header CRC mismatch user.frodo: EA10BDA4 E4C3FF07 Here's the output from the version command in case you want to see how the system was compiled: mailstore04.bates.edu version name : Cyrus IMAPD version: v2.4.2 857bf79 2010-10-19 vendor : Project Cyrus support-url: http://cyrusimap.web.cmu.edu os : Linux os-version : 2.6.18-194.17.4.el5PAE environment: Built w/Cyrus SASL 2.1.22 Running w/Cyrus SASL 2.1.22 Built w/Sleepycat Software: Berkeley DB 4.3.29: (July 12, 2010) Running w/Sleepycat Software: Berkeley DB 4.3.29: (July 12, 2010) Built w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 Running w/OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008 Built w/zlib 1.2.3 Running w/zlib 1.2.3 CMU Sieve 2.4 TCP Wrappers mmap = shared lock = fcntl nonblock = fcntl idle = poll imapd.conf from backend: configdirectory: /var/lib/imap partition-default: /home/imap defaultpartition: default sievedir: /var/lib/imap/sieve admins: cyrus sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN GSSAPI altnamespace: true hashimapspool: true tls_cert_file: /etc/pki/tls/certs/mailstore04.bates.edu-cert.pem tls_key_file: /etc/pki/tls/certs/mailstore04.bates.edu-key.pem tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt proxyservers: murderdev.bates.edu mupdate_server: murderdev.bates.edu mupdate_username: backend imapd.conf from murderdev: configdirectory: /var/lib/imap partition-default: /home/imap sievedir: /var/lib/imap/sieve admins: cyrus sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN GSSAPI altnamespace: true hashimapspool: true tls_cert_file: /etc/pki/tls/certs/murderdev.bates.edu-cert.pem tls_key_file: /etc/pki/tls/certs/murderdev.bates.edu-key.pem tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt mupdate_server: murderdev.bates.edu mupdate_username: backend imapd.conf from frontend: configdirectory: /var/lib/imap partition-default: /home/imap admins: cyrus backend sasl_pwcheck_method: saslauthd sasl_mech_list: PLAIN GSSAPI altnamespace: true hashimapspool: true tls_cert_file: /etc/pki/tls/certs/postoffice05.bates.edu-cert.pem tls_key_file: /etc/pki/tls/certs/postoffice05.bates.edu-key.pem tls_ca_file: /etc/pki/tls/certs/ca-bundle.crt mupdate_server: murderdev.bates.edu mupdate_username: backend attachment: rspell.vcf smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
On 11/02/2010 01:41 PM, Robert Spellman wrote: I'm looking into upgrading our cyrus configuration to 2.4.2, and have been playing with a test configuration that mimics our production environment. The test configuration consists of three servers: 1 backend server (mailstore04) 1 murder server (murderdev) 1 front end server (postoffice05) All servers are running CentOS from a clean installation, along with all available patches. I login via cyradm to the backend server mailstore04. I create a user, along with a Trash, Sent and Junk folder I modify the permissions to allow me full access to the user's folders I send an email to the new user, and view it from my account. I then try to delete the account. Here's what I'm seeing in the syslog on the backend: Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Junk Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Sent Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Trash Nov 2 12:32:52 mailstore04 imap[6444]: Remove of supposedly empty directory /home/imap/f/user/frodo failed: Directory not empty Out of curiosity, what was left behind in /home/imap/f/user/frodo ? snip attachment: boutilpj.vcf Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
Here's the output from a ls -lR starting in the user's mailbox. In this case, the Junk folder was left. Sometimes other folders are left around, so it's not consistent. [cy...@mailstore04 frodo]$ pwd /home/imap/f/user/frodo [cy...@mailstore04 frodo]$ ls -lR .: total 40 -rw--- 2 cyrus mail 1066 Nov 2 12:32 1. -rw--- 1 cyrus mail 940 Nov 2 12:32 cyrus.cache -rw--- 1 cyrus mail 196 Nov 2 12:31 cyrus.header -rw--- 1 cyrus mail 224 Nov 2 12:32 cyrus.index drwx-- 2 cyrus mail 4096 Nov 2 12:31 Junk ./Junk: total 24 -rw--- 1 cyrus mail 4 Nov 2 12:31 cyrus.cache -rw--- 1 cyrus mail 196 Nov 2 12:31 cyrus.header -rw--- 1 cyrus mail 128 Nov 2 12:32 cyrus.index On 11/2/10 12:51 PM, Patrick Boutilier wrote: On 11/02/2010 01:41 PM, Robert Spellman wrote: I'm looking into upgrading our cyrus configuration to 2.4.2, and have been playing with a test configuration that mimics our production environment. The test configuration consists of three servers: 1 backend server (mailstore04) 1 murder server (murderdev) 1 front end server (postoffice05) All servers are running CentOS from a clean installation, along with all available patches. I login via cyradm to the backend server mailstore04. I create a user, along with a Trash, Sent and Junk folder I modify the permissions to allow me full access to the user's folders I send an email to the new user, and view it from my account. I then try to delete the account. Here's what I'm seeing in the syslog on the backend: Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Junk Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Sent Nov 2 12:32:52 mailstore04 imap[6444]: Deleted mailbox user.frodo.Trash Nov 2 12:32:52 mailstore04 imap[6444]: Remove of supposedly empty directory /home/imap/f/user/frodo failed: Directory not empty Out of curiosity, what was left behind in /home/imap/f/user/frodo ? snip Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ attachment: rspell.vcf smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
On Tuesday, November 02, 2010 05:56:31 pm Robert Spellman wrote: Here's the output from a ls -lR starting in the user's mailbox. In this case, the Junk folder was left. Sometimes other folders are left around, so it's not consistent. [cy...@mailstore04 frodo]$ pwd /home/imap/f/user/frodo [cy...@mailstore04 frodo]$ ls -lR .: total 40 -rw--- 2 cyrus mail 1066 Nov 2 12:32 1. -rw--- 1 cyrus mail 940 Nov 2 12:32 cyrus.cache -rw--- 1 cyrus mail 196 Nov 2 12:31 cyrus.header -rw--- 1 cyrus mail 224 Nov 2 12:32 cyrus.index drwx-- 2 cyrus mail 4096 Nov 2 12:31 Junk ./Junk: total 24 -rw--- 1 cyrus mail 4 Nov 2 12:31 cyrus.cache -rw--- 1 cyrus mail 196 Nov 2 12:31 cyrus.header -rw--- 1 cyrus mail 128 Nov 2 12:32 cyrus.index Could it have anything to do with delayed expunges being set? Kind regards, Jeroen van Meeuwen -- Senior Engineer, Kolab Systems AG e: vanmeeu...@kolabsys.com t: +316 42 801 403 w: http://www.kolabsys.com pgp: 9342 BF08 Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
I think you are close. As soon as I closed my email client, the folders were deleted, along with the ability to recreate the mailbox. Thanks for pointing me in the right direction. I suppose in a normal, production environment, I don't go around deleting and then recreating mailbox quite so often. Rob Could it have anything to do with delayed expunges being set? Kind regards, Jeroen van Meeuwen -- smime.p7s Description: S/MIME Cryptographic Signature Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
Re: cyrus-imapd v2.4.2 on CentOS issue with deleting users
On Tue, Nov 02, 2010 at 02:08:17PM -0400, Robert Spellman wrote: I think you are close. As soon as I closed my email client, the folders were deleted, along with the ability to recreate the mailbox. Thanks for pointing me in the right direction. I suppose in a normal, production environment, I don't go around deleting and then recreating mailbox quite so often. Ok - your IMAP client was holding the lock open so the cleanup didn't finish. Basically 2.4.x delays deletion of a mailbox until all the current users have closed it - which is why the files were still around. This may be pretty broken with subfolders when deleting an entire user though - I'll have to think about it for a bit... Basically, it was correct to refuse to recreate the mailbox because the old mailbox was still held open by another connection. I'm leaning towards adding kill currently open connections for this user logic inside Cyrus actually, because replication failover and murder XFER have similar issues with a still-open connection existing as they clean up. Bron. Bron. Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/
