GSSAPI authentication ceased working
I'm currently setting up a new imap server to replace my old one.
Yesterday I had GSSAPI authentication running, today it ceased working.
I did quite some configuration in the meantime mostly on the LDAP
server, but nothing I'd readily associate with cyrus-imap authentication.
I appreciate any ideas for more systematic troubleshooting.
Regards,
- lars.
The setup:
KDC and LDAP is a sever called hel. The KDC uses LDAP as backend.
Cyrus-Imap (v2.2.13-Debian-2.2.13-14+b3) runs on hermod.
What worked yesterday:
kinit cyrus
imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr
cyradm --user cyrus --auth GSSAPI --server hermod.mgr
What still works today:
kinit cyrus
Diagnostics:
# kinit cyrus
hermod:~# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: cy...@mgr
Valid starting ExpiresService principal
01/02/09 16:41:41 01/03/09 02:41:41 krbtgt/m...@mgr
renew until 01/03/09 16:41:41
Kerberos 4 ticket cache: /tmp/tkt0
klist: You have no tickets cached
hermod:~# imtest -v -u cyrus -a cyrus -p imap -r MGR hermod.mgr
S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+b3 server ready
C: C01 CAPABILITY
S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID
NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT
THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=GSSAPI
AUTH=NTLM AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR
S: C01 OK Completed
Authentication failed. generic failure
Security strength factor: 0
C: Q01 LOGOUT
* BYE LOGOUT received
Q01 OK Completed
Connection closed.
hermod: /var/log/auth.log
Jan 2 17:07:54 hermod imtest: GSSAPI Error: Unspecified GSS failure. Minor
code may provide more information (Decrypt integrity check failed)
hel: /var/log/syslog
Jan 2 16:07:54 hel krb5kdc[1652]: TGS_REQ (7 etypes {18 17 16 23 1 3 2})
172.16.6.5: PROCESS_TGS: authtime 0, unknown client for imap/hermod@mgr,
Decrypt integrity check failed
Jan 2 16:07:54 hel last message repeated 3 times
What I tried:
Since Decrypt integrity check failed means wrong password I recreated the
principal imap/hermod.mgr and replaced the keytab file with the new key. I
also removed the ldapdb auxprop, which I had installed in the meantime, but
nothing helped.
If I remove the ticket for cyrus, I receive:
Jan 2 17:13:36 hermod imtest: GSSAPI Error: Unspecified GSS failure. Minor
code may provide more information (No credentials cache found)
as I would expect.
Cyrus Home Page: http://cyrusimap.web.cmu.edu/
Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki
List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
ldapdb auxprop configuration
I'm trying set up cyrus-imap using the ldapdb auxprop. I guess I've the LDAP part up and running, but somehow imap does not really request for authentication. So probably I still have something messed in the configuration, which apparently has changed with respect to my last install a couple of years ago. Any ideas for systematic troubleshooting are welcome. Regards, - lars. This is the sasl related part of the imap configuration: hermod:~# grep sasl /etc/imapd.conf | grep -v '^#' | grep -v '^\s*$' sasl_mech_list: PLAIN DIGEST-MD5 CRAM-MD5 sasl_pwcheck_method: auxprop sasl_auxprop_plugin: ldapdb sasl_ldapdb_uri: ldaps://hel.mgr sasl_ldapdb_id: mailadmin sasl_ldapdb_pw: * sasl_ldapdb_mech: DIGEST-MD5 sasl_auto_transition: no The following is running as expected: hermod:~# ldapwhoami -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -H ldaps://hel.mgr Enter LDAP Password: SASL/DIGEST-MD5 authentication started SASL username: u:cyrus SASL SSF: 128 SASL data security layer installed. dn:uid=cyrus,ou=mailbox,dc=mgr and of course: ldapsearch -U mailadmin -X u:cyrus -W -Y DIGEST-MD5 -b ou=mailbox,dc=mgr (uid=cyrus) returns the password of cyrus, which is kept as plaintext inside the LDAP repositiory. ldapsearch returns the base64 encoded plain password. However using this same password the following happens: hermod:~# imtest -v -u cyrus -a cyrus -p imap -m DIGEST-MD5 hermod.mgr S: * OK hermod.mgr Cyrus IMAP4 v2.2.13-Debian-2.2.13-14+b3 server ready C: C01 CAPABILITY S: * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS ID NO_ATOMIC_RENAME UNSELECT CHILDREN MULTIAPPEND BINARY SORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES ANNOTATEMORE IDLE STARTTLS AUTH=DIGEST-MD5 AUTH=CRAM-MD5 SASL-IR S: C01 OK Completed C: A01 AUTHENTICATE DIGEST-MD5 S: + bm9uY2U9IlBMREhNY0JjbG1XOUt2dk5FQWQrb0R5cmZ3YjY3cHcyb1VIWHhacDE0dXc9IixyZWFsbT0iaGVybW9kLm1nciIscW9wPSJhdXRoLGF1dGgtaW50LGF1dGgtY29uZiIsY2lwaGVyPSJyYzQtNDAscmM0LTU2LHJjNCxkZXMsM2RlcyIsbWF4YnVmPTQwOTYsY2hhcnNldD11dGYtOCxhbGdvcml0aG09bWQ1LXNlc3M= Please enter your password: C: dXNlcm5hbWU9ImN5cnVzIixyZWFsbT0iaGVybW9kLm1nciIsbm9uY2U9IlBMREhNY0JjbG1XOUt2dk5FQWQrb0R5cmZ3YjY3cHcyb1VIWHhacDE0dXc9Iixjbm9uY2U9IkVZR2hkY1UvZy9vU0J5VkNsMkhSVWt3NWVuMTlOR3puWU9PQjZuSUpPams9IixuYz0wMDAwMDAwMSxxb3A9YXV0aC1jb25mLGNpcGhlcj1yYzQsbWF4YnVmPTEwMjQsZGlnZXN0LXVyaT0iaW1hcC9oZXJtb2QubWdyIixyZXNwb25zZT00Yjk3OWJhMTU0NWUzZDBkMTJiYWNlNjY4NTk4YjhjZA== failure: prot layer failure The detailed log of slapd has the following for this request: slap_listener_activate(10): slap_listener(ldaps:///) conn=15 fd=24 ACCEPT from IP=172.16.6.5:53956 (IP=0.0.0.0:636) connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 connection_read(24): unable to get TLS client DN, error=49 id=15 conn=15 fd=24 TLS established tls_ssf=128 ssf=128 connection_get(24): got connid=15 connection_read(24): checking for input on id=15 ber_get_next ber_get_next on fd 24 failed errno=0 (Success) connection_closing: readying conn=15 sd=24 for close connection_close: conn=15 sd=24 conn=15 fd=24 closed (connection lost) So apparently imapd-ldapdb connects and establishes SSL. For the rest I'm unsure, but it seems like it does not talk to LDAP anymore and terminates, i.e. there is no authentication happening. The result is the same for trying telnet localhost imap2 and a login for cyrus. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
login issues
All mail users are in sasldb. But now the problem is that I get more and more outlook (MS) clients that can't do CRAM-MD5. So, I guess the imapd.conf I use needs some tweaking. What do I have to change so that outlook express users can login? The users are just -mail- users and have NO UNIX account on the system. ==imapd.conf configdirectory: /opt/csw/var/cyrus/config partition-default: /opt/csw/var/cyrus/mail sievedir: /opt/csw/var/cyrus/sieve admins: cyrus unixhierarchysep: no altnamespace: no munge8bit: yes sasl_pwcheck_method: saslauthd # sieve_allowplaintext: yes # sasl_minimum_layer: 0 # sasl_mech_list: PLAIN LOGIN # autocreatequota: -1 # createonpost: yes tls_ca_file: /opt/csw/ssl/private/imap/server.pem tls_cert_file: /opt/csw/ssl/private/imap/server.pem tls_key_file: /opt/csw/ssl/private/imap/server.pem === -- Dick Hoogendijk -- PGP/GnuPG key: 01D2433D + http://nagual.nl/ | SunOS sxce snv104 ++ + All that's really worth doing is what we do for others (Lewis Carrol) Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: login issues
-- dick hoogendijk d...@nagual.nl is rumored to have mumbled on 2. Januar 2009 21:04:37 +0100 regarding login issues: All mail users are in sasldb. But now the problem is that I get more and more outlook (MS) clients that can't do CRAM-MD5. So, I guess the imapd.conf I use needs some tweaking. What do I have to change so that outlook express users can login? We use the following for maximum compatibility, but it requires that all those SASL modules are actually installed: allowanonymouslogin: no allowplaintext: no sasl_pwcheck_method: auxprop sasl_mech_list: DIGEST-MD5 CRAM-MD5 PLAIN NTLM LOGIN That way Outlook can either use NTLM or some SSL/TLS secured connection. -- Sebastian Hagedorn - RZKR-R1 (Flachbau), Zi. 18, Robert-Koch-Str. 10 Zentrum für angewandte Informatik - Universitätsweiter Service RRZK Universität zu Köln / Cologne University - Tel. +49-221-478-5587 pgpZ3G9tQxn82.pgp Description: PGP signature Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ldapdb auxprop configuration
Thanks Dan, To make sure that the ldapdb plugin is installed correctly: # cat /usr/lib/sasl2/pluginview.conf # pluginviewer | grep ldapdb hermod:/# grep sasl /etc/imapd.conf | grep -v '^#' | grep -v '^\s*$' | sed 's/^sasl_//' /usr/lib/sasl2/pluginviewer.conf hermod:/# saslpluginviewer -a Installed auxprop mechanisms are: ldapdb sasldb List of auxprop plugins follows Plugin ldapdb , API version: 4 supports store: yes Plugin sasldb , API version: 4 supports store: yes Didn't know this tool so far. Should it say something different? Does your /var/log/auth.log or /var/log/syslog give you anything useful? At least it's not too useful to me ... (after setting sasl_log_level: 7) /var/log/auth.log: Jan 2 22:31:15 hermod cyrus/imap[3432]: DIGEST-MD5 server step 1 Jan 2 22:31:15 hermod imtest: DIGEST-MD5 client step 2 Jan 2 22:31:17 hermod imtest: DIGEST-MD5 client step 2 Jan 2 22:31:17 hermod cyrus/imap[3432]: DIGEST-MD5 server step 2 /var/log/syslog: Jan 2 22:31:15 hermod cyrus/master[3432]: about to exec /usr/lib/cyrus/bin/imapd Jan 2 22:31:15 hermod cyrus/imap[3432]: executed Jan 2 22:31:15 hermod cyrus/imap[3432]: accepted connection Jan 2 22:31:17 hermod cyrus/master[3425]: process 3432 exited, signaled to death by 11 Jan 2 22:31:17 hermod cyrus/master[3425]: service imap pid 3432 in BUSY state: terminated abnormally You may want to experiment with the ldapdb_starttls and ldapdb_rc options (see sasl's options.html doc). See 'man ldap.conf' for options that you can place in ldaprc. If you do choose to use starttls, you'll need to replace ldaps://hel.mgr with ldap://hel.mgr. I tried sasl_ldapdb_uri: ldap://hel.mgr sasl_ldapdb_starttls: try and it comes out the same; slapd logs a successful STARTTLS. I tried: sasl_ldapdb_rc: /etc/ldap/ldap.conf which yields sending short packages in both cases. This slapd debug output is from a STARTTLS variant: TLS: can't accept: A TLS packet with unexpected length was received.. connection_read(16): TLS accept failure error=-1 id=8, closing connection_closing: readying conn=8 sd=16 for close connection_close: conn=8 sd=16 conn=8 fd=16 closed (TLS negotiation failure) But still imtest fails with failure: prot layer failure. There is no activity in slapd before the password is entered in imtest. Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: ldapdb auxprop configuration
Lars Hanke wrote: hermod:/# saslpluginviewer -a Installed auxprop mechanisms are: ldapdb sasldb List of auxprop plugins follows Plugin ldapdb , API version: 4 supports store: yes Plugin sasldb , API version: 4 supports store: yes Didn't know this tool so far. Should it say something different? No, that confirms that ldapdb is installed. Does your /var/log/auth.log or /var/log/syslog give you anything useful? /var/log/syslog: Jan 2 22:31:15 hermod cyrus/master[3432]: about to exec /usr/lib/cyrus/bin/imapd Jan 2 22:31:15 hermod cyrus/imap[3432]: executed Jan 2 22:31:15 hermod cyrus/imap[3432]: accepted connection Jan 2 22:31:17 hermod cyrus/master[3425]: process 3432 exited, signaled to death by 11 Jan 2 22:31:17 hermod cyrus/master[3425]: service imap pid 3432 in BUSY state: terminated abnormally 'signaled to death by 11' is a big red flag... your imapd process is seg faulting. It's possibly caused by an old SASL/OpenLDAP reentrant bug (are you running an old version of libldap?). You can specify a debug_command in your imapd.conf to generate a back trace. See: https://langhorst.com/cgi-bin/dwww//usr/share/doc/cyrus21-common/README.Debian.debug.gz - Dan Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
Now see, I've had almost exactly the opposite experience. Reiserfs seemed to start out well and work consistently until the filesystem reached a certain size (around 160GB, ~30m files) at which point backing it up would start to take too long and at around 180GB would take nearly a week. This forced us to move to ext3 and it doesn't seem to be degrade that way. We did, however, also move from a single partition to 8 of them, so that obviously has some effect as well. As you noted, changing two variables at once doesn't help you determine which was the problem! Multiple partitions will definitely allow more parallelism, which definitely helps speed things up, which is one of the other things we have done over time. Basically we went from a few large volumes to hundreds of 300G(data)/15G(meta) volumes. One of our machines has 40 data volumes + 40 meta data volumes + the standard FS mounts. $ mount | wc -l 92 We've found that splitting the data up into more volumes + more cyrus instances seems to help as well because it seems to reduce overall contention points in the kernel + software (eg filesystem locks spread across multiple mounts, db locks are spread across multiple dbs, etc) Also one thing I did fail to mention, was that for the data volumes, you should definitely be using the notail mount option. Unfortunately that's not the default, and I think it probably should be. Tails packing is neat for saving space, but it reduces the average meta-data density, which makes stating lots of files in a directory a lot slower. I think that's what you might have been seeing. Of course you also mounted noatime,nodiratime on both? I think that's another problem with a lot of filesystem benchmarks, not finding out what the right mount tuning options are for your benchmark. Arguing that the default should be fine is clearly wrong, because every sane person uses noatime, so you're already doing some tuning, so you should find out what's best for the filesystem you are trying. For the record, we use: noatime,nodiratime,notail,data=ordered On all our reiserfs volumes. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Storage Sizing: IOPS per mailbox
When sizing a storage device for a large cyrus server, the typical question asked by storage vendors is what is the IOPS required per mailbox M$$ Exchange has this concept of IOPS. and they suggest 1.5 IOPS per mailbox ( heavy users ) If I use postfix and cyrus , on my imap server ( pure IMAP server .. All spam filtering , outgoing mails , authentication etc happens on different servers ) If the storage is used only for imap storage , what is the typical IOPS requirement per user We will probably assume 30-50 mails a day of average 100k , and an email client checking for new mail every 5minutes Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
Re: choosing a file system
On Sat, 2009-01-03 at 13:21 +1100, Rob Mueller wrote: Now see, I've had almost exactly the opposite experience. Reiserfs seemed to start out well and work consistently until the filesystem reached a certain size (around 160GB, ~30m files) at which point backing it up would start to take too long and at around 180GB would take nearly a week. This forced us to move to ext3 and it doesn't seem to be degrade that way. We did, however, also move from a single partition to 8 of them, so that obviously has some effect as well. As you noted, changing two variables at once doesn't help you determine which was the problem! Multiple partitions will definitely allow more parallelism, which definitely helps speed things up, which is one of the other things we have done over time. Basically we went from a few large volumes to hundreds of 300G(data)/15G(meta) volumes. One of our machines has 40 data volumes + 40 meta data volumes + the standard FS mounts. $ mount | wc -l 92 We've found that splitting the data up into more volumes + more cyrus instances seems to help as well because it seems to reduce overall contention points in the kernel + software (eg filesystem locks spread across multiple mounts, db locks are spread across multiple dbs, etc) Running multiple cyrus instances with different dbs ? How do we do that. I have seen the ultimate io-contention point is the mailboxes.db file. And that has to be single. Do you mean dividing the users to different cyrus instances. That is a maintenance issue IMHO. I had the feeling whatever optimizations done at the FS level would give us a max of 5-10% benefit. We migrated from ext3 to reiserfs on our cyrus servers with 30k mailboxes. I am not sure I saw a great benefit in terms of the iowait. At peak times I always see a iowait of 40-60% But the new Solid-State-Disks seem very promising. They are claimed to give 30x the throughput of a 15k rpm disk. If IO improves by 30 times that should make all these optimizations unnecessary. As my boss used to tell me ... Good hardware always compensates for not-so-good software. Also one thing I did fail to mention, was that for the data volumes, you should definitely be using the notail mount option. Unfortunately that's not the default, and I think it probably should be. Tails packing is neat for saving space, but it reduces the average meta-data density, which makes stating lots of files in a directory a lot slower. I think that's what you might have been seeing. Of course you also mounted noatime,nodiratime on both? I think that's another problem with a lot of filesystem benchmarks, not finding out what the right mount tuning options are for your benchmark. Arguing that the default should be fine is clearly wrong, because every sane person uses noatime, so you're already doing some tuning, so you should find out what's best for the filesystem you are trying. For the record, we use: noatime,nodiratime,notail,data=ordered On all our reiserfs volumes. Rob Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html Cyrus Home Page: http://cyrusimap.web.cmu.edu/ Cyrus Wiki/FAQ: http://cyrusimap.web.cmu.edu/twiki List Archives/Info: http://asg.web.cmu.edu/cyrus/mailing-list.html
