Re: Strange SASL Authentication Issue
Zitat von Robert Krig : On 01/13/2012 09:52 AM, lst_ho...@kwsoft.de wrote: Zitat von Robert Krig : On 01/11/2012 08:38 PM, Gary Smith wrote: Restarting postfix, saslauthd and authdaemon seems to get it working again, at least for a while. Are you using pam_mysql by chance? Yes, I am. Too bad, pam_mysql is known to leak memory. We have used it some time ago and the only option to get it "stable" was with "saslauthd -n 0". Regards Andreas So far it's been running stable, memorywise by using the "-c" flag for cacheing. Is there any downside to the "-n 0" flag? I had read about before, but I wanted to see if cacheing alone made a difference. The downside of "-n 0" is that for every authentication request a new process is spawned and ended afterwards so the memory leak will not sum up. This will hit performance limits because of init costs (process startup, db-connection etc.) if your authentication rate is very high. For small/mid-size systems it should not matter that much. Regards Andreas smime.p7s Description: S/MIME Cryptographic Signature
Re: Strange SASL Authentication Issue
On 01/13/2012 09:52 AM, lst_ho...@kwsoft.de wrote: > Zitat von Robert Krig : > >> >> >> On 01/11/2012 08:38 PM, Gary Smith wrote: Restarting postfix, saslauthd and authdaemon seems to get it working again, at least for a while. >>> Are you using pam_mysql by chance? >> >> Yes, I am. > > Too bad, pam_mysql is known to leak memory. We have used it some time > ago and the only option to get it "stable" was with "saslauthd -n 0". > > Regards > > Andreas > > So far it's been running stable, memorywise by using the "-c" flag for cacheing. Is there any downside to the "-n 0" flag? I had read about before, but I wanted to see if cacheing alone made a difference.
Re: Strange SASL Authentication Issue
Zitat von Robert Krig : On 01/11/2012 08:38 PM, Gary Smith wrote: Restarting postfix, saslauthd and authdaemon seems to get it working again, at least for a while. Are you using pam_mysql by chance? Yes, I am. Too bad, pam_mysql is known to leak memory. We have used it some time ago and the only option to get it "stable" was with "saslauthd -n 0". Regards Andreas smime.p7s Description: S/MIME Cryptographic Signature
Re: Strange SASL Authentication Issue
On 01/11/2012 08:38 PM, Gary Smith wrote: >> Restarting postfix, saslauthd and authdaemon seems to get it working again, >> at least for a while. >> > Are you using pam_mysql by chance? Yes, I am.
RE: Strange SASL Authentication Issue
> -Original Message- > From: owner-postfix-us...@postfix.org > [mailto:owner-postfix-us...@postfix.org] On Behalf Of Robert Krig > Sent: Thursday, January 12, 2012 3:03 AM > To: Postfix users > Subject: Re: Strange SASL Authentication Issue > > I think I might have identified the problem. Apparently in some > situations SASLauthd can produce memory leaks. This is what I've > gathered from other people's posts. > > Now I'm not certain if this is what was causing my particular issue > (too soon to tell), but it's worth checking out. You might test this by restarting saslauthd once the problem appears. If that causes the symptom to stop, you have stronger evidence that the problem is there. -MSK
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 13:11:12 Wietse Venema wrote: > Robert Krig: This is just a quick heads up for people who might be experiencing similar issues. I think I might have identified the problem. Apparently in some situations SASLauthd can produce memory leaks. This is what I've gathered from other people's posts. Now I'm not certain if this is what was causing my particular issue (too soon to tell), but it's worth checking out. I noticed that the RAM usage on my mailserver seemed to continually grow over time, and the date when the authentication issues arose, seem to coincide with the ram usage growing beyond the upper limit of the Server's resources. Anyways, I added the "-c" parameter to my saslauthd options. This is supposed to cache the authentication credentials. So far, my ram usage seems to stay more or less constant. It's still too early to tell if this solved it or not. Since my issue is very sporadic. But I thought I'd just mention it, in case someone here has similar issues.
RE: Strange SASL Authentication Issue
> Restarting postfix, saslauthd and authdaemon seems to get it working again, > at least for a while. > Are you using pam_mysql by chance?
Re: Strange SASL Authentication Issue
Robert Krig: > On Wednesday 11 January 2012 11:38:13 Wietse Venema wrote: > > > You have a problem that starts at some unpredictable moment, and > > that causes SOME ACCOUNTs to fail PERSISTENTLY after it starts. > > > > This is typical of one PERSISTENT process (like saslauthd or mysqld) > > having some corruption of some kind, that PERSISTS until you restart > > that PERSISTENT process. > > > > The Postfix SMTP server is not a persistent process. It is just > > the messenger of the bad news. > > > > I am done helping you. Good luck. > > > > Wietse > > Sorry, didn't mean to sound rude or ungrateful. I think I simply > misunderstood > you the first time. Anyways, thanks you for help. No offense taken, you just appeared to be focused on the wrong part of the problem (the fact that it turns on at unpredicable times). I hope that the search for the culprit (saslauthd, mysqld or perhaps the NSCD daemon, that's one I forgot to mention) will get the problem resolved. Wietse
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 11:38:13 Wietse Venema wrote: > You have a problem that starts at some unpredictable moment, and > that causes SOME ACCOUNTs to fail PERSISTENTLY after it starts. > > This is typical of one PERSISTENT process (like saslauthd or mysqld) > having some corruption of some kind, that PERSISTS until you restart > that PERSISTENT process. > > The Postfix SMTP server is not a persistent process. It is just > the messenger of the bad news. > > I am done helping you. Good luck. > > Wietse Sorry, didn't mean to sound rude or ungrateful. I think I simply misunderstood you the first time. Anyways, thanks you for help.
Re: Strange SASL Authentication Issue
Robert Krig: > On Wednesday 11 January 2012 10:15:19 Wietse Venema wrote: > > > > > Some accounts fail persistently, if I recall correctly. > > Sorry, I think you misunderstood me. Let me explain again. You have a problem that starts at some unpredictable moment, and that causes SOME ACCOUNTs to fail PERSISTENTLY after it starts. This is typical of one PERSISTENT process (like saslauthd or mysqld) having some corruption of some kind, that PERSISTS until you restart that PERSISTENT process. The Postfix SMTP server is not a persistent process. It is just the messenger of the bad news. I am done helping you. Good luck. Wietse
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 10:15:19 Wietse Venema wrote: > > Some accounts fail persistently, if I recall correctly. Sorry, I think you misunderstood me. Let me explain again. Our 4 webservers CONSTANTLY send registration emails to new users via a local php-mailer on each webserver instance which connects to our central postfix mail server. This works, except on rare occasions when it fails without explanation. This is what I find so puzzling. It's not consistent behaviour, and I haven't yet found what causes it, or been able to reproduce it. It seems to happen at random. Sometimes it can be weeks before it happens again. Once it happens, only THAT account is unable to authenticate until I restart all relevant daemons. So, a user registers on our website via one of our webservers. In this case www1, www2, www3 or www4. All our webservers use the same account to send a registration confirmation email to the new user. In this case "nore...@example.com" This happens via a local phpmailer instance on each webserver. The phpmailer simply sends the registration mail via smtp to our postfix server using the nore...@example.com account. Either way, I'll try what you suggested and see if I can't find anymore clues. On our postfix server, we have lots of other accounts and domains. But the weirdest thing is, when THAT account stops working, all others still continue to work as normal. That's what makes it so difficult to catch that there is a problem. If the mail server itself suddenly completely stopped working, we would notice this immediately. But as it is, it's only the webservers trying to send registration emails via that one account.
Re: Strange SASL Authentication Issue
Robert Krig: > On Wednesday 11 January 2012 09:08:03 Wietse Venema wrote: > > Fortunately, the Postfix SMTP server is a short-lived process that > > runs for a few minutes at a time without ever changing the system > > configuration. Every new Postfix SMTP server process is like a > > new-born with a blank memory of its past. > > > > Therefore, if SASL logins fail, especially when they fail persistently, > > then either the SASL client has changed, or the SASL server > > infrastructure **outside POSTFIX** has changed. > > But thats just it, they don't fail persistently. I mean, it all works fine, > until all of a sudden it doesn't anymore and only for these accounts. The > other accounts continue to work fine. Some accounts fail persistently, if I recall correctly. This happens outside the non-persistent Postfix SMTP server, which maintains no memory of what has happened previously. This part is really simple. Now, let's look where errors can persist: > /usr/lib/sasl2/smtpd.conf > pwcheck_method: saslauthd The non-persistent Postfix SMTP server talks to the Cyrus SASL library, which talks to the persistent saslauthd process. > /etc/conf.d/saslauthd > SASLAUTHD_OPTS="-m /var/run/saslauthd -r -a pam" The persistent saslauthd process talks to the PAM subsystem. > /etc/authlib/authdaemonrc > authmodulelist="authmysql" And PAM talks to the persistent MySQL server. Somewhere in this chain, the information about some accounts gets messed up, and the corruption is persistent. I therefore suggest looking at any one of the PERSISTENT processes in this chain, instead of the non-persistent processes from Postfix. Wietse
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 09:08:03 Wietse Venema wrote: > Fortunately, the Postfix SMTP server is a short-lived process that > runs for a few minutes at a time without ever changing the system > configuration. Every new Postfix SMTP server process is like a > new-born with a blank memory of its past. > > Therefore, if SASL logins fail, especially when they fail persistently, > then either the SASL client has changed, or the SASL server > infrastructure **outside POSTFIX** has changed. But thats just it, they don't fail persistently. I mean, it all works fine, until all of a sudden it doesn't anymore and only for these accounts. The other accounts continue to work fine. > > This would be a good time to provide configuration information about > how Postfix interfaces to the SASL server infrastructure **outside > POSTFIX**. > > There are two such possible infrastructures: Dovecot or Cyrus SASL. > This choice is made with the smtpd_sasl_type parameter. > > Examine the output from: > > # postconf smtpd_sasl_type smtpd_sasl_type = cyrus > If this is "cyrus", you need to report what's in the smtpd.conf > file, whose location depends on how your distributor has tweaked > the details of the SASL server infrastructure **outside POSTFIX**. > This file could be located in /usr/local/lib/sasl2, in /etc/postfix, > or any number of other places. /usr/lib/sasl2/smtpd.conf ## pwcheck_method: saslauthd mech_list: plain login saslauthd_path: /var/run/saslauthd/mux log_level: 7 ## /etc/authlib/authmysqlrc ### MYSQL_SERVER localhost MYSQL_PORT 3306 MYSQL_USERNAME postfix_user MYSQL_PASSWORD postfixpassword MYSQL_DATABASE postfix_db MYSQL_USER_TABLEmailbox MYSQL_CRYPT_PWFIELD password MYSQL_UID_FIELD 5000 MYSQL_GID_FIELD 5000 MYSQL_LOGIN_FIELD username MYSQL_HOME_FIELD"/home/vmail" MYSQL_NAME_FIELDname MYSQL_MAILDIR_FIELD maildir MYSQL_QUOTA_FIELD quota ### /etc/authlib/authdaemonrc ### authmodulelist="authmysql" authmodulelistorig="authuserdb authpam authpwd authshadow authpgsql authldap authmysql authcustom authpipe" daemons=5 authdaemonvar=/var/spool/authdaemon DEBUG_LOGIN=2 DEFAULTOPTIONS="" LOGGEROPTS="" ### /etc/conf.d/saslauthd ### SASLAUTHD_OPTS="-m /var/run/saslauthd -r -a pam" ### By the way, I'm running Arch Linux, in case thats relevant. (You never know)
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 08:08:34 I wrote: > On Wednesday 11 January 2012 07:45:46 Robert Krig wrote: > > Whats weird is that the problem gets fixed by simply > > restarting the services. > > Try it without restarting Postfix next time, just your > saslauthd and anything it needs for data (e.g., mysqld, > postmaster, whatever.) You do not have a Postfix issue. Forgot to mention Courier authdaemond, mentioned in the OP. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 07:45:46 Robert Krig wrote: > On Wednesday 11 January 2012 07:14:14 Wietse Venema wrote: > > Why do you believe that there is a problem with SASL > > authentication between the PHP application and Postfix? > > Because the only error that shows up in the log file is this: > ## > postfix/smtpd[7310]: connect from www2.domain.com[xx.xx.xx.xx] > > postfix/smtpd[7310]: warning: www2.domain.com[xx.xx.xx.xx]: SASL > LOGIN authentication failed: authentication failure > > postfix/smtpd[7310]: lost connection after RSET from > www2.domain.com[xx.xx.xx.xx] > > postfix/smtpd[7310]: disconnect from www2.domain.com[xx.xx.xx.xx] Postfix is the messenger, the relay between the authenticating client and the SASL backend. It is only reporting what happened. BTW, unless you actually own "domain.com" (you surely do not) you should not use it as an example. Example.com (.net, .org) and others in gTLDs and many ccTLDs have been set aside for examples. snip > > Your posting provides no concrete symptoms (logging!) that would > > allow the list to help you towards a solution. It is not unusual > > for people to confuse authentication and encryption. > > > > http://www.postfix.org/DEBUG_README.html#mail. > > > > DO NOT TURN ON VERBOSE LOGGING until asked to do so. The default > > Postfix logging may look like useless garbage to you, but it > > provides a lot of detail that gets drowned out out when you open > > the firehose. > > I've enabled debug logging only for the affected hosts, so that my > log files don't get overwhelmed with useless noise. Still useless and not going to help. Either the authentication succeeds or not. You won't find anything useful in Postfix verbose logs. And the logging you did share this time did not indicate a Postfix problem. > Like I said, it's weird. If the affected clients could not send any > mail it would be one thing, but why they seem to work fine for > weeks and then once in a while simply refuse to authenticate > properly, is beyond me. It must be a problem in the SASL backend and/or its data source. > Could it have something to do with > smtpd_recipient_restrictions = permit_mynetworks, > > permit_sasl_authenticated, > > reject_unauth_destination > which I have in my main.cf? No. > The affected hosts are in my mynetworks list. As far as I > understand it, this would mean that the hosts which are listed in > "mynetworks" do not HAVE to authenticate. Correct. > The phpmailer clients in > this case are configured to try and authenticate with the proper > username and password. If they attempted without authentication, and as you say, they are listed in mynetworks, they would succeed. > Is there a possibility that there is a race condition of some sort? No, or at least not something relevant to this list, which is for Postfix support. > We have 4 webservers. www1, www2, www3, www4. They all use the same > username and password to authenticate and send mail via the same > account. Could there be a problem if they try to authenticate > simultaneously? Check the SASL documentation and logs. > Or would it be better to remove the "permit_mynetworks" line, so > that they are forced to authenticate properly? That is a policy decision for you to make. > Whats weird is that the problem gets fixed by simply restarting > the services. Try it without restarting Postfix next time, just your saslauthd and anything it needs for data (e.g., mysqld, postmaster, whatever.) You do not have a Postfix issue. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
Re: Strange SASL Authentication Issue
Robert Krig: > On Wednesday 11 January 2012 07:14:14 Wietse Venema wrote: > > > Why do you believe that there is a problem with SASL authentication > > between the PHP application and Postfix? > > Because the only error that shows up in the log file is this: > ## > postfix/smtpd[7310]: connect from www2.domain.com[xx.xx.xx.xx] > > postfix/smtpd[7310]: warning: www2.domain.com[xx.xx.xx.xx]: SASL LOGIN > authentication failed: authentication failure Fortunately, the Postfix SMTP server is a short-lived process that runs for a few minutes at a time without ever changing the system configuration. Every new Postfix SMTP server process is like a new-born with a blank memory of its past. Therefore, if SASL logins fail, especially when they fail persistently, then either the SASL client has changed, or the SASL server infrastructure **outside POSTFIX** has changed. This would be a good time to provide configuration information about how Postfix interfaces to the SASL server infrastructure **outside POSTFIX**. There are two such possible infrastructures: Dovecot or Cyrus SASL. This choice is made with the smtpd_sasl_type parameter. Examine the output from: # postconf smtpd_sasl_type If this is "dovecot", you need to check the Dovecot authentication server logs. If this is "cyrus", you need to report what's in the smtpd.conf file, whose location depends on how your distributor has tweaked the details of the SASL server infrastructure **outside POSTFIX**. This file could be located in /usr/local/lib/sasl2, in /etc/postfix, or any number of other places. I suggest doing: # find / -name smtpd.conf and reporting the contents of any files thus found. Wietse
Re: Strange SASL Authentication Issue
On Wednesday 11 January 2012 07:14:14 Wietse Venema wrote: > Why do you believe that there is a problem with SASL authentication > between the PHP application and Postfix? > Because the only error that shows up in the log file is this: ## postfix/smtpd[7310]: connect from www2.domain.com[xx.xx.xx.xx] postfix/smtpd[7310]: warning: www2.domain.com[xx.xx.xx.xx]: SASL LOGIN authentication failed: authentication failure postfix/smtpd[7310]: lost connection after RSET from www2.domain.com[xx.xx.xx.xx] postfix/smtpd[7310]: disconnect from www2.domain.com[xx.xx.xx.xx] ## For comparison, this is what it normally looks like: ## postfix/smtpd[7310]: connect from www2.domain.com[xx.xx.xx.xx] postfix/smtpd[7310]: A406B202D9: client=www2.domain.com[xx.xx.xx.xx], sasl_method=LOGIN, sasl_username=nore...@domain.com postfix/cleanup[7309]: A406B202D9: message- id=<7d3559e19e3c13f1aa342b9d5a33a...@www.domain.com> postfix/qmgr[9970]: A406B202D9: from=, size=733, nrcpt=1 (queue active) postfix/smtpd[7310]: disconnect from www2.domain.com[xx.xx.xx.xx] postfix/smtp[7360]: A406B202D9: to=, relay=mx- ha01.web.de[xxx.xx.xxx.xxx]:25, delay=0.21, delays=0.08/0.02/0.05/0.06, dsn=2.0.0, status=sent (250 OK id=1Rg1kQ-0002ax-00) postfix/qmgr[9970]: A406B202D9: removed ## > Your posting provides no concrete symptoms (logging!) that would > allow the list to help you towards a solution. It is not unusual > for people to confuse authentication and encryption. > > http://www.postfix.org/DEBUG_README.html#mail. > > DO NOT TURN ON VERBOSE LOGGING until asked to do so. The default > Postfix logging may look like useless garbage to you, but it provides > a lot of detail that gets drowned out out when you open the firehose. > > Wietse I've enabled debug logging only for the affected hosts, so that my log files don't get overwhelmed with useless noise. Like I said, it's weird. If the affected clients could not send any mail it would be one thing, but why they seem to work fine for weeks and then once in a while simply refuse to authenticate properly, is beyond me. Could it have something to do with smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination which I have in my main.cf? The affected hosts are in my mynetworks list. As far as I understand it, this would mean that the hosts which are listed in "mynetworks" do not HAVE to authenticate. The phpmailer clients in this case are configured to try and authenticate with the proper username and password. Is there a possibility that there is a race condition of some sort? We have 4 webservers. www1, www2, www3, www4. They all use the same username and password to authenticate and send mail via the same account. Could there be a problem if they try to authenticate simultaneously? Or would it be better to remove the "permit_mynetworks" line, so that they are forced to authenticate properly? Whats weird is that the problem gets fixed by simply restarting the services.
Re: Strange SASL Authentication Issue
Robert Krig: > I've got a weird issue on one of my postfix installations that I can't > explain. > > My postfix setup uses MySQL as an authentication backend, and the > accounts are managed via Postfixadmin. > > All of our webservers use phpmailer to send out registration notices to > users who register on our site. Now, there are plenty of other accounts > on this mail server. Once in a while what happens, and ONLY with this > one mail account which is used to send out registration emails, is that > for no apparent reason SASL Authentication will stop working, but ONLY > for that one account. All other accounts are happy to go about their Why do you believe that there is a problem with SASL authentication between the PHP application and Postfix? Your posting provides no concrete symptoms (logging!) that would allow the list to help you towards a solution. It is not unusual for people to confuse authentication and encryption. http://www.postfix.org/DEBUG_README.html#mail. DO NOT TURN ON VERBOSE LOGGING until asked to do so. The default Postfix logging may look like useless garbage to you, but it provides a lot of detail that gets drowned out out when you open the firehose. Wietse
Strange SASL Authentication Issue
I've got a weird issue on one of my postfix installations that I can't explain. My postfix setup uses MySQL as an authentication backend, and the accounts are managed via Postfixadmin. All of our webservers use phpmailer to send out registration notices to users who register on our site. Now, there are plenty of other accounts on this mail server. Once in a while what happens, and ONLY with this one mail account which is used to send out registration emails, is that for no apparent reason SASL Authentication will stop working, but ONLY for that one account. All other accounts are happy to go about their business. It's really weird. I have no idea how to reproduce it, or what is causing it. Needless to say, it is very annoying since we only notice it a day or two later, when we see a lot of "user waiting for email registration" messages in the admin section of our website. Restarting postfix, saslauthd and authdaemon seems to get it working again, at least for a while. This has me totally baffeled. I've turned on detailed debug logging in postfix for just our webservers, so it might be a while before this hits me again. But how weird. So far the only message in the Logs that appears whenever this happens is: warning: www1.domain.com[xx.xxx.xxx]: SASL LOGIN authentication failed: authentication failure It's the same message for all our webservers which act as clients to this postfix server. What I don't understand is, how can authentication suddenly stop working for only this account, while all the other ones still keep working and authenticating? Like I've said, I've enabled debug logging for specific hosts. But that might take a while to produce some insightful results. In the meantime, does anyone have the faintest idea what might be causing a problem here?
