Re: Postfix using all CPU after nightly mail submission
Hi Wietse, I limited my postfix installation to default_process_limit 5, 4, 3, 2, and even 1, and still saw the same effects. I am thinking it might be either my opendkim milter (which applies the DKIM signature for each mail) or SASL as these are the only other processes on the server. Are you aware of any issues with either related to I/O? I have not seen any configuration settings for opendkim to do any performance throttling. I am still dumbfounded how this continues to occur. I am not sending mail in large quantity - maybe 7,000-8,000 total - just in a short amount of time. The I/O shouldn't be THAT high... at least not to leave the server unresponsive... the mail client connects to my server every evening (around midnight) and sends mails in a burst fashion within an hour or so. I did as you suggested and opened a console on the VMWare host, did a tail of the mail log, and it sent mail for a good 5-10 minutes before finally becoming unresponsive. I tried to Ctrl-C out of tail, nothing. I've done the same monitoring with top and still see no culprit for the sudden halt. I check syslog and other logs on the server and see no crashes or panics. Any other ideas what might be causing this? Further debugging I can do? Thanks On Fri, Jan 19, 2018 at 2:26 PM, Wietse Venema <wie...@porcupine.org> wrote: > Zach Sheppard: > > Wietse: > > > > I have not made any changes to rsyslog.conf. All it does it redirect all > > mail log messages to one log in /var/log/mail which I rotate with a cron > > script nightly. However, I do agree that it really could be the only > other > > process that could be hanging the server. > > > > I'm not able to determine what program is consuming the CPU because I > can't > > login to the console when this occurs. The only way I can recover the > > machine is by forcibly powering off. > > I suspect that heavy I/O from Postfix and syslog is too much for > your VM. > > To diagnose the problem, run screen(1) on a stable machine, and > then open a login session into the VM while it still responsive. > Then come back to that screen session when things go bad. You're > likely to find that when the VM is very slow, all time is spent in > the VM's kernel, and the host's VMM. > > Note that VMs, while fine for CPU-bound jobs, can introduce serious > CPU overhead for things that do massive amounts of I/O like Postfix > plus syslog. > > If you can't get a better VM, you can reduce the impact from a > 'large' mailing by reducing the number of concurrent Postfix SMTP > server and client processes. > > # postconf default_process_limit=10 > # postfix reload > > Wietse > -- This message may contain confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Don't send "sender non-delivery notification" to one sender
Is it possible to exempt one sender from receiving non-delivery notifications? We have a DMARC policy that forces DKIM headers on all mail and when Postfix sends a non-delivery notification to this bulk mail sender (a fake e-mail alias) it fails because of our DMARC policy. I have set notify_classes= to not receive any e-mails about errors/bounces but it looks like the non-delivery notifications still occur. How could I block one sender from receiving those notifications? Thanks
Re: Postfix using all CPU after nightly mail submission
Hi Viktor: I configured that internally generated mail to be filtered through our DKIM milter. This was to allow deliveries of error messages from the server to my inbox. However, I see above how this is not ideal since that can just be delivered to the root user inbox. I have turned off that setting. After making several tweaks from suggestions in this thread I saw much better performance last night. On Fri, Jan 19, 2018 at 2:25 PM, Viktor Dukhovni <postfix-us...@dukhovni.org > wrote: > On Fri, Jan 19, 2018 at 02:04:05PM -0500, Zach Sheppard wrote: > > > I have not made any changes to rsyslog.conf. All it does it redirect all > > mail log messages to one log in /var/log/mail which I rotate with a cron > > script nightly. However, I do agree that it really could be the only > other > > process that could be hanging the server. > > Your milter could be another culprit, and you've unwisely configured > filtering of internally generated mail, including notices about > problems. So the first thing is to undo the filtering of internally > generated mail. See also my other comments about your config. > > As for syslog, for MTAs I prefer syslog-ng, to rsyslog, and make > sure that /dev/log is configured as a "dgram" not "stream" socket. > Also make sure that log writes are not synchronous. > > -- > Viktor. > -- This message may contain confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Re: Postfix using all CPU after nightly mail submission
Wietse: I have not made any changes to rsyslog.conf. All it does it redirect all mail log messages to one log in /var/log/mail which I rotate with a cron script nightly. However, I do agree that it really could be the only other process that could be hanging the server. I'm not able to determine what program is consuming the CPU because I can't login to the console when this occurs. The only way I can recover the machine is by forcibly powering off. # /etc/rsyslog.confConfiguration file for rsyslog. # # For more information see # /usr/share/doc/rsyslog-doc/html/rsyslog_conf.html # MODULES # module(load="imuxsock") # provides support for local system logging module(load="imklog") # provides kernel logging support #module(load="immark") # provides --MARK-- message capability # provides UDP syslog reception #module(load="imudp") #input(type="imudp" port="514") # provides TCP syslog reception #module(load="imtcp") #input(type="imtcp" port="514") ### GLOBAL DIRECTIVES ### # # Use traditional timestamp format. # To enable high precision timestamps, comment out the following line. # $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # # Set the default permissions for all log files. # $FileOwner root $FileGroup adm $FileCreateMode 0640 $DirCreateMode 0755 $Umask 0022 # # Where to place spool and state files # $WorkDirectory /var/spool/rsyslog # # Include all config files in /etc/rsyslog.d/ # $IncludeConfig /etc/rsyslog.d/*.conf ### RULES ### # # First some standard log files. Log by facility. # auth,authpriv.* /var/log/auth.log *.*;auth,authpriv.none -/var/log/syslog #cron.* /var/log/cron.log daemon.*-/var/log/daemon.log kern.* -/var/log/kern.log lpr.* -/var/log/lpr.log mail.* -/var/log/mail/mail.log user.* -/var/log/user.log # # Logging for the mail system. Split it up so that # it is easy to write scripts to parse these files. # #mail.info -/var/log/mail.info #mail.warn -/var/log/mail.warn #mail.err /var/log/mail.err # # Some "catch-all" log files. # *.=debug;\ auth,authpriv.none;\ news.none;mail.none -/var/log/debug *.=info;*.=notice;*.=warn;\ auth,authpriv.none;\ cron,daemon.none;\ mail,news.none -/var/log/messages # # Emergencies are sent to everybody logged in. # *.emerg :omusrmsg:* On Fri, Jan 19, 2018 at 1:20 PM, Wietse Venema <wie...@porcupine.org> wrote: > Zach Sheppard: > > However, whenever Postfix gets a large e-mail load it processes the > e-mails > > for around 30-45 minutes and then consistently uses around 70-80% of the > > CPU effectively locking up the entire system so much so that I can't even > > login and debug without either disabling the network card in VMWare or > > rebooting the system totally. The VM has 2 CPU cores and 4GB RAM > allocated > > to it which I thought was plenty for this application. > > Did you determine out WHAT PROGRAM is consuming CPU? As Postfix is > totally I/O limited, I suspect that there may be some other system > component that is messing up. > > For example, it is well known that one badly configured syslog > service can use up lots more CPU than all of Postfix combined. > > Wietse > -- This message may contain confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Re: Postfix using all CPU after nightly mail submission
nd unix - n n - 2 pipe
flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store
${nexthop} ${user} ${extension}
mailman unix - n n - - pipe
flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
${nexthop} ${user}
-- end of postfinger output --
On Fri, Jan 19, 2018 at 9:06 AM, Bastian Blank <
bastian+postfix-users=postfix@waldi.eu.org> wrote:
> On Fri, Jan 19, 2018 at 08:52:39AM -0500, Zach Sheppard wrote:
> > I see no errors or alarms in the mail log nor syslog. The only odd
> message
> > I see is a series of ^@ symbols before the system freezes:
>
> Please read http://www.postfix.org/DEBUG_README.html#mail about how to
> report problems.
>
> > Jan 15 00:42:41 mailrelay opendkim[1643]: 385B38094E: DKIM-Signature
> field
> > added (s=mailrelay, d=oconee.k12.sc.us)
> > Jan 15 00:42:41 mailrelay postfix/qmgr[5601]: 385B38094E: from=<
> > ...@oconee.k12.sc.us>, size=2282, nrcpt=1 (queue active)
> > Jan 15 00:42:41 mailrelay postfix/smtpd[6128]: 8BFCC80968:
> > client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN,
> sasl_username=AAA@
> > oconee.k12.sc.us
>
> You have a broken system, nothing reports "DDD.DD.DD.DD" as ip address.
>
> > This message may contain confidential information and is intended only
> for
> > the individuals named. If you are not the named addressee you should not
> > disseminate, distribute or copy this e-mail. Please notify the sender
> > immediately by e-mail if you have received this e-mail by mistake and
> > delete this e-mail from your system. If you are not the intended
> recipient
> > you are notified that disclosing, copying, distributing or taking any
> > action in reliance on the contents of this information is strictly
> > prohibited.
>
> Please remove this statement. This mailing-list is already public
> archived.
>
> Bastian
>
> --
> Those who hate and fight must stop themselves -- otherwise it is not
> stopped.
> -- Spock, "Day of the Dove", stardate unknown
>
Postfix using all CPU after nightly mail submission
Hello, We use a Postfix installation on Debian 9 in our network to serve as a send-only SMTP server for a very large application in our school district. This application does not have its own built-in SMTP server and requires a relay to send notification e-mails to students, teachers, and parents topping out at around 10,000 e-mails per day. Our hosted e-mail solution (G Suite) has a sending cap that we hit with this application so we setup this Postfix installation to serve as an alternative. However, whenever Postfix gets a large e-mail load it processes the e-mails for around 30-45 minutes and then consistently uses around 70-80% of the CPU effectively locking up the entire system so much so that I can't even login and debug without either disabling the network card in VMWare or rebooting the system totally. The VM has 2 CPU cores and 4GB RAM allocated to it which I thought was plenty for this application. I see no errors or alarms in the mail log nor syslog. The only odd message I see is a series of ^@ symbols before the system freezes: Jan 15 00:42:41 mailrelay opendkim[1643]: 385B38094E: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:41 mailrelay postfix/qmgr[5601]: 385B38094E: from=< ...@oconee.k12.sc.us>, size=2282, nrcpt=1 (queue active) Jan 15 00:42:41 mailrelay postfix/smtpd[6128]: 8BFCC80968: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:41 mailrelay postfix/cleanup[6375]: 8BFCC80968: message-id=< 734906173.4337.1515994961560.javamail....@oconee.k12.sc.us> Jan 15 00:42:41 mailrelay opendkim[1643]: 8BFCC80968: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:41 mailrelay postfix/qmgr[5601]: 8BFCC80968: from=< ...@oconee.k12.sc.us>, size=2370, nrcpt=1 (queue active) Jan 15 00:42:41 mailrelay postfix/smtpd[6128]: B66B38096E: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:41 mailrelay postfix/cleanup[6375]: B66B38096E: message-id=< 902964141.4338.1515994961733.javamail....@oconee.k12.sc.us> Jan 15 00:42:41 mailrelay opendkim[1643]: B66B38096E: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:41 mailrelay postfix/qmgr[5601]: B66B38096E: from=< ...@oconee.k12.sc.us>, size=3705, nrcpt=1 (queue active) Jan 15 00:42:41 mailrelay postfix/smtpd[6128]: E39EA8096F: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:41 mailrelay postfix/cleanup[6375]: E39EA8096F: message-id=< 1233892182.4339.1515994961918.javamail....@oconee.k12.sc.us> Jan 15 00:42:41 mailrelay opendkim[1643]: E39EA8096F: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:42 mailrelay postfix/qmgr[5601]: E39EA8096F: from=< ...@oconee.k12.sc.us>, size=2281, nrcpt=1 (queue active) Jan 15 00:42:42 mailrelay postfix/smtpd[6128]: 25EFC80970: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:42 mailrelay postfix/cleanup[6375]: 25EFC80970: message-id=< 2097536797.4340.1515994962141.javamail....@oconee.k12.sc.us> Jan 15 00:42:42 mailrelay opendkim[1643]: 25EFC80970: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:42 mailrelay postfix/qmgr[5601]: 25EFC80970: from=< ...@oconee.k12.sc.us>, size=2486, nrcpt=1 (queue active) Jan 15 00:42:42 mailrelay postfix/smtpd[6128]: 5329380972: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:42 mailrelay postfix/cleanup[6375]: 5329380972: message-id=< 951323567.4341.1515994962327.javamail....@oconee.k12.sc.us> Jan 15 00:42:42 mailrelay opendkim[1643]: 5329380972: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:42 mailrelay postfix/qmgr[5601]: 5329380972: from=< ...@oconee.k12.sc.us>, size=2326, nrcpt=1 (queue active) Jan 15 00:42:42 mailrelay postfix/smtpd[6128]: 8EF0980973: client=unknown[DDD.DD.DD.DD], sasl_method=LOGIN, sasl_username=AAA@ oconee.k12.sc.us Jan 15 00:42:42 mailrelay postfix/cleanup[6375]: 8EF0980973: message-id=< 484411487.4342.1515994962572.javamail....@oconee.k12.sc.us> Jan 15 00:42:42 mailrelay opendkim[1643]: 8EF0980973: DKIM-Signature field added (s=mailrelay, d=oconee.k12.sc.us) Jan 15 00:42:42 mailrelay postfix/qmgr[5601]: 8EF0980973: from=< ...@oconee.k12.sc.us>, size=2408, nrcpt=1 (queue active)^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^ @^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
