Re: Alert of unusually large queue
On 10/16/12 8:15 AM, Wietse Venema wrote:
Kaleb Hosie:
We host a mail server which runs Postfix and there has been a few
times where one of our clients computers becomes infected with
malware and the password is compromised.
How this has come to my attention is because every once in a while,
I will login to the mail server and see an unusually large mail
queue which is all being sent to one domain.
Is it possible to monitor the queue automatically and have it send
me an alert if the postfix queue reaches over a certain threshold?
To fight symptoms, run a cron job every 10 minutes or so:
#!/bin/sh
postqueue -p | awk '
BEGIN { limit = 10240 }
/^-- .+ Kbytes in .+ Request/ { queue_len = $5}
END { if (queue_len > limit)
print "Queue size", queue_len | "mail -s 'Queue size problem' root"
}
'
To throttle clients that send too much mail, see postfwd, policyd
and the like.
Wietse
Another method would be to use SNMP monitoring, which we have setup and
works quite nicely as our monitoring system will send email alerts, sms
messages, and phone calls. On the mail server (centos 5/6) there is a
script that is grabbing the size of the mailq:
mon_queue.sh
#!/bin/bash
# assume queue length of zero
count=0
# Place the output of postqueue into the array variable
# Output should look something like:
# -- 285 Kbytes in 20 Requests.
# (without the leading "# ")
queuelength=( $(/usr/sbin/postqueue -p | tail -n 1) )
# Make sure the array has six elements, we are interested in the
# fifth element (index 4 because arrays in BASH are 0-based)
if (( ${#queuelength[*]} == 6 ))
then
if (( queuelength[4] > 0 ))
then
count=${queuelength[4]}
else
count=0
fi
elif (( ${#queuelength[*]} == 4 )) && [[ ${queuelength[*]} = "Mail
queue is empty" ]]
then
count=0
else
# unknown output from postqueue
count="0"
fi
echo ${count}
exit
Then inside snmpd.conf add the line:
exec postqueue /usr/bin/sudo /path/to/script/mon_queue.sh
Now you should have an snmp oid of something like
.1.3.6.1.4.1.2021.8.1.101.1 which can be used to query. Then setup your
SNMP monitoring server with thresholds for queue size, and enjoy having
a historical perspective on queues.
RE: Alert of unusually large queue
> -Original Message- > From: owner-postfix-us...@postfix.org [mailto:owner-postfix- > us...@postfix.org] On Behalf Of Jan P. Kessler > Sent: 22 October 2012 15:44 > To: postfix-users@postfix.org > Subject: Re: Alert of unusually large queue > > > >> I'm not sure, if sending an e-mail about a "full mailqueue"-condition > >> is the best way to go ;-) > > depends > > > > if you have no bulk-mail on your server it will tak enot too long to > > find a good value to adjust the "50" and as example if i have > > 500 queued messages i like to look if there is soemthing going wrong > > > What I meant was, that there is a good chance, that you will not receive > this notification, because whatever condition causes your mails to stuck > in the queue could stop that notification, too ;-) > > As mentioned by other posters you should set up a real monitoring > system, that periodically checks your queue or generates an alert (e.g. > snmp trap) on the server which does not rely on the mechanism that you > are trying to monitor (here smtp). > > cheers, jpk That's a good point, it might be worthwhile looking into something like a php script that interfaces with an SMS API. I've seen that done in the past. Kind regards, James Day (IT Engineer)
Re: Alert of unusually large queue
Jan P. Kessler: > As mentioned by other posters you should set up a real monitoring > system, that periodically checks your queue or generates an alert (e.g. > snmp trap) on the server which does not rely on the mechanism that you > are trying to monitor (here smtp). To monitor an SMTP server, try to send a test message into it, and raise an alarm if that test message is not delivered to mailbox or smtp within some deadline. Wietse
Re: Alert of unusually large queue
>> I'm not sure, if sending an e-mail about a "full mailqueue"-condition is >> the best way to go ;-) > depends > > if you have no bulk-mail on your server it will tak enot too long > to find a good value to adjust the "50" and as example if i have > 500 queued messages i like to look if there is soemthing going > wrong What I meant was, that there is a good chance, that you will not receive this notification, because whatever condition causes your mails to stuck in the queue could stop that notification, too ;-) As mentioned by other posters you should set up a real monitoring system, that periodically checks your queue or generates an alert (e.g. snmp trap) on the server which does not rely on the mechanism that you are trying to monitor (here smtp). cheers, jpk
Re: Alert of unusually large queue
Am 20.10.2012 22:08, schrieb Jan P. Kessler: > Hey guys, > >> if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is" >> `$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; >> fi > > I'm not sure, if sending an e-mail about a "full mailqueue"-condition is > the best way to go ;-) > > cheers, Jan > perhaps use nagios, xymon etc monitor prog so i.e the monitor client actions some mailq watch script which results go to the monitor server, which alerts you via mail and/or sms -- Best Regards MfG Robert Schetterer [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich
Re: Alert of unusually large queue
On Oct 20, 2012, at 4:08 PM, Jan P. Kessler wrote: > Hey guys, > >> if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is" >> `$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; >> fi > > I'm not sure, if sending an e-mail about a "full mailqueue"-condition is > the best way to go ;-) Any of these checks could be handled by Nagios or anything else that can easily execute a remote command. Or tie it into your snmp daemon… But yeah, a problem with a giant queue that piles up between cron'd intervals could certainly lead to some missed alerts. :) Charles > cheers, Jan >
Re: Alert of unusually large queue
Am 20.10.2012 22:08, schrieb Jan P. Kessler: > Hey guys, > >> if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is" >> `$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; >> fi > > I'm not sure, if sending an e-mail about a "full mailqueue"-condition is > the best way to go ;-) depends if you have no bulk-mail on your server it will tak enot too long to find a good value to adjust the "50" and as example if i have 500 queued messages i like to look if there is soemthing going wrong the only optimizing for me would be to send another notify if the count goes down and prevent sending multiple notfies after reaching the configured limit signature.asc Description: OpenPGP digital signature
Re: Alert of unusually large queue
Hey guys, > if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is" > `$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; fi I'm not sure, if sending an e-mail about a "full mailqueue"-condition is the best way to go ;-) cheers, Jan
Re: Alert of unusually large queue
Am 16.10.2012 15:20, schrieb James Day:
> I use the following to do just that. I'm sure there is a better way but I
> fudged this together myself
>
> Script 1:
>
> #!/bin/bash
>
> /usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}' >
> /etc/postfix/mailq_count
>
> Script 2:
>
> #!/bin/bash
>
> mailq_count="/bin/cat /etc/postfix/mailq_count"
>
> if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is"
> `$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; fi
>
>
> These run as cron jobs every few minutes.
thank you for that
i optimized this to one script without temp-file
mailq_count=`expr $mailq_count + 0`
this makes sure that we have a number if queue is empty
otherwise: /usr/local/bin/watch-queue.sh: line 4: [: -gt: unary operator
expected
_
#!/bin/bash
mailq_count=`/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}'`
mailq_count=`expr $mailq_count + 0`
if [ $mailq_count -gt 50 ]; then
echo "Mail count on Server is $mailq_count"
fi
signature.asc
Description: OpenPGP digital signature
Re: Alert of unusually large queue
On Tue, Oct 16, 2012 at 09:15:54AM -0400, Wietse Venema wrote: > Kaleb Hosie: > > We host a mail server which runs Postfix and there has been a > > few times where one of our clients computers becomes infected > > with malware and the password is compromised. > > > > How this has come to my attention is because every once in a > > while, I will login to the mail server and see an unusually > > large mail queue which is all being sent to one domain. > > > > Is it possible to monitor the queue automatically and have it > > send me an alert if the postfix queue reaches over a certain > > threshold? > > To fight symptoms, run a cron job every 10 minutes or so: snip > To throttle clients that send too much mail, see postfwd, > policyd and the like. +1, you need to be proactive against this kind of thing. In addition to client rate limiting, you should use content filtering of your submission stream. The vast majority of such ratware will have URIBL-listed content in the spew, so SpamAssassin URIBL lookups are likely to be very effective. This is the growing threat against email, given the overall success of DNSBLs against the previous generation of ratware. -- http://rob0.nodns4.us/ -- system administration and consulting Offlist GMX mail is seen only if "/dev/rob0" is in the Subject:
RE: Alert of unusually large queue
>From: owner-postfix-us...@postfix.org [mailto:owner-postfix-us...@postfix.org]
>On Behalf Of Kaleb Hosie
>Sent: 16 October 2012 13:50
>To: postfix-users@postfix.org
>Subject: Alert of unusually large queue
>
>We host a mail server which runs Postfix and there has been a few times where
>one of our clients computers becomes infected with malware and the password is
>compromised.
>
>How this has come to my attention is because every once in a while, I will
>login to the mail server and see an unusually large mail queue which is all
>being sent to one domain.
>
>Is it possible to monitor the queue automatically and have it send me an alert
>if the postfix queue reaches over a certain threshold?
>
>Thanks everyone
I use the following to do just that. I'm sure there is a better way but I
fudged this together myself
Script 1:
#!/bin/bash
/usr/bin/mailq | /usr/bin/tail -n1 | /usr/bin/gawk '{print $5}' >
/etc/postfix/mailq_count
Script 2:
#!/bin/bash
mailq_count="/bin/cat /etc/postfix/mailq_count"
if [ `$mailq_count` -gt 50 ]; then echo "Mail count on Server is"
`$mailq_count`|/usr/sbin/sendmail -f r...@example.com repo...@example.com ; fi
These run as cron jobs every few minutes.
Hope that helps.
Kind regards,
James Day
(IT Engineer)
Re: Alert of unusually large queue
Kaleb Hosie:
> We host a mail server which runs Postfix and there has been a few
> times where one of our clients computers becomes infected with
> malware and the password is compromised.
>
> How this has come to my attention is because every once in a while,
> I will login to the mail server and see an unusually large mail
> queue which is all being sent to one domain.
>
> Is it possible to monitor the queue automatically and have it send
> me an alert if the postfix queue reaches over a certain threshold?
To fight symptoms, run a cron job every 10 minutes or so:
#!/bin/sh
postqueue -p | awk '
BEGIN { limit = 10240 }
/^-- .+ Kbytes in .+ Request/ { queue_len = $5}
END { if (queue_len > limit)
print "Queue size", queue_len | "mail -s 'Queue size problem' root"
}
'
To throttle clients that send too much mail, see postfwd, policyd
and the like.
Wietse
