Hoi folx,
this is long and detailed, but maybe it also helps others in the same
situation. Also comments from other german qmail users hit by the same
problems are welcome, maybe we can get hold of the spammer together.
This weekend and also yesterday we had a nasty problem:
Some spamming software was used to forge sender addresses of two domains
hosted on our mail servers.
- the spam was injected on various relay open mail servers throughout
Germany (similar IP address blocks, so I think they were detected
by network scans)
- Received lines look like it were local injects on that machines (all Linux
hosts as far as I've checked) so they might have been hacked
- the spam advertised US products, so I don't think the spammers are
from Germany nor that they are among our customers.
- the addresses do not exist nor do similar ones. They used the domains
"link-m.de" and "i-dial.de" with different users
The problems:
- our "portal" mailserver was hit by about 100000 failure notices
within a very short period of time
- both domains are forwarded via smtproutes to other mailservers
(both also qmail)
So these were under heavy load, too, and one of them also generated
failure copies to a remote postmaster account, which made it even
worse for that server.
When we noticed the problems (Saturday night *sigh*) we already had
about 50000 emails in the queue on the portal server. The smtp port
was mostly unusable, as some hundred mailservers tried to inject
the failure notices. I first blocked (via tcpserver) the open relays
(hacked servers?) injecting "direct bounced" as soon as I detected them.
That was not really a solution.
The queue filled rapidly up, as the server couldn't get
the mails through to the other two servers and they were coming in
faster than qmail-send could schedule them for delivery.
I then reduced the number of parallel smtpds (via tcpserver) to give
qmail-send a chance and set up local delivery (via virtualdomains)
for that domains into Maildirs for "legit" users (.qmail-default) and
null deliveries (only a # line in the .qmail-user file) for the non
existing users to get rid of the emails in the queue.
After that I set up a serialmail delivery of the legit emails to
the two other hosts.
That helped getting the situation somehow "under control" and after
3 hours or so we had a kinda stable situation and I increased the number
of parallel smptds again.
However a few questions/actions remain:
Actions:
- I'll patch qmail-smtpd to make use of a badrcpto file, to get rid of
the emails at an earlier stage and hopefully faster to keep the
smtp port usable
- I'll use Russells big-todo-patch and hope that qmail-send will be
more productive in similar situations
Questions:
- what would be the most effective way to get rid of some 10000 emails
in the queue if all I know is the addressee?
- how can I detect such a situation *fast* besides doing a qmail-qstat
every e.g. 15 minutes and comparing the results against some limits
and issue some alarm (of course not via email ;-))
- any other proposals for the "in the middle of an attack" steps besides
those I have used?
- any other ideas, comments on how to prevent/handle that kind of
problems?
Thanks a lot,
\Maex
P.S. We're still analyzing the bounces we've saved away and will contact
all the admins of the "injecting" servers to get hold of the
spammers. However I fear that if he really is from US the chances of
legal actions against them are minimal :-(((
--
SpaceNet GmbH | http://www.Space.Net/ | Yeah, yo mama dresses
Research & Development | mailto:[EMAIL PROTECTED] | you funny and you need
Joseph-Dollinger-Bogen 14 | Tel: +49 (89) 32356-0 | a mouse to delete files
D-80807 Muenchen | Fax: +49 (89) 32356-299 |
