Final
Update:
My mail server is
running perfectly smooth processing mail as soon as it arrives while running
Declude Antivirus and three DNS blacklists using Imails built-in
antispam. I have tried to re-enabled Declude junkmail, but it
immediately starts to backlog the spool directory again and mail processing is
slow. Perhaps someone can see something that I do not in this log
file… This is a sample connection with debug logging
on.
I have a service
contract with Declude and Imail for what that’s worth, but I’m not going to
get anywhere with that unless I can help point them in the right direction so
they deem me worthy to forward my issue onto someone with
experience.
I’m currently
processing what appears to be 100,000 messages per day with 3000 accounts (90%
of that incoming mail). My previous counts of 200,000 only appear when
declude junkmail is running, so they are not accurate.
Will
-----Original
Message-----
From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Will
Sent: Monday, August 01, 2005 9:05
AM
To:
[email protected]
Subject: RE: [Declude.JunkMail] Declude
Woes
Just an
update…
Last
week I had disabled Declude and cleaned out my spool directory. For the
next day I had not experienced any issues. My spooled messages are
staying well under 500 and mail is flowing exactly how I want it to (without
Declude). Now, we can get away with not having Declude Junkmail for a
bit, but I really needed virus scanning, so I re-enabled Declude
Antivirus. After I verified everything was running smoothly I downloaded
F-prot and removed Mcafee before the weekend and things have also been running
very smooth. Just a note, I had run Declude Antivirus for years without
any issues and I only seem to run into these overflow problems when I
introduce Declude Junkmail.
Since
the major issues I had last week I have made three changes… I have modified
the queue manager to process 100 messages at a time. I have disabled
Declude Junkmail. I have now moved from Mcafee to F-Prot. The only
one of these changes that has ensured that mail does not go into overflow and
backup is the disabling of Declude Junkmail. I’m not so sure it’s a DNS
issue because the Imail spam filters run perfectly fine, which I am now using
in place of Declude. They do not do as good a job identifying spam, but
they are better than nothing.
Things
to think about:
I was
interested in the statement that the mail server can be setup to not bounce
messages if they are detected as SPAM. This still confuses me, because
the BOUNCEONLYIFYOUMUST statement in the $default$.junkmail file appears to
actually bounce the messages. As it is, all my filters are set to WARN
and I don’t understand why changing them to BOUNCEONLYIFYOUMUST will
help.
I do not
believe I have bulk mailers on my system, but I will not discount it.
Does anyone have a good suggestion for determining this? After looking
through my logs and running Imail log analyzer none of my own IP’s stand out
for message counts and the invalid user entries I find are from outside
sources outside of my network. These connections are always denied and
mail does not get processed onto our server. I will think about Declude
HiJack.
I still
want to use Declude Junkmail, so I will test some of the suggestions put into
place such as whitelisting my IP range and disabling outbout virus scanning,
but other than these two suggestions, I don’t see any major changes here that
will affect me.
Will
Friday:
SpamContent
13400
SpamPhrase
19107
SpamFeatures
1283
SpamUrlDomain
35018
LocalDeliver
92331
RemoteDeliver
10526
Saturday:
SpamContent
10089
SpamPhrase
20739
SpamFeatures
1090
SpamUrlDomain
27194
LocalDeliver
83297
RemoteDeliver
9008
Sunday:
SpamContent
10425
SpamPhrase
18741
SpamFeatures
793
SpamUrlDomain
25756
LocalDeliver
81033
RemoteDeliver
10142
-----Original
Message-----
From:
[EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
On Behalf Of Matt
Sent: Friday, July 29, 2005 2:03
PM
To:
[email protected]
Subject: Re: [Declude.JunkMail] Declude
Woes
Will,
What you
probably have are users bulk-mailing through your server (evidenced by the GSE
files), issues with running both IMail Antispam and Declude, and choosing a
much slower than optimal virus scanner as your one an only engine in a
CPU-challenged environment. There could well be issues with
fragmentation and not enough disks as well.
The reports that you shared
are somewhat inconclusive, primarily because running Declude plus IMail
Antispam will skew these numbers. So there is no telling what the true
volume is. If you run Declude, you should turn IMail Antispam off and
just rely on Declude alone.
Your config looks fine except that you
should strongly consider spending about $50 on an F-Prot license and using
that instead of McAfee since you are CPU challenged.
The fact that you
are an ISP also speaks volumes to the need of restricting bulk-mailing among
your client base. Not only can they redline your server, they can also
get you blacklisted. Adding a gateway will do nothing for this part of
the issue. You need to look into integrating Declude Hijack, but be
prepared to sort through issues in the week that follows integrating
it.
To save CPU, you could whitelist your userbase in Declude by their
IP space by listing them in your global.cfg. You could also turn off
outbound virus scanning with Declude Virus Pro (not sure if that also applies
to Declude Standard).
IMO, this isn't an issue of sizing or a bug in
Declude, this is all about optimizing your configuration where it counts most,
and putting in place reasonable protections so that bulk-mailing isn't allowed
to go through your mail server. Jumping to a gateway will just be
another partial fix and it might not provide enough juice to prevent you from
backing up, especially if the load is primarily the result of customers
bulk-mailing through your server.
If you did everything that I
identified here, you would save about 1/2 to 2/3 of your current CPU
utilization. That might not cut it however since you were clearly
demanding more than 100% before, but I believe it
should.
Matt
Will wrote:
Well
thank you to everyone who responded to my need for perspective and
assistance.
I
believe I will be in need of a mail gateway if I am to continue to use
Declude.
Also,
does anyone know how to configure an Imail/Declude setup to not bounce spam
messages?
Will
-----Original
Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Michael
Jaworski
Sent: Friday, July
29, 2005 1:11 PM
To: [email protected]
Subject: RE: [Declude.JunkMail] Declude
Woes
I can
second the need for a gateway defense when under attack. We run a Windows
shop and were being crushed under multiple dictionary attacks for
two domains on a daily basis. I took the daunting task to build our first
Linux box running Postfix. The first box was a tough start though I had
a employee who had Linux experience. We are running Postfix on OpenBSD 3.6
with MySql for dynamic update ability. (I am still working on grabbing
additions, updates and deletions from SmarterMail admins so we can throw all
our domains in Postfix and update in realtime) After a few weeks we added a
second box in the event the first box went down. The second box was a breeze
since it was basically a duplication. Both mx records now point at the two
boxes. The hardware was old 500Mhz and 1ghz cpu with 512mbs of ram each. The
1ghz is primary and takes 75% of the load without much effort with plenty of
free memory. The whole setup allows the main server running
SmarterMail/Declude Pro/Sniffer/F-Prot to respond quickly to POP, web mail and
smtp traffic requests.
The
Linus approach only should cost you some time and old equipment as the
software is free. Our experience over the last two years showed it was worth
climbing the short Linux learning cliff. And it is true ... they run
forever
One
important note not related to using a gateway: We never bounce spam e-mail
back to the "sender". The backscatter traffic can kill you and skew your
reports.
Michael
Jaworski
Puget Sound Network, Inc.
(206) 217-0400
(800)
599-9485
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
On Behalf Of Will
Sent: Friday, July 29, 2005 9:38
AM
To: [email protected]
Subject: RE: [Declude.JunkMail] Declude
Woes
Well,
I’m back at it today.
Yesterday I
disabled Declude early in the day and started working mail back into the
spool directory from the overflow directory. This was a long process,
but by the end of the day I had gone from a backlog of 150,000 files in the
spool and 134,000 in the overflow directory to about 1500 (that includes
logs). During this time I needed to stop and restart the queue manger
a number of time. I did this to allow me to delete all the .gse files,
which I figured would save me time discarding them. However, by the
time I got down to 1500 files and started to watch the spool it started to
increase in size again; climbing to 4000 within a matter of minutes. I
stopped and restarted the queuemanager and these files were then
processed. I verified they were actually getting processed by sending
test messages to myself. At this point I was pretty upset and
confused. I looked through the sys logs and found nothing out of the
ordinary, queuemanger would simply stop. I set all the queuemanager
setting back to default and tried again without luck. I had to stop
and restart it every few minutes to get it to process a few thousand
messages. Finally, I purchased an Imail service agreement and upgraded
to 8.21. Magically, it worked. The queumanger started to deliver
messages as soon as they arrived. My thought immediately went into
conspiracy mode. It seems like this has happened before where we had a
perfectly workable solution and something completely confusing happened and
an upgrade magically fixes it!
Anyway… I
re-enabled declude and let it run overnight. Now I have a backlog
again. There are mostly D*.SMD files in the spool right now with all
their delivery Q* files in the overflow directory (*shakes fist at overflow
directory*). Time to start the process again today. I’m
disabling declude to get those messages out and one thing to note, after I
have stopped the smtp server and added smtpd.exe backing into the delivery
application, I still have about 20+ declude.exe processes. I have
stopped and started it again as well as the queuemanager and they are still
there. In fact they are creating more declude.exe processes as I
watch. I’m trying to kill them, but they just keep coming back… having
to restart so I can start processing mail.
We are
an ISP and here are some random examples of some of our Imail daily reports
to give you an idea of what kind of traffic we see:
--
=====================================================
MailPure custom filters for Declude JunkMail Pro.
http://www.mailpure.com/software/
=====================================================
