Re: [Mimedefang] More on filter_helo
Philip Prindeville wrote: What if the milter interface were to evolve to have an additional hook for other commands (like VRFY/EXPN/RTRN, etc) and you wanted to base how you handled those commands on what you had seen in the HELO? It would be handled the same way as filter_sender is: The filter_vrfy routine would be given the VRFY argument and the HELO argument. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] The campaign to save filter_helo
On Thu, 9 Nov 2006, David F. Skoll [EMAIL PROTECTED] wrote:
Dirk the Daring wrote:
I use filter_helo and am quite happy with it. I successfully reject
obviously fraudulent HELOs at filter_helo.
At least, you *think* you do.
If you test it, you'll discover they're only rejected at MAIL FROM: time.
No, I'm fairly sure about this.
When I re-wrote my filter to take advantage of filter_helo, I also
inserted quite a few logging statements. Mainly to insure that my filter
did what I wanted it to do.
My examination of my logs quite clearly shows that when filter_helo
ended with a return('REJECT'), the connection progressed no further. I
have plenty of examples of this.
Also, after some analysis, I found I was able to reject some 50% of
foreign (not on my network) SMTP connections by the end of filter_helo.
Between sendmail's GREETPAUSE, RATECONTROL and CONNCONTROL; and using
filter_helo to detect obviously fraudulent HELOs, I dropped half the
spammers that much sooner.
I've already removed filter_helo from the svn version of MIMEDefang;
I appreciate all the work you do, and I've always been very happy with
MIMEDefang. I'm constantly referring people looking for a better anti-SPAM
solution to the RP website.
I really wish you'd reconsider removal of filter_helo. My personal
anti-SPAM philosophy is Reject early, reject often and filter_helo helps
me do that.
you can just move your test unchanged into filter_sender.
That can be done and will work, but it allows spammers to waste that
much more of my mail relay's CPU and my network's bandwidth. If I know
they're fraudulent at HELO, why let them lie to me again at MAIL FROM ?
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] The campaign to save filter_helo
Dirk the Daring wrote: When I re-wrote my filter to take advantage of filter_helo, I also inserted quite a few logging statements. Mainly to insure that my filter did what I wanted it to do. Doh... oh, well. I have reconsidered and reinstated filter_helo. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] More on filter_helo
David F. Skoll wrote: Philip Prindeville wrote: What if the milter interface were to evolve to have an additional hook for other commands (like VRFY/EXPN/RTRN, etc) and you wanted to base how you handled those commands on what you had seen in the HELO? It would be handled the same way as filter_sender is: The filter_vrfy routine would be given the VRFY argument and the HELO argument. Regards, David. So... duplicating filter_helo() in filter_vrfy(), filter_expn(), filter_rtrn(), and any other command that can come immediately after a HELO is somehow better. Ok. I guess I'm just not smart enough to see how. -Philip ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] More on filter_helo
Philip Prindeville wrote: So... duplicating filter_helo() in filter_vrfy(), filter_expn(), filter_rtrn(), and any other command that can come immediately after a HELO is somehow better. Because of the way MIMEDefang works, the filter_vrfy() may be called in a completely different Perl process than the filter_helo() was, so you still would need to pass the HELO information as a parameter if you want to use it in filter_vrfy(). That's why filter_recipient() includes all the info in filter_sender(), and filter_sender() includes all of filter_helo(), etc. -- David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] filter_helo called after mail from?
Jonas Eckerman [EMAIL PROTECTED]:
We do our HELO checks in filter_relay,
How do you do that?
I thought that neither the $Helo variable nor the commands file (from
wich the helo string can be read) was available that early in the
mimedefang process.
It comes in as a parameter in filter_relay. Pseudo-code (my real code
has lots of calls to site-specific logging etc.):
[...]
# 2002/11/25 Anne Bennett: our machines:
$re_localhost= '127\.0\.0\.1';
$re_our_networks = '132\.205\.\d+\.\d+';
$re_our_domains =
'(?:([\w\-\.]+\.)?(concordia\.ca|concordia\.montreal\.qc\.ca|myconcordia\.ca))';
[...]
sub filter_relay($$$)
{
my ( $relayip, $relayname, $helo ) = @_;
[...]
# 2003/09/04 Anne Bennett: strip square brackets from helo string if any:
$stripped_helo = $helo;
$stripped_helo =~ s/^\[+//;
$stripped_helo =~ s/\]+$//;
# Reject any (external) HELO/EHLO that pretend to be one of ours.
# 2002/11/25 Anne Bennett: log only.
# 2003/06/23 Sylvain Robitaille: in production with rejection.
if ( ( $relayip !~ /^($re_our_networks|$re_localhost)$/ )
( $stripped_helo =~ /^($re_our_domains)$/i ) )
{
[reject with:]
... IP $relayip faked HELO/EHLO with our name '$helo';
}
# 2003/05/05 Sylvain Robitaille: Also check for obviously forged
# external numeric HELO/EHLO strings (numeric string does not match
# relay's IP address).
# 2003/06/13 Sylvain Robitaille: In practice the faked numeric HELO
# strings can trigger too many false positives, given that many
# sending sites may use NAT or double-homed hosts.
# 2003/09/04 Anne Bennett: however no one should be faking as
# one of our networks.
if ( ( $relayip !~ /^($re_our_networks|$re_localhost)$/ )
( $stripped_helo =~ /^\d+\.\d+\.\d+\.\d+$/ )
( $stripped_helo =~ /^$re_our_networks|$re_localhost$/ ) )
{
[just testing, hey, I forgot to check this and put it into production!]
[hmph, doesn't seem to catch anything]
... IP $relayip faked numeric HELO/EHLO with our IP '$helo';
}
...
}
Note: the code above actually hasn't caught much lately. I think it did
help when we first put it in, though.
Anne.
--
Ms. Anne Bennett, Senior Sysadmin, ENCS, Concordia University, Montreal H3G 1M8
[EMAIL PROTECTED]+1 514 848-2424 x2285
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] The campaign to save filter_helo
Thanks from me too... At 09:35 AM 11/9/2006, you wrote: Dirk the Daring wrote: When I re-wrote my filter to take advantage of filter_helo, I also inserted quite a few logging statements. Mainly to insure that my filter did what I wanted it to do. Doh... oh, well. I have reconsidered and reinstated filter_helo. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang John Jaeger - Billings, Montana EMail To: mailto:[EMAIL PROTECTED] Home Page : http://www.jjgb.com PGP: RSA Key ID: 0xAAEC7751 http://www.jjgb.com/public_files/RSA_Key.zip Our liberty is protected by four boxes... The ballot box, the jury box, the soap box, and the cartridge box. - Anonymous Soap Box didn't work, now using the Cartridge Box 3/20/2003 ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] filter_helo called after mail from?
Anne Bennett wrote: [HELO info] comes in as a parameter in filter_relay. You must be running an old version of MIMEDefang, because that hasn't been the case since version 2.43. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Cannot get spamassassin to use bayes
Date: Tue, 17 Oct 2006 15:37:40 +0200
From: Andrea Venturoli [EMAIL PROTECTED]
Reply-To: mimedefang@lists.roaringpenguin.com
As per subject, I'm running the most recent stable versions of
sendmail+mimedefang+spamassassin on FreeBSD 5.4. Due to the latest
overwhelming increase in spam getting in I tryed enabling bayesian
filters: I trained spamassassin and manually checked that it works quite
effectively; however I cannot get it to apply bayes when called from
mimedefang.
I've put:
use_bayes 1
bayes_path /usr/local/etc/mail/spamassassin
in /usr/local/etc/mimedefang/sa-mimedefang.cf.
This will cause you to have one Bayesian database for all users.
Bayesian testing works best when done for each user individually. In
order to do this, you may want to store the bayes stuff in an SQL
database. However, I have to warn you that this can seriously bloat
MySQL databases. Mine are currently running at just under 4 GB and
99% of that is attributable to using it to store SpamAssassin bayes
stuff.
Having gotten the warning out of the way... You will need to be
using stream_by_recipient() to be able to process each user
individually. Also you will need
$SASpamTester-signal_user_changed({ username = $user });
prior to the call to spam_assassin_check(); In order to get the
$SASpamTester variable iniitialised, you will need this code in your
filter (it used to be in the sample filter, but has since been
removed):
# The next lines force SpamAssassin modules to be loaded and rules
# to be compiled immediately. This may improve performance on busy
# mail servers. Comment the lines out if you don't like them.
if ($Features{SpamAssassin}) {
spam_assassin_init()-compile_now(1) if defined(spam_assassin_init());
}
You will also need to add some lines to your
/etc/mail/sa-mimedefang.cf file:
bayes_store_module Mail::SpamAssassin::BayesStore::MySQL
bayes_sql_dsn DBI:mysql:spamassassin:localhost
bayes_sql_username spamuser
bayes_sql_password XXX
bayes_auto_learn_threshold_nonspam 0.4
bayes_auto_learn_threshold_spam 10.0
bayes_ignore_to users@spamassassin.apache.org
You will need to setup a bunch of tables as well. See the README
files in the sql directory of the SpamAssassin distribution on how to
setup the database tables.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] The campaign to save filter_helo
Dirk: Any chance of seeing this fantastic filter_helo that blocks out 50% of your SPAM? Thanks in advance, Mark Mark van Proctor Information Systems Analyst acQuire Technology Solutions Pty Ltd Australia Chile Canada United Kingdom == This email (including all attachments) is the sole property of acQuire Technology Solutions Pty Ltd and may be confidential. If you are not the intended recipient, you must not use or forward the information contained in it. This message may not be reproduced or otherwise republished without the written consent of the sender. If you have received this message in error, please delete the email and notify the sender. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] filter_helo saved
On Thu, 9 Nov 2006, David F. Skoll [EMAIL PROTECTED] wrote: I have reconsidered and reinstated filter_helo. Thank you very much, David. I really appreciate your work on MIMEDefang. Dirk ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] sendmail and filter_helo interaction
Jim McCullars and I have been discussing filter_helo offlist, and
David's observation (supported by Jim's experimentation) that if
filter_helo returns a REJECT, the connection is not immediately rejected,
but rather is rejected after MAIL FROM.
It happens that I have been using a heavily-logged filter, and examined
some test connections that Jim made to my mailserver. Some interesting
results came out of that. I'm using sendmail v8.13.8 and MIMEDefang v2.57
on Solaris 9 with Perl v5.8.6.
It is true that even if filter_helo returns REJECT, the connection is
not immediately dropped. The sending host can still issue another command,
such as MAIL FROM.
However, it is also true that if the sending host does issue another
command, like MAIL FROM, that there is no corresponding MILTER call.
MIMEDefang never sees a call to filter_sender if filter_helo returns a
REJECT.
So it *appears* like the connection is maintained, but it also seems
that the SMTP conversation *effectively ends* if filter_helo returns
REJECT. The connecting host can issue another command, but it will be
ignored.
I've theorized that if the connecting host issues a RSET followed by
another (valid) HELO, the connection can proceed and be successful. This
might be why the connection is not immediately dropped. Also, I use
FEATURE(`delay_checks'), which may have something to do with it.
Something else also came out of the experiments. If a connecting host
violates GREETPAUSE (sends before presentation of the banner), its HELO is
still passed via MILTER call to MIMEDefang.
However, if RATECONTROL is tripped, there is no MILTER call. Presumbly,
this is true of CONNCONTROL.
Finally, I got asked about my filter_helo code. My current filter has
a *lot* of logging statements, because I've been experimenting and need
to meter its function. I'm also not a Perl hacker, so I doubt my code
is the most-efficient way to write this. Here it is...feel free to
use/adapt the code as may suit your needs, or ignore my code and write
your own:
# Some global variables used by the filter* functions
###
# Declare a hash of
# - Key: IP addresses we consider internal
# - Value: Flag as to if host is exempt from AV scans (0=No, 1=Yes)
###
%OurHosts=( 127.0.0.1, 0,
192.168.2.2, 0,
10.2.3.4, 0 );
###
# Declare a hash of
# - Key: Domain Names we host
# - Value: A flag as to if the Domain should be
# receiving E-Mail (0=No, 1=Yes)
###
%OurDomains=( mydomain.tld, 1,
otherdomain.tld, 1,
notadomain.tld, 0 );
[..other code..]
#***
# %PROCEDURE: filter_helo
# %ARGUMENTS:
# IP address of remote host; hostname of remote host; HELO string
# presented by remote host
# %RETURNS:
# 2-5 element array (see documentation)
# %DESCRIPTION:
# Called after SMTP connection has been established and sending host has
# given a HELO statement, but not MAIL FROM: or RCPT TO:
#***
sub filter_helo ()
{
# Read the parameters passed to the function
my($hostip, $hostname, $helo) = @_;
# Local string for Domain name processing
my($domainstring);
# Local variable for string indexing
my($subindex)=0;
# Search the list of our hosts using the $hostip argument
if ( exists($OurHosts{$hostip}) )
{
# Recognize our internal host
md_syslog('info', Internal Host $hostip HELO $helo);
# Don't look at it further
return('CONTINUE', 'ok');
}
else
{
# Foreign host
md_syslog('info', Foreign Host $hostip HELO $helo);
}
# Check if the HELO is an IP address
if ($helo =~ /^(\[?)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(\]?)$/ )
{
# HELO looks like an IP - the comparison will split the string
# into 3 variables; $1 will have [ or be undefined, $2
will
# have the IP address without any brackets, $3 will have
# ] or be undefined ($1 and $3 are undefined if HELO
lacked
# square brackets)
md_syslog('info', IP HELO $helo);
# Check #0
# The IP address portion should *not* be identical to the
# original HELO string - if it is, the original HELO
lacked
# brackets and therefore is invalid (this is safer than
# trying to evaluate $1 and $3 directly, as they may be
# undefined, or have garbage from a
Re: [Mimedefang] sendmail and filter_helo interaction
On Thu, 2006-11-09 at 23:06 -0500, Dirk the Daring wrote: # Check #3 # HELO should not contain localhost How effective is this for you? Do you run into false positives? # Check #4 # If the HELO is an FQDN, the index and rindex of . will not be the same # This catches the spammer using domain.tld (which will slip # by Check #2) I check that the HELO must have a ., but I haven't gone any further than that. Does this work well for you? Any false positives? Richard signature.asc Description: This is a digitally signed message part ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] sendmail and filter_helo interaction
Dirk the Daring wrote:
# Check #4
# If the HELO is an FQDN, the index and rindex of . will not
be the same
# This catches the spammer using domain.tld (which will slip
# by Check #2)
if ( index($helo, .) == rindex($helo, .) )
{
# Reject connection - invalid HELO
md_syslog('alert', Non-FQDN HELO $helo by Host $hostip);
return('REJECT', INVALID HELO/EHLO: $helo is
not FQDN);
}
As I wrote previously, my entire filter is heavily logged. My
analysis of those logs indicates that only about 50% of foreign
mailhosts connecting to my network get past HELO. Based on the
I-think-reasonable assumption that no legitimate mail server would be
tripped up by GREETPAUSE, RATECONTROL, CONNCONTROL or the tests I have
in filter_helo, my conclusion is that those 50% are spammers, and I'm
effectively stopping them by the end of HELO.
Given that I don't think check #4 is valid, I'm not sure I believe your
claim. For one, depending on the configuration I'm using, you might end
up rejecting my email, because my mail server's hostname is the
registered domain name (rudd.cc) ... and I'm not a spammer.
(I don't recall any prohibition on a host's name being just its
registered domain, domain.tld)
I'm also curious why you're using a lot of index/rindex calls instead of
regular expressions (I'm not enough of an expert to know if one is
honestly faster than the other). For the above one, why not:
$helo =~ /^[^\.]+\.[^\.]+$/
(from the start of the string, one or more non-dots, followed by 1 dot,
followed by one or more non-dots, and then the end of the string; you
can only match this expression if you have exactly 1 dot in the strong)
Or,
(($helo =~ /\./) ($helo !~ /\..+\./)
(contains at least one dot, AND does not contain: a dot, at least any
one other character, and then another dot, anywhere in the string;
again, you can only match these two expressions if you have exactly one
dot in the string)
$helo =~ /\./
also works for your index of . isn't -1 check.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
