[Mimedefang] Spammers - was Re: md_check_against_smtp_server and md_graphdefang_log
--- On Tue, 3/26/13, Les Mikesell lesmikes...@gmail.com wrote: Besides which, real spammers are much more likely to take the trouble to set up SPF properly than an ordinary person who just wants to send you a message that you'd want to see. 1) When a spammer uses SPF, recipients KNOW the spammer domains and servers and automatically block them. Such is most effective with a shared reporting namespace of spammers (like spamhaus or spamcop). Unless they're registering domains with stolen credit cards, burning alot of domains like that is going to get expensive, even with the cheapest registration services. 2) Spammers seem to be avoiding spoofing domains that have proper SPF records set up, while not embracing SPF for their own domains. (cf. http://spf-all.com/ - where only 0.6% of domains having SPF records used +all -- 139,000 of 88 million.) ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] md_check_against_smtp_server() returned an empty response
This is unrelated to the other thread. I'm troubleshooting an issue where md_check_against_smtp_server() intermittently hangs for 5 minutes. It's *always* 300 seconds exactly (ignoring sub-second precision). The error is always the returned an empty response from get_smtp_return_code(). I'm using this function for doing callbacks against the sender's address. I look up the MX records using Net::DNS. Before you hate me too much, I cache positive responses for 7 days and negative responses for 1 hour. The delays discussed above are around the actual md_check_against_smtp_server() call and *nothing else*. To get this far, I had to instrument my filter with logging before and after every non-trivial block of code. Does anyone have any idea why the 15 second IO::Socket::INET timeout isn't applying in these cases? Thanks, Richard signature.asc Description: This is a digitally signed message part ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server() returned an empty response
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 27 Mar 2013, Richard Laager wrote: I'm troubleshooting an issue where md_check_against_smtp_server() intermittently hangs for 5 minutes. It's *always* 300 seconds exactly (ignoring sub-second precision). I'm using this function for doing callbacks against the sender's address. I look up the MX records using Net::DNS. Before you hate me too :-| Does anyone have any idea why the 15 second IO::Socket::INET timeout isn't applying in these cases? tar-pitting? IMHO, the timeout kicks in when there are no TCP alive packets. But when the other partner sends keep alive packets, but does not respond simply. - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) iQEVAwUBUVK4+p8mjdm1m0FfAQL84wgAkFbNP1dGeHiQ1RhilKnPpyGdxHQ4GwSQ bFwK2IUhn3WfBSg0n9xHZZpZidsbsc8rwap3ExKEvJwgPgJJF4iVo5y3MvHTK0cQ HTyN85eovgmsdxdnBk3aa7zJoc9QKfCMj4soTNtmaMMVYcaFXS82hpUwluf6ukZD z0stXY4pD+IngHcs+LXlUKa5o7ysU+/t3p12no+/yZqpEty97fJq8v2gk0HqbwF0 RQG+WaBNJNJ9mmSwAfGjgLV6m5K/9EHVXyv2e9Nmm+Mc5yGY6j10Rb+Flm4erN/9 PTSCNmrb+hGoviubzAAKGhVTSTvrZX4uIowg37VrDijknZpc52ekfA== =BsnM -END PGP SIGNATURE- ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
Am 27.03.2013 01:43, schrieb James Curtis:
I made the modification to /etc/sysconfig/mimedefang (still not sure why this
is referred to as 'running with the -t option')
MX_RECIPIENT_CHECK=yes
That sysconfig setting tells the init script to start MIMEdefang
with the -t option. Running with the -t option is low-level
speak, running with MX_RECIPIENT_CHECK enabled would be the
corresponding high-level expression.
-Code
sub filter_recipient
{
my($recip, $sender, $ip, $host, $first, $helo, $rcpt_mailer, $rcpt_host,
$rcpt_addr) = @_;
return md_check_against_smtp_server($sender, $recip, localhost,
192.168.1.10);
}
Code-
I now get the following response before the data phase that rejects the user!
550 5.1.1 t...@testdomain.com... User unknown
-
Yea! It's working
Good.
I tried changing the code as suggested below:
--Code-
sub filter_recipient
{
my ($retval, $code, $dsn, $text) = md_check_against_smtp_server($sender,
$recip, localhost, 192.168.1.10);
# if ($retval eq Reject) {
# md_graphdefang_log('notauser', $recip, $sender);
# return action_discard
#}
#else{
# md_graphdefang_log('valid', $retval, $code);
# }
}
-Code--
I told you not to remove the first line
my($recip, $sender, ...) = @_;
Without that line, the $sender and $recip variables will not be set,
so your md_check_against_smtp_server call will try to check an empty
address.
I also told you to compare $retval to REJECT in all capitals in
your if statement because that's what md_check_against_smtp_server
will return. As it stands, the comparison will never be true so
the else branch will always be run.
And finally I told you to insert a return statement before the
closing brace. The code above will return an empty result, which
the caller doesn't expect.
Btw, return action_discard is not appropriate in filter_recipient
either. It should be something like return('REJECT', 'You lose!');.
Having tried with the remarked statement and having it fail, I remarked it
back to just ther ecommended line and it still fails all email address',
whether valid or not valid with this response:
501 5.5.4 Invalid Address
And quite rightly so. The empty address is indeed invalid as a
recipient. :-)
In sum, try something like this: (Sorry for the line wraps.)
-Code
sub filter_recipient
{
my($recip, $sender, $ip, $host, $first, $helo, $rcpt_mailer,
$rcpt_host, $rcpt_addr) = @_;
my($retval, $code, $dsn, $text) =
md_check_against_smtp_server($sender, $recip, localhost, 192.168.1.10);
if ($retval eq REJECT) {
md_graphdefang_log('notauser', $recip, $sender);
return ('REJECT', 'go away');
}
else {
md_graphdefang_log('valid', $retval, $code);
return ('CONTINUE', 'ok');
}
}
Code-
HTH
T.
--
Tilman Schmidt
Phoenix Software GmbH
Bonn, Germany
signature.asc
Description: OpenPGP digital signature
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Spammers - was Re: md_check_against_smtp_server and md_graphdefang_log
On Tue, 26 Mar 2013 23:14:15 -0700 (PDT) kd6...@yahoo.com wrote: 1) When a spammer uses SPF, recipients KNOW the spammer domains and servers and automatically block them. Eventually. But when spammers register domains and throw them away after a few hours' use, it can be difficult to keep up. Such is most effective with a shared reporting namespace of spammers (like spamhaus or spamcop). Unless they're registering domains with stolen credit cards, burning alot of domains like that is going to get expensive, even with the cheapest registration services. Spam gangs sell spam software and time on botnets to suckers. If you're charging someone a few hundred bucks to do a spam run, adding $6 for domain name registration doesn't significantly affect the economics. 2) Spammers seem to be avoiding spoofing domains that have proper SPF records set up, That's not my feeling. Taking a look at a couple of hours' worth of data from one of our scanners, we see: SPF Pass: = 18531 accepted as ham 14622 considered spam so the correlation between SPF pass and non-spam is very mild indeed. SPF Softfail: = 1243 accepted as ham 1343 considered spam Again quite mild correlation. SPF Fail: = 68 accepted as ham 553 considered spam Better correlation, but still a significant FP rate if we outright rejected SPF Fail mail. Furthermore, if we look at some of the SPF Fails, we see things like this (obfuscated to protect the guilty): 2013-03-27T08:30:48.487010-04:00 colo4 CanIt[22686]: r2RCUlQB029099: what=accepted, stream=u...@example.com, realm=example-com, country_code=NL, linktype=Ethernet or modem, nrcpts=1, os=Linux, osver=2.6.x, prob=0.0001, relay=x.y.z.a, score=4.9, sender=noci...@example.org, tests=T_RP_MATCHES_RCVD:-0.01;SPF(fail:5);DKIM(none:0);RBL(rp-good:-0.1), subject=Scheduled Task Endpoint Security Scan - Completed That's from a network monitoring system that sends (by default) from the domain of the system manufacturer. Since most people don't bother changing defaults, millions of these alert mails go out every day and fail SPF. Welcome to the IT industry... a giant pile of fail. Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
Date: Wed, 27 Mar 2013 12:19:30 schm...@phoenixsoftware.de
I told you not to remove the first line
my($recip, $sender, ...) = @_;
Without that line, the $sender and $recip variables will not be set,
so your md_check_against_smtp_server call will try to check an empty
address.
I also told you to compare $retval to REJECT in all capitals in
your if statement because that's what md_check_against_smtp_server
will return. As it stands, the comparison will never be true so
the else branch will always be run.
And finally I told you to insert a return statement before the
closing brace. The code above will return an empty result, which
the caller doesn't expect.
Btw, return action_discard is not appropriate in filter_recipient
either. It should be something like return('REJECT', 'You lose!');.
Having tried with the remarked statement and having it fail, I remarked it
back to just ther ecommended line and it still fails all email address',
whether valid or not valid with this response:
501 5.5.4 Invalid Address
And quite rightly so. The empty address is indeed invalid as a
recipient. :-)
In sum, try something like this: (Sorry for the line wraps.)
-Code
sub filter_recipient
{
(redacted for brevity) ...
}
}
Code-
Thanks for the thorough explanation. I had spent some time looking at the
mimedefang.pl script (specifically the interaction between
md_check_against_smtp_server and watching how it calls get_smtp_return_code)
and I was getting close.
.
I put in the code as you provided, but it still doesn't log anything (either
valid or notauser) into the md_graphdefang_log. However I did find this in my
/var/log/maillog
Mar 27 09:59:40 monitor mimedefang.pl[23405]: md_graphdefang_log called outside
of message context
-Bill
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
On 3/27/2013 10:05 AM, James Curtis wrote: However I did find this in my /var/log/maillog Mar 27 09:59:40 monitor mimedefang.pl[23405]: md_graphdefang_log called outside of message context # man mimedefang-filter [...] md_graphdefang_log($event, $v1, $v2) [...] Note that md_graphdefang_log should not be used in filter_relay, filter_sender or filter_recipient. The global variables it relies on are not valid in that context. [...] I work around this in my filters by making a my_graphdefang_log() that doesn't access the variables that aren't yet defined in the above contexts. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
27 Mar 2013 10:59:45 -0400 -mattg Note that md_graphdefang_log should not be used in filter_relay, filter_sender or filter_recipient. The global variables it relies on are not valid in that context. [...] This has been a very long thread to say, You can't get there from here :-) I work around this in my filters by making a my_graphdefang_log() that doesn't access the variables that aren't yet defined in the above contexts. I am glad that I have been able to get the md_check_against_smtp_server added to remove the backscatter. I am really glad that I've learned some context for Perl (some concepts from BASIC apply, but that's about it). Can't imagine the amount of errors I'll incur when creating a new sub-routine for my_graphdefang_log(), but it will be a fun time. I'm assuming that I start with the code from md_graphdefang_log and remove the variables. Cheers, (can you hear the sound of my O'Reily Perl for Beginners book opening?) Thanks for all the help and consideration during this process from all involved. -Bill Curtis ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
[Mimedefang] What about DKIM
Hey all, Now that we've see/talked some stats on SPF... I'd be interested to know what anyone might have to offer on DKIM usefulness. -Ben -- Ben Kamen - O.D.T., S.P. -- eMail: b...@benjammin.net http://www.benjammin.net Fortune says: A man either lives life as it happens to him, meets it head-on and licks it, or he turns his back on it and starts to wither away. -- Dr. Boyce, The Menagerie (The Cage), stardate unknown - - NOTICE: All legal disclaimers sent to benjammin.net/benkamen.net or any of it's affiliated domains are rendered null and void on receipt of communications and will be handled/considered as such. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] What about DKIM
On Wed, 27 Mar 2013 12:22:37 -0500 Ben Kamen bka...@benjammin.net wrote: Now that we've see/talked some stats on SPF... I'd be interested to know what anyone might have to offer on DKIM usefulness. DKIM is useful for letting you know that a message has been relayed through a responsible organization's server. I don't think it's very useful as a spam/ham indicator. Plenty of validly-signed mail is spam (think Yahoo!) and some ham ends up with broken DKIM signatures (think broken boilerplate-appending software.) The up-and-coming thing is DMARC, which will probably enjoy good press the way SPF and DKIM did for a few years until it too is found to be not very useful. :) DMARC is intended to close two loopholes: It lets domain owners *specify* what you should do on SPF fail or DKIM fail, and it gives domain owners feedback about failed SPF/DKIM so a domain owner can know that he/she's the victim of spoofing. DMARC falls flat because it does not in any way protect what the user sees as the From field in a mail reader, so phishers can happily spoof mail and still be DMARC-compliant. http://www.dmarc.org/ Regards, David. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] What about DKIM
--- On Wed, 3/27/13, Ben Kamen bka...@benjammin.net wrote: Now that we've see/talked some stats on SPF... I'd be interested to know what anyone might have to offer on DKIM usefulness. Not widely used. Also, Yahoo, who started DK, doesn't even do its ADSP extension coding correctly: They have an entry but CNAME it to something that never resolves to a TXT-RR, so it's broken. Some other domains do it correctly, but I see it in perhaps one of 50 transactions at most. It's much harder to set up. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Spammers - was Re: md_check_against_smtp_server and md_graphdefang_log
On Wed, Mar 27, 2013 at 8:49 AM, David F. Skoll d...@roaringpenguin.com wrote: Welcome to the IT industry... a giant pile of fail. But hey, think of all the jobs that would go away if things were designed right in the first place so you could just drop in a mass produced appliance. I'm pretty sure I'll retire first... -- Les Mikesell lesmikes...@gmail.com ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
Am 27.03.2013 15:59, schrieb Matt Garretson: # man mimedefang-filter [...] md_graphdefang_log($event, $v1, $v2) [...] Note that md_graphdefang_log should not be used in filter_relay, filter_sender or filter_recipient. The global variables it relies on are not valid in that context. [...] OMG *blush* Although I checked the manpage several times in the course of this thread I never noticed that paragraph. Well, you live and learn. Thanks, Matt, for pointing that out. -- Tilman Schmidt Phoenix Software GmbH Bonn, Germany signature.asc Description: OpenPGP digital signature ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
On Wed, 2013-03-27 at 19:45 +0100, Tilman Schmidt wrote: Am 27.03.2013 15:59, schrieb Matt Garretson: Note that md_graphdefang_log should not be used in filter_relay, filter_sender or filter_recipient. The global variables it relies on are not valid in that context. [...] Although I checked the manpage several times in the course of this thread I never noticed that paragraph. Well, you live and learn. This is a silly restriction that I wish was lifted. Yes, the subject will be blank if you call it earlier, but that's unavoidable. I don't actually use graphdefang, though. So that may be coloring my thinking. In my filter, I have code that does more-or-less the same thing but without this restriction. It works great. -- Richard ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
27 Mar 2013 14:11:08 -0500 Although I checked the manpage several times in the course of this thread I never noticed that paragraph. Well, you live and learn. Hindsight is 20/20, I found this thread from 2008, indicating the same issue, and possible resolution. http://lists.roaringpenguin.com/pipermail/mimedefang/2008-September/034277.html Thanks again for all the help. ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] md_check_against_smtp_server and md_graphdefang_log
On 3/27/2013 12:41 PM, James Curtis wrote:
Can't imagine the amount of errors I'll incur when creating a new
sub-routine for my_graphdefang_log(), but it will be a fun time. I'm
assuming that I start with the code from md_graphdefang_log and
remove the variables.
Precisely. I'd suggest adding the new function to your filter instead
of modifying mimedefang.pl . Also, note this excerpt from the
mimedefang.pl man page:
TESTING FILTERS
You are strongly recommended to test your filter before
installing it in /etc/mail/mimedefang-filter. To test the filter,
save it in a file (e.g. test-filter) and run this command:
mimedefang.pl -f test-filter -test
This tests the filter for syntactic correctness. If it
passes, you can install it as a production filter. (Note that the
test tests only for correct Perl syntax; it doesn’t make sure your
filter does something sensible.)
As a simple example, you could start with something like this, which is
basically a stripped-down md_graphdefang_log():
sub my_graphdefang_log($;$$)
{
return unless defined($GraphDefangSyslogFacility);
my $event = shift;
my $value1 = shift;
my $value2 = shift;
$value1 = unless defined($value1);
$value2 = unless defined($value2);
# Make values safe for graphdefang
$event = percent_encode_for_graphdefang($event);
$value1 = percent_encode_for_graphdefang($value1);
$value2 = percent_encode_for_graphdefang($value2);
md_syslog($GraphDefangSyslogFacility|info,MYLOG,_NOMSGID_, .
$event,$value1,$value2,_NOSENDER_,_NORECIP_,_NOSUBJ_);
}
That should work from filter_relay() onwards. It's up to you what you
want to log, as long as you don't try to access any variables that are
not defined in the current state of the milter.
___
NOTE: If there is a disclaimer or other legal boilerplate in the above
message, it is NULL AND VOID. You may ignore it.
Visit http://www.mimedefang.org and http://www.roaringpenguin.com
MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com
http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
Re: [Mimedefang] Spammers - was Re: md_check_against_smtp_server and md_graphdefang_log
On 27 March 2013 15:49, David F. Skoll d...@roaringpenguin.com wrote: That's from a network monitoring system that sends (by default) from the domain of the system manufacturer. Since most people don't bother changing defaults, millions of these alert mails go out every day and fail SPF. Welcome to the IT industry... a giant pile of fail. This reminds me of a well-known classical author from my country who was quoted saying: “I know I'm dumb but when I look around I gain some courage!” :) -- mișu ___ NOTE: If there is a disclaimer or other legal boilerplate in the above message, it is NULL AND VOID. You may ignore it. Visit http://www.mimedefang.org and http://www.roaringpenguin.com MIMEDefang mailing list MIMEDefang@lists.roaringpenguin.com http://lists.roaringpenguin.com/mailman/listinfo/mimedefang
