Re: FCrDNS and localhost

[Benny Pedersen]

On Fri, June 5, 2009 23:55, mouss wrote:
> why bother yourself with SPF since nobody remote should call himself
> "localhost". localhost is a reserved domain.

will you wake up one day and beat me in my foot ? :)))

localhost check does not rule out that spf check can be usefull

-- 
http://localhost/ 100% uptime and 100% mirrored :)

Re: New slew of spams

[Benny Pedersen]

On Fri, June 5, 2009 20:05, Rob McEwen wrote:

> I highly recommend scoring RDNS_NONE at much higher than "0.1", and
> scoring RCVD_IN_PBL at much higher than 0.9

meta SPAM_LOCAL (RDNS_NONE && RCVD_IN_PBL)
describe SPAM_LOCAL Meta: it hits both RDNS_NONE and RCVD_IN_PBL
score SPAM_LOCAL 5.0

-- 
http://localhost/ 100% uptime and 100% mirrored :)

Re: New slew of spams

[Benny Pedersen]

On Fri, June 5, 2009 19:58, Jeremy Morton wrote:
> http://pastebin.com/m586e296c
http://cbl.abuseat.org/lookup.cgi?ip=93.5.36.134

do you use zen.spamhaus.org in exim ?

http://www.wpbl.info/cgi-bin/detail.cgi?ip=93.5.36.134

if the ip is not sending ham to you block the ip localy

-- 
http://localhost/ 100% uptime and 100% mirrored :)

Re: FCrDNS and localhost

[mouss]

Adam Katz a écrit :
> John Hardin wrote:
>> So that data comes from /etc/hosts. How does that materially affect the
>> FCrDNS sanity test?
> 
> By definition, FCrDNS uses DNS lookups.  Unless you're using dnsmasq,
> the entries in /etc/hosts are ignored during DNS lookups. 

This is wrong.

FCrDNS lookup uses a DNS resolver, which could use whatever you setup on
your system, including /etc/hosts.

for example, postfix uses the system resolver, which can be configured
via nsswitch.conf to query /etc/hosts and/or DNS.

> Unless I'm
> mistaken, no FCrDNS implementation ever queries /etc/hosts (nor should
> it). 

Are you saying that all those widely used servers that do this are wrong?

> This means FCrDNS will conclude that localhost does not resolve


I don't know why you say that localhost doesn't resolve. It resolves on
all systems I have ever used. my basic DNS "template" files include
resolution for reserved domains and IP blocks (so that resolving private
IPs doesn't go over the network uselessly...).

> and that 127.0.0.1 has no rDNS (excepting cases where the admins have
> manually placed such entries into the local DNS).

which is the default in *BSD and other "distributions".

Re: FCrDNS and localhost

[mouss]

Adam Katz a écrit :
> Matus UHLAR - fantomas wrote:
>> 181.188.252.222.in-addr.arpa domain name pointer localhost.
>>
>> That is why FcRDNS is being used everywhere...
>>
>> localhost has address 127.0.0.1 => fail.
> 
> Actually, localhost doesn't resolve via DNS;

I don't know where you're taking this from:

$ host localhost 127.0.0.1
Using domain server:
Name: 127.0.0.1
Address: 127.0.0.1#53
Aliases:

localhost.netoyen.net has address 127.0.0.1


> it has no A record, nor
> any other record type.  It resolves locally without using DNS; see
> your /etc/hosts file.  Similarly, 1.0.0.127.in-addr.arpa. has no PTR
> record indicating it should be called localhost.
> 

It does here. we BSD users love DNS ;-p

>> if anyone uses reverse DNS name without forward-confirming it, it's their
>> own fault and they can take all consequencies from such stupid setup. afaik
>> some reverse-checking services are more strict about invalid than about
>> nonexisting hostnames. And I recommend to behave like that.
>>
>> SA (usually) uses hostname passed by MTA, so if an MTA is affected by this
>> bug, blame MTA, not SA. And I'm not sure if the hostname is used by any
>> checks that would cause positive (oor lower negative) score.
> 
> Sadly, too many servers are set up improperly in this context, so I
> doubt I'm in the minority when I say that I don't use this metric to
> single-handedly block mail.
> 
> My khop-general.sa.khopesh.com channel contains:
> 
> # Sendmail's FCrDNS, see http://www.sendmail.org/faq/section3#3.38
> header   KHOP_MAYBE_FORGED   Received =~ /\(may be forged\)/
> describe KHOP_MAYBE_FORGED   Relay IP's reverse DNS does not resolve to IP
> scoreKHOP_MAYBE_FORGED   0.8 # 20050802, raised 0.15->0.8 20090603
> 
> # Violates rfc2821?  See http://en.wikipedia.org/wiki/FCrDNS#Uses
> headerKHOP_HELO_FCRDNS   X-Spam-Relays-Untrusted =~ /^[^\]]+
> rdns=(\S+) helo=(?!\1)\S/
> describe  KHOP_HELO_FCRDNS   Relay HELO differs from its IP's reverse DNS
> score KHOP_HELO_FCRDNS   0.4 # 20090603
> 
> 
>> Maybe SPF, I expect someone to comment on this...
> 
> Same problem as above: "localhost" is not actually a domain.

it _is_.

> 
> $ host -t TXT localhost.
> localhost has no TXT record
> $ host -t TXT localhost.localdomain.
> localhost.localdomain has no TXT record
> 

In contrast, "localdomain" is not a valid TLD.

> I suppose I could place such an entry in my local DNS server...
> Actually, I like that idea.  Don't forget to also create an A record!
> 
> You'll want TXT record  "v=spf1 ip4:127.0.0.0/8 -all"  for both
> localhost. and localhost.localdomain.
> 

why bother yourself with SPF since nobody remote should call himself
"localhost". localhost is a reserved domain.

MIME_NO_TEXT

[John Hardin]

All:

Sorry that the last iteration of the MIME_NO_TEXT rules (see the "word doc 
spam" message I just resent) didn't get sent to the list - it should have 
gone to the list but I didn't notice the discussion had gone off-list.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  It is not the place of government to make right every tragedy and
  woe that befalls every resident of the nation.
---
 Tomorrow: the 65th anniversary of D-Day

Re: word doc spam

[John Hardin]

On Tue, 2 Jun 2009, Yet Another Ninja wrote:


On 6/2/2009 7:55 PM, John Hardin wrote:


 Oh, sorry, I got that backwards checking for _not_ PHP... Never mind
 those last rules.

 The mailer is going to be easy to change (even randomly) in a spam tool.
 I'd suggest that it's not valid to check that for this test,


Could be but all the hits I saw with the .png and .rtf files had the PHP 
X-mailer in them.


Perhaps this, then?


header __CTYPE_MULTIPART_ANY  Content-Type =~ /multipart\/\w/i
header __XM_PHP  X-Mailer =~ /^PHP\s?v?\/?\d\./
ifplugin Mail::SpamAssassin::Plugin::MIMEHeader
  mimeheader __ANY_TEXT_ATTACH Content-Type =~ /text\/\w+/i
  meta MIME_NO_TEXT (__CTYPE_MULTIPART_ANY && !__ANY_TEXT_ATTACH)
  scoreMIME_NO_TEXT 1.00
  describe MIME_NO_TEXT No text body parts
  meta MIME_PHP_NO_TEXT (MIME_NO_TEXT && __XM_PHP)
  scoreMIME_PHP_NO_TEXT 2.00
  describe MIME_PHP_NO_TEXT No text body parts, X-Mailer: PHP
endif

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Of the twenty-two civilizations that have appeared in history,
  nineteen of them collapsed when they reached the moral state the
  United States is in now.  -- Arnold Toynbee
---
 4 days until the 65th anniversary of D-Day

Re: New slew of spams

[rich...@buzzhost.co.uk]

On Fri, 2009-06-05 at 20:33 +0200, Raymond Dijkxhoorn wrote:
> Hi!
> 
> >> http://pastebin.com/m586e296c
> >>
> >> As you can see they tend to hit a couple of blacklists, but don't get a
> >> high enough score to be marked as spam.  What do your SpamAssassin
> >> analyses give of this e-mail, and any tips as to how I can get these
> >> marked as spam?
> 
> > But;
> >
> > 93.5.36.134  listed in b.barracudacentral.org.
> > 93.5.36.134  listed in XBL NJABL
> > 93.5.36.134  listed in PBL (SPAMHAUS)
> > 93.5.36.134  listed in cbl.abuseat.org.
> >
> > So they could have been blocked ?
> 
> Perhaps now, but most of them end up after the first runs ... ;)
> Most likely at time of the run they were not listed (yet).
> 
> Bye,
> Raymond.
> 
Even in the breakdown you've posted they are listed on at least one
black list. Personally, I would have dropped them on connecting IP
before wasting spamassassin on scanning them - but that opens a can of
worms and people have differing views on doing that.

Re: [SA] Identifying Source of False Positives -- RESOLVED

[Rich Shepard]

On Fri, 5 Jun 2009, Adam Katz wrote:


Since that regex matches nothing, I assume you meant it to be
m'^[^\n]+\n\s*$'s  or  m'^[^\n]+\n\s*$'ms


Adam,

  I didn't write this. It apparently came with the local.cf file a few years
ago.

Rich

--
Richard B. Shepard, Ph.D.   |  IntegrityCredibility
Applied Ecosystem Services, Inc.|Innovation
 Voice: 503-667-4517  Fax: 503-667-8863

Re: [SA] Identifying Source of False Positives -- RESOLVED

[Adam Katz]

Rich Shepard wrote:
> # for empty message bodies:
> body   EMPTY_BODY   m'^[^\n]+\n\s*$'
> describe   EMPTY_BODY   Message has subject but no body
> score  EMPTY_BODY   2.5

Egads ... that's an unbounded multi-line regex (that little plus sign is
quite CPU-intensive).  I don't understand its intent, either ... it
looks for a line that includes linebreaks but with no multi-line flag.
Ignoring that bug, it wants a nonzero line followed by either a blank
line or a line filled only with spaces.  How does this characterize an
empty body?  What does this have to do with the presence of a subject?

Since that regex matches nothing, I assume you meant it to be
m'^[^\n]+\n\s*$'s  or  m'^[^\n]+\n\s*$'ms

With a trailing s, that rule matches one-line emails that end in a blank
line (which are quite common).

With a trailing ms, that rule matches any email with a paragraph in it
(like this one), which is almost every single email.

It appears you wanted something like this:

body __EMPTY_BODY  !~ m'\w\n\w's
meta SUBJ_NO_BODY  __EMPTY_BODY && __HAS_SUBJECT
describe SUBJ_NO_BODY  Message has subject but no body
scoreSUBJ_NO_BODY  2.5

Or perhaps like this:

body EMPTY_BODY!~ m'\w\n\w's
describe EMPTY_BODYMessage has no text in body
scoreEMPTY_BODY2.5

Also, that score seems pretty high, and I wonder about your intent.  If
you're trying to use it to catch image-only spam, please use the other
rules we've proposed on the list, like MIME_IMAGE_ONLY.

Re: New slew of spams

[Raymond Dijkxhoorn]

Hi!


http://pastebin.com/m586e296c

As you can see they tend to hit a couple of blacklists, but don't get a
high enough score to be marked as spam.  What do your SpamAssassin
analyses give of this e-mail, and any tips as to how I can get these
marked as spam?



But;

93.5.36.134  listed in b.barracudacentral.org.
93.5.36.134  listed in XBL NJABL
93.5.36.134  listed in PBL (SPAMHAUS)
93.5.36.134  listed in cbl.abuseat.org.

So they could have been blocked ?


Perhaps now, but most of them end up after the first runs ... ;)
Most likely at time of the run they were not listed (yet).

Bye,
Raymond.

Re: New slew of spams

[Adam Katz]

Jeremy Morton wrote:
> I've suddenly started getting a new slew of spams that are making their
> way through my SpamAssassin filter.  Here's an example of one:
> 
> http://pastebin.com/m586e296c
> 
> As you can see they tend to hit a couple of blacklists, but don't get a
> high enough score to be marked as spam.  What do your SpamAssassin
> analyses give of this e-mail, and any tips as to how I can get these
> marked as spam?

Install iXhash and activate Razor2.

Additionally, I recommend this rule to bump up the iXhash scores (note
that the meta line wraps here but should not in your config file):

meta IXHASH_CHECK GENERIC_IXHASH || NIXSPAM_IXHASH || CTYME_IXHASH ||
HOSTEUROPE_IXHASH
describe IXHASH_CHECK BODY: MD5 checksum matches known spam
score IXHASH_CHECK0 2 0 2

Re: New slew of spams

[John Hardin]

On Fri, 5 Jun 2009, Jeremy Morton wrote:

I've suddenly started getting a new slew of spams that are making their 
way through my SpamAssassin filter.  Here's an example of one:


http://pastebin.com/m586e296c


Look for the MIME_NO_TEXT ruleset I posted a few days ago.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  Rights can only ever be individual, which means that you cannot
  gain a right by joining a mob, no matter how shiny the issued
  badges are, or how many of your neighbors are part of it.  -- Marko
---
 Tomorrow: the 65th anniversary of D-Day

Re: [sa] New slew of spams

[Charles Gregory]

On Fri, 5 Jun 2009, Jeremy Morton wrote:
I've suddenly started getting a new slew of spams that are making their way 
through my SpamAssassin filter.  Here's an example of one:

http://pastebin.com/m586e296c


These are examples of the new variant on 'image only' spams, having only a 
rtf file attachment, instead of an image. Check the archives and you will

find rules to tag messages with 'octet-stream mime part but no text part'.
Quite effective.

- Charles

Re: New slew of spams

[rich...@buzzhost.co.uk]

On Fri, 2009-06-05 at 18:58 +0100, Jeremy Morton wrote:
> Hi,
> 
> I've suddenly started getting a new slew of spams that are making their 
> way through my SpamAssassin filter.  Here's an example of one:
> 
> http://pastebin.com/m586e296c
> 
> As you can see they tend to hit a couple of blacklists, but don't get a 
> high enough score to be marked as spam.  What do your SpamAssassin 
> analyses give of this e-mail, and any tips as to how I can get these 
> marked as spam?
> 
> Best regards,
> Jeremy Morton (Jez)

But;

93.5.36.134  listed in b.barracudacentral.org. 
93.5.36.134  listed in XBL NJABL 
93.5.36.134  listed in PBL (SPAMHAUS) 
93.5.36.134  listed in cbl.abuseat.org. 

So they could have been blocked ?

Re: Question on add-to-blacklist

[Larry Starr]

On Wednesday 03 June 2009, Jari Fredriksson wrote:
> > On Tuesday 02 June 2009, Michael Scheidell wrote:
> > What "optional" fields are you refering to?
> >
> > I have seen this, on the spamassassin WIKI:
> >
> > CREATE TABLE awl (
> >  username varchar(100) NOT NULL default '',
> >  email varchar(200) NOT NULL default '',
> >  ip varchar(10) NOT NULL default '',
> >  count int(11) default '0',
> >  totscore float default '0',
> >  PRIMARY KEY  (username,email,ip)
> > ) TYPE=MyISAM;
> >
> > Is there a better reference?
>
>  CREATE TABLE `awl` (
>   `username` varchar(100) NOT NULL DEFAULT '',
>   `email` varchar(200) NOT NULL DEFAULT '',
>   `ip` varchar(10) NOT NULL DEFAULT '',
>   `count` int(11) DEFAULT '0',
>   `totscore` float DEFAULT '0',
>   `lastupdate` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE
> CURRENT_TIMESTAMP, PRIMARY KEY (`username`,`email`,`ip`)
> ) ENGINE=InnoDB ;

Thanks, I can see that the 'lastupdate' field would prove handy over time.

I'm noticing that most of the scripts, that I've found for processing AWL 
data, including 'convert_awl_dbm_to_sql' will not run, for me, as written.

They seem to, invariably, use "tie %hash" and then use something like:

 my @k = grep(!/totscore$/,keys(%h));

to build a list of keys.   My existing AWL has something over 10Million 
records, and my system only contains 4G of RAM.

I've been rewriting the loops in the scripts that I'm trying to use from:
  my @k = grep(!/totscore$/,keys(%h));
  for my $key (@k) {

to:
  while(my ($key, $v) = each %h)

This may be, a bit slower, but scales better to large data volumes.

-- 
Larry G. Starr - lar...@fullcompass.com or sta...@globaldialog.com
Software Engineer: Full Compass Systems LTD.
Phone: 608-831-7330 x 1347  FAX: 608-831-6330
===
There are only three sports: bullfighting, mountaineering and motor
racing, all the rest are merely games! - Ernest Hemmingway

Re: New slew of spams

[Rob McEwen]

Jeremy Morton wrote:
> I've suddenly started getting a new slew of spams that are making
> their way through my SpamAssassin filter.  Here's an example of one:
>
> http://pastebin.com/m586e296c
>
> As you can see they tend to hit a couple of blacklists, but don't get
> a high enough score to be marked as spam.  What do your SpamAssassin
> analyses give of this e-mail, and any tips as to how I can get these
> marked as spam? 

I highly recommend scoring RDNS_NONE at much higher than "0.1", and
scoring RCVD_IN_PBL at much higher than 0.9

If you don't feel comfortable having these combine to score higher than
threshold, then consider bumping each of these up at least by a whole a
point or two... and then add a metarule that might add an additional
point or two if BOTH of these have been triggered.

An occassional legit e-mail will have RDNS_NONE, and an occassional
legit e-mail will have RCVD_IN_PBL. But even extreme fewer legit emails
will have hits on BOTH of these. So I'd suggest scoring the combination
of the two either just above threshold, or (at the least...) just below
threshold.

-- 
Rob McEwen
http://dnsbl.invaluement.com/
r...@invaluement.com
+1 (478) 475-9032

New slew of spams

[Jeremy Morton]

Hi,

I've suddenly started getting a new slew of spams that are making their 
way through my SpamAssassin filter.  Here's an example of one:


http://pastebin.com/m586e296c

As you can see they tend to hit a couple of blacklists, but don't get a 
high enough score to be marked as spam.  What do your SpamAssassin 
analyses give of this e-mail, and any tips as to how I can get these 
marked as spam?


Best regards,
Jeremy Morton (Jez)

Re: Bayes learning trusted networks mailing list email

[RW]

On Fri, 05 Jun 2009 10:24:31 -0400
Micah Anderson  wrote:

 If I understand things properly, because I've got these
> setup in my trusted_networks, then these previous hops will be
> checked in RBLs, so the spam is more detectable.

That doesn't really help. If you think about it, tests that run on
untrusted headers will run whether or not you put the list servers into
your trusted network. The tests that run on the trusted boundary are
whitelisting rules (plus a few rules that will soon get moved to the
internal boundary). You might get some benefit from putting the list
servers into the internal network, but the chances are that the list is
already blocking on zen, and maybe DUL lists and SPF.

> What I am unsure of is if I am poisoning my bayes by reporting these
> messages that make it through as spam. Should I be just deleting them?
> The tokens that are legitimate that will end up as collateral damage
> are going to be the list footers, the list administration messages,
> and potentially other pieces.
> 
> I'm hoping I can identify why my bayes database is so bad (it thinks
> everything is BAYES_00 now), and if this is why I will want to change
> my training behavior.

It's really hard for BAYES to work on in-list spams because they
contain so many strong ham tokens. What I would suggest is to use
a separate address and Bayes database for the lists and train it on all
spam, but only learn ham that doesn't hit BAYES_00. I use sieve to
select some in-list candidates for learning (with dspam rather than SA).

You might also configure BAYES to ignore some of the list headers.

Things like challenge-response messages and out-of-office replies are
best handled with simple filtering or custom SA tests.

Re: Identifying Source of False Positives -- RESOLVED

[Rich Shepard]

On Fri, 5 Jun 2009, Bowie Bailey wrote:


In that case, you should be able to track down the issue by comparing the
two files. Is the EMPTY_BODY rule defined in the old local.cf file? If
so, what does it say?


Bowie,

  Yes, it was in the old local.cf:

# for empty message bodies:
body   EMPTY_BODY   m'^[^\n]+\n\s*$'
describe   EMPTY_BODY   Message has subject but no body
score  EMPTY_BODY   2.5

  It apparently used to work, but isn't with the new SA to which I upgraded
a few months ago.

Thanks,

Rich

--
Richard B. Shepard, Ph.D.   |  IntegrityCredibility
Applied Ecosystem Services, Inc.|Innovation
 Voice: 503-667-4517  Fax: 503-667-8863

Re: I never got WrongMx working and have no idea why.

[Matus UHLAR - fantomas]

> > On 04.06.09 18:04, Steven W. Orr wrote:
> > > The following file came in and we can see that it did not work. The
> > > mail came through mx2.zoneedit.com

> On Fri, 5 Jun 2009 16:31:05 +0200
> Matus UHLAR - fantomas  wrote:
> > Maybe the plugin was unable to find out the destination domain. Can
> > you try inserting X-Envelope-To: or Delivered-To: header with the
> > envelope recipient to the mail?

On 05.06.09 16:51, RW wrote:
> That's unlikely to help, the address appeared several times in the
> email, so "all_to_addrs" should have found it, assuming that "xxx" was
> consistent and correct.
> 
> From a quick look at the plugin, it looks like it takes the domain from
> the username or falls back to the first valid address in
> "all_to_addrs". So it could fail if the username contains a domain,
> and that domain has different MX records.

that may be the problem.

I was somewhere trying to provide the destination address to spamc -u 
but didn't seem to work...
-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson.

Re: Identifying Source of False Positives -- RESOLVED

[Bowie Bailey]

Rich Shepard wrote:
The empty body problem is a more difficult problem.  Have procmail 
save a
copy of the raw message somewhere and take a look at it.  Make sure 
there
is a blank line between the headers and the body.  Run 'spamassassin 
-D'
on this saved message and look for anything unusual in the debug 
output.


  This seems to have been resolved by replacing the old
/etc/mail/spamassassin/local.cf with the new version. Many fewer rules 
and

other entries, but I no longer see the EMPTY_BODY test adding 2.5 to the
scores.


In that case, you should be able to track down the issue by comparing 
the two files.  Is the EMPTY_BODY rule defined in the old local.cf 
file?  If so, what does it say?


--
Bowie

Re: Identifying Source of False Positives -- RESOLVED

[Rich Shepard]

On Tue, 2 Jun 2009, Rich Shepard wrote:


 I started doing this today. Each of the false positive messages was
exported from alpine to a file, and I ran sa-learn on that file telling it
the text is ham.


  Today the mail and logwatch summary reports appeared in my inbox and there
were no false positives in the holding cell. This may have resolved the
issue of missing messages, but I'll continue to monitor and train SA on the
ham that was mistakenly labeled as spam.


The empty body problem is a more difficult problem.  Have procmail save a
copy of the raw message somewhere and take a look at it.  Make sure there
is a blank line between the headers and the body.  Run 'spamassassin -D'
on this saved message and look for anything unusual in the debug output.


  This seems to have been resolved by replacing the old
/etc/mail/spamassassin/local.cf with the new version. Many fewer rules and
other entries, but I no longer see the EMPTY_BODY test adding 2.5 to the
scores.

Thank you all very much,

Rich

--
Richard B. Shepard, Ph.D.   |  IntegrityCredibility
Applied Ecosystem Services, Inc.|Innovation
 Voice: 503-667-4517  Fax: 503-667-8863

Re: I never got WrongMx working and have no idea why.

[RW]

On Fri, 5 Jun 2009 16:31:05 +0200
Matus UHLAR - fantomas  wrote:

> On 04.06.09 18:04, Steven W. Orr wrote:
>
> > The following file came in and we can see that it did not work. The
> > mail came through mx2.zoneedit.com
> 
> Maybe the plugin was unable to find out the destination domain. Can
> you try inserting X-Envelope-To: or Delivered-To: header with the
> envelope recipient to the mail?



That's unlikely to help, the address appeared several times in the
email, so "all_to_addrs" should have found it, assuming that "xxx" was
consistent and correct.

From a quick look at the plugin, it looks like it takes the domain from
the username or falls back to the first valid address in
"all_to_addrs". So it could fail if the username contains a domain,
and that domain has different MX records.

Re: two databases

[Micah Anderson]

* Michael Grant  [2009-06-05 10:26-0400]:
> On Fri, Jun 5, 2009 at 16:08, Micah Anderson  wrote:
> > Michael Grant  writes:
> >
> >> I did not realize one could store the bayes scores in sql.
> >>
> >> So I'd store the bayes scores on a third server and let both mxes use
> >> the same database.
> >
> > I did this, but my bayes in mysql and pointed two different spamd
> > machines at it, but I had severe problems that I could not resolve. I
> > posted to the list[0] about the problems.
> >
> > The basic problem was that as soon as I fired up the second server it
> > immediately starts blocking on the bayes work. Average scantimes go from
> > 1-2 seconds up to 35+ and the max children get eaten up by blocking on
> > the bayes work to the point where its pointless because too many
> > processes are blocked. Disabling the bayes_sql stuff on one of the
> > machines dropped the scantimes back to their expected average of 1-2
> > seconds (but of course none of the BAYES tests will fire and
> > autolearning fails).
> >
> > My mysql server is its own machine, it was local to the first spamd
> > (local LAN) and remote to the second (over the net). I eliminated any
> > hostname lookup problems, obviously couldn't eliminate network latency,
> > but that shouldn't have caused such a severe result. I'm running with
> > InnoDB tables, so I shouldn't have any row-level locking issues... in
> > any case I might have had some issues because my MySQL database needed
> > to be optimized, but I was not able to determine how and now I just run
> > one of the spamd's without bayes, which is not too bad because my bayes
> > database seems to be totally worthless at the moment. :P
> >
> > micah
> >
> > 0. http://permalink.gmane.org/gmane.mail.spam.spamassassin.general/113673
> >
> >
> 
> Wow.  I did not get around to setting this up yet.  But on the MySQL
> front, did you try enabling the query cache by adding this to the
> mysql command line?
> 
> --maximum-query_cache_size=1M

I presume this setting is the same in my.cnf:
query_cache_limit   = 1048576

I dont recall all the things I tried, but it seems worth trying again,
this time with a fresh approach. 

> Also, a tool I used a lot to help debug this sort of issue was mytop.

I've never had too much luck with mytop, but I have found the
tuning-primer.sh to work well: http://www.day32.com/MySQL/

micah


signature.asc
Description: Digital signature

Re: two databases

[Rick Macdougall]

Michael Grant wrote:

On Fri, Jun 5, 2009 at 16:08, Micah Anderson  wrote:

Michael Grant  writes:


I did not realize one could store the bayes scores in sql.

So I'd store the bayes scores on a third server and let both mxes use
the same database.

I did this, but my bayes in mysql and pointed two different spamd
machines at it, but I had severe problems that I could not resolve. I
posted to the list[0] about the problems.

The basic problem was that as soon as I fired up the second server it
immediately starts blocking on the bayes work. Average scantimes go from
1-2 seconds up to 35+ and the max children get eaten up by blocking on
the bayes work to the point where its pointless because too many
processes are blocked. Disabling the bayes_sql stuff on one of the
machines dropped the scantimes back to their expected average of 1-2
seconds (but of course none of the BAYES tests will fire and
autolearning fails).



I found that the bayes lookup occurred, then the connection was closed, 
then a second connection attempt was made to do bayes learning but it 
attempted to use the same socket.


Because the socket on the remote server hadn't closed yet, the process 
hung until closed, then preceded.


What I ended up doing was having two spamd machines use DBI.pm (which I 
found on the spamassassin wiki and that makes SA use persistent 
connections) and have auto-learning ON on those two machines.


The other two machines run with bayes enabled but with auto-learning OFF.

For me this solved all my problems.

Please note how ever that this occurred to me using 3.0.x and I've just 
been upgraded ever since with out checking to see if the re-connection 
issue has been solved since everything *just works* as it is currently 
configured.


HTHs,

Rick

FreeMail.bl installation instructions

[Micah Anderson]

The FreeMail.pm installation instructions are a little thin:

### Install:
#
# Please add loadplugin to init.pre (so it's loaded before cf files!):
#
# loadplugin Mail::SpamAssassin::Plugin::FreeMail FreeMail.pm

My understanding, and please correct me if I am wrong, is that you
actually need to do this:

# 1. Install FreeMail.pm in /etc/spamassassin
#
# 2. Add the following loadplugin to init.pre:
#
# loadplugin Mail::SpamAssassin::Plugin::FreeMail FreeMail.pm
#
# 2. Download http://sa.hege.li/FreeMail.cf to /etc/spamassassin
#
# 3. Download http://sa.hege.li/freemail_domains.cf to /etc/spamassassin

I knew about the FreeMail.cf because I've used SA plugins before, but I
had no idea about the domain list. Might be good to make these
instructions a little more explicit, so that others will also win.

Micah

Re: two databases

[d . hill]

Quoting Micah Anderson :


any case I might have had some issues because my MySQL database needed
to be optimized, but I was not able to determine how and now I just run
one of the spamd's without bayes, which is not too bad because my bayes
database seems to be totally worthless at the moment. :P


http://dev.mysql.com/doc/refman/5.0/en/optimize-table.html

I have a cronjob set up that does an optimize table on all the SA  
tables every 24 hours to make sure everything is in line.

Re: I never got WrongMx working and have no idea why.

[Matus UHLAR - fantomas]

On 04.06.09 18:04, Steven W. Orr wrote:
> In my  /etc/mail/spamassassin, I have two files, wrongmx.cf and wrongmx.pm
>
> The cf file looks like this:
> loadplugin  WrongMX wrongmx.pm
>
> header  WRONGMX eval:wrongmx()
> describeWRONGMX Sent to lower pref MX when higher pref MX was up.
> tflags  WRONGMX net
> score   WRONGMX 1.0
>
> My dns MX record looks like this:
>
> ;; ANSWER SECTION:
> syslang.net.9738IN  MX  100 mx2.zoneedit.com.
> syslang.net.9738IN  MX  0 syslang.net.
>
> The following file came in and we can see that it did not work. The mail 
> came through mx2.zoneedit.com

Maybe the plugin was unable to find out the destination domain. Can you try
inserting X-Envelope-To: or Delivered-To: header with the envelope recipient
to the mail?

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
BSE = Mad Cow Desease ... BSA = Mad Software Producents Desease

Re: I never got WrongMx working and have no idea why.

[Matus UHLAR - fantomas]

> On Thu, 4 Jun 2009 18:04:35 -0400 (EDT)
> "Steven W. Orr"  wrote:
> 
> > My dns MX record looks like this:
> > 
> > ;; ANSWER SECTION:
> > syslang.net.9738IN  MX  100 mx2.zoneedit.com.
> > syslang.net.9738IN  MX  0 syslang.net.
> > ...
> > The pm file is the latest. This trap has never fired and I'm about to
> > give up on it and shut it off. I just have to think that I must be
> > doing something wrong. Anyone?

On 05.06.09 00:47, RW wrote:
> I can't really see the point your using this plugin. All you need is a
> one-line custom rule looking for mx2.zoneedit.com in received headers.

and another rule comparing the time mail was received - it the difference
was .g. above 5 minutes, it's quite possible that the primary MX was _not_
up ehwn the secondary acepted the mail.

> Presumably the advantage of the plugin is that it automatically detects
> that a server is a backup. You already know what your backup is
> called, and presumably you control your mx settings.

Well, he may have many domeins, some out of his control, and some may have
different backup MX servers...

-- 
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
How does cat play with mouse? cat /dev/mouse

Re: two databases

[Michael Grant]

On Fri, Jun 5, 2009 at 16:08, Micah Anderson  wrote:
> Michael Grant  writes:
>
>> I did not realize one could store the bayes scores in sql.
>>
>> So I'd store the bayes scores on a third server and let both mxes use
>> the same database.
>
> I did this, but my bayes in mysql and pointed two different spamd
> machines at it, but I had severe problems that I could not resolve. I
> posted to the list[0] about the problems.
>
> The basic problem was that as soon as I fired up the second server it
> immediately starts blocking on the bayes work. Average scantimes go from
> 1-2 seconds up to 35+ and the max children get eaten up by blocking on
> the bayes work to the point where its pointless because too many
> processes are blocked. Disabling the bayes_sql stuff on one of the
> machines dropped the scantimes back to their expected average of 1-2
> seconds (but of course none of the BAYES tests will fire and
> autolearning fails).
>
> My mysql server is its own machine, it was local to the first spamd
> (local LAN) and remote to the second (over the net). I eliminated any
> hostname lookup problems, obviously couldn't eliminate network latency,
> but that shouldn't have caused such a severe result. I'm running with
> InnoDB tables, so I shouldn't have any row-level locking issues... in
> any case I might have had some issues because my MySQL database needed
> to be optimized, but I was not able to determine how and now I just run
> one of the spamd's without bayes, which is not too bad because my bayes
> database seems to be totally worthless at the moment. :P
>
> micah
>
> 0. http://permalink.gmane.org/gmane.mail.spam.spamassassin.general/113673
>
>

Wow.  I did not get around to setting this up yet.  But on the MySQL
front, did you try enabling the query cache by adding this to the
mysql command line?

--maximum-query_cache_size=1M

Also, a tool I used a lot to help debug this sort of issue was mytop.

Michael Grant

Bayes learning trusted networks mailing list email

[Micah Anderson]

I get a significant amount of spam that comes through mailing lists that
I am legitimately subscribed to, either they are the administration
emails asking me if I want to approve the "email" or not, or they are
messages that make it through the list.

These messages are either hitting ALL_TRUSTED, because they come from
mailing lists on my networks, or are tagged with a clear
untrusted-relays list. In otherwords, I've got my trusted_networks setup
so that SA knows about networks that I trust to be sending legitimate
email (they are not spam originators), but obviously spam gets through,
but the spam comes from hops previous to these networks. If I understand
things properly, because I've got these setup in my trusted_networks,
then these previous hops will be checked in RBLs, so the spam is more
detectable. For example, the debian servers do send some spam to me, but
the Received: headers in the emails are correct, so if the server's
address is in trusted_networks, then SA will look up the address debian
got the email from in RBLs.  

What I am unsure of is if I am poisoning my bayes by reporting these
messages that make it through as spam. Should I be just deleting them?
The tokens that are legitimate that will end up as collateral damage are
going to be the list footers, the list administration messages, and
potentially other pieces.

I'm hoping I can identify why my bayes database is so bad (it thinks
everything is BAYES_00 now), and if this is why I will want to change my
training behavior.

thanks,
micah

Re: two databases

[Micah Anderson]

Michael Grant  writes:

> I did not realize one could store the bayes scores in sql.
>
> So I'd store the bayes scores on a third server and let both mxes use
> the same database.

I did this, but my bayes in mysql and pointed two different spamd
machines at it, but I had severe problems that I could not resolve. I
posted to the list[0] about the problems.

The basic problem was that as soon as I fired up the second server it
immediately starts blocking on the bayes work. Average scantimes go from
1-2 seconds up to 35+ and the max children get eaten up by blocking on
the bayes work to the point where its pointless because too many
processes are blocked. Disabling the bayes_sql stuff on one of the
machines dropped the scantimes back to their expected average of 1-2
seconds (but of course none of the BAYES tests will fire and
autolearning fails).

My mysql server is its own machine, it was local to the first spamd
(local LAN) and remote to the second (over the net). I eliminated any
hostname lookup problems, obviously couldn't eliminate network latency,
but that shouldn't have caused such a severe result. I'm running with
InnoDB tables, so I shouldn't have any row-level locking issues... in
any case I might have had some issues because my MySQL database needed
to be optimized, but I was not able to determine how and now I just run
one of the spamd's without bayes, which is not too bad because my bayes
database seems to be totally worthless at the moment. :P

micah

0. http://permalink.gmane.org/gmane.mail.spam.spamassassin.general/113673

Re: for discussion FQDN of *.lan vs *.home

[Karsten Bräckelmann]

On Fri, 2009-06-05 at 16:28 +0300, Henrik K wrote:
> On Fri, Jun 05, 2009 at 12:19:59PM +0100, RW wrote:

> > This test only looks at the last hop, so I don't see your concern.
> > 
> > Actually it should be the last hop into the internal network,
> > presumably it's one of the tests that's fixed in SVN. IMO it should
> > also test for "auth= "
> 
> Right, it's internal in SVN.. though isn't in sa-update yet.

*External*. :)


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: for discussion FQDN of *.lan vs *.home

[Henrik K]

On Fri, Jun 05, 2009 at 12:19:59PM +0100, RW wrote:
> > 
> > header HELO_LH_HOME X-Spam-Relays-Untrusted =~ /^[^\]]+ 
> > helo=\S+\.(?:home|lan) /i
> 
> This test only looks at the last hop, so I don't see your concern.
> 
> Actually it should be the last hop into the internal network,
> presumably it's one of the tests that's fixed in SVN. IMO it should
> also test for "auth= "

Right, it's internal in SVN.. though isn't in sa-update yet.

Re: for discussion FQDN of *.lan vs *.home

[Karsten Bräckelmann]

On Fri, 2009-06-05 at 06:20 -0400, Michael Scheidell wrote:
> I posted a bug, you can discuss here and I guess vote or discuss on 
> bugzilla:

No voting. And please keep the discussion on the list.

> Way too many people are using .lan (local area network) as their 
> internal, local lan.
> 
> I agree if FIRST untrusted does a 'helo *.lan' you should score it high, 
> but if they have an internal server that does a helo *.lan to their 
> external (bastian or smart host) and it uses a valid FQDN, you should 
> not score it so high.
> 
> header HELO_LH_HOME X-Spam-Relays-Untrusted =~ /^[^\]]+ 
> helo=\S+\.(?:home|lan) /i
  ^^^

The rule is anchored at the beginning of the internal header, and
excludes the closing square bracket in his matching. Thus it only
matches the last (the one handing off to your MX) untrusted relay.


-- 
char *t="\10pse\0r\0dtu...@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}

Re: Barracuda Blacklist

[Andy Dorman]

BUZZHOST_STINGER wrote:

On Sun, 2009-05-31 at 14:39 -0600, LuKreme wrote:

On 29-May-2009, at 07:32, Andy Dorman wrote:
1. I could not find out WHY our IPs (we have a block of 32 for the  
cluster of servers that my email was being sent from) were being  
listed


I do have to add this would be a lie. A call to Barracuda support and
they will happily go through their evidence database with you. 


If your block was on the reputation list, it would have been because you
were sending spam. It's really that simple.



Sorry, I was not explicit enough when saying I could not find out WHY we were 
blocked.  After spending a couple of hours at their site trying to figure out 
how to register our service, I gave up.  At the time I did not know emailreg.org 
was a part of Baracudda.  And if you look on their site (I just made a fresh 
pass) their "Contact us" is an email form.


As for sending spam and getting listed, the crazy thing is that we don't send 
ANY email other than mail like this (which actually originates from a different 
IP block) and the email that is addressed to our customers and is passed by our 
filters (SA plus some other metrics we have added).  And that email only goes to 
our customer's destination email servers.  Of course, a small % of that will be 
spam (except when someone does something stupid on their white list and puts 
something like yahoo.com in it).




The 'join emailreg.org' is the usual old B/S they give to non
co-operative mongs that call them. 



Ahhh.  That makes sense in a twisted sort of way.

So, I hope this explains that I was not lying.  I just did not see a 
fast/obvious way to get info on how our IP got black listed.  And by the time I 
had more info, the blacklist had been lifted, again with no apparent reason.


--
Andy Dorman
Ironic Design, Inc.
AnteSpam.com, HomeFreeMail.com, ComeHome.net

Re: for discussion FQDN of *.lan vs *.home

[RW]

On Fri, 05 Jun 2009 06:20:12 -0400
Michael Scheidell  wrote:


> I agree if FIRST untrusted 

FWIW the terms first and last should always be used in the client ->
spamassassin direction.

> does a 'helo *.lan' you should score it
> high, but if they have an internal server that does a helo *.lan to
> their external (bastian or smart host) and it uses a valid FQDN, you
> should not score it so high.
> 
> header HELO_LH_HOME X-Spam-Relays-Untrusted =~ /^[^\]]+ 
> helo=\S+\.(?:home|lan) /i

This test only looks at the last hop, so I don't see your concern.

Actually it should be the last hop into the internal network,
presumably it's one of the tests that's fixed in SVN. IMO it should
also test for "auth= "

Re: for discussion FQDN of *.lan vs *.home

[Michael Scheidell]

sorry, bugzilla link:

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6124


--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation

   * Certified SNORT Integrator
   * 2008-9 Hot Company Award Winner, World Executive Alliance
   * Five-Star Partner Program 2009, VARBusiness
   * Best Anti-Spam Product 2008, Network Products Guide
   * King of Spam Filters, SC Magazine 2008

_
This email has been scanned and certified safe by SpammerTrap(r). 
For Information please see http://www.secnap.com/products/spammertrap/

_

for discussion FQDN of *.lan vs *.home

[Michael Scheidell]

I posted a bug, you can discuss here and I guess vote or discuss on 
bugzilla:


Way too many people are using .lan (local area network) as their 
internal, local lan.


I agree if FIRST untrusted does a 'helo *.lan' you should score it high, 
but if they have an internal server that does a helo *.lan to their 
external (bastian or smart host) and it uses a valid FQDN, you should 
not score it so high.


header HELO_LH_HOME X-Spam-Relays-Untrusted =~ /^[^\]]+ 
helo=\S+\.(?:home|lan) /i


3.714 points is pretty high.

score HELO_LH_HOME 2.602 3.169 2.689 3.714

in this case client used the 'default' FQDN on their exchange server 
(yes, stupid, not RFC compliant)  they have a real FQDN that matches 
their ip, but for some reason, microsoft does not make it abundantly 
clear how important the FQDN setting in exchange is.


Score a little lower, or maybe score *.lan and *.home a little different
split it into two rules.


--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation

   * Certified SNORT Integrator
   * 2008-9 Hot Company Award Winner, World Executive Alliance
   * Five-Star Partner Program 2009, VARBusiness
   * Best Anti-Spam Product 2008, Network Products Guide
   * King of Spam Filters, SC Magazine 2008

_
This email has been scanned and certified safe by SpammerTrap(r). 
For Information please see http://www.secnap.com/products/spammertrap/

_

Re: Barracuda Blacklist

[BUZZHOST_STINGER]

On Sun, 2009-05-31 at 14:39 -0600, LuKreme wrote:
> On 29-May-2009, at 07:32, Andy Dorman wrote:
> > 1. I could not find out WHY our IPs (we have a block of 32 for the  
> > cluster of servers that my email was being sent from) were being  
> > listed

I do have to add this would be a lie. A call to Barracuda support and
they will happily go through their evidence database with you. 

If your block was on the reputation list, it would have been because you
were sending spam. It's really that simple.

The 'join emailreg.org' is the usual old B/S they give to non
co-operative mongs that call them. 

Much that I hate Barracuda for their digital shoplifting, they are
unlike the money grabbers at $pamhaus and the cbl assholes. You can call
them and they will tell you the date, time and quantity of the crap your
IP has sent. Their data is good - very good. They even keep details of
how many of their users marked the message as spam. So please go right
ahead an call them and raise a 'BBL NON CUSTOMER CASE'.

UK CONTACT: Adam Light +44 (0)1256 300102
US CONTACT: Jan Gobble ++1 408 342 5300 

What is the IP range you have? I don't see it in the thread any place.

Re: Controlling spamd logging from spamc

[Martin Gregorie]

On Thu, 2009-06-04 at 18:32 -0400, Jeff Mincy wrote: 
> From: Martin Gregorie 
> 
> Wouldn't it be easier to run another spamd on a different machine for
> rule development and testing?  Or perhaps just running as a different
> 'test' user, and then ignore log messages for that user in the statistics.
> 
I'm about to set that up today and get it integrated with cvs.

> >   Would anybody else find this a useful feature too?
> 
> I've sometimes wanted the other way - eg get more debugging output for
> a particular message.
> 
I think these are two sides of the same coin: if spamc could pass
debugging control flags to spamd via a message wrapper then it would be
simple to add the ability to control logging as well.

Martin