[sniffer] New Rulebot F001

2006-03-06 Thread Pete McNeil
Hello Sniffer folks,

  The first of the new rulebots is coming online.

  Rulebot F001 creates IP rules for sources that consistently fail
  many tests while also reaching the cleanest of our spamtraps.

  The rules will appear in group 63.

  The bot is playing catchup a bit (since there have been few IP rules
  at all since we disabled the old bots).

  The algorithms used in this bot have been tested manually for 2
  weeks with no false positives.

  Expect an increase in your rulebase size while F001 catches up with
  current spamtrap data.

Thanks,

_M

Pete McNeil (Madscientist)
President, MicroNeil Research Corporation
Chief SortMonster (www.sortmonster.com)
Chief Scientist (www.armresearch.com)


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


[sniffer] New rulebase compilers online.

2006-03-06 Thread Pete McNeil
Hello Sniffer Folks,

  I have just completed work to upgrade the rulebase compiler bots.
  They are now significantly more efficient. As a result you will be
  seeing updates more frequently.

  Previous lag was between 40-120 minutes.

  Current lag (sustained) is  5 minutes.

  More timely updates should equate to lower spam leakage for new
  spam.

  You do not need to take any action on this. This note is for your
  information only.

Thanks,

_M

Pete McNeil (Madscientist)
President, MicroNeil Research Corporation
Chief SortMonster (www.sortmonster.com)
Chief Scientist (www.armresearch.com)


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


RE: [sniffer] New Rulebot F001

2006-03-06 Thread Jay Sudowski - Handy Networks LLC
There's been at least one FP ;)

--
Rule - 861038
NameF001 for Message 2888327: [216.239.56.131]
Created 2006-03-02
Source  216.239.56.131
Hidden  false
Blocked false
Origin  Automated-SpamTrap
TypeReceivedIP
Created By  [EMAIL PROTECTED]
Owner   [EMAIL PROTECTED]
Strength2.08287379496965
False Reports   0
From Users  0
[FPR:B]

The rule is below threshold, and/or badly or broadly coded so it will be
removed from the core rulebase.


My concern with automated IP rule coding is that we use Sniffer because
it's extremely accurate.  Coding rules linked to IPs, particularly IPs
that are used by google or any large ISP to send large amounts of
(mostly legitimate) email is contrary to what Sniffer is great at, which
is tagging spam that no one else is.

Is response code 63 going to be utilized for any other purposes?  If
not, I will let Declude know to weight these responses lower than normal
Sniffer.

- Jay 
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil
Sent: Monday, March 06, 2006 3:00 PM
To: sniffer@sortmonster.com
Subject: [sniffer] New Rulebot F001

Hello Sniffer folks,

  The first of the new rulebots is coming online.

  Rulebot F001 creates IP rules for sources that consistently fail
  many tests while also reaching the cleanest of our spamtraps.

  The rules will appear in group 63.

  The bot is playing catchup a bit (since there have been few IP rules
  at all since we disabled the old bots).

  The algorithms used in this bot have been tested manually for 2
  weeks with no false positives.

  Expect an increase in your rulebase size while F001 catches up with
  current spamtrap data.

Thanks,

_M

Pete McNeil (Madscientist)
President, MicroNeil Research Corporation
Chief SortMonster (www.sortmonster.com)
Chief Scientist (www.armresearch.com)


This E-Mail came from the Message Sniffer mailing list. For information
and (un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re: [sniffer] New Rulebot F001

2006-03-06 Thread Darin Cox
We just reviewed this morning's logs and had a few false positives.  Not
sure if these are due to the new rulebot, but it's more than we've had for
the entire day for the past month.

Rules
--
873261
866398
856734
284831
865663

Darin.


- Original Message - 
From: Jay Sudowski - Handy Networks LLC [EMAIL PROTECTED]
To: sniffer@SortMonster.com
Sent: Monday, March 06, 2006 3:13 PM
Subject: RE: [sniffer] New Rulebot F001


There's been at least one FP ;)

--
Rule - 861038
NameF001 for Message 2888327: [216.239.56.131]
Created 2006-03-02
Source  216.239.56.131
Hidden  false
Blocked false
Origin  Automated-SpamTrap
TypeReceivedIP
Created By  [EMAIL PROTECTED]
Owner   [EMAIL PROTECTED]
Strength2.08287379496965
False Reports   0
From Users  0
[FPR:B]

The rule is below threshold, and/or badly or broadly coded so it will be
removed from the core rulebase.


My concern with automated IP rule coding is that we use Sniffer because
it's extremely accurate.  Coding rules linked to IPs, particularly IPs
that are used by google or any large ISP to send large amounts of
(mostly legitimate) email is contrary to what Sniffer is great at, which
is tagging spam that no one else is.

Is response code 63 going to be utilized for any other purposes?  If
not, I will let Declude know to weight these responses lower than normal
Sniffer.

- Jay
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil
Sent: Monday, March 06, 2006 3:00 PM
To: sniffer@sortmonster.com
Subject: [sniffer] New Rulebot F001

Hello Sniffer folks,

  The first of the new rulebots is coming online.

  Rulebot F001 creates IP rules for sources that consistently fail
  many tests while also reaching the cleanest of our spamtraps.

  The rules will appear in group 63.

  The bot is playing catchup a bit (since there have been few IP rules
  at all since we disabled the old bots).

  The algorithms used in this bot have been tested manually for 2
  weeks with no false positives.

  Expect an increase in your rulebase size while F001 catches up with
  current spamtrap data.

Thanks,

_M

Pete McNeil (Madscientist)
President, MicroNeil Research Corporation
Chief SortMonster (www.sortmonster.com)
Chief Scientist (www.armresearch.com)


This E-Mail came from the Message Sniffer mailing list. For information
and (un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html



This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re[2]: [sniffer] New Rulebot F001

2006-03-06 Thread Pete McNeil

On Monday, March 6, 2006, 3:13:53 PM, Jay wrote:

JSHNL There's been at least one FP ;)

JSHNL --
JSHNL Rule - 861038
JSHNL NameF001 for Message 2888327: [216.239.56.131]
JSHNL Created 2006-03-02
JSHNL Source  216.239.56.131
JSHNL Hidden  false
JSHNL Blocked false
JSHNL Origin  Automated-SpamTrap
JSHNL TypeReceivedIP
JSHNL Created By  [EMAIL PROTECTED]
JSHNL Owner   [EMAIL PROTECTED]
JSHNL Strength2.08287379496965
JSHNL False Reports   0

Yes, sorry about the confusion. The original announcement happened
about 3 days before that FP. The note was a resend this afternoon so
that Karen (Tink) could update the web site with recent news.

In fact, both of those notes were resends... The originals didn't make
it because I transposed the s and n near the t in sortmonster.

Sorry again for the confusion.

_M



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re: [sniffer] New rulebase compilers online.

2006-03-06 Thread Matt

Pete,

Does this mean that you are somehow supporting incremental rule base 
updates, or is it that the compiler is just much faster so we will get 
the same number of updates, but generally get them 40-120 minutes 
earlier in relation to the data that generated them?


Either way, definitely an improvement.  The closer to real-time we can 
get, the better.


Thanks,

Matt



Pete McNeil wrote:


Hello Sniffer Folks,

 I have just completed work to upgrade the rulebase compiler bots.
 They are now significantly more efficient. As a result you will be
 seeing updates more frequently.

 Previous lag was between 40-120 minutes.

 Current lag (sustained) is  5 minutes.

 More timely updates should equate to lower spam leakage for new
 spam.

 You do not need to take any action on this. This note is for your
 information only.

Thanks,

_M

Pete McNeil (Madscientist)
President, MicroNeil Research Corporation
Chief SortMonster (www.sortmonster.com)
Chief Scientist (www.armresearch.com)


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


 




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re[2]: [sniffer] New Rulebot F001

2006-03-06 Thread Pete McNeil
On Monday, March 6, 2006, 3:42:50 PM, Darin wrote:

DC We just reviewed this morning's logs and had a few false positives.  Not
DC sure if these are due to the new rulebot, but it's more than we've had for
DC the entire day for the past month.

DC Rules
DC --
DC 873261
DC 866398
DC 856734
DC 284831
DC 865663

Three of these are from F001 and have been removed.

865663 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.233.166.182
 http://www.dnsstuff.com/tools/ptr.ch?ip=64.233.166.182

856734 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.249.82.200
 http://www.dnsstuff.com/tools/ptr.ch?ip=64.249.82.200

873261 - http://www.dnsstuff.com/tools/ip4r.ch?ip=207.217.120.227
 http://www.dnsstuff.com/tools/ptr.ch?ip=207.217.120.227


I haven't yet processed the fps, only looked up the rules.

There are currently 32820 rules authored by the F001 bot.

Hope this helps,

_M





This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


RE: Re[2]: [sniffer] New Rulebot F001

2006-03-06 Thread Colbeck, Andrew
Pete,

One of these was EarthLink [207.217.120.227], and one of these was
Google Mail [64.233.166.182].

SpamBag lists the EarthLink address as a source of bogus bounces, and I
posit that this would be the source of the mail to the spamtraps that
would trigger the F001 bot.

I would like to state that I don't need Message Sniffer to identify
servers that send bogus postmaster notifications.  This would be
entirely due to false positives such as the three examples above.

Given that spammers clearly recycle their email database as a
fake-mailfrom database, any spamtrap address will get bogus bounces and
therefore, the spamtraps will flag legitimate senders' IP addresses in
Rule 63.

I don't expect nor want you to discuss the details of the spamtraps as
the point of one class of your spamtraps is that their methods are
secret.  However, Matt has described a subset of the filters various
Decluders have used to filter out postmaster bounces and other reflected
noise, and I can certainly chip in on that conversation offline.

Andrew.


 -Original Message-
 From: [EMAIL PROTECTED] 
 [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil
 Sent: Monday, March 06, 2006 3:18 PM
 To: Darin Cox
 Subject: Re[2]: [sniffer] New Rulebot F001
 
 On Monday, March 6, 2006, 3:42:50 PM, Darin wrote:
 
 DC We just reviewed this morning's logs and had a few false 
 positives.  
 DC Not sure if these are due to the new rulebot, but it's more than 
 DC we've had for the entire day for the past month.
 
 DC Rules
 DC --
 DC 873261
 DC 866398
 DC 856734
 DC 284831
 DC 865663
 
 Three of these are from F001 and have been removed.
 
 865663 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.233.166.182
  http://www.dnsstuff.com/tools/ptr.ch?ip=64.233.166.182
 
 856734 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.249.82.200
  http://www.dnsstuff.com/tools/ptr.ch?ip=64.249.82.200
 
 873261 - http://www.dnsstuff.com/tools/ip4r.ch?ip=207.217.120.227
  http://www.dnsstuff.com/tools/ptr.ch?ip=207.217.120.227
 
 
 I haven't yet processed the fps, only looked up the rules.
 
 There are currently 32820 rules authored by the F001 bot.
 
 Hope this helps,
 
 _M
 
 
 
 
 
 This E-Mail came from the Message Sniffer mailing list. For 
 information and (un)subscription instructions go to 
 http://www.sortmonster.com/MessageSniffer/Help/Help.html
 


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re: Re[2]: [sniffer] New Rulebot F001

2006-03-06 Thread Darin Cox
Thanks, Pete.

Darin.


- Original Message - 
From: Pete McNeil [EMAIL PROTECTED]
To: Darin Cox sniffer@SortMonster.com
Sent: Monday, March 06, 2006 6:17 PM
Subject: Re[2]: [sniffer] New Rulebot F001


On Monday, March 6, 2006, 3:42:50 PM, Darin wrote:

DC We just reviewed this morning's logs and had a few false positives.  Not
DC sure if these are due to the new rulebot, but it's more than we've had
for
DC the entire day for the past month.

DC Rules
DC --
DC 873261
DC 866398
DC 856734
DC 284831
DC 865663

Three of these are from F001 and have been removed.

865663 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.233.166.182
 http://www.dnsstuff.com/tools/ptr.ch?ip=64.233.166.182

856734 - http://www.dnsstuff.com/tools/ip4r.ch?ip=64.249.82.200
 http://www.dnsstuff.com/tools/ptr.ch?ip=64.249.82.200

873261 - http://www.dnsstuff.com/tools/ip4r.ch?ip=207.217.120.227
 http://www.dnsstuff.com/tools/ptr.ch?ip=207.217.120.227


I haven't yet processed the fps, only looked up the rules.

There are currently 32820 rules authored by the F001 bot.

Hope this helps,

_M





This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re[2]: [sniffer] New rulebase compilers online.

2006-03-06 Thread Pete McNeil
On Monday, March 6, 2006, 6:09:43 PM, Matt wrote:

M Pete,

M Does this mean that you are somehow supporting incremental rule base 
M updates, or is it that the compiler is just much faster so we will get
M the same number of updates, but generally get them 40-120 minutes 
M earlier in relation to the data that generated them?

The latter. Incremental updates are coming with the V3 engine. We will
have real time reporting and tuning before that.

The new behavior for the compiler bots is to seek out any eligible
rulebases that match the profile of the previously compiled rulebase
and to use the cached data to build the new rulebase provided it is
discovered within a short enough period (a matter of seconds). This is
called replication. Replication happens in seconds. Compiling a
rulebase takes between 5 and 35 minutes depending on the complexity.

While I have seen occasional spikes, I generally now see unfinished,
eligible rulebase counts in the low teens and estimated lag in the
single digits.

M Either way, definitely an improvement.  The closer to real-time we can
M get, the better.

:-)

_M


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html


Re[4]: [sniffer] New Rulebot F001

2006-03-06 Thread Pete McNeil
On Monday, March 6, 2006, 7:24:20 PM, Andrew wrote:

snip

CA I would like to state that I don't need Message Sniffer to
CA identify servers that send bogus postmaster notifications.  This
CA would be entirely due to false positives such as the three
CA examples above.

CA Given that spammers clearly recycle their email database as a
CA fake-mailfrom database, any spamtrap address will get bogus bounces and
CA therefore, the spamtraps will flag legitimate senders' IP addresses in
CA Rule 63.

CA I don't expect nor want you to discuss the details of the
CA spamtraps as the point of one class of your spamtraps is that
CA their methods are secret.  However, Matt has described a subset of
CA the filters various Decluders have used to filter out postmaster
CA bounces and other reflected noise, and I can certainly chip in on
CA that conversation offline.

In addition to all previous IP rule false positives, any new false
positives will be kept in the rulebase to prevent any repeats.

Regarding outscatter, we do create rules where we can to eliminate
known outscatter - when the bounce contains sufficient information to
identify it clearly as originating from malware or known spam.

However, the trap F001 is using are pre-processed with mediation rules
to blind the system from these kinds of messages. These rules are
not complete (perhaps never will be) but they are pretty good and
getting better.

With each new case we will be refining what cannot be seen by bots or
even people from these sources.

_M



This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html