Re: Re[4]: [sniffer] Rash of false positives

[Darin Cox]

Hi Pete,
 
I'll send the logs for the past two days separately 
to support (at).  We do run snf2check on every downloaded rulebase, so that 
shouldn't be an issue.
 
The one thing I didn't think to do was to revert to 
an old rulebase, but we only keep the previous, so it would have already been 
too late when we saw the problem this morning.
 
Thanks,
Darin.
 
 
- Original Message - 
From: Pete 
McNeil 
To: Darin Cox 
Sent: Tuesday, November 08, 2005 4:03 PM
Subject: Re[4]: [sniffer] Rash of false positives

On Tuesday, November 8, 2005, 3:25:20 PM, Darin wrote:



  
  

  >

  Hi Pete,
   
  There was a consistent stream of false positives over 
  the mentioned time period, not just a blast at a particular time. 
   They suddenly started at 5pm (shortly after a 4:30pm rulesbase 
  update), and were fairly evenly spread from 5pm - 11pm and 6am - 10am 
  today (not many legitimate emails came in between 11pm and 6am)...spanning 
  4 other rulebase updates at 8:40pm, 12am, 3am, and 6:20am.  There 
  were a number of different rules involved, and over 45 false positives in 
  that time period.

This is highly unusual -- I didn't remove many rules, and normally only one 
or two would be responsible. If you found that a large number of rules were 
responsible then something else happend and we need to look at that... I'd need 
to see your SNF logs from that period since the changes (removals anyway) in the 
rulebase were very small and unrelated - that just doesn't line up with your 
description.

One thing does-- in the past if snf2check was not used to check a new 
download then a corrupted rulebase could cause SNF to produce erratic results... 
since snf2check has been in place we have not seen this. Is it possible that a 
bad rulebase file got pressed into service on your system? -- probably a look at 
the logs would help there too since this kind of failure is accompanied by very 
specific oddities in the logs.

Hope this helps,

_M
This E-Mail came from the Message Sniffer mailing list. For 
information and (un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html 

Re[4]: [sniffer] Rash of false positives

[Pete McNeil]

On Tuesday, November 8, 2005, 3:25:20 PM, Darin wrote:




>


Hi Pete,
 
There was a consistent stream of false positives over the mentioned time period, not just a blast at a particular time.  They suddenly started at 5pm (shortly after a 4:30pm rulesbase update), and were fairly evenly spread from 5pm - 11pm and 6am - 10am today (not many legitimate emails came in between 11pm and 6am)...spanning 4 other rulebase updates at 8:40pm, 12am, 3am, and 6:20am.  There were a number of different rules involved, and over 45 false positives in that time period.





This is highly unusual -- I didn't remove many rules, and normally only one or two would be responsible. If you found that a large number of rules were responsible then something else happend and we need to look at that... I'd need to see your SNF logs from that period since the changes (removals anyway) in the rulebase were very small and unrelated - that just doesn't line up with your description.

One thing does-- in the past if snf2check was not used to check a new download then a corrupted rulebase could cause SNF to produce erratic results... since snf2check has been in place we have not seen this. Is it possible that a bad rulebase file got pressed into service on your system? -- probably a look at the logs would help there too since this kind of failure is accompanied by very specific oddities in the logs.

Hope this helps,

_M





This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: Re[2]: [sniffer] Rash of false positives

[Darin Cox]

Hi Pete,
 
There was a consistent stream of false positives 
over the mentioned time period, not just a blast at a particular 
time.  They suddenly started at 5pm (shortly after a 4:30pm rulesbase 
update), and were fairly evenly spread from 5pm - 11pm and 6am - 10am today (not 
many legitimate emails came in between 11pm and 6am)...spanning 4 other 
rulebase updates at 8:40pm, 12am, 3am, and 6:20am.  There were a 
number of different rules involved, and over 45 false positives in that time 
period.
 
Since the problem was evidently corrected with 
the 10am rulebase, you will probably need to look back at what 
happened starting with the 4:30pm rulebase.  I doubt looking at 
the current rulebase will help since the problem has now been corrected, 
but I'm sure you archive them and can look back to see what process breakdown 
allowed this to happen.
 
I'm familiar with the panic procedure, but since 
there was such a broad base of false positives across a number of rules, adding 
panic rules for all of them just didn't make sense.  Disabling Sniffer 
entirely would have been the action we would have taken.
 
Let me know what you find out.
 
I completely understand the learning curve with new 
staff, but the quality of the rules is imperative.  Anything you can do to 
keep that quality high is much appreciated.
 
Thanks,
Darin.
 
 
- Original Message - 
From: Pete 
McNeil 
To: Darin Cox 
Sent: Tuesday, November 08, 2005 2:49 PM
Subject: Re[2]: [sniffer] Rash of false positives

On Tuesday, November 8, 2005, 11:02:09 AM, Darin wrote:



  
  

  >

  Hi Pete,
   
  The rash of false positives seems to have stopped 
  with the last sniffer rulebase update at 10am ET.  It had started 
  with a rulebase update at 4:30pm ET yesterday, and continued through the 
  updates at 8:40pm, 12am, 3am, and 6:20am today.
   
  I'd still like to know what happened, and how we can 
  avoid it in the future.
   

I've been bound up in some performance tuning today so I've not had a chance 
to follow this thread until now. When I first looked in on it I scanned the 
false positive submissions and almost none of them matched any active rules.

I know that a couple of rules were pulled out after review last night late .. 
they had been picked up by some FPs in SURBL & others that matched up with 
spamtrap submissions. It's possible that these are what you experienced. I won't 
know unless you can give me some log entries to go with those messages since 
those entries will tell me the rule IDs.

As for having it happen again - that's very unlikely since ever time we pull 
a rule out due to FPs or potential FPs (the rules that were pulled had not 
caused any FPs yet but were expected to... one was rr.com IIRC, it was pulled 
only a couple hours after it's creation).

A lot of things have to go wrong to cause an FP problem like you are 
reporting.

Please look up our rule-panic procedure which is designed to mitigate these 
problems immediately for you if they happen:

http://www.sortmonster.com/MessageSniffer/Help/FalsePositivesHelp.html#RulePanic

We can't guarantee that rule-panics won't happen, but we can make them 
exceedingly rare and non-repeatable.

I will be processing your FP submissions shortly.

Hope this helps,

_M
This E-Mail came from the Message 
Sniffer mailing list. For information and (un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html 

Re[2]: [sniffer] Rash of false positives

[Pete McNeil]

On Tuesday, November 8, 2005, 10:19:20 AM, Darrell wrote:

Dsic> I too have had to submit a lot more false positives lately.  I also second
Dsic> that false positive processing seems to be a lot slower than previously.

We have introduced a number of new rule coding procedures (and people)
as well as a number of new spamtraps and usertraps.

These (collectively) can contribute to new FPs in a number of obvious
and not-so-obvious ways. New people and procedures require some
shake-out time. New spamtraps can re-introduce content that awakens
old rules which may no-longer be valid. I've seen a mix of both cases
recently.

FP processing has been slower and faster, depending upon the day. I
apologize for this personally since I'm the primary FP person... I'm
also the primary developer and maintainer of the systems... so the
explanation follows: I have been a bit overloaded lately working on
improvements to the system, training, support, and R&D work to apply
to upcoming versions. On occasion this has prevented me from keeping
the schedule I would like w/ regard to FP processing.

All of that said, the general trend of the rate of FP reports (today
not withstanding) has been downward or flat. I expect the rate of
false positives to decrease slowly over time as our new folks gain
experience and we develop more powerful tools to help them.

I will work harder to maintain a more regular schedule on FP
processing while still pushing development and support work forward.

Thanks,

_M




This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re[2]: [sniffer] Rash of false positives

[Pete McNeil]

On Tuesday, November 8, 2005, 11:02:09 AM, Darin wrote:




>


Hi Pete,
 
The rash of false positives seems to have stopped with the last sniffer rulebase update at 10am ET.  It had started with a rulebase update at 4:30pm ET yesterday, and continued through the updates at 8:40pm, 12am, 3am, and 6:20am today.
 
I'd still like to know what happened, and how we can avoid it in the future.
 





I've been bound up in some performance tuning today so I've not had a chance to follow this thread until now. When I first looked in on it I scanned the false positive submissions and almost none of them matched any active rules.

I know that a couple of rules were pulled out after review last night late .. they had been picked up by some FPs in SURBL & others that matched up with spamtrap submissions. It's possible that these are what you experienced. I won't know unless you can give me some log entries to go with those messages since those entries will tell me the rule IDs.

As for having it happen again - that's very unlikely since ever time we pull a rule out due to FPs or potential FPs (the rules that were pulled had not caused any FPs yet but were expected to... one was rr.com IIRC, it was pulled only a couple hours after it's creation).

A lot of things have to go wrong to cause an FP problem like you are reporting.

Please look up our rule-panic procedure which is designed to mitigate these problems immediately for you if they happen:

http://www.sortmonster.com/MessageSniffer/Help/FalsePositivesHelp.html#RulePanic

We can't guarantee that rule-panics won't happen, but we can make them exceedingly rare and non-repeatable.

I will be processing your FP submissions shortly.

Hope this helps,

_M





This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] Rash of false positives

[Darin Cox]

Hi Pete,
 
The rash of false positives seems to have 
stopped with the last sniffer rulebase update at 10am ET.  It had started 
with a rulebase update at 4:30pm ET yesterday, and continued through the updates 
at 8:40pm, 12am, 3am, and 6:20am today.
 
I'd still like to know what happened, and how we 
can avoid it in the future.
 
Thanks,
Darin.
 
 
- Original Message - 
From: Darin Cox 
To: sniffer@SortMonster.com 
Sent: Tuesday, November 08, 2005 8:45 AM
Subject: [sniffer] Rash of false positives

Hi Pete,
 
What's going on over there?  We had somewhere 
between 5 and 10 times the usual number of Sniffer false positives this 
morning.  They are across the board, so it's not just one rule that's 
catching them, or a particular set of senders or receivers.
 
Hopefully you can get it under control 
soon.
 
It would also be extremely helpful if you could 
speed up the false positive processing.  Lately it seems to take 2-4 days 
for the rules to be adjusted, which usually means more of the same are caught 
and submitted over that time.  I believe speeding up that process would 
result in fewer to process all around.
 
Thanks,
Darin.
 
 

Re: [sniffer] Rash of false positives

[Darin Cox]

I've submitted about 45 so far this morning.  I normally submit at most a
half dozen each morning.

Darin.


- Original Message - 
From: "Darrell ([EMAIL PROTECTED])" <[EMAIL PROTECTED]>
To: 
Sent: Tuesday, November 08, 2005 10:19 AM
Subject: Re: [sniffer] Rash of false positives


I too have had to submit a lot more false positives lately.  I also second
that false positive processing seems to be a lot slower than previously.

Darrell
 
Check out http://www.invariantsystems.com for utilities for Declude,
mxGuard, And Imail.  IMail/Declude Overflow Queue Monitoring, SURBL/URI
integration, MRTG Integration, and Log Parsers.


Scott Fisher writes:

> I don't know if I would call it a rash, but over the last week, I've
submitted about 30 false positives. That's far more than average.
> I've developed a feeling that Message Sniffer has become "too tight".
>
> - Original Message - 
>   From: Darin Cox
>   To: sniffer@SortMonster.com
>   Sent: Tuesday, November 08, 2005 8:54 AM
>   Subject: Re: [sniffer] Rash of false positives
>
>
>   We're seeing a continual stream of false positives.  It's taking all of
our time just to keep up with it at the moment.  If something isn't done
soon, we're going to have to disable sniffer.
>
>   Darin.
>
>
>   - Original Message - 
>   From: Computer House Support
>   To: sniffer@SortMonster.com
>   Sent: Tuesday, November 08, 2005 9:34 AM
>   Subject: Re: [sniffer] Rash of false positives
>
>
>   Dear Darin,
>
>   Thanks for the heads up.  It's going to take me about 45 minutes to
check the 9000 messages that were blocked by Sniffer last night, but I'll
let you know if we experienced the same thing.
>
>
>   Michael Stein
>   Computer House
>   www.computerhouse.com
>
> - Original Message - 
> From: Darin Cox
> To: sniffer@SortMonster.com
> Sent: Tuesday, November 08, 2005 8:45 AM
> Subject: [sniffer] Rash of false positives
>
>
> Hi Pete,
>
> What's going on over there?  We had somewhere between 5 and 10 times
the usual number of Sniffer false positives this morning.  They are across
the board, so it's not just one rule that's catching them, or a particular
set of senders or receivers.
>
> Hopefully you can get it under control soon.
>
> It would also be extremely helpful if you could speed up the false
positive processing.  Lately it seems to take 2-4 days for the rules to be
adjusted, which usually means more of the same are caught and submitted over
that time.  I believe speeding up that process would result in fewer to
process all around.
>
> Thanks,
>
> Darin.
>



This E-Mail came from the Message Sniffer mailing list. For information and
(un)subscription instructions go to
http://www.sortmonster.com/MessageSniffer/Help/Help.html


This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] Rash of false positives

[Darrell (supp...@invariantsystems.com)]

I too have had to submit a lot more false positives lately.  I also second 
that false positive processing seems to be a lot slower than previously. 


Darrell

Check out http://www.invariantsystems.com for utilities for Declude, 
mxGuard, And Imail.  IMail/Declude Overflow Queue Monitoring, SURBL/URI 
integration, MRTG Integration, and Log Parsers. 



Scott Fisher writes: 


I don't know if I would call it a rash, but over the last week, I've submitted 
about 30 false positives. That's far more than average.
I've developed a feeling that Message Sniffer has become "too tight". 

- Original Message - 
  From: Darin Cox 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 8:54 AM
  Subject: Re: [sniffer] Rash of false positives 



  We're seeing a continual stream of false positives.  It's taking all of our time just to keep up with it at the moment.  If something isn't done soon, we're going to have to disable sniffer. 

  Darin. 



  - Original Message - 
  From: Computer House Support 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 9:34 AM
  Subject: Re: [sniffer] Rash of false positives 



  Dear Darin, 

  Thanks for the heads up.  It's going to take me about 45 minutes to check the 9000 messages that were blocked by Sniffer last night, but I'll let you know if we experienced the same thing. 



  Michael Stein
  Computer House
  www.computerhouse.com 

- Original Message - 
From: Darin Cox 
To: sniffer@SortMonster.com 
Sent: Tuesday, November 08, 2005 8:45 AM
Subject: [sniffer] Rash of false positives 



Hi Pete, 

What's going on over there?  We had somewhere between 5 and 10 times the usual number of Sniffer false positives this morning.  They are across the board, so it's not just one rule that's catching them, or a particular set of senders or receivers. 

Hopefully you can get it under control soon. 

It would also be extremely helpful if you could speed up the false positive processing.  Lately it seems to take 2-4 days for the rules to be adjusted, which usually means more of the same are caught and submitted over that time.  I believe speeding up that process would result in fewer to process all around. 

Thanks, 

Darin. 





This E-Mail came from the Message Sniffer mailing list. For information and 
(un)subscription instructions go to 
http://www.sortmonster.com/MessageSniffer/Help/Help.html

Re: [sniffer] Rash of false positives

[Scott Fisher]

I don't know if I would call it a rash, but over 
the last week, I've submitted about 30 false positives. That's far more than 
average.
I've developed a feeling that Message Sniffer has 
become "too tight".
 
- Original Message - 

  From: 
  Darin Cox 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 8:54 
  AM
  Subject: Re: [sniffer] Rash of false 
  positives
  
  We're seeing a continual stream of false 
  positives.  It's taking all of our time just to keep up with it at the 
  moment.  If something isn't done soon, we're going to have to disable 
  sniffer.
  Darin.
   
   
  - Original Message - 
  From: Computer House Support 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 9:34 AM
  Subject: Re: [sniffer] Rash of false positives
  
  Dear Darin,
   
  Thanks for the heads up.  It's going to take me 
  about 45 minutes to check the 9000 messages that were blocked by Sniffer last 
  night, but I'll let you know if we experienced the same thing.
   
   
  Michael SteinComputer House
  www.computerhouse.com
   
  
- Original Message - 
From: 
Darin Cox 
To: sniffer@SortMonster.com 
Sent: Tuesday, November 08, 2005 8:45 
AM
Subject: [sniffer] Rash of false 
positives

Hi Pete,
 
What's going on over there?  We had 
somewhere between 5 and 10 times the usual number of Sniffer false positives 
this morning.  They are across the board, so it's not just one rule 
that's catching them, or a particular set of senders or 
receivers.
 
Hopefully you can get it under control 
soon.
 
It would also be extremely helpful if you could 
speed up the false positive processing.  Lately it seems to take 2-4 
days for the rules to be adjusted, which usually means more of the same are 
caught and submitted over that time.  I believe speeding up that 
process would result in fewer to process all around.
 
Thanks,
Darin.
 
 

Re: [sniffer] Rash of false positives

[Darin Cox]

No, we automatically update with every notification 
of a new rulebase.
 
Looking further, they started just before 5pm ET 
yesterday.  So far, it's about 10 times the usual number of Sniffer false 
positives.  We've sent quite a few this morning to false (at) for 
processing.
Darin.
 
 
- Original Message - 
From: Paul Lushinsky 
To: sniffer@SortMonster.com 
Sent: Tuesday, November 08, 2005 10:10 AM
Subject: Re: [sniffer] Rash of false positives

After reviewing all the blocked messages for the past 2 days on 2 different 
servers, I found no false positives. Do you happen to have an old rule base from 
several days again ? If so, try that to see if it temporarily resolves the false 
positives.
 
-Original 
  Message-From: "Darin Cox" <[EMAIL PROTECTED]>To: Date: 
  Tue, 8 Nov 2005 08:45:39 -0500Subject: [sniffer] Rash of false 
  positives
  Hi Pete,
   
  What's going on over there?  We had 
  somewhere between 5 and 10 times the usual number of Sniffer false positives 
  this morning.  They are across the board, so it's not just one rule 
  that's catching them, or a particular set of senders or receivers. 
  
   
  Hopefully you can get it under control 
  soon.
   
  It would also be extremely helpful if you could 
  speed up the false positive processing.  Lately it seems to take 2-4 days 
  for the rules to be adjusted, which usually means more of the same are caught 
  and submitted over that time.  I believe speeding up that process would 
  result in fewer to process all around.
   
  Thanks,
  Darin.
   
   

Re: [sniffer] Rash of false positives

[Paul Lushinsky]

After reviewing all the blocked messages for the past 2 days on 2 
different servers, I found no false positives. Do you happen to have an old 
rule base from several days again ? If so, try that to see if it temporarily 
resolves the false positives.
 
-Original 
Message-From: "Darin Cox" <[EMAIL PROTECTED]>To: 
Date: Tue, 8 Nov 2005 08:45:39 -0500
Subject: [sniffer] Rash of false positives
Hi Pete,
 
What's going on over there?  We had 
somewhere between 5 and 10 times the usual number of Sniffer false positives 
this morning.  They are across the board, so it's not just one rule 
that's catching them, or a particular set of senders or receivers.

 
Hopefully you can get it under control 
soon.
 
It would also be extremely helpful if you could 
speed up the false positive processing.  Lately it seems to take 2-4 
days for the rules to be adjusted, which usually means more of the same are 
caught and submitted over that time.  I believe speeding up that 
process would result in fewer to process all around.
 
Thanks,
Darin.
 
 

Re: [sniffer] Rash of false positives

[Darin Cox]

We're seeing a continual stream of false 
positives.  It's taking all of our time just to keep up with it at the 
moment.  If something isn't done soon, we're going to have to disable 
sniffer.
Darin.
 
 
- Original Message - 
From: Computer 
House Support 
To: sniffer@SortMonster.com 
Sent: Tuesday, November 08, 2005 9:34 AM
Subject: Re: [sniffer] Rash of false positives

Dear Darin,
 
Thanks for the heads up.  It's going to take me about 
45 minutes to check the 9000 messages that were blocked by Sniffer last night, 
but I'll let you know if we experienced the same thing.
 
 
Michael SteinComputer House
www.computerhouse.com
 

  - Original Message - 
  From: 
  Darin Cox 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 8:45 
  AM
  Subject: [sniffer] Rash of false 
  positives
  
  Hi Pete,
   
  What's going on over there?  We had 
  somewhere between 5 and 10 times the usual number of Sniffer false positives 
  this morning.  They are across the board, so it's not just one rule 
  that's catching them, or a particular set of senders or 
receivers.
   
  Hopefully you can get it under control 
  soon.
   
  It would also be extremely helpful if you could 
  speed up the false positive processing.  Lately it seems to take 2-4 days 
  for the rules to be adjusted, which usually means more of the same are caught 
  and submitted over that time.  I believe speeding up that process would 
  result in fewer to process all around.
   
  Thanks,
  Darin.
   
   

Re: [sniffer] Rash of false positives

[Computer House Support]

Dear Darin,
 
Thanks for the heads up.  It's going to take me about 
45 minutes to check the 9000 messages that were blocked by Sniffer last night, 
but I'll let you know if we experienced the same thing.
 
 
Michael SteinComputer House
www.computerhouse.com
 

  - Original Message - 
  From: 
  Darin Cox 
  To: sniffer@SortMonster.com 
  Sent: Tuesday, November 08, 2005 8:45 
  AM
  Subject: [sniffer] Rash of false 
  positives
  
  Hi Pete,
   
  What's going on over there?  We had 
  somewhere between 5 and 10 times the usual number of Sniffer false positives 
  this morning.  They are across the board, so it's not just one rule 
  that's catching them, or a particular set of senders or 
receivers.
   
  Hopefully you can get it under control 
  soon.
   
  It would also be extremely helpful if you could 
  speed up the false positive processing.  Lately it seems to take 2-4 days 
  for the rules to be adjusted, which usually means more of the same are caught 
  and submitted over that time.  I believe speeding up that process would 
  result in fewer to process all around.
   
  Thanks,
  Darin.
   
   

[sniffer] Rash of false positives

[Darin Cox]

Hi Pete,
 
What's going on over there?  We had somewhere 
between 5 and 10 times the usual number of Sniffer false positives this 
morning.  They are across the board, so it's not just one rule that's 
catching them, or a particular set of senders or receivers.
 
Hopefully you can get it under control 
soon.
 
It would also be extremely helpful if you could 
speed up the false positive processing.  Lately it seems to take 2-4 days 
for the rules to be adjusted, which usually means more of the same are caught 
and submitted over that time.  I believe speeding up that process would 
result in fewer to process all around.
 
Thanks,
Darin.