[sniffer] My issues with the General category, looking for a better solution
Pete and other Sniffer Customers, I've been having a lot of issues with false positives in the General category, and I'm in search of a better way to handle such things after making little progress without a large time commitment to the issue that this creates. The General category seemingly primarily consists of E-mail that comes from spam reports by Sniffer 's customers, and didn't' hit one of Sniffer's spam traps. Since I only monitor a certain range of E-mail that just barely manages to fail my system, I often times find that such messages that are tagged with Sniffer General and fall in this range are what I consider to be false positives, and originate from bulk mail providers such as CheetahMail, DartMail, etc., or come directly from first-parties such as Amazon, Target, eDiets, etc. Recently I undertook a large undertaking of identifying the bulk-mail providers by both IP block and reverse DNS entries so that I could segregate this content from the other stuff, and also defeat other filters that I use in my Declude setup that produce somewhat random results, but weren't intended to target E-mail of this variety (such as BADHEADERS, SPAMHEADERS, GIBBERISH, BASE64SUB and others). I then assigned a base score for each of these providers in 4 levels based on the trustworthiness of the provider, some are automatically held or deleted on my system. This gives me a predictable base score on top of which scores from Sniffer, SpamCop and SURBL are primarily the deciding factor in causing the E-mails to be held. Unfortunately, this exposed a large number of false positives primarily in Sniffer-General, but also in Sniffer-Experimental that were sneaking in under the limit or were otherwise not found when the E-mail's were not being segregated. It is my quest to fix these issues as they account for over 3/4 of all of my false positives. Marcus' own statistics suggest only about an 80% accuracy for this group of rules. I've narrowed down what I feel is really at issue here, so let me summarize and then discuss: 1) Sniffer customers reporting advertising related E-mail that comes from companies with first-party relationships with the recipients (though mostly never gave direct permission to add them to lists). 2) Overbroad rules generated by Sniffer. This includes things such as tagging a bulk-mail provider's domain for a violation of one of their customer's, and generating rules from things like tracking links or image hosts, and occasionally phrase and more broadly coded filters (such as *offers@). 3) Rules that target things that other rules that I have asked to be blocked cause repeated false positives despite my efforts to stop such things from occurring. As far as the first item goes, this is primarily an issue with the fact that everyone has different standards for what they consider to be spam, and we are most likely to disagree about things that fall into this gray category where first-party relationships between the sender and recipient often exist, but with varying levels of abuse that results from many different types of circumstances. For instance, many really hate Orbit, Travelocity, Expedia and Hotwire ads, but they are sent, from what I can tell, exclusively to their customers. It's the topic and the frequency that makes people consider it to be spam, but they do all honor opt-outs from what I can tell, and just today for instance, a customer of mine reported a very low value Orbit ad as a false positive. I have had experiences where I have asked that rules be blocked for the same source on three different occasions because seemingly as fast as Pete removes them according to his rules, new ones appear. I do maintain my own whitelist for such things, but I also make it a practice to report such things to Sniffer because I am not sure what rule might have tripped and what other issues might be caused by such rules if they aren't removed from my rulebase. My whitelist is specifically targeted and doesn't always prevent future rules from causing issues on my system. I am also hesitant to request white rules because spammers will domain stuff in order to exploit such things or throw off URL parsers. So the net effect of all of this is that whitelisting is only partially successful and it takes me considerable time to report, whitelist and monitor on a continual basis. I'm sure that I am pissing off some other people by submitting FP's that defeat their FN reports. I think there needs to be a change in the way that this is handled and I have a couple of ideas. The first idea would be to implore other Sniffer customers to not report E-mail that they might find objectionable, but have no proof of it being sent to people that don't have a first-party relationship with the company or newsletter, or no proof of the company not honoring opt-outs. When I get such reports from my customers, I unsubscribe them and have never had an issue doing so. Naturally I don't unsubscri
Re[4]: [sniffer] Few questions
On Wednesday, December 15, 2004, 6:54:01 PM, Marc wrote: MH> Pete, MH> FWIW, it appears that I just had a bad download. I re-downloaded it, and MH> it's running w/o errors. Thx. One other quick note/reminder. Use the snf2check utility on your downloaded rulebase files before putting them in service. This will ensure that you have a complete file that is not corrupted. Best, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[2]: [sniffer] Download server is really slow..
According to the logs there was a run on the server at this time... apparently quite a few servers downloading at the top of the hour - all competing. If you use a scheduled task for getting your rulebase files, please stagger your download schedule according to the chart here: http://www.sortmonster.com/MessageSniffer/Help/LogsHelp.html#When The chart is based on the first letter of your license ID. Scheduling updates at these times will ensure that we don't have a pile-up where everyone gets to the server at the top of the hour or some other common time. Folks who are triggering updates based on our update notifications are already making the most efficient use of resources because our rulebase compiler system schedules updates in a nice even flow. Hope this helps, Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Download server is really slow..
Me too [EMAIL PROTECTED] - Original Message - From: "Chuck Schick" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Wednesday, December 15, 2004 7:44 PM Subject: [sniffer] Download server is really slow.. > Anyone else having that problem? > > Chuck Schick > Warp 8, Inc. > (303)-421-5140 > www.warp8.com > > > This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html > --- > [This E-mail scanned for viruses by EWOL using Declude Virus] > > --- [This E-mail scanned for viruses by EWOL using Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Download server is really slow..
Anyone else having that problem? Chuck Schick Warp 8, Inc. (303)-421-5140 www.warp8.com This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[4]: [sniffer] Few questions
On Wednesday, December 15, 2004, 6:54:01 PM, Marc wrote: MH> Pete, MH> FWIW, it appears that I just had a bad download. I re-downloaded it, and MH> it's running w/o errors. Thx. Great! That makes sense too - unfortunately there's no sure way to separate the two cases (corrupted file or bad authentication) with the current file structure - so we generalize it to authenticating the file in general and throw that error if it fails to decrypt & rehash properly. (You threw me off when you mentioned renaming the file ;-) Best, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Few questions
Pete, FWIW, it appears that I just had a bad download. I re-downloaded it, and it's running w/o errors. Thx. --- Marc MH> I downloaded the sniffer demo a couple of days ago and finally installed it MH> to run as an external test w/Declude today. I ran it all morning w/o any MH> problems. This afternoon, I downloaded a new version of the snf file: MH> http://www.sortmonster.com/MessageSniffer/Demo/snfrv2r3.snf and it appeared MH> to be a newer file (larger size and newer date/time stamp). But when I MH> swapped the old file with the new by renaming them I got a ERROR_RULE_AUTH MH> error in the log. Am I doing something wrong? I am a bit confused - the demo rulebase should have been the same name in both cases. At any rate, renaming a rulebase file breaks the encryption used to validate the contents. Each license ID and authentication code constitute a unique matched pair - neither work without the other, and the contents of the rulebase file are checked with a hash that is seeded with license ID and authentication string. The license ID for the demo is snfrv2r3. Please be sure you have the most recent software and rulebase from our try-it page. MH> This may be a moot point (at least for me) since our order was faxed to you MH> earlier today and I hope to be setting up an auto-update routine tomorrow. MH> But, I was curious as I'm trying to understand how sniffer works. That's great! I'll check on your order's progress soon. If you have further questions please let us know. Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Few questions
ATTENTION ROB OF ZELLMAN PRINTING: Turn off read receipts. Fix the problem with your server rejecting replies to the very read receipts you request. John Tolmachoff Engineer/Consultant/Owner eServices For You > -Original Message- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > Behalf Of ~ ROB @ ZELLEM ~ > Sent: Wednesday, December 15, 2004 1:25 PM > To: [EMAIL PROTECTED] > Subject: Re: Re[2]: [sniffer] Few questions > > hey guys.. > This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: Re[2]: [sniffer] Few questions
Rob: 1. Could you please turn off the read receipt when you post to this (or any other) list. 2. You do not get the email notification with the trial version. Chuck Schick Warp 8, Inc. (303)-421-5140 www.warp8.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of ~ ROB @ ZELLEM ~ Sent: Wednesday, December 15, 2004 2:25 PM To: [EMAIL PROTECTED] Subject: Re: Re[2]: [sniffer] Few questions hey guys.. when you talk about getting emails about the file being old.. well i have the file for a week now and did not get any kind of email about this. All i did was download the file and put it in my server config like the howto said. any ideas? i mean am i supposed to register some place? - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Marc Hilliker" <[EMAIL PROTECTED]> Sent: Wednesday, December 15, 2004 4:11 PM Subject: Re[2]: [sniffer] Few questions > On Wednesday, December 15, 2004, 2:42:55 PM, Marc wrote: > > MH> Pete, > > MH> I downloaded the sniffer demo a couple of days ago and finally > installed it > MH> to run as an external test w/Declude today. I ran it all morning > MH> w/o > any > MH> problems. This afternoon, I downloaded a new version of the snf > MH> file: http://www.sortmonster.com/MessageSniffer/Demo/snfrv2r3.snf > MH> and it > appeared > MH> to be a newer file (larger size and newer date/time stamp). But > MH> when I swapped the old file with the new by renaming them I got a > ERROR_RULE_AUTH > MH> error in the log. Am I doing something wrong? > > I am a bit confused - the demo rulebase should have been the same name > in both cases. At any rate, renaming a rulebase file breaks the > encryption used to validate the contents. Each license ID and > authentication code constitute a unique matched pair - neither work > without the other, and the contents of the rulebase file are checked > with a hash that is seeded with license ID and authentication string. > > The license ID for the demo is snfrv2r3. Please be sure you have the > most recent software and rulebase from our try-it page. > > MH> This may be a moot point (at least for me) since our order was > MH> faxed > to you > MH> earlier today and I hope to be setting up an auto-update routine > tomorrow. > MH> But, I was curious as I'm trying to understand how sniffer works. > > That's great! I'll check on your order's progress soon. > > If you have further questions please let us know. > > Thanks, > _M > > > > > > > This E-Mail came from the Message Sniffer mailing list. For > information > and (un)subscription instructions go to > http://www.sortmonster.com/MessageSniffer/Help/Help.html > This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[4]: [sniffer] Few questions
On Wednesday, December 15, 2004, 4:24:45 PM, ~ wrote: ~RZ~> hey guys.. ~RZ~> when you talk about getting emails about the file being old.. well i have ~RZ~> the file for a week now and did not get any kind of email about this. ~RZ~> All i did was download the file and put it in my server config like the ~RZ~> howto said. ~RZ~> any ideas? i mean am i supposed to register some place? Yes. You can find registration instructions our buy-it page: http://www.sortmonster.com/MessageSniffer/Buy-It.html Once you register we will assign you a license ID and our system will begin sending you an email each time your rulebase is updated. Best, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Few questions
hey guys.. when you talk about getting emails about the file being old.. well i have the file for a week now and did not get any kind of email about this. All i did was download the file and put it in my server config like the howto said. any ideas? i mean am i supposed to register some place? - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Marc Hilliker" <[EMAIL PROTECTED]> Sent: Wednesday, December 15, 2004 4:11 PM Subject: Re[2]: [sniffer] Few questions On Wednesday, December 15, 2004, 2:42:55 PM, Marc wrote: MH> Pete, MH> I downloaded the sniffer demo a couple of days ago and finally installed it MH> to run as an external test w/Declude today. I ran it all morning w/o any MH> problems. This afternoon, I downloaded a new version of the snf file: MH> http://www.sortmonster.com/MessageSniffer/Demo/snfrv2r3.snf and it appeared MH> to be a newer file (larger size and newer date/time stamp). But when I MH> swapped the old file with the new by renaming them I got a ERROR_RULE_AUTH MH> error in the log. Am I doing something wrong? I am a bit confused - the demo rulebase should have been the same name in both cases. At any rate, renaming a rulebase file breaks the encryption used to validate the contents. Each license ID and authentication code constitute a unique matched pair - neither work without the other, and the contents of the rulebase file are checked with a hash that is seeded with license ID and authentication string. The license ID for the demo is snfrv2r3. Please be sure you have the most recent software and rulebase from our try-it page. MH> This may be a moot point (at least for me) since our order was faxed to you MH> earlier today and I hope to be setting up an auto-update routine tomorrow. MH> But, I was curious as I'm trying to understand how sniffer works. That's great! I'll check on your order's progress soon. If you have further questions please let us know. Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re[2]: [sniffer] Few questions
On Wednesday, December 15, 2004, 2:42:55 PM, Marc wrote: MH> Pete, MH> I downloaded the sniffer demo a couple of days ago and finally installed it MH> to run as an external test w/Declude today. I ran it all morning w/o any MH> problems. This afternoon, I downloaded a new version of the snf file: MH> http://www.sortmonster.com/MessageSniffer/Demo/snfrv2r3.snf and it appeared MH> to be a newer file (larger size and newer date/time stamp). But when I MH> swapped the old file with the new by renaming them I got a ERROR_RULE_AUTH MH> error in the log. Am I doing something wrong? I am a bit confused - the demo rulebase should have been the same name in both cases. At any rate, renaming a rulebase file breaks the encryption used to validate the contents. Each license ID and authentication code constitute a unique matched pair - neither work without the other, and the contents of the rulebase file are checked with a hash that is seeded with license ID and authentication string. The license ID for the demo is snfrv2r3. Please be sure you have the most recent software and rulebase from our try-it page. MH> This may be a moot point (at least for me) since our order was faxed to you MH> earlier today and I hope to be setting up an auto-update routine tomorrow. MH> But, I was curious as I'm trying to understand how sniffer works. That's great! I'll check on your order's progress soon. If you have further questions please let us know. Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Few questions
Pete, I downloaded the sniffer demo a couple of days ago and finally installed it to run as an external test w/Declude today. I ran it all morning w/o any problems. This afternoon, I downloaded a new version of the snf file: http://www.sortmonster.com/MessageSniffer/Demo/snfrv2r3.snf and it appeared to be a newer file (larger size and newer date/time stamp). But when I swapped the old file with the new by renaming them I got a ERROR_RULE_AUTH error in the log. Am I doing something wrong? This may be a moot point (at least for me) since our order was faxed to you earlier today and I hope to be setting up an auto-update routine tomorrow. But, I was curious as I'm trying to understand how sniffer works. Thank you, --- Marc This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
