[sniffer] Message Sniffer question
If I remember correctly. I have an email account with a Imail program alias, that when it gets a mail from Message Sniffer triggers an update. It's still getting mail and triggering updates. I'm thinking this isn't need with Sniffer v3 anymore? Scott Fisher | IT Director FARM PROGRESS COMPANIES | 255 38th Avenue, Suite P | St. Charles, IL 60174-5410 630/462-2323 | Fax 630/462-2957 | mailto:sfis...@farmprogress.com sfis...@farmprogress.com http://www.farmprogress.com/ www.FarmProgress.com This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. Although Farm Progress Companies has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments. image001.gif
Re: [sniffer]A design question - how many DNS based tests?
I use about 100 dnsbl/rbl/rhsbl list of varying weights and reliabilities. How many matter... I'd have to say the shining star is CBL. Hits 45% of the spam with a very low false positive rate. The relay RBLs days are way behind them, The proxy RBLs most useful days are behind them The DUL RBLs I don't think have ever been comprehensive/correct enough to be as useful as they should be in the day of the spam zombie. The spam source RBL's (other than CBL) are a little over-zealous to me causing me some false positives problems, thus lower than weight. They seem to be on the downtrend too. Oddly Fiveten Spam (127.0.0.2) has had a big jump in the last two months catching 60% of the spam although with a 1 % false positive rate. I have 2 1/4 years of my spam test results posted at All tests: http://it.farmprogress.com/declude/Testsbymonth.html Spam tests: http://it.farmprogress.com/declude/spamtestbymonth.html ham tests: http://it.farmprogress.com/declude/hamtestsbymonth.html - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Message Sniffer Community sniffer@sortmonster.com Sent: Tuesday, June 06, 2006 8:26 AM Subject: [sniffer]A design question - how many DNS based tests? Hello Sniffer Folks, I have a design question for you... How many DNS based tests do you use in your filter system? How many of them really matter? Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
Re: [sniffer] Sniffer, MDLP, and invURIBL?
the %WEIGHT% passes the current message weight from Declude to INVURIBL. Used with SKIPWEIGHT option in invuribl.exe.config the %REMOTEIP% passes the sender's IP from Declude to INVURIBL. Used to whitelist IPs in senderipwhitelist.txt invuribl will find false positives, but is a very effective test. The INVURIBL weighting is determined with your setting in invuribl.exe.config I personally use multi.surbl.org and multi.uribl.com Name servers checked against sbl.spamhaus.org URI's "A" record checked agains sbl.spamhaus.org, cn-kr.blackholes.us and russia.blackholes.us - Original Message - From: Joe Wolf To: sniffer@SortMonster.com Sent: Saturday, February 25, 2006 11:05 AM Subject: [sniffer] Sniffer, MDLP, and invURIBL? I'm currently running Sniffer via Declude and use MDLP. Great! Since all the talk about invURIBL on the Imail list I thought I'd give it a try. The only problem I have is that it doesn't seem to be compatible with MDLP. invURIBL assigns its own weight to each message. The global.cfg line is as follows: INV-URIBL external weight "X:\INVURIBL\INVURIBL.exe %WEIGHT% %REMOTEIP%" 0 0 I'm not an expert but the %WEIGHT% must pass the weight determined by invURIBL to Declude. I don't know what the variables of the weighting system are. I'm worried that I may start getting a bunch of false positives since MDLP can't manage the weighting of invURIBL. Would appreciate any advice from anyone that knows more about this than I do! Thanks, Joe
Re: Re[2]: [sniffer] False Positive - no reaction?
I like this idea more than the email notification. I really don't need more emails. - Original Message - From: Andy Schmidt [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Tuesday, February 21, 2006 10:16 AM Subject: RE: Re[2]: [sniffer] False Positive - no reaction? Hi Pete, I agree that the email notification is tricky - because you might respond to spam - and, you may NOT respond to someone who did not use an authorized address. On the other hand, if I KNEW there was an auto-response and I did NOT get a response, it would be an indication to me, the user, that I must have done something wrong. So - in a sense - no response is also a message I can act on. The only other suggestion I have is to create a 24 hour 'queue' display on the web site. All you need to show is a column of the sender domain names of the email (not the entire sender email address). If I submit a false positive I can confirm that it made it into your queue by checking the web page. This way, you don't need to send automated emails. Best Regards Andy Schmidt Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, February 21, 2006 11:04 AM To: Andy Schmidt Subject: Re[2]: [sniffer] False Positive - no reaction? On Tuesday, February 21, 2006, 10:16:11 AM, Andy wrote: AS Sorry - didn't mean to be pushy. I just thought that false AS positives are worse than missed spam, so I had assumed that they AS would always be at the top of the queue. It is a very tough balancing act. Don't feel bad at all - you're not being pushy. The current goal is to respond in less than 24 hours and if possible to review twice per day. Yesterday a number of urgent tasks toppled that schedule. The first review happened (at around 0600) but there were no FPs at that time. I'm working to increase the review cycle... there are just a lot of things going on right now. Just so everyone knows, we do hear - loud and clear - that responding to FPs is important, and we have been much better about it over the recent past. I expect that service aspect to improve moving forward along with other things. AS I can wait (PS - would have calmed my nerves, if there had been some AS automatic ticket number response that reassured me that my email AS was received. The web site makes it sound as if there's a million AS reasons why a false positive might not be accepted - so an automatic AS confirmation might be a good self-service tool. That's a good point. I'll look at that possibility when I rewrite the false processing bot. We're getting a lot of spam lately at our false@ address and I would want to make sure that there was no outscatter. I can tell the bot to only respond to validated senders, but then there is the issue of email reliability in the response... what if you don't get the response I mean. ... There are still folks that occasionally (some frequently) send false reports from unauthorized addresses --- those would not get a response... I'm overthinking this now %^b When I get to the false processing bot I will add a response mechanism. Thanks! _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Rash of false positives
I don't know if I would call it a rash, but over the last week, I've submitted about 30 false positives. That's far more than average. I've developed a feeling that Message Sniffer has become "too tight". - Original Message - From: Darin Cox To: sniffer@SortMonster.com Sent: Tuesday, November 08, 2005 8:54 AM Subject: Re: [sniffer] Rash of false positives We're seeing a continual stream of false positives. It's taking all of our time just to keep up with it at the moment. If something isn't done soon, we're going to have to disable sniffer. Darin. - Original Message - From: Computer House Support To: sniffer@SortMonster.com Sent: Tuesday, November 08, 2005 9:34 AM Subject: Re: [sniffer] Rash of false positives Dear Darin, Thanks for the heads up. It's going to take me about 45 minutes to check the 9000 messages that were blocked by Sniffer last night, but I'll let you know if we experienced the same thing. Michael SteinComputer House www.computerhouse.com - Original Message - From: Darin Cox To: sniffer@SortMonster.com Sent: Tuesday, November 08, 2005 8:45 AM Subject: [sniffer] Rash of false positives Hi Pete, What's going on over there? We had somewhere between 5 and 10 times the usual number of Sniffer false positives this morning. They are across the board, so it's not just one rule that's catching them, or a particular set of senders or receivers. Hopefully you can get it under control soon. It would also be extremely helpful if you could speed up the false positive processing. Lately it seems to take 2-4 days for the rules to be adjusted, which usually means more of the same are caught and submitted over that time. I believe speeding up that process would result in fewer to process all around. Thanks, Darin.
Re: [sniffer] Version 3.05.10
web site has 3.0.5.11 - Original Message - From: Robert Grosshandler [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Friday, October 21, 2005 3:00 PM Subject: [sniffer] Version 3.05.10 Are there release notes for this somewhere? What changed between .09 and .10? Do I need to worry? Inquiring minds want to know. Rob --- [This E-mail scanned for viruses by Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Large amounts of spam still getting through
I just assumed it was a defective spamming software. - Original Message - From: John T (Lists) [EMAIL PROTECTED] To: sniffer@SortMonster.com Sent: Saturday, October 15, 2005 2:10 PM Subject: RE: Re[2]: [sniffer] Large amounts of spam still getting through I wonder is that is some kind Outlook vulnerability. John T eServices For You -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robert Grosshandler Sent: Saturday, October 15, 2005 10:43 AM To: sniffer@SortMonster.com Subject: RE: Re[2]: [sniffer] Large amounts of spam still getting through We're seeing the header info in the body problem. It seems to be always spam. Another way it manifests itself is that Declude can't alter the Subject line properly. The folks at Declude tell us that they're aware of it, and that they are just waiting for more pre altered by Declude examples to code for it. Rob M. Stein wrote: By the way, has anyone seen the spam that gets through that has the header info in the body of the mail message instead of where it's supposed to be? How is that possible? --- [This E-mail scanned for viruses by Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] auto update tmp files
I dump them once a day. - Original Message - From: Bonno Bloksma To: sniffer@SortMonster.com Sent: Monday, September 19, 2005 8:21 AM Subject: [sniffer] auto update tmp files Hi, Ok, I had auto update pretty much in the air. Seems all I needed was a program alias that fired the script. ;-) There's just one thing, I end up with alot of "tmpID.tmp" files in my spool directory. Any way of deleting those automagically? I could simply delete all tmp.tmp files in my midnight run. Would that be a problem? The only program alias I have is the sniffer update. Met vriendelijke groet, Bonno Bloksma hoofd systeembeheer tio hogeschool toerisme en hospitality julianalaan 9 / 7553 ab hengelo t 074 255 06 10 / f 074 255 06 16 [EMAIL PROTECTED] / www.tio.nl
Re: [sniffer] Spam blocks loading me up with spam
I'm also taking out the: 200.49.32.xxx to 200.49.47.xxx addresses with my IPFILE. Most of them were taken out in Feb with SBL 17983. The trouble on this spammer for me, is they aren't listed anywhere (with the 299.49.50.XXXs and are probably burning through domain names faster than the SURBLs can really be effective. So unless I get an SURBL hit or a Sniffer hit they are leaking through. Hopefully with Pete's new rules, this will be stopped. 200.49.32.0/24200.49.32.0/24moved 06-15-05SBL17983200.49.33.0/24200.49.33.0/24starsoftmails.comadded 02-17-05SBL17983200.49.34.0/24200.49.34.0/24moved 06-15-05SBL17983200.49.35.0/24200.49.35.0/24moved 06-15-05SBL17983200.49.36.0/24200.49.36.0/24moved 06-15-05SBL17983200.49.37.0/24200.49.37.0/24afdtc.comadded 02-17-05SBL17983200.49.38.0/24200.49.38.0/24afdtc.comadded 02-17-05SBL17983200.49.39.0/24200.49.39.0/24afdaa.comadded 02-17-05SBL17983200.49.40.0/24200.49.40.0/24moved 06-15-05SBL17983200.49.41.0/24200.49.41.0/24moved 06-15-05SBL17983200.49.42.0/24200.49.42.0/24moved 06-15-05SBL17983200.49.43.0/24200.49.43.0/24awwsc.comadded 02-17-05SBL17983200.49.44.0/24200.49.44.0/24arvvv.commoved 05-29-05SBL17983200.49.45.0/24200.49.45.0/24starofferzone.comadded 02-17-05SBL17983200.49.46.0/24200.49.46.0/24fdcmm.comadded 02-17-05SBL17983200.49.47.0/24200.49.47.0/24bicsc.comadded 02-17-05SBL17983 - Original Message - From: Darrell ([EMAIL PROTECTED]) To: sniffer@SortMonster.com Sent: Thursday, June 16, 2005 6:44 PM Subject: Re: [sniffer] Spam blocks loading me up with spam Scott, Not to many incoming for me - about 200 out of about 125K messages. One thing to note is the ones I am getting are around that block but even lower like 200.49.44.x. Darrell ---Check out http://www.invariantsystems.com for utilities for Declude And Imail. IMail Queue Monitoring, Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. - Original Message - From: Scott Fisher To: sniffer@SortMonster.com Sent: Thursday, June 16, 2005 6:04 PM Subject: [sniffer] Spam blocks loading me up with spam Am I the only one getting blasted by these spam from these IP blocks? Sniffer seems a little behind on catching these. 200.49.48.0/24200.49.48.0/24 200.49.49.0/24200.49.49.0/24mowz2.com200.49.50.0/24200.49.50.0/24qckcstmr.com 200.49.51.0/24200.49.51.0/24srvdupfrsh.com200.49.52.0/24200.49.52.0/24aahtv.com200.49.53.0/24200.49.53.0/24aakai.com 200.49.54.0/24200.49.54.0/24aakib.com200.49.55.0/24200.49.55.0/24aakli.com200.49.56.0/24200.49.56.0/24aafix.com200.49.57.0/24200.49.57.0/24e.com 200.49.58.0/24200.49.58.0/24200.49.59.0/24200.49.59.0/24 Domain names andlinks seem to be five chars beginning with aa. Theyalsoseem to be progressing through theIP blocks. i think they started in on the June 15th and have been spamming pretty consistantly.
[sniffer] Spam Submissions - same spam
A question: If I have the same spam sent to multiple recipients, should I be submitting more than one copy to [EMAIL PROTECTED]?
Re: [sniffer] My issues with the General category, looking forabettersolution
I would tend to agree with you that these are false positives. Eweek, Infoworld, Birthday Express, Best Buy, Chadwicks cause regular spam tagging here. If it is a company I've heard of and the links and such point back to that company, I usually give it the benefit of the doubt. - Original Message - From: Matt To: [EMAIL PROTECTED] Sent: Friday, December 17, 2004 6:35 AM Subject: Re: [sniffer] My issues with the General category, looking forabettersolution Greg,Yes, I should have inserted a "probably" or otherwise taken more care with my words. I didn't mean for my reply to be contentious.Anyway, here's a sample of what I am talking about. I've isolated most major bulk-mail providers from the rest of my Hold E-mail which constitutes about 2% of all blocked mail on my system. From midnight though 7 a.m. (7 hours) I had 49 messages held that were from these providers. Of those 49 messages, 42 were false positives, partly due to my own fault in weighting Big Foot Interactive to auto-hold, but of the 42, 26 were tagged by Sniffer-General. Those were from the following companies: Circuit City Sur La Table (a wine shop) eWEEK Daily Inbox Harry and David Things RememberedWhat I think is happening is that people buy something online at Circuit City, and then Circuit City automatically adds them to their E-mail list, and then someone that doesn't like the practice of default opt-in reports it to Sniffer and it is added to the General category. The same thing probably happened to most of the list above except for Daily Inbox which is not related to commerce. There is also a possibility of some harvesting or not honoring opt-outs, but the sample above is not nearly as bad or suggestive of such as most Sniffer-General hits.I have also found that SpamCop and SenderDB-Block have similar issues, often having what I personally consider to be false positives on first-party advertising such as the list above (at least one of the three paid a role in virtually all of the 42 false positives this morning). I get the feeling that SpamCop has either dirty spamtraps (old dead accounts or catch-alls used as spamtraps) or there are enough submissions of this stuff for them to tag these sources, and I have a feeling that Alligate which powers SenderDB has bayesian filtering that isn't friendly to advertising content or is triggered by other things like SpamCop, or shared IP's are causing the hits on some of it.Am I just one of a few that considers these things to be false positives? Do others just not really care if this stuff gets blocked? I'm not sure, but I don't want to keep reporting these things as FP's only to piss off the people that are reporting them, and I don't wish for the people that consider them to be spam to impact my system in the way that it is currently if I can help it, and I hope there is an easier way to approach this. Note that I expect no miracles, I just thought this was something that might be fruitful to discuss.MattSystem Administrator wrote: on 12/16/04 5:36 PM, Matt wrote: The reason why you aren't seeing these is because you aren't weighting Sniffer General at your subject tagging or hold weight, so it takes multiple hits for the false positives to show up on your system. Wow, I didn't realize you knew so much about my system. By the way, is 33 more or less than 30? I've always thought 33 (sniffer-general weight on my system) was more than 30 (subject tagging weight), but if you are telling me it is less, ... Greg This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: Re[2]: [sniffer] Version 2-3.1 Official Release
Does the cfg file need to be renamed with your license id also? - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] [EMAIL PROTECTED] Sent: Thursday, October 28, 2004 4:13 PM Subject: Re[2]: [sniffer] Version 2-3.1 Official Release On Thursday, October 28, 2004, 3:57:02 PM, andyb wrote: at Hi, at I have a support contract. Can you tell me what I need to do to at upgrade? Is there a step by step somewhere? To upgrade to the latest version of Message Sniffer: 1. Make a backup copy of your current sniffer executable. [licensid.exe] 2. Download the latest distribution from our Try-It page. 3. Rename the executable from the distribution [snfrv2r3.exe] to match your license ID [licensid.exe]. 4. Replace the old .exe with the new one. If you would like to use some of the new features that require the .cfg file, and you don't already have a .cfg file in the directory with your .exe then you can copy the .cfg file from the distribution and the modify it as needed. If you already have a .cfg file in use then you should use the new .cfg file from the distribution as a reference and create a new .cfg file that does what you want. If you don't have a .cfg file and you don't have a need for any of those features then you don't need to do anything - but you might want to copy the .cfg file just in case you want to use it later. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Today
I think Sniffer is finding the bulk of the new spam's here. For people like me that require two or more test hits to get to a hold weight, the other DNSBL tests are definitely lagging. - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Jorge Asch [EMAIL PROTECTED] Sent: Tuesday, September 21, 2004 7:58 PM Subject: Re: [sniffer] Today On Tuesday, September 21, 2004, 6:13:23 PM, Jorge wrote: JA Something happened today? I've had a couple of quiet days lately, today JA I've received over 40+ message of spam that went undetected by JA MessageSniffer... Spam storm - We've been pounding new rules for about two days. We're still at it. A good place to look is always here: http://www.sortmonster.com/MessageSniffer/Performance/ChangeRates.jsp Days Ago Adjustments --- 01118 11147 2777 3810 Today so far (not over yet) 1118 new rules. That's nearly one new rule per minute all day long! (53.23 secs/rule). Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Stock obfuscation question
Are there any rules in place to deal with this obfuscation? Sec. tion 2. 7, A o, f the Sec, urities A, ct of 19. 33 and Se.ction 2. 1B of the Se. curities Excha. nge A, ct of 19. 34. Scott Fisher Director of IT Farm Progress Companies This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[4]: [sniffer] Charset
-Mad, How set up is Message Sniffer to determine if an e-mail in a foreign language is spam and then code for it. I dutifully submit my Spanish spam to the spam at sortmonster.com address. It's a very, very small percentage of my overall spam, but it consistently lands in my battleground grey-weight ranges. I only ask, because I have seen the amount of non-English spam trending upwards. I've noticed spam here in Russian, German, Spanish, Korean, Portuguese and Chinese. - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: Michiel Prins [EMAIL PROTECTED] Sent: Friday, August 20, 2004 7:04 AM Subject: Re[4]: [sniffer] Charset On Friday, August 20, 2004, 2:35:35 AM, Michiel wrote: MP Pete, even your message had a chaset header: MP Content-Type: text/plain; charset=us-ascii Yes, a tricky gadget indeed. MP I think you'll generate more FP's if you do something like that than FN's MP you might have now. Aren't there spamassassin config files that detect this MP spam? Just to be clear - we're not precisely talking about spam per-se. Rather we're talking about stating that all traffic on a particular system should be only in one language as a matter of policy... The distinction is small I suppose, but in my mind important. In filtering spam we're usually trying to target only messages that are unsolicited commercial email, pornography, or somehow harmful... With this other approach instead of trying to defeat what we don't want, we are trying to only accept what we do want... Not so much putting up blocks, more like putting up a huge block and punching holes. There are some SA filters that do this kind of thing... Ultimately I think it boils down to filtering out anything with a charset that is not wanted. If we achieve this by attrition (rather than attempting to capture all of the charsets at once) then we will achieve a strong result quickly at a relatively low cost and we might avoid potential false positives that are out there. MHO, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Charset
We don't want any violent Mad Scientists! [EMAIL PROTECTED] 8/20 11:59a On Friday, August 20, 2004, 11:20:44 AM, Vivek wrote: VK On Aug 20, 2004, at 10:36 AM, Jorge Asch wrote: Well, since 100% of my users speak english/spanish I can safely bet that NONE of my mail should have strange character sets. So I can assume if they do, they must be spam. VK Be careful about that. I've gotten pure English email from folks in VK various parts of the world who's default character set was other than VK one I'd expect. Charset != Language. Along these lines, I saw spam today that was in english but used one of the character sets that were recently blocked by request (Only locally - no such thing will happen in the core system so nobody has to worry). I violently agree - blocking on character sets can be dangerous, so if you request these rules to be added be sure you watch for unexpected false positives afterward. ;-) _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: Re[2]: [sniffer] Charset
I'll chime in on the subject too. I've finally managed to get the spam in Chinese under control on my system, but for a while I really wished Message Sniffer has language based filters. I.e. Result 40 Chinese Result 41 Cyrillic Result 42 Spanish Result 43 Germain We could then turn on or off the languages we didn't want. From my foray with dealing with Chinese, it certainly much easier said than done. Chinese was doable, I've had no luck stopping my Spanish spam. Then again, you might be better at it than I. [EMAIL PROTECTED] 8/19 9:52a On Thursday, August 19, 2004, 10:11:45 AM, Jorge wrote: JA Michiel Prins wrote: Can't you use the content filter of your mail server to detect if the charset is used? JA I've tried, but it's not 100% effective I recall the earlier conversations about this. We have not had a lot of call for generally blocking foreign character sets so that project has not received much attention. Another issue with this is that many of our customers are not in the US and so defining foreign is often problematic. We can more easily establish local black rules for you. When you have an example of a character set you would like to block, please send us a note to support@ with your license ID in the subject line and the words Local black rule please Explain in your note that you want us to block the character set(s) in the message. Attach the message to your note. We will verify your license ID and then create local black rules for the character sets we find in the message. Over a short time this should have the effect you are looking for. Hope this helps, _M PS: We do filter foreign spam that is submitted to us at spam@ using the same rules that we follow for other messages. That is, we don't treat them as foreign - only as spam in general. Russian spam in particular has rapidly become heavily obfuscated - though there are usually patterns that can be found to block the messages. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Declude configuration
Here's what I use: My subject tag weight is 100 points. So in general sniffer weighs in at 90%. The greymail (60) needs some taming. I run a greymail whitelist to credit back 42 points to those that I don't consider spam. That's the only code you really need to be wary of. You may tend to get a few more false positives in the experimental (62) category, so if you wanted to weight that one less. On the other hand it is likely to catch the newest spam. I find having return code 0 helps. When I encounter a spam that has the sniffer-notfound in the headers, it gets forwarded to sortmonster. SNIFFER-NOTFOUNDexternal 000 D:\IMail\Declude\Sniffer\sniffer2.exe code 0 0 SNIFFER-TRAVEL external 047 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-INSURANCE external 048 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-AV-PUSH external 049 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-WAREZ external 050 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-SPAMWAREexternal 051 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-SNAKEOILexternal 052 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-SCAMS external 053 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-PORNexternal 054 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-MALWARE external 055 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-ADVERTISING external 056 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-SCHEMES external 057 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-CREDIT external 058 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-GAMBLINGexternal 059 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-GREYMAILexternal 060 D:\IMail\Declude\Sniffer\sniffer2.exe code 42 0 SNIFFER-OBFUSCATION external 061 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-EXPERIMENTALexternal 062 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 SNIFFER-GENERAL external 063 D:\IMail\Declude\Sniffer\sniffer2.exe code 90 0 Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 06/14/04 11:14AM I am new to Sniffer, and have it up and running with the basic line looking for a nonzero return code. I would now like to start setting different weights for different return codes. Does some one have a example configuration I can use? John Tolmachoff Engineer/Consultant/Owner eServices For You This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] automate sniffer updates
I'd like to automate the sniffer updates. I currently run scheduled updates, but if I could update upon e-mail notification, my spam window would be closed a little bit more. Has anyone had any luck doing these? Scott Fisher Director of IT Farm Progress Companies This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Possible blip?
2 thoughts from me: 1. Right on on the Nigerian scams, possible keeping these rules longer. As I was forwarding out a Nigerian scam to the spam mailbox, I too wondered how long the Nigerian rules were kept in play. I might also add Nigeria's twin sister the International Lottery spam and Stock Spams might also be kept longer. I noticed an increase in the Stock spams this week. 2. I've been tracking different character sets for a couple of weeks, the Chinese, Cyrillic and Korean look promising. I get false hits on Greek, Thai, and Vietnamese Headers. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 12:42PM Pete, Our Hold range has returned to more normal territory on Thursday. Here's the stats from the week as a whole on what has been very consistent traffic. Out of all E-mail processed, both good and bad, the %Hold represents what scored between 10-24 points on our system and needed review, the %Sniffer represents all Sniffer hits except for Gray, the %Spam is what we scanned and didn't deliver (generally about 99.8% of spam is caught at a score of 10 which this is based on), and the Sniffer/Spam is the percentage of Sniffer hits as a portion of messages scoring 10 or more. Day %Hold%Sniffer%SpamSniffer/Spam Mon: 1.86% 77.27% 80.37% 96.14% Tue: 2.83% 74.53% 79.37% 93.39% Wed: 2.13% 77.60% 79.66% 97.41% Thur:1.95% 76.50% 80.66% 94.84% The only change that we made to our system was to add two smaller domains later in the week, and we introduced filters for Cyrillic and Chinese languages on Wednesday morning which have cut our hold file down by 0.38 percentage points on Thursday, which explains how our %Hold is lower on than on Wednesday with a lower Sniffer hit rate on spam. I did note two high volume untagged static spammers on Tuesday that we blacklisted locally, and that combined with the increase in Sniffer change rates (spam storm) might account for the changes that I saw. I am wondering though about the recommendations that you have made for possibly fine tuning our rule base. Again though, please keep in mind that I still feel that performance is overall very, very good. One of my thoughts regarding minimum rule strengths and grace periods is that all groups aren't necessarily the same. For instance Nigerian scams are low volume and sporadic, and my system performs the worst on these things. Maybe lower rule strengths and longer grace periods makes much more sense for the Phishing category than it does for many other categories for instance. Is that possible? I also looked up the rule strengths on your site and found that about 50%, or maybe more, have a strength below 1, and maybe lowering that is worth testing out so long as I don't massively increase the number of records. I do think though that I would like to test out extending the grace period. Most of my false positives are not on things that this would affect, and that might give niche sources a little extra coverage if I understand things correctly. I'll follow your directions and contact you directly regarding any affirmative changes, but I thought it might be beneficial to keep this discussion public since some other stats hounds might find this information to be of use :) If you can glean anything from the numbers that I gave you, please add your thoughts. Thanks, Matt Pete McNeil wrote: At 05:00 PM 5/19/2004, you wrote: snip/ I haven't yet upgraded to the most recent release, I'm still on the prior beta. I'll probably do that this evening. I tend to wait on upgrades until there has been enough time for bugs to surface unless I am already looking for a fix. I'm sure that the extra verification of the rulebase will help prevent the potential of problems, and I guess this has the possibility of being caused by a bit of corrupted data, though that's probably reaching. There were no substantive changes from the beta to the production version. Largely just a removal of monitoring code. Again, regardless if there was a blip, Sniffer still does a wonderful job of tagging lots and lots of E-mail, just not quite as much as the day before. Last night I was able to adjust the rule strength analysis window back to it's original settings. About 5 days of data were lost - but those days will be recovered quickly. Please let me know if this adjustment improved your conditions. I've noted that on a number of other lists there seem to be posts about a sudden increase in spam over the past few days. We are definitely seeing this also - approximately a 25% or more increase in new rule additions in the past 4 days: http://www.sortmonster.com/MessageSniffer/Performance/ChangeRates.jsp Specifically note from about 4 days ago... Days Ago Adjustments --- 0356 1508 2391
Re: [sniffer] Possible blip?
Interesting. Are you searching for 2 character pairs with GB2312? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 01:46PM Scott, Regarding my Cyrillic and Chinese filters, I did a review of a full week's held spam, looking for foreign languages and patterns to tag. I found from other research that the primary Chinese characterset, GB2312, contains the Western Latin characterset, and so someone could send an E-mail with this characterset defined and still have English as the message. Because of this I do more than just look for the offending characterset, I've built a combo filter that looks for both high bit characters such as ¥ as well as body or header hits for encoding of GB2312 (Chinese/Korean) or Windows-1251 (Cyrillic). I also have Declude END statements for appearances of US-ASCII and ISO-8859-1, so messages like this one that are referencing such patterns won't trip the filter. It seems to be stopping about 80% to 90% of the stuff, but I'm guessing that the stuff that is getting through didn't hit one of the high bit characters in my filter and I might need to simply expand my list a bit. Unfortunately I have no idea what characters are most common, so I'm just eyeballing it from sources. I had one false positive on a Yahoo Groups posting that referenced 163.com, a Chinese free Web mail provider that inserts Chinese language footers. The message was in English, but encoded in GB2312 and didn't indicate any sign of English besides the actual text. Because of this, I might throw in an exception for the word the (followed by a space) just as a test to see if text in English is present, but I have to review that. This message was also BASE64 encoded and that might be an appropriate exception??? The last pattern that I might look at is using the new MailPolice test for identifying Web-mail providers, and excepting them from the filter because they have issues with encoding languages I've found. Hope this helps. Matt Scott Fisher wrote: 2 thoughts from me: 1. Right on on the Nigerian scams, possible keeping these rules longer. As I was forwarding out a Nigerian scam to the spam mailbox, I too wondered how long the Nigerian rules were kept in play. I might also add Nigeria's twin sister the International Lottery spam and Stock Spams might also be kept longer. I noticed an increase in the Stock spams this week. 2. I've been tracking different character sets for a couple of weeks, the Chinese, Cyrillic and Korean look promising. I get false hits on Greek, Thai, and Vietnamese Headers. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 12:42PM Pete, Our Hold range has returned to more normal territory on Thursday. Here's the stats from the week as a whole on what has been very consistent traffic. Out of all E-mail processed, both good and bad, the %Hold represents what scored between 10-24 points on our system and needed review, the %Sniffer represents all Sniffer hits except for Gray, the %Spam is what we scanned and didn't deliver (generally about 99.8% of spam is caught at a score of 10 which this is based on), and the Sniffer/Spam is the percentage of Sniffer hits as a portion of messages scoring 10 or more. Day %Hold%Sniffer%SpamSniffer/Spam Mon: 1.86% 77.27% 80.37% 96.14% Tue: 2.83% 74.53% 79.37% 93.39% Wed: 2.13% 77.60% 79.66% 97.41% Thur:1.95% 76.50% 80.66% 94.84% The only change that we made to our system was to add two smaller domains later in the week, and we introduced filters for Cyrillic and Chinese languages on Wednesday morning which have cut our hold file down by 0.38 percentage points on Thursday, which explains how our %Hold is lower on than on Wednesday with a lower Sniffer hit rate on spam. I did note two high volume untagged static spammers on Tuesday that we blacklisted locally, and that combined with the increase in Sniffer change rates (spam storm) might account for the changes that I saw. I am wondering though about the recommendations that you have made for possibly fine tuning our rule base. Again though, please keep in mind that I still feel that performance is overall very, very good. One of my thoughts regarding minimum rule strengths and grace periods is that all groups aren't necessarily the same. For instance Nigerian scams are low volume and sporadic, and my system performs the worst on these things. Maybe lower rule strengths and longer grace periods makes much more sense for the Phishing category than it does for many other categories for instance. Is that possible? I also looked up the rule strengths on your site and found that about 50%, or maybe more, have a strength below 1, and maybe lowering that is worth testing out so long as I don't massively increase the number of records. I do think though
Re: [sniffer] OT: Language filtering in Declude, wasPossibleblip?
Wouldn't it be better to reverse the order? Run the subject and header tests on the majority of the mail. Then run the body with a TESTSFAILED END NOTCONTAINS CHINESE. You should end up with less body searches this way. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 04:28PM I think you might have possibly identified the group of required characters. I'll give that a try. I'm not sure if any Cyrillic stuff has been passing through but this bears watching as well and I might have to change my list there as well. I am also tagging BIG5, however almost all spam comes in GB2312. Here's what I'm searching for in the CHINESE filter: # CHINESE v1.0.0 SKIPIFWEIGHT25 MAXWEIGHT10 TESTSFAILEDENDNOTCONTAINSHIGHBIT SUBJECTENDCONTAINScharset=gb2312 SUBJECTENDCONTAINScharset=gb2312 SUBJECTENDCONTAINScharset=big5 SUBJECTENDCONTAINScharset=big5 HEADERS10CONTAINS=?gb2312?b? HEADERS10CONTAINS=?big5?b? HEADERS10CONTAINScharset=gb2312 HEADERS10CONTAINScharset=gb2312 HEADERS10CONTAINScharset=big5 HEADERS10CONTAINScharset=big5 BODY10CONTAINScharset=gb2312 BODY10CONTAINScharset=3dgb2312 BODY10CONTAINScharset=big5 BODY10CONTAINScharset=3dbig5 BODY10CONTAINScontent=zh-cn BODY10CONTAINScontent=3dzh-cn The END statements for the subject are meant as a precaution, although it's probably not necessary with the HIGHBIT filter ending on US-ASCII and ISO-8859-1 (plus a language definition hit for 'content=en-us'). I do believe that you can apply a similar technique to spam in Spanish, but since the characterset is the same as English, you would be searching for those 'content=' markers in combination with special characters (a short list in this case). We hardly see any Spanish spam, or at least held Spanish spam so I'm doing nothing about it. Spanish is of course a lot more common in US E-mail. It may be that some Spanish spam isn't identified as Spanish since that's not necessary for proper display in most E-mail clients, but I have seen no proof of that. Matt Scott Fisher wrote: Interesting. I generally just punish people if GB2312 ?BIG5 or such are in the headers. This is overwhelmingly SPAM, but like you siad there are English in some of those messages. It looks like the GB2312 Chinese characters will have A B0 to F7 as it's highbyte. and an A0 to FF as it's lowbyte. If the GB2312 Chinese is present, I would think most every character should be one of these: °±²³ µ¶· *º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷ Checking some of my e-mails confirms that. The bad news is that requires another body filter. It's too bad there wasn't a BODY256 filter type where only the first 256 bytes would be checked. That would certainly be enough to score up these, and wouldn't be a CPU hog. I'm not certain that I'd want to throw another body filter at my few Chinese spams. How often do you get a body indication of GB2312 / Cyrillic charactersets with no header indication? It's an interesting subject because I those few Chinese spams that get through to three of my accounts frustrate me. Got any tips for Spanish spam? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 03:17PM No, just one, but it won't score unless there is a header or body indication of the GB2312 or Windows-1251 charactersets. I'm using a combo filter in Declude where the HIGHBIT filter is non-scoring, and the CHINESE and CYRILLIC filters contain a line that says: TESTSFAILED END NOTCONTAINS HIGHBIT I'm pretty sure that the CHINESE and CYRILLIC filters will always hit where appropriate unless the HIGHBIT test doesn't hit. I have about 65 different high bit characters in that filter presently, all copied from spam. If Scott was around, I would ask him how the NONENGLISH test is tripped because that might accomplish the same goals, however I'm not sure if it also scores the definition of a characterset, in which case it would have false positives in this scenario. Matt Scott Fisher wrote: Interesting. Are you searching for 2 character pairs with GB2312? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 01:46PM Scott, Regarding my Cyrillic and Chinese filters, I did a review of a full week's held spam, looking for foreign languages and patterns to tag. I found from other research that the primary Chinese characterset, GB2312, contains the Western Latin characterset, and so someone could send an E-mail with this characterset defined and still have English as the message. Because
