Re: [sniffer] New Rulebot F001
Lowest result code wins with Sniffer, 63 is the highest score currently, and these rules are going in a place where formerly they were only IP's,so you shouldn't need to adjust anything. I would imagine that refinement should improve accuracy in the IP rules, though I don't believe that it will be near perfect. I do however want to voice my general and ongoing concern about automation and extracting IP's from spamtraps. This can be done, but one must be very careful to remove legitimate or compromised hosts, and most that don't bother to do so are even worse than SpamCop when it comes to listing ISP's and the like. For a good picture of whether a host is spammy, one should also look at all of the good traffic, and make sure that there is a huge sample of data to work with. Alternatively, one should be checking for things such as the host being legitimate (does it answer with a name that matches the reverse DNS or HELO that it gave you, does it have "mail" in the name, etc.). Also, it makes sense to have different qualification mechanisms for zombie spam and static spammers since their heuristics are quite different and can be targeted more effectively and more accurately with mechanisms built to their patterns. I do fear that automation of this sort, unless it is done in a very reserved manner (throwing out what can't be almost absolutely confirmed), will result in foreign hosts being caught, and large ISP's/E-mail providers much in the same way as they have been. CBL takes the reserved approach and is therefore much, much more accurate than SpamCop, yet their results aren't that far off the last I checked. CBL primarily targets zombies with their methods, and they do this because it is much easier to find a sign of an illegitimate host (that also hit a spamtrap). Matt Jay Sudowski - Handy Networks LLC wrote: There's been at least one FP ;) -- Rule - 861038 NameF001 for Message 2888327: [216.239.56.131] Created 2006-03-02 Source 216.239.56.131 Hidden false Blocked false Origin Automated-SpamTrap TypeReceivedIP Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength2.08287379496965 False Reports 0 From Users 0 [FPR:B] The rule is below threshold, and/or badly or broadly coded so it will be removed from the core rulebase. My concern with automated IP rule coding is that we use Sniffer because it's extremely accurate. Coding rules linked to IPs, particularly IPs that are used by google or any large ISP to send large amounts of (mostly legitimate) email is contrary to what Sniffer is great at, which is tagging spam that no one else is. Is response code 63 going to be utilized for any other purposes? If not, I will let Declude know to weight these responses lower than normal Sniffer. - Jay -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Monday, March 06, 2006 3:00 PM To: sniffer@sortmonster.com Subject: [sniffer] New Rulebot F001 Hello Sniffer folks, The first of the new rulebots is coming online. Rulebot F001 creates IP rules for sources that consistently fail many tests while also reaching the cleanest of our spamtraps. The rules will appear in group 63. The bot is playing catchup a bit (since there have been few IP rules at all since we disabled the old bots). The algorithms used in this bot have been tested manually for 2 weeks with no false positives. Expect an increase in your rulebase size while F001 catches up with current spamtrap data. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] New rulebase compilers online.
Pete, Does this mean that you are somehow supporting incremental rule base updates, or is it that the compiler is just much faster so we will get the same number of updates, but generally get them 40-120 minutes earlier in relation to the data that generated them? Either way, definitely an improvement. The closer to real-time we can get, the better. Thanks, Matt Pete McNeil wrote: Hello Sniffer Folks, I have just completed work to upgrade the rulebase compiler bots. They are now significantly more efficient. As a result you will be seeing updates more frequently. Previous lag was between 40-120 minutes. Current lag (sustained) is < 5 minutes. More timely updates should equate to lower spam leakage for new spam. You do not need to take any action on this. This note is for your information only. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] New RuleBot F002 Online
Pete, In light of current and prolonged issues, this seems like a good and safe tactic. I would appreciate it however if maybe you could place the rules in another result code since this result code is not as accurate as some others are and some of us weight it lower than others. Thanks, Matt Pete McNeil wrote: Hello Sniffer Folks, Rulebot F002 has been placed online. This rulebot captures and creates geocities web links from the "chatty" campaigns. This is largely a time saver for us humans... we will focus our attention more on abstracts for these campaigns now that F002 will be capturing the raw links. Rules from F002 will produce a 60 result code (Ungrouped). The engine is following a standard protocol that we have used for months. I expect no false positives from this one. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) Chief Scientist (www.armresearch.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] New RuleBot F002 Online
Pete, I would definitely like to see rules classified for what they are based on instead of the content, but certainly I don't expect to see that without a major new release. Rules such as those based phrases, IP's, domains, patterns, and viruses all have different accuracies and issues. If you were also to group them in a similar way, we could tag multiple rules for a single message so that for instance a phrase and a domain both hit on the same message. My logs show that I average 3 matches for every final result. If this becomes a plan, I would proceed very carefully since doing it in a way that could cause a lot of cross-over pollution would make comboing such things for a higher score unwise. I would in fact recommend creating something like 4 groups; 1) IP's, 2) Domains, E-mail addresses & Links, 3) Patterns (like domain patterns and obfuscation), and 4) Content. There shouldn't be any crossover of FP's in such a thing, so multiple hits would be stronger. In relation to the placement of RuleBot F002 results, I would just favor pretty much anything but the 60 and 63 groups because they are scored lower due to FP's on my system, and it has generally been said by others that this is the case on theirs as well. F002 has the appearance of being hyper-accurate, and it would help if it was placed in a group with other hyper accurate results. Even placing it in 61 (Experimental) would be preferred over 60. Thanks, Matt Pete McNeil wrote: On Friday, March 10, 2006, 3:41:00 PM, Darin wrote: DC> Totally agree. I'd like to see some separation between rules created by DC> newer rulebots and preexisting rules. That way if there becomes an issue DC> with a bot, we can turn off one group quickly and easily. There is no way to do this without completely reorganizing the result codes or defeating the competitive ranking mechanisms. If you feel strongly about it I can move these rule groups to lower numbers on your local rulebase or make some other numbering scheme - but I don't recommend it. Moving these rule groups to lower numbers would cause them to win competitions with other rules where they would normally not win. At some point in the future we might renumber the rule groups again, but I like to avoid this since there are so many folks that just don't get the message (no matter what we do to publish it) when we make changes like this and so any large scale changes tend to cause confusion for very long periods. For example: I still, on occasion, have questions about the gray-hosting group which has not existed for quite a long time. So far there has not been one FP reported on bot F002 and extremely few on F001 - the vast majority of those associated with the very first group of listings prior to the last two upgrades for the bot. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Message loop
Pete, I tried replying to some FP reports and I received back some loop reports from your gateway: Failed to deliver to '[EMAIL PROTECTED]' mail loop: too many hops (too many 'Received:' header fields) Reporting-MTA: dns; server75.appriver.com Original-Recipient: rfc822;<[EMAIL PROTECTED]> Final-Recipient: system;<[EMAIL PROTECTED]> Action: failed Status: 5.0.0 Received: from [10.238.11.79] (HELO inbound.appriver.com) by server75.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 204450520 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:36:23 -0400 Received: by inbound.appriver.com (CommuniGate Pro PIPE 5.0.6) with PIPE id 40116463; Wed, 19 Apr 2006 16:36:22 -0400 Received: from [69.20.119.203] (HELO server70.appriver.com) by inbound.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 40116469 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:36:18 -0400 Received: by server70.appriver.com (CommuniGate Pro PIPE 5.0.6) with PIPE id 20584892; Wed, 19 Apr 2006 16:31:39 -0400 Received: from [12.6.93.226] (HELO exchange01.apprivercorp.com) by server70.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 20584879 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:31:35 -0400 Received: from server75.appriver.com ([207.97.224.142]) by exchange01.apprivercorp.com with Microsoft SMTPSVC(6.0.3790.1830); Wed, 19 Apr 2006 15:32:57 -0500 Received: from [10.238.11.110] (HELO inbound.appriver.com) by server75.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 204446070; Wed, 19 Apr 2006 16:31:35 -0400 Received: by inbound.appriver.com (CommuniGate Pro PIPE 5.0.6) with PIPE id 43046685; Wed, 19 Apr 2006 16:31:33 -0400 Received: from [207.97.227.84] (HELO www03.appriver.com) by inbound.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 43046648; Wed, 19 Apr 2006 16:31:27 -0400 Received: from outbound.appriver.com [10.238.11.133] by www03.appriver.com with ESMTP (SMTPD32-8.15) id ADFB2787015E; Wed, 19 Apr 2006 16:30:51 -0400 Received: from [10.238.11.92] (HELO server87.appriver.com) by outbound.appriver.com (CommuniGate Pro SMTP 5.0.2) with SMTP id 75970524 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:30:43 -0400 Received: by server87.appriver.com (CommuniGate Pro PIPE 5.0.6) with PIPE id 153351626; Wed, 19 Apr 2006 16:30:43 -0400 Received: from [216.88.36.96] (HELO mnr1.microneil.com) by server87.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 153351606 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:30:34 -0400 Received: by mnr1.microneil.com (Postfix, from userid 93) id 578433F20C0; Wed, 19 Apr 2006 16:30:34 -0400 (EDT) X-SortMonster-MessageSniffer-Rules: snfrv2r3-v2-3.2 0-69638-850--m 0-113972-1237-3710-m 0-113972-1237-12801-m 0-113972-1237-23632-m 0-113972-1237-31312-m 0-69638-850--f X-SortMonster-MessageSniffer-Result: 0 Received: from SortMonster.com (unknown [216.88.36.181]) by mnr1.microneil.com (Postfix) with ESMTP id 0DFE03F20BE for <[EMAIL PROTECTED]>; Wed, 19 Apr 2006 16:30:34 -0400 (EDT) Received: from outbound.appriver.com [207.97.229.125] by SortMonster.com with ESMTP (SMTPD32-6.05) id ADE1CCC009A; Wed, 19 Apr 2006 16:30:25 -0400 Received: from [10.238.11.52] (HELO inbound.appriver.com) by outbound.appriver.com (CommuniGate Pro SMTP 5.0.2) with ESMTP id 75970346 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:30:24 -0400 Received: by inbound.appriver.com (CommuniGate Pro PIPE 5.0.6) with PIPE id 98719021; Wed, 19 Apr 2006 16:30:22 -0400 Received: from [66.109.52.12] (HELO mailpure.com) by inbound.appriver.com (CommuniGate Pro SMTP 5.0.6) with ESMTP id 98718972 for [EMAIL PROTECTED]; Wed, 19 Apr 2006 16:30:10 -0400 Received: from [192.168.0.100] [24.29.42.95] by mailpure.com with ESMTP (SMTPD32-8.15) id ADD0457300A2; Wed, 19 Apr 2006 16:30:08 -0400 Message-ID: <[EMAIL PROTECTED]> Date: Wed, 19 Apr 2006 16:31:16 -0400 From: Matthew Bramble <[EMAIL PROTECTED]> Organization: MailPure User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Sniffer FP Support <[EMAIL PROTECTED]> Subject: Re: hb064pkq - [EMAIL PROTECTED] References: <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> In-Reply-To: <[EMAIL PROTECTED]> Content-Type: multipart/alternative; boundary="040105010502030206010001" X-MailPure: X-MailPure: Spam Score: 0 X-MailPure: Scan Time: 19 Apr 2006 at 16:30:09 EST X-MailPure: Spool File: D9DD0457300A2541E.SMD X-MailPure: Server Name: X-MailPure: SMTP Sender: [EMAIL PROTECTED] X-MailPure: Received From: (Private IP) [192.168.0.100] X-MailPure: Country Chain: X-MailPure: X-MailPure: Spam and virus blocking services provided by MailPure.com X-MailPure: X-Policy: sortmonster.com X-Note: This Email was scanned by AppRiver SecureTide X-Note: Spam Tests Failed: X-Country-Path: PRIV
Re: [sniffer]A design question - how many DNS based tests?
I have 46 RBL's configured, though 16 are configured to score differently on last hop and prior hops. I would say that more than 35 of these are things that I would not like to lose. I weight most RBL's at around half of my Hold weight in Declude. False positives on my system typically hit about 5 different tests of various types before they get enough weight to be blocked. Sniffer is the test most often a part of false positives, being a contributing factor in about half of them. About 3/4 of all FP's (things that are blocked by my system) are some form of automated or bulk E-mail. That's not to say that other tests are more accurate; they are just scored more appropriately and tend to hit less often, but the FP issues with Sniffer have grown due to cross checking automated rules with other lists that I use, causing two hits on a single piece of data. For instance, if SURBL has an FP on a domain, it is possible that Sniffer will pick that up too based on an automated cross reference, and it doesn't take but one additional minor test to push something into Hold on my system. IMO, the more tests, the better. It's the best way to mitigate FP's. I don't look to Sniffer as anything more than a contributer to the overall score. Sniffer can't block a message going to my system on it's own due to it's weighting. I think it's more important to be accurate than to hit more volume, and handling false positive reports with Sniffer is cumbersome for both me and Sniffer. I would hope that any changes seek to increase accuracy above all else. Sniffer does a very good job of keeping up with spam, and it's main issues with leakage are caused by not being real-time, but that's ok with me. At the same time Sniffer is the test most often a part of false positives, being a contributing factor in about half of them. About 3/4 of all FP's (things that are blocked by my system) are some form of automated or bulk E-mail. That's not to say that other tests are more accurate; they are just scored more appropriately and tend to hit less often, but the FP issues with Sniffer have grown due to cross checking automated rules with other lists that I use, causing two hits on a single piece of data, and the growth of the Sniffer userbase which has become more likely to report first-party advertising as spam, either manually or through an automated submission mechanism. Matt Pete McNeil wrote: Hello Sniffer Folks, I have a design question for you... How many DNS based tests do you use in your filter system? How many of them really matter? Thanks! _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
Re: [sniffer]FP suggestions
d reverse DNS entries for bulk mailers so that I can treat them differently on my system. Some I do block by default, but others that send a majority of legitimate messages I balance in my system so that they don't fail technical tests (such as Declude's BADHEADERS, SPAMHEADERS and HELOBOGUS) since they are not appropriate for non-zombies, and at least two hits on things like Sniffer, SURBL and SpamCop are required before something gets blocked. The essence of the issue here is that while one mailing might be spam, it doesn't make sense to paint all of the provider's mailings as spam. Sniffer still hits a lot of what passes through my system, but they are not blocked nearly as often as before. I recall reporting places like roving.com, icebase.net, cheetahmail.com and other well known providers to Sniffer as false positives in the past. I recognize that Sniffer has a lot of clients that don't care if such things get blocked, and do care a lot if spam leaks through, and it is tough to target individual lists as opposed to the entire provider. Another subset of this is third-party tracking services that sometimes get tagged based on the fact that some of their clients do spam, and then there is the subset that advertises third-parties with direct links to their domains where those third-parties have spammed, i.e. Entertainment Books, Omaha Steaks, University of Phoenix Online, etc. You clearly pulled those domains from spam samples, but they cross-contaminate opt-in lists through advertising links. Based on past discussions, we may well differ on our opinions about how to deal with this, but it isn't workable to continually report such things if they continually get listed in other ways, and that's why I decided sometime ago to track these providers myself. Regarding those things that are submitted to Sniffer as spam that I don't consider to be spam, that's another very tough cookie to crack. If one customer reports E-mails from Harry & David to Sniffer as spam, and I report it as ham...who is right? I think that this comes down to having an official definition of spam and communicating it. Spamhaus has a definition that isn't workable in the real-world because it requires affirmative confirmation of being put on a bulk mail list by companies that you do business with, and as we know, virtually no one follows this so closely, yet many want these E-mails and everyone has the ability to unsubscribe. My definition of spam, like Spamhaus, is that it is both bulk and unsolicited, however we differ when it comes to defining unsolicited. I try to allow any first-party communications, advertising or otherwise, so long as they are directly related to the actions that created the relationship (i.e. no third-party offers), and so long as they honor unsubscriptions without jumping through hoops (like remembering obscure logins to stop messages). There are of course some grey areas around the edges such as lists that mix opt-ins with harvested/bought lists, but they are fairly rare and I can't suggest a pure way to deal with such things outside of hard work and manual review, though I would discourage against collection methods that can cause pollution such as automated submissions which for instance can report things like Harry & David because the admin blacklisted them locally, and then they show up as blocked spam that Sniffer didn't hit and are submitted. I think this may be a contributing factor. I would prefer that people manually report, and that they know the rules for what to report (i.e. the spam definition). I didn't intend to draw you into this discussion within another thread or at this time, but I do think that Sniffer would benefit from some more focus on the FP issues. I hope this helps and I am willing to lend some more ideas or opinions if you want to bounce some of your own off of me or the list. Thanks, Matt
Re: [sniffer]Re[2]: [sniffer]FP suggestions
Pete, An X-Header would be very, very nice to have. I understand the issues related to waiting to see if something comes through, and because of that, I would maybe suggest moving on your own. Sniffer doesn't need to be run on every single message in a Declude system. Through weight based skipping, many administrators (especially the ones that could make the most use of this) could skip processing Sniffer once a certain weight is reached, and in turn that would save enough load that it should easily make up for needing to re-write the message to the disk with the modified headers. On external tests that allow for weight skipping on my system, I was skipping around 50% of messages before lightening the load with pre-scanning. Sniffer could do weight skipping with Declude by accepting the %WEIGHT% variable in the command line. SNIFFER-IP external 063 "C:\IMail\Declude\Sniffer\customer-code.exe license-code WH=26 WL=-5 CW=%WEIGHT%" 5 0 ...etc. The WH setting says don't run if equal to or greater than, the WL says don't run if equal to or less than, and the CW passes in the weight from Declude at the time of calling Sniffer. It still launches Sniffer, but it could be stopped immediately before any heavy lifting is done. The best solution of course would be for Declude to allow for weight-based skipping in the config without calling the executable, but I started asking about that back in the Scott days and I am not holding out hope for that happening soon considering. The most realistic option would seem to then have Sniffer do the heavy lifting of rewriting itself, and save some CPU and disk I/O by improving efficiencies with something as simple as weight-based skipping. I'm pretty sure the net result would be less CPU and disk I/O overall if both were done. Another alternative may be to create a separate executable (with weight-based skipping) that would only deal with adding headers from the text file that Sniffer drops in the directory. There would be less benefit overall to keeping this all in one app, but it would target the primary need. This could easily be written by one of us in _vbscript_ as a proof of concept. I have considered doing this before, but it isn't at the top of my priorities. BTW, you could maybe even encode links in the headers for FP reporting through a Web interface, completely removing the forwarding mechanism from the mix, though you wouldn't have the opportunity to see the messages which may not be good as a whole. Matt Pete McNeil wrote: Hello Scott, Wednesday, June 7, 2006, 10:08:58 AM, you wrote: For me the pain of false positives submissions is the research that happens when I get a "no rule found" return. I then need to find the queue-id of the original message and then find the appropriate Sniffer log and pull out the log lines from there and then submit it. Almost always in these cases, a rule is removed. If this process could be improved that would really be a time saver. This depends on the email system you are using. On some systems (MDaemon, and postfix, for example) X- headers from SNF can be emitted into the message. When we see these we can identify the rules directly without asking for the extra research. It would be nice if Declude would offer a mechanism to pick up the optional .xhdr file SNF can generate and include it in the X headers that it already adds to the message. I know this begs the question, why not have SNF add the headers for SmarterMail and IMail platforms, and the reason is that it would require writing an additional copy of the message to disk. Since these systems tend to be io bound already (Declude/IMail anyhow) the performance penalty would be prohibitive. If Declude picks up .xhdr from SNF directly then it can be included in the ONE rewrite Declude makes anyway. I've asked them about this and other improved integration opportunities for a while now (many months), and I get favorable responses, but no action so far. I guess we will see :-) _M
Re: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Pete, Since the %WEIGHT% variable is added by Declude, it might make sense to have a qualifier instead of making the values space delimited. Errors in Declude could cause values to not be inserted, and not everyone will want to skip at a low weight. I haven't seen any bugs with %WEIGHT% since shortly after it was introduced, but you never know. I have seen some issues with other Declude inserted variables though. One other thing that I came across with the way that Declude calls external apps...you can't delimit the data with things like quotes. There is no mechanism for escaping a functional quote from a quote that should appear in the data that you pass to it...so don't use quotes as delimiters :) Matt Pete McNeil wrote: Hello Matt, Wednesday, June 7, 2006, 3:37:36 PM, you wrote: Pete, An X-Header would be very, very nice to have. I understand the issues related to waiting to see if something comes through, and because of that, I would maybe suggest moving on your own. I've got it on the list to have a message rewriting option... it's just not as high as some others. I hadn't thought about the weight gating utility - though that seems like something that would be useful in general for external tests... "weightgate -5 %WEIGHT% 20 " 5 0 is executed if %WEIGHT% is in the range [-5,20] and the exit code of is returned. That seems like a pretty simple utility to knock out - perhaps I will ;-) Also, on the FP reporting links idea, that would break the process - it's important for us to see the message for many reasons, and it's important for the FP resolution process to be interactive. _M
Re: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]Re[2]: [sniffer]FP suggestions
Pete, I think that you just broke Scott's record with his two hour feature request with your own a two hour program :) Anyone remember those days??? Thanks, Matt Pete McNeil wrote: Hello Matt, Wednesday, June 7, 2006, 4:22:05 PM, you wrote: Pete, Since the %WEIGHT% variable is added by Declude, it might make sense to have a qualifier instead of making the values space delimited. I don't want to mix delimiters... everything so far is using spaces, so it makes sense to continue that way IMO. Errors in Declude could cause values to not be inserted, and not everyone will want to skip at a low weight. I haven't seen any bugs with %WEIGHT% since shortly after it was introduced, but you never know. I have seen some issues with other Declude inserted variables though. Well, errors are always a possibility, but in this case it _should_ be reasonably safe. For example, if this is used to gate SNF, then a missing %WEIGHT% would result in trying to launch a program with the same name as the authentication string, and it is highly unlikely that would be found, so the result would be the "program not found" error code. That's not perfect because it's a nonzero result, but it is safe in that it is not likely to launch another program. One other thing that I came across with the way that Declude calls external apps...you can't delimit the data with things like quotes. There is no mechanism for escaping a functional quote from a quote that should appear in the data that you pass to it...so don't use quotes as delimiters :) Not a problem... I just whipped together a utility called WeightGate.exe that can be downloaded here (for now): http://www.messagesniffer.com/Tools/WeightGate.exe Suppose you wanted to use it in Declude to skip running SNF if your weight was already ridiculously low (perhaps white listed) or already so high that you want to save the extra cycles. Then you might do something like this: SNF external nonzero "c:\tool\WeightGate.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx" 10 0 (hopefully that didn't wrap, and if it did you will know what I meant ;-) To test this concept out you might first create a copy of WeightGate.exe callled ShowMe.exe (case matters!) and then do something like this: SNF external nonzero "c:\tool\ShowMe.exe -50 %WEIGHT% 30 c:\SNF\sniffer.exe authenticationxx" 10 0 The result of that would be the creation of a file c:\ShowMe.log that contained all of the parameters ShowMe.exe was called with -- that way you wouldn't have to guess if it was correct. ShowMe.exe ALWAYS returns zero, so this _should_ be safe ;-) If you run WeightGate on the command line without parameters it will tell you all about itself and it's alter ego ShowMe.exe. That description goes like this (I may fix the typo(s) later): WeightGate.exe (C) 2006 ARM Research Labs, LLC. This program is distributed AS-IS, with no warranty of any kind. You are welcome to use this program on your own systems or those that you directly support. Please do not redistribute this program except as noted above, however feel free to recommend this program to others if you wish and direct them to our web site where they can download it for themselves. Thanks! www.armresearch.com. This program is most commonly used to control the activation of external test programs from within Declude (www.declude.com) based on the weigth that has been calculated thus far for a given message. As an added feature, if you rename this program to ShowMe.exe then it will emit all of the command line arguments as it sees them to a file called c:\ShowMe.log so that you can use it as a debugging aid. If you are seeing this message, you have used this program incorrectly. The correct invocation for this program is: WeightGate , ,... Where: = a number representing the lowest weight to run . = a number representing the actual weight to evaluate. = a number representing the highest weight to run . = the program to be activated if is in range. , = arguments for . If is in the range [,] then WeightGate will run and pass all of , ,... to it. Then WeightGate will collect the exit code of and return it as WeightGate's exit code. If WeightGate gets the wrong number of parameters it will display this message and return FAIL_SAFE (zero) as it's exit code. If is not in range (less than or greater than ) then WeightGate will NOT launch and will return FAIL_SAFE (zero) as it's exit code. As a deubgging aid, I was called with the following arguments: arg[0] = WeightGate
Re: [sniffer]FP suggestions
Darin, Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text. I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source. Matt Darin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
Re: [sniffer]WeightGate source, just in case...
Pete, Just two more cents for the masses... If people use this for two different external tests in Declude, they need to create two differently named executables because Declude will assume the calling executable to be part of the same test and only run it once (or possibly create an error depending on one's configuration). This may not be necessary if you have different test types defined, i.e. nonzero, weight, external, and bitmask, but better safe than sorry. Also, I noted that the Subjects on this list are being repeated. I saw that you changed to a new server, but I also noted that there is no space after "[sniffer]" in the Subject and thought that maybe this is what is throwing things off. Maybe adding that space will correct the issue??? Matt Pete McNeil wrote: Hello Sniffer Folks, The WeightGate utility I posted was done in a real hurry, so I thought I'd post source here just in case anybody wants to review it and in case of any trouble ;-) Also for any educational value it may have for others interested in writing similar kinds of utilities. Source attached in HTML form. Thanks, _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing "Forward". I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. I have been telling Outlook users to copy and paste the headers into a forwarded message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. Matt --- Begin Message --- Of course I'm sending the full message as an attachment. You can do that with Outlook by attaching and item, then browsing your mail folders for the message to attach. And yes, that's how you do it with Outlook Express as well. I don't use Thunderbird or Netscape mail, but I would assume you still need to attach the original message to avoid the headers being lost. What I was referring to was a little more involved than that... namely the possibility of it not matching a rule because the attachment was encoded differently. For example, I've seen mail go through that baes64 encoded an attached email that was not originally base64 encoded. From Pete's responses, it sounded like "no rule found" really did mean no rule was matched. Especially since he has a separate code for "rule already removed". FPs we send are always from same day, or, at the very least, within 24 hours. Darin. - Original Message - From: Matt To: Message Sniffer Community Sent: Wednesday, June 07, 2006 11:46 PM Subject: Re: [sniffer]FP suggestions Darin,Outlook will strip many of the headers when forwarding. Outlook Express needs to forward the messages using "Forward As Attachment" in order to insert the full original headers. Thunderbird/Netscape Mail will work just by forwarding. If you paste the full source in a message, you should send as plain text.I have many FP's that come back as having no rules found, but these are more likely to be from rules that were already removed. So I wouldn't jump to a conclusion that the rule was not found because of formatting unless you are not sending the full unadulterated original message source. I would imagine that it would mostly be IP rules that aren't found when not forwarding the full original source.MattDarin Cox wrote: It is unclear - we receive FPs that have traveled through all sorts of clients, quarantine systems, changed hands various numbers of times, or not (to all of those)... Right now I don't want to make that research project a high priority. Understood. That's true it wouldn't change, but submitting the message directly would not be correct - the dialogue is with you, and in any case, additional trips through the mail server also modify parts of the header and sometimes parts of the message (tag lines, disclaimers, etc)... Hmmm... with attaching the original message, I guess it still makes more sense to deliver to us first for now. Just looking for an alternative that gets you the message as close as possible to the original form as possible. Maybe we'll write a script to copy and forward the D*.SMD file as an attachment to you for FPs at some point in the future. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> --- End Message --- # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: [sniffer][Fwd: Re: [sniffer]FP suggestions]
Darin, Thunderbird allows you to choose the default forwarding method as either inline or as attachment. It might actually default to inline, I can't remember, but whenever it does message/rfc822 attachments, it is as a whole unlike some other clients that edit it down to the bare minimum of what the consider to be useful like addressing, subject date and MIME stuff if appropriate. I'm definitely guilty of being a Netscape diehard, and I'm very happy that the Mozilla project brought things back to life again. I fully understand the attachment trick with Outlook thanks to the confirmations. This will be easier than having people cut and paste the headers in. This doesn't happen much, but there is nothing worse than getting a spam report without header info. I also understand the encoding issues with forwarding in Outlook/OE. It's a shame that this happens. Maybe having a copy of Thunderbird around for this purpose might fit in where this is an issue. Sounds like adding Sniffer headers would be the best solution for this issue on a wider basis since you definitely can't convince every admin not to submit using Outlook/OE. Soon I'm going to code up my Sniffer FP reports to be automatically triggered when a message is reprocessed from my spam review system, so I won't have to even bother with the source any more. That should only take a couple of hours, and it would be time well spent. I always fix issues and whitelist locally where appropriate, but I also report to Sniffer for the benefit of all in addition to making sure that a FP rule will not tag something outside of the scope of what I whitelisted, and I have to report in order to be able to see what the content of the rule was. Customers do most of the reprocessing now, I just do the back end stuff. Matt Darin Cox wrote: Thunderbird and Netscape just takes the full original source and attaches it as a message/rfc822 attachment. I forwarded this message back to the list by just pressing "Forward". Interesting that they include the headers with a simple forward, without specifying forward as attachment. I haven't ever seen that behaviour before in a mail client. Seems like a few forwards would create a very bloated message with all of the old headers. I'm pretty sure that Outlook Express works simply by just pressing Forward As Attachment, or at least it gives me enough of the original, including the full headers, to determine how to block the spam. Yes it does. However you've missed the point. The issue is not how to get the headers. It is how to keep an email client from encoding the message and headers differently, so that Sniffer can properly identify the rule that caught the message. Please excuse me for wanting more detail about the Outlook attachment trick, but would you mind attaching this message to a response so that I could look at the headers and such? Sorry, I don't use Outlook. But I can tell you the steps to take in Outlook 2003 (other versions are almost exactly the same). I have my Outlook users follow these with no problem. 1. Create a new email message 2. Click the arrow beside the paperclip icon, select item instead of file from the dropdown 3. Browse mailboxes from the popup dialog to select the message to attach. 4. Viola, original message and headers attached. There was a discussion about Outlook's behavior with Scott some time ago. Apparently Microsoft was pressured by customers to remove headers when forwarding because they felt that they were a security/privacy risk. No one told them that Outlook was a security/privacy risk on it's own :) ...but that's another story. I would probably feel different if I had the need for groupware though, but digs at Microsoft are irresistible sometimes. I don't remember that discussion, and am not sure we're talking about the same thing. If you attach the original message via the steps above, you get the full original message, headers and body. We have a number of customers who send spam reports this way, mostly on Outlook 2002 and 2003. Darin # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: [sniffer]Re[2]: [sniffer]WeightGate source, just in case...
Pete, My understanding was that Declude treats different arguments to an executable as just being other forms of that executable so it only processes it once. I'm not positive one way or another. It's worth testing though. Matt Pete McNeil wrote: Hello Matt, Wednesday, June 7, 2006, 11:52:56 PM, you wrote: Pete, Just two more cents for the masses... If people use this for two different external tests in Declude, they need to create two differently named executables because Declude will assume the calling executable to be part of the same test and only run it once (or possibly create an error depending on one's configuration). This may not be necessary if you have different test types defined, i.e. nonzero, weight, external, and bitmask, but better safe than sorry. I think this might not be correct. IIRC, the design spec for that feature was that if the command line was different in the test then it would be executed again and if the command line was identical it would not. This was to allow for calling the same program with different parameters. I'm pretty sure that's how it works --- it might be worth a few tests if you're sure it's not that way, but I strongly suspect that if one of the parameters are different in the test line (inside the quotes) then it will be executed again as a different test. Also, I noted that the Subjects on this list are being repeated. I saw that you changed to a new server, but I also noted that there is no space after "[sniffer]" in the Subject and thought that maybe this is what is throwing things off. Maybe adding that space will correct the issue??? It does look a little weird. Sometimes it's normal though. I'll see if I can identify anything odd in the settings. _M
[sniffer] Re: New SPAM pain
Pete surely won't mind after you post your observations :) Matt Darrell ([EMAIL PROTECTED]) wrote: If Pete doesn't mind I will post my observations in regards to the product. I run both products (CommTouch and Sniffer). Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. John Shacklett writes: I'm dying to start a thread and talk about Sniffer's stance on CommTouch, but I can resist. Instead, I would like to point out that eight clearly spam messages have made it through to my Inbox [or Outlook Junk Folder] so far this week that appear to have skinned clear through Sniffer. First ones I've seen in > >Are we undergoing a new phase or campaign that I can make adjustments for? -- John # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. I would be nice too if you talked with Declude about allowing for the insertion of headers, or even if you did this on your own. I believe the D* file may be editable when the external app is launched. That would make recovery of this so much easier for me (minutes instead of hours of work). Thanks, Matt Pete McNeil wrote: Hello Darin, Monday, October 16, 2006, 5:17:26 PM, you wrote: > Anyone else seeing a sudden increase in FPs? We normally report a few each day, but we're seeing a 10x increase in FPs for the past three days. Not sure if this is it, but there was an image segment rule that went in over the weekend and resulted in an unusual number of false positives today. The rule was removed. IIRC the rule id was: 1174356 Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Significant increase in false positives
There is no doubt that having Declude handle xhdr files would be optimal. I might add that an option to exclude the header on non-hits would also be wise. David Barker appears open to some feature requests of late, and I would think that you could make this happen. Not everyone has capacity limitations, so the internal functionality would probably suit the needs of many of your users also, and cover all non-SA systems instead of just Declude. Regarding this rule, the binary segment is non-searchable. My only solution would be to write some _vbscript_ that parsed the Sniffer log for hits and move the files from my CopyFile directory back into Declude's Proc. I'm guessing that someone could also do some grepping for this, but that ain't a strength of mine. I could do this in minutes though if I had headers to search on. Thankfully this rule only hit about 1,000 times this weekend as a final match (I'm ignoring those that weren't final matches since those would have hit anyway). My gateway gets rid of most image spams, so I would expect a comparably higher rate for others. Regarding false positives in general. I don't expect Sniffer to be perfect due to the way that rules are generated, but I have two suggestions. 1) One would be to test all new rules on a small sub-set of E-mail that covers the most common patterns such as attachments and E-mail/webmail clients with various formats including forwards and replies. This would likely stop the worst of the worst in terms of FP issues like the one earlier this year that was hitting on most base64 code. I envision hundreds of test messages and not thousands, so this should be practical. 2) The second suggestion is one that I have mentioned many times before in private involving being able to tag messages on multiple types of hits for a stronger result. The separation would need to be on the type rule so that all rule types would be isolated from one another. For instance, phrase, pattern, IP and domain rules could be put in different codes and allowed to be scored in combination. It would also be equally as important to treat rules from user submissions different from those generated from spam traps since these rules are not nearly as universal. I currently average just under 3 matches per message that Sniffer hits, and I would imagine that there is a lot of mixing between these types. This would allow many that are scoring Sniffer lower than our block weight to then score these multiple classification hits higher. This wouldn't be useful though unless it was seperated by types like I listed since I often find multiple hits under the current rulebase format. Thanks, Matt Pete McNeil wrote: Hello Matt, Monday, October 16, 2006, 10:03:04 PM, you wrote: > Pete, Would you please clarify this a bit. Declude of course doesn't record the rule in the headers, so this is difficult to figure out. Knowing the pattern may help identify the problematic messages. Also knowing the start time and end time of the rule would also help. The rule was coded for a binary segment in an image file. Here is the rule information: Rule - 1174356 Name image spam binary segment as text !1AQaq"2 Created 2006-10-14 Source !1AQaq"2 Hidden false Blocked false Origin Spam Trap Type Simple Text Created By [EMAIL PROTECTED] Owner [EMAIL PROTECTED] Strength 3.20638481603822 False Re
[sniffer] Re: Increase in spam
I have a moderately large number of domains and accounts that I protect, and in the past, whenever someone indicated an increase in spam, my system was always at normal levels and I chalked it up to some change on the end of the one reporting the issue (such as a nobody alias or being baptized by a brute force /dictionary spam attack). Over the last two weeks though, my connection traffic to my gateway is up about 90%, and what gets through my gateway to Declude has risen around 20%, yet there has been no noticeable increase in good E-mail. I'm guessing that one of the zombie spammers has become more aggressive, at least with tarpitting avoidance and retries, while clearly there are a lot of new static spam blocks (Scott Richter for instance is back with a vengeance). In relation to yesterday's reported leakage, I am guessing that Sniffer isn't the only protection, and that RBL's are a part of that system. If so, the Network Solutions issues yesterday did cause issues with resolving off of blacklists as it has been reported, and that could explain the extra leakage. Matt Darin Cox wrote: We saw a sudden ~50% increase on July 16th, but only fluctuations and moderate growth since then. On weekdays we're now at 80% spam, 95% or better on weekends. Darin. - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Message Sniffer Community" Sent: Wednesday, October 18, 2006 9:23 AM Subject: [sniffer] Re: Increase in spam Hello K, Wednesday, October 18, 2006, 8:52:17 AM, you wrote: I've been seeing a massive increase in spam over the last 2 days getting through with minimal scores. Could this be due to the drawback of the filter involved with false positives, or something else? It's hard to pin down, but not likely to be the pulled rule. We have seen a relative increase in new spam campaigns over the past 2 days preceded by a lull. That may be what you're noticing. I've attached a graph to illustrate. _M
[sniffer] Re: SPAM Problems
Sorry about the OT here, but I feel compelled to add just a little follow up on the topic of pre-scanning and Alligate. Alligate is IMO definitely the way to go. As Paul pointed out, greylisting everything (i.e. ORF) has drawbacks and I wouldn't use a solution that greylisted everything. I worked with Brian Milburn of Alligate for months to help him create a method of providing selective greylisting so that most legitimate E-mail is not greylisted. I also helped him create a method of storing triplicates for use with greylisting that only track base domains and not the full sender and recipient, thus substantially reducing what needs to be greylisted if it does trigger selective greylisting. I received nothing in return except for a very capable product that benefited my system greatly. Brian is also a lot like Pete and R. Scott Perry. Setting things up optimally is not going to be an out of the box type of experience. I have both offered some free assistance in private and public to those that are dealing with Alligate, and Brian can also provide some support for new setups. There is of course a limit to my time for things like this. I have also occasionally consulted on such things at the request of others. So while it can be a hard nut to crack, especially if one is not familiar with the architecture or concepts of a pre-scanning gateway, there is help out there, and it is definitely worth while. I formerly used ORF for tarpitting and address validation, but going to Alligate for this was the best move that I have made since picking up Declude and Sniffer. Note that Alligate Gateway is not a replacement for Sniffer, Declude or any other deep scanning solution, it is merely a tool for handling validation and some blocking of the most obvious and easiest to detect spam, primarily with passive means of blocking (greylisting and tarpitting), and without needing to throw a lot of CPU at it. I handle over 1 million connections per day and Alligate averages about 5% CPU at peak times. Only 7% of the connections result in delivery of a message to my deep-scanning layer using a configuration that is not aggressive. There is only one zombie spammer at present that will survive greylisting. Matt Dave Marchette wrote: I agree with the pre-scanning concept. IMgate, ORF and Alligate are all good, but it just depends upon your level of comfort with each type of environment these run in. Each takes several days of fine tuning and log babysitting (even though the vendors tell you it is plug and play- it's not). We've tested all three and prefer Alligate (thanks Matt!) but any way you look at it, if you are running even moderate volume then pre-scanning is the next step in the evolution of protection. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Technical Support Sent: Monday, October 23, 2006 7:28 AM To: Message Sniffer Community Subject: [sniffer] Re: SPAM Problems We also use ORF by VamSoft on IIS to pre-process. We do not use the grey listing. We tried it, and it is great at eliminating spam, but it can delay mail for hours, which is a problems for most email users. Instead of grey listing, we have found ORF's tar-pitting very effective. We set some tests at the ORF level, but don't block on them (because there is no "weighting"). We also have some spam trap email addresses. Fail a test or hit a spam trap and we tar-pit. Instead of sending us 100 spams a minute they can only send one per minute. We can pick up x-records with Declude and not have to re-run the tests on the iMail server, still using Declude to score the messages based on the prior tests. ORF even has a built-in interface for sniffer. It is simpler and preferable to process everything on the iMail server, but when you want to off-load processing to stretch your iMail / Declude investment, this arrangement can do the trick. Paul Fuhrmeister [EMAIL PROTECTED] -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of David Waller Sent: Monday, October 23, 2006 5:15 AM To: Message Sniffer Community Subject: [sniffer] Re: SPAM Problems Filippo, We had a similar problem. Due to the huge volumes of spam we found our mail server becoming less able to deal with email. Imail/Declude/Sniffer is expensive in processor terms when processing email and we found the best was to pre-process mail filtering using Greylisting (we used Vamsoft in IIS SMTP but others exist). This has dramatically reduced the load on our server and seems to stop the bulk of spammers and mail harvesters Hope this helps. David # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PRO
[sniffer] Re: Version 2-3.5 Release -- Faster Engine
Kudos Pete! Just wanted to say thanks. Matt Pete McNeil wrote: Hello SNF Folks, The plan was to hold off until the next major release, however in light of recent increases in spam traffic we are pushing out a new version with our faster engine included. All other upgrades are will wait for the major release ;-) The scanning engine upgrade results in a 2x speed increase that hopefully will help with the higher volumes we are seeing now. Version 2-3.5 also rolls up 2-3.2i1 which included the timing and file locking upgrades. You can find version 2-3.5 here: http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions Thanks, _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Increase in spam
My connection traffic doubled over the course of two weeks. This started on about the 9th, and seems to have peaked and leveled off last week. The increase was mostly due to a new brute force spammer (a.k.a. dictionary attack), but static spam seems to have also increased by about 20%. I believe the static spam increase is Scott Richter lighting his companies back up, mostly at carolina.net, in addition to the brute force zombie spam and massive increase in backscatter that these attacks are causing. About 10% of connections to our gateways are the result of backscatter, with 30% of that coming from FrontBridge alone (bigfish.net). The new brute force spamming is not just simply higher volume, it also has more targets than before. This has helped to expose several customer's accounts that had a domain aliased with a catch-all and forwarded to an account that we were protecting. Previously, we never saw this since those domains weren't being attacked. I have no clue as to why anyone is still providing catch-alls, especially mail forwarding services like BulkRegister. It just seems like a good way to limit the capacity of a server by 75% or more. Matt Pete McNeil wrote: Hello Andrew, Wednesday, October 25, 2006, 1:33:20 PM, you wrote: For another organization's graph of spam trends as received by them, check out the updated graphs at TQM cubed: http://tqmcube.com/tide.php Their graph shows a sharp uptick at the end of June 2006. ...and a new upward trend around 0917. That's consistent with what we've seen. _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Uploading problems
Try WPUT http://sourceforge.net/projects/wput/ Matt K Mitchell wrote: At 11:16 PM 12/7/2006 -0700, Jay Sudowski - Handy Networks LLC wrote: Give this a try: http://www.ncftp.com/download/ Just did about 5 minutes ago. It won't run without specifying a destination directory, and sortmonster ftp won't allow any directory settings. Thanks though :o)
[sniffer] Re: FTP server / firewall issues - Resolved.
Darin, There are many people with firewall or client configuration issues that cause problems with FTP, however HTTP rarely experiences issues and is definitely easier to support. As far as efficiency goes, since the rulebases will all be zipped, there is little to be gained from on-the-fly improvements to FTP (and there are some for HTTP as well). In such a case, I would consider it to be effectively a wash, nothing gained, nothing lost (measurably). Matt Darin Cox wrote: Thanks, Pete. Appreciate you taking the time to explain what's happening in more detail. I'm curious as to why FTP is more difficult than HTTP to debug, deploy, secure, and scale, though. I tend to think of them on equal footing, with the exception of FTP being faster and more efficient to transfer files in my experience. Thanks for the link to save some time. Much appreciated. Darin. - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: "Message Sniffer Community" Sent: Friday, January 05, 2007 9:47 PM Subject: [sniffer] Re: FTP server / firewall issues - Resolved. Hello Darin, Friday, January 5, 2007, 6:23:22 PM, you wrote: Hi Pete, Why the change? Many reasons. HTTP is simpler to deploy and debug, simpler to scale, less of a security problem, etc... Also, the vast majority of folks get their rulebase files from us with HTTP - probably for many of the reasons I mentioned above. FTP is more efficient for transferring files than HTTP. Not necessarily ;-) Can we request longer support for FTP to allow adequate time for everyone to schedule, test, and make the change? I'm not in a hurry to turn it off at this point, but I do want to put it out there that it will be turned off. I remember trying dHTTP initially when this was set up, but it wasn't working reliably, plus FTP is more efficient, so we went that way. wget may work better when we have time to try it. Also, what's this about gzip? Is the rulebase being changed to a .gz file? Compression is a good move to reduce bandwidth, but can we put in a plug for a standard zipfile? Gzip is widely deployed and an open standard on all of the platforms we support. We're not moving to a compressed file -- the plan is to change the scanning engine and the rulebase binary format to allow for incremental updates before too long - so for now we will keep the file format as it is. Apache easily compresses files on the fly when the connecting client can support a compressed format. The combination of wget and gzip handle this task nicely. As a result, most achieve the benefits of compression during transit almost automatically. Do you have scripts already written to handle downloads the way you want them now? If so, how about a link? We have many scripts on our web site: http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.AutoUpdates My personal favorite is: http://www.sortmonster.com/MessageSniffer/Help/UserScripts/ImailSnifferUpdateTools.zip I like it because it's complete as it is, deploys in minutes with with little effort, generally folks have no trouble achieving the same results, and an analog of the same script is usable on *nix systems where wget and gzip are generally already installed. There are others of course. Hope this helps, _M
[sniffer] Re: Integration with Mailenable
Yeah, filtering services suck! Matt Chris Bunting wrote: Merak mail server has been great for us, we have 10,000 users, and have not had any problems with it over the 5+ years we have been using it... It's been rock-solid. Don't waste your money on the anti-virus/anti-spam filtering services... just use message sniffer with a content filter and you are set. Thank You, Chris Bunting Lancaster Networks Direct: 717-278-6639 Office: 888-LANCNET x703 MS Certified Systems Engineer IP Telephony Expert Lancaster Networks 1085 Manheim Pike Lancaster PA 17601 www.lancasternetworks.com -- Corporate Technology Solutions... Specializing in 3com NBX Telephony Solutions IT Services - Phone Systems - Digital CCTV -- The information in this e-mail is confidential and may be privileged or subject to copyright. It is intended for the exclusive use of the addressee(s). If you are not an addressee, please do not read, copy, distribute or otherwise act upon this email. If you have received the email in error, please contact the sender immediately and delete the email. The unauthorized use of this email may result in liability for breach of confidentiality, privilege or copyright. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Jay Sudowski - Handy Networks LLC Sent: Thursday, March 15, 2007 10:27 PM To: Message Sniffer Community Subject: [sniffer] Re: Integration with Mailenable Stay Away From MailEnable. There are so many exploits out there for MailEnable, and there are more exploits found monthly, if not weekly. At one particular interval, MailEnable had to re-release the same patch several times in the *same* week because it kept on not actually fixing the root of the issue. If you run MailEnable, odds are that you will end up exploited, even if you stay on the of the patches. On top of that, MailEnable is just simply a CPU and IO hog, much more so than other other mail server I have ever seen. By default, they use entirely text based configuration files, which on occasion get truncated to zero during periods of high activity on the server. In the past year, we have assisted our customers move 20,000+ mailboxes away from MailEnable, mostly all to SmarterMail. Do not waste your time and money with MailEnable. -Jay -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Phillip Cohen Sent: Thursday, March 15, 2007 12:22 PM To: Message Sniffer Community Subject: [sniffer] Integration with Mailenable We are finally going to replace our old Vopmail server. Looking at Mailenable Enterprise. Will Sortmonster work with that program? Is anyone using Mailenable? If so how is it and if it works with Sortmonster how did you use them together. THanks, Phil # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Integration with Mailenable
There is in fact a Domain Keys plug-in for SmarterMail listed on their downloads page: http://www.smartertools.com/Products/SmarterMail/DL/v4.aspx Personally I'm not a fan of any present sender identification implementation. Both SPF and Domain Keys are primarily associated with spam by volume, and SPF can at cause one's customers issues when they do things like use alternative SMTP servers or find themselves behind an SMTP proxy at a hotel or T-Mobile HotSpot...but I digress. I think that both IMail and SmarterMail are decent products, but neither one of them is perfect. SmarterMail certainly has a lower cost of entry. I would trust Jay's experience with MailEnable considering his extensive experience. Matt Jay Sudowski - Handy Networks LLC wrote: Hi Phil - Good question. We integrate Sniffer into SmarterMail via Declude. However, SmarterMail does have the capability to run a program against a message before it is delivered. We have some customers that use a batch file to call f-prot and get virus scanning integrated into their mail server on the cheap. I believe it would likely be possible to make use of the same functionality to call Sniffer directly, and thus avoid having to purchase Declude. I have just never had a need to attempt this. As for domain keys, I don't believe so. However, you can setup SPFyou're your domains simply by adding the appropriate DNS records to said domains zone files. -Jay -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Phillip Cohen Sent: Friday, March 16, 2007 12:01 PM To: Message Sniffer Community Subject: [sniffer] Re: Integration with Mailenable Jay, Thanks for the heads up on Mailenable. I took a look at SmarterMail and it looks pretty good. How does it interface with Message Sniffer or does it require and external gateway such as EWall? How has support been with it and how have they been as far as updates. Also does it have "domain keys" capability and SPF support for sending mail to yahoo.com etc... Thanks, Phil At 07:26 PM 3/15/2007, you wrote: Stay Away From MailEnable. There are so many exploits out there for MailEnable, and there are more exploits found monthly, if not weekly. At one particular interval, MailEnable had to re-release the same patch several times in the *same* week because it kept on not actually fixing the root of the issue. If you run MailEnable, odds are that you will end up exploited, even if you stay on the of the patches. On top of that, MailEnable is just simply a CPU and IO hog, much more so than other other mail server I have ever seen. By default, they use entirely text based configuration files, which on occasion get truncated to zero during periods of high activity on the server. In the past year, we have assisted our customers move 20,000+ mailboxes away from MailEnable, mostly all to SmarterMail. Do not waste your time and money with MailEnable. -Jay -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Phillip Cohen Sent: Thursday, March 15, 2007 12:22 PM To: Message Sniffer Community Subject: [sniffer] Integration with Mailenable We are finally going to replace our old Vopmail server. Looking at Mailenable Enterprise. Will Sortmonster work with that program? Is anyone using Mailenable? If so how is it and if it works with Sortmonster how did you use them together. THanks, Phil # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: How to incorporate a white list?
Agreed, however reverse DNS is not a universal solution as things like RR accounts will come from the same base domain as RR spam zombies, and you would otherwise have to track down each unique reverse DNS entry. I would test a connection to the SMTP server instead. Most of these servers will at least respond. So if a domain like yahoo.com, gmail.com, rr.com, etc. is found in the reverse DNS for a new IP rule, you would then check to see if it at least responded to a port 25 connection, and if it did, skip that rule. Note that I score IP rules at half the weight of the others. There are more common issues with international ISP's and webmail providers than with things like yahoo.com, gmail.com, rr.com, etc. Many don't get a lot of international traffic so they don't notice it. Matt Andy Schmidt wrote: Hi, Unless I'm mistaken, rule 1370762 was targeting the same address range. If I may make a suggestion: Before the spam-trap robots are allowed to block major, well-known and easily recognizable email providers, how about the robot script pulls a WHOIS and a Reverse DNS and runs that data against a table of "can't block" entities - or at least spits those out for "human review". If that can't be done, then how about the robots issue an hourly report of "suspect" IPs. A no-brainer script can pull matching WHOIS and RevDNS for quick human review and overriding (if necessary). I would rather those obvious bad rules are caught before or very quickly after they go live. There is always some delay before I get first reports until I realize that this is a "real" problem. Then I have to try to get headers from end-users before I can dig into logs... Hours and hours pass (especially if it's overnight events). In the meantime the problem escalates all around me. Thanks, Andy -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Tuesday, April 03, 2007 11:09 AM To: Message Sniffer Community Subject: [sniffer] Re: How to incorporate a white list? Hello Andy, Tuesday, April 3, 2007, 9:36:17 AM, you wrote: Hi Phil, Yes, it seems as if some Sniffer rules, e.g., 1367683, is broadly targeting Google's IPs. I've submitted 3 false positive reports since last night, at least two of them were Google users, one located in the U.S. and the other in the Netherlands! This IP rule has been pulled. FP processing will happen shortly. _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: How to incorporate a white list?
Pete, CBL has a proven 99.97% accuracy and on some systems over a 40% hit rate on traffic, yet their methods are rather simple and easy to implement. If an IP hits your spamtrap, and it has either no reverse DNS entry or it has a dynamic reverse DNS entry, it is added, if it doesn't, it isn't added. They have a few other mechanisms that I am aware of, but the above will take care of almost everything related to spam zombies. Your current whitelisting method will take care of the few exceptions to this. There is rather simple code that can test for standard types of dynamic reverse DNS entries with both numbers and hex encoded values, and exceptions for names that might include things like "mail" or "mx#" in the names. If you want to expand this to static spammers, you merely introduce other pre-qualifications such as having a Mail From domain or HELO that matches the payload domain in the body. I figure that for the most part however that you are tagging static spammers with other rules that take presidence over the IP rules, and that this would be minimally beneficial in comparison to spam zombies. The source of the false positives hitting your spam traps are most likely due to AFF (Advance Fee Fraud) and some phishing, which use free accounts on legitimate servers to send their spam, and an increasing precidence of hacked E-mail accounts being used by zombie spammers. The first method would avoid listing such servers in almost every circumstance, and we certainly wouldn't ever see things like yahoo.com, gmail.com and rr.com mail servers listed like we see with some degree of regularity under the current method. Matt Pete McNeil wrote: Hello Andy, Tuesday, April 3, 2007, 5:15:12 PM, you wrote: Hi Jonathan: That's exactly the problem. These particular rules were blocking Google mail servers - NOT specific content. To clarify, it was blocking precisely one IP. The F001 bot only tags a single IP at a time (not ranges, ever), and only after repeated appearances at clean spamtraps where the message also fails other tests (often including content problems like bad headers, obfuscation, heuristic scam text matching etc.) The rule was in place from 20070326. The first reported false positives arrived today (just after midnight). The rule was removed just less than 12 hours from that report (due to scheduling and heavy spam activity this morning that requiring my immediate attention). The report was ordinary (not a rule panic). As is the case with all FPs, the rule cannot be repeated (without special actions). Obviously, as already discussed in the past, it IS necessary that these IP-based blocks are put under a higher scrutiny. I'm not suggesting that the "automatic" bots should be disabled, I'm just proposing that "intelligence" must be incorporated that will use RevDNS and WHOIS to identify POSSIBLY undesirable blocks and to flag those for human review by Sniffer personnel so that they don't end up poisoning mail severs of all their clients. While interesting, these mechanism are not foolproof nor trivial to implement. Also - our prior research has taught us that direct human involvement in IP rule evaluation leads to far more errors we can allow. Once upon a time, IP rules were created in very much the way you describe -- candidate IPs were generated from spamtraps and the live spam corpus and then reviewed (manually and automatically) against RevDNS, WHOIS, and other tools. At that time, IP rules had the absolute worst reliability of any test mechanism we provided. Upon further R&D, we created the F001 bot that is in place and now the error rate has been significantly reduced and our people are able to focus on things that computers can't do better. Please don't get me wrong, I'm definitely not saying that the F001 bot can't be improved - it certainly can, and will if it survives. What I am saying is that it is accurate enough now that any improvements in accuracy would be non-trivial to implement. Our current development focus is on developing the suite of applications and tools that will allow us to complete the alpha and beta testing of the next version of SNF*. That work has priority, and given that these events are rare and easily mitigated we have not deemed it necessary to make enhancements to the F001 bot a higher priority. The following factors make it relatively easy to mitigate these IP FP events (however undesirable): Rule panics can make these rules immediately inert, FP report/response times are sufficiently quick, The IP rule group is sequestered at the lowest priority so that it can easily be weighted lower than other tests. Also, it is likely that the F001 bot and IP rules group will be eliminated once the next SNF version is sufficiently deployed because one of the major enhancements of the new engine is a mul
[sniffer] Re: Downloads are not working....
Appriver, who is somehow involved with Sniffer, is having a ridicolous problem with sending messages over and over again (once every few seconds). They pulled their contact information from their site but didn't take down their servers. I suspect this is putting strain on them and if Sniffer uses their bandwidth for downloads, that could explain things. Matt Chuck Schick wrote: Speeds are really slow and the connection is lost before completionEverything checks out good on our end. Is something going on with the sortmonster end of things? Chuck Schick Warp 8, Inc. (303)-421-5140 www.warp8.com # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Downloads are not working....
Pete McNeil wrote: I'm not sure what the actual issue is (I will get that data later), however I've just been informed that it should be resolved in the next 20 minutes or so. The issue was that they were redelivering messages over and over again. One customer got one message over 500 times in an hour! I suspect that our gateways were blocking some of this automatically, and I also tried to block it at the router level but it kept popping out of other address blocks. Matt
[sniffer] Re: Appriver issue
I have something that I would also like to clear up. When I indicated that AppRiver had removed it's contact page, it likely just wasn't operating at the time that I was attempting to access it. Considering their issues, it would not be a surprise to see other issues like this caused, but it seemed suspicious since their home page was working and not their contact page. I did note that it was working by the time that it was pointed out that it was up. In no way did I ever believe that Pete or Sniffer had any direct involvement in the system that created these problems, and in no way should this reflect badly on Pete or Sniffer as far as I am concerned. I was slightly miffed after getting off the phone with them where their reaction quite clearly indicated that they were aware of the issue. I suggested that they take their servers off-line due to the issues that were being caused, but I was probably barking up the wrong tree. The servers weren't taken off line for another hour or so, or maybe this is when the delivery servers caught up with the queued E-mail destined for my client. I'm not sure why they didn't act on this sooner. When you have a loop, it is important to stop it, and their multi-homing made it difficult for others to block. One user received about 500 copies of the same message (and also called them), and there were other examples that we saw which were much more limited. I do hope that they didn't choose to introduce new software at 11 a.m. ET on the busiest E-mail day of the week, and that this was only when the problems surfaced... Everyone that deals with significant volumes of E-mail has issues from time to time, and I wouldn't draw conclusions about AppRiver based on just this one circumstance. I would imagine that it is hard to plan for how to deal with a broad scale looping issue, and I'm sure this was a learning experience for them. Matt David Moore wrote: I think what Peter is try to say is that Sort monster is hosted at Appriver and Appriver had an issue and therefore so did Sort monster. http://www.dnsstuff.com/tools/dnsreport.ch?&domain=sortmonster.com Regards David Moore [EMAIL PROTECTED] J.P. MCP, MCSE, MCSE + INTERNET, CNE. www.adsldirect.com.au for ADSL and Internet www.romtech.com.au for PC sales Office Phone: (+612) 9453 1990 Fax Phone: (+612) 9453 1880 Mobile Phone: +614 18 282 648 POSTAL ADDRESS: PO BOX 190 BELROSE NSW 2085 AUSTRALIA. - This email message is only intended for the addressee(s) and contains information that may be confidential, legally privileged and/or copyright. If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email, or taking any action in reliance on its contents by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Kevin Rogers Sent: Saturday, 19 May 2007 11:59 AM To: Message Sniffer Community Subject: [sniffer] Re: Appriver issue Thanks for the explanation, and I wasn't trying to blame you - just wanted more info is all. We use Sniffer, but not Appriver. You said that if we don't use Appriver, we shouldn't have been affected, but you also seemed to say that if one of the recipient's of my user's email uses Appriver that might've caused a problem. And also that *some* of Sniffer users might have experienced the problem as well. It sounds like things are still being worked out. I just wanted some kind of verification that they were aware of the problem, were working on it, that they were in some way sorry about what happened...you know - the usual stuff. And I know that you are not an official rep of Appriver or anything, but presently you're all we have in that role ;) Thanks Kevin Pete McNeil wrote: Hello Kevin, Friday, May 18, 2007, 8:52:47 PM, you wrote: Pete - Thanks for the reply, but I guess I don't understand what you're saying. "Some packet loss" and "rulebase downloads to slow down for a time" don't reflect what happened to me yesterday and apparently not what happened to one of the other posters either when he said that Appriver was having a problem "with sending messages over and over again". I received over (at last count) 35,000 messages (almost all of which were bounced replies, from one email from one of our users who sent an email to about 70 people) yesterday. And I had already gone to http://www.armresearch.com/ yesterday and there was nothing there. There
[sniffer] Re: Error Messages since WeightGate
hat can run before the mystery heap is depleted. However, some people have reported better results by raising the value to 2048KB -- that's one of the problems with undocumented resources (there's no way to know for sure which value is better or why). We recommend going to http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 and changing the registry entry to use a value of "256" or "2048" (NOTE: Microsoft recommends 512 in that article; if you use 512, make sure not to have IMail's MaxQueProc registry entry set to more than 30). Matt Keith Johnson wrote: Darrell, Did you alter your heap size 3rd entry? If so, did you go to 1024 or other. I found this article by crossing a Declude page, appears to be what I need to go after. http://support.microsoft.com/default.aspx?scid=kb;EN-US;q142676 -Keith _ From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED]) Sent: Sun 6/10/2007 2:31 PM To: Message Sniffer Community Subject: [sniffer] Re: Error Messages since WeightGate After looking into it I am on board with what Pete said about the "heap" issue. It makes sense to me that its the heap issue since were launching weight gate -> SNF. Effectively doubling the amount of processes being launched. Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. Keith Johnson wrote: Darrell, You are right, a reboot will take care of it for a season, then it comes back out of the blue. Very strange indeed. Keith _ From: Message Sniffer Community on behalf of Darrell ([EMAIL PROTECTED]) Sent: Sat 6/9/2007 9:36 PM To: Message Sniffer Community Subject: [sniffer] Re: Error Messages since WeightGate Keith, I was having the same problems last week. Just came out of the blue and was across several of our servers as well. Same error verbatim. FWIW - I also use weightgate. I rebooted the servers I was seeing this issue on and the problem has not returned. Very odd you mentioned that as I thought this was isolated to just me. Darrell --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. Keith Johnson wrote: It appears since installing WeightGate we have been receiving a lot of the below Application PopUps indicating an error: The application failed to initialize properly 0xc142. Click on OK to terminate the application The application entry is our Sniffer .exe. Today alone I saw over 300. I thought it was an isolated issue. However, it is happening across all our servers. We are running the latest Sniffer in Persistent mode. We never saw these prior to WeightGate. Has anyone seen this before? Below is the actual entry in Event Log. -Keith Event Type: Information Event Source: Application Popup Event Category: None Event ID: 26 Date: 6/9/2007 Time: 12:12:35 AM User: N/A Computer: NAIMAIL2 Description: Application popup: rrctp2ez.exe - Application Error : The application failed to initialize properly (0xc142). Click on OK to terminate the application. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> -- --- Check out http://www.invariantsystems.com for utilities for Declude, Imail, mxGuard, and ORF. IMail/Declude Overflow Queue Monitoring, SURBL/URI integration, MRTG Integration, and Log Parsers. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To un
[sniffer] Re: Error Messages since WeightGate
Here's a better page from someone at Microsoft all about the desktop heap. This one suggests that you can change the limit from 48 MB to a value as much as 450 MB. You will probably normally not need more than the total number of processes that Declude can use times the amount of memory allocated per session, so if you have 512 MB/session, and have 100 processes defined in Declude, you would need about 50 MB, but adding something like weightgate to an app that has latency could very well increase the needs even more. http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx Matt Matt wrote: Keith, When I looked at this several years ago, this is what I came up with: Windows allows a total of 48 MB in the heap, and each service started process uses the third setting in the chain, or 512 KB by default, and there is about 10 MB that gets used for other things. Based on what Scott Perry wrote concerning this in a obscure page on the Declude site about Declude Queue, there can only be a total of 77 service started processes before having issues, and you can assume that there will be one for Declude up to my limit of 40, and also often times another process whether it is a virus scanner or external filter application in JunkMail. Windows apparently starts to barf when the limit is reached, and applications can go into a bad state, only partially launching and becoming corrupted. This has a high association with load, but the true association seems to be the number of processes, which typically correspond to load but not necessarily. This is probably also what has caused McAfee to barf on occasion on my server with similar errors. McAfee has a decent amount of latency compared to most other things that Declude launches except of course for Eradispam due to the timeout issues. There are two camps on what to do with the mystery heap, aka desktop heap. Some have indicated on the IMail and Declude lists in the past that setting it to 2048 would resolve some issues with IMail's SMTP and also the 16 bit version of F-Prot when run from Declude which is awfully slow and CPU intensive. That change however would reduce the number of service started processes that were possible by a factor of 4. Scott suggests that reducing it to maybe 256 would help in high traffic servers, though this is a limit that you wouldn't want to pass because it could cause instability. FYI, the error messages will contribute to heap usage, so these must be cleared, and when you have a bunch of these, it will limit what you can run, and in fact make the problem worse. If you are using Declude as a service, that certainly takes one process off the top that used to count towards the heap, but it's likely what is running concurrently that is causing the issue along with error dialogs. Weightgate certainly adds to this issue, as well as other plugins and virus scanners. The best solution for a high volume server that wants to do weight skipping would be for either Sniffer or Declude to skip based on both a high and a low weight within the config. I have been asking for over three years for this and have even recently documented a solution for Declude that would be backwards compatible with current configs should they opt to do this. Here's a quote from the old Declude site authored by Scott: *Flaw #1 - Server crashing: Microsoft's Mystery Heap* Fortunately, not many people experience this problem. However, it is listed first because it is more serious than the other flaw. This one can back up mail for hours/days, and crash the server. The problem here is that each process that is started by a service uses a certain (unknown) amount of an undocumented type of memory that Windows allocates. Without knowing how much of the mystery heap is used, or how much is left, or how much is available when the system starts, it's impossible to know when you will run out. When you DO run out, Windows does a *terrible* job in handling it. Instead of preventing the program from loading and recording an error to the event log, Windows will keep the program half-loaded (the error almost always occurs while loading .DLLs) and pop up an error message saying that it can't start the program. When this happens, unless you happen to be at the server, you won't have a chance to close the box. So, another one will soon pop up as another SMTP process is started. By the time you find out, there could be hundreds or thousands of the pop-up boxes. Since Microsoft doesn't clear them automatically, when the original 30 SMTP processes end, there still isn't enough of this mystery heap left, because Microsoft is using it to display these error messages. So
[sniffer] Dead Sniffer processes piling up.
Pete, I found this morning an instance where suddenly the number of processes on my system shot from around 50 to as many as 300, and after that peak, it settled down and rode the 150 level. All of the hung processes are Sniffer being called by Declude. I also had about 10 errors waiting to be cleared from another application, but probably because of the way that Sniffer works (as a service or something related), the Sniffer processes are just hung without a prompt. I saw this last week also. I have Declude set for 200 processes, so it probably reached 300 when the first 100 hung, and then it stayed with those 100 hung. Is there anything that can be done in Sniffer to kill off these hung processes in an automated and proactive manner? I recently upgraded to the latest version and I was probably a version or two behind, and I don't recall this happening before. Thanks, Matt
[sniffer] Re: Dead Sniffer processes piling up.
Pete, I have left all of those processes active for troubleshooting, and they are still there and definitely Sniffer. Process Explorer even shows what command line the executable was run with so I was able to do some digging in the logs for specifics. I found that Declude was recording errors related to Sniffer, and Sniffer was not logging the messages or associated events at all. Here's what Declude is showing: 06/14/2007 09:11:01.665 q3d2b00d49610.smd ERROR: External program SNIFFER-IP didn't finish quick enough; terminating. 06/14/2007 09:11:01.665 q3d2b00d49610.smd Couldn't get external program exit code Now it could be that Declude is causing issues by trying to terminate Sniffer. FYI, I don't have a stack up of any additional files in my Sniffer directory, and the service is started and everything seems fine outside of this one period of time when my server got wallopped. What happened was that one customer with over 1,000 addresses received 950 E-mails from a ConstantContact customer in spam run from harvested addresses. They can deliver fast enough that it likely stressed my system for a moment, and triggered this behavior. It could also be that the content of these messages caused an issue with Sniffer. My server has 8 cores in it, and if it reached 100% CPU, it only did so for a moment in time. This very likely could be associated with heap issues, but I did double my heap memory the other day, and normally it doesn't cause processes like this to just hang in the background doing nothing. That other application that hung about 10 times during this period is what suggests that it could be a heap issue because I know that app to be the first to go under stress (it is not a service). That app does an average of over 50 DNS lookups and has a lot more latency than Sniffer does, so it is remarkable that Sniffer hung 100 times and that app only hung 10 times. That suggests to me that maybe something better could be done in terms of cleaning up these processes. I'll keep the server in this state until the evening in the event that you want to take a look at it. Thanks, Matt Pete McNeil wrote: Hello Matt, Thursday, June 14, 2007, 12:44:32 PM, you wrote: > I also had about 10 errors waiting to be cleared from another application, but probably because of the way that Sniffer works (as a service or something related), the Sniffer processes are just hung without a prompt. I saw this last week also. I have Declude set for 200 processes, so it probably reached 300 when the first 100 hung, and then it stayed with those 100 hung. Is there anything that can be done in Sniffer to kill off these hung processes in an automated and proactive manner? I recently upgraded to the latest version and I was probably a version or two behind, and I don't recall this happening before. It seems very unlikely that SNF instances would be hung -- they will either time-out themselves or be killed off by Declude. Please let us know if there are any errors in your SNF log. Also - check the SNF working directory to make sure you don't have a lot of old job files hanging around. That can cause SNF instances to relax their timing based on the assumption there is a high system load -- with relaxed timing they will stay around longer waiting for results. If you find that you do have a lot of old job files hanging around then you should clean them out to get things going normally again. Stop SMTP Wait for all jobs to finish Stop your persistent instance Remove all left-over job files (QUE, WRK, FIN, ABT, XXX, SVR) Restart your persistent instance Restart SMTP Also, presuming you have a persistent instance - make sure that is still running. If that had failed for some reason then you might be running now in peer-server mode which will be a bit slower than persistent mode. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Spammers turning to PDF attachments?
I saw some AFF (419/Nigerian) stuff that came in PDF format. That one is going to be a real pain if they keep it up since those messages come from legitimate webmail providers, and lack all but a few words of text in the body. It's hard to get a consistent pattern out of that. I'm not nearly as worried about the zombies. Matt Colbeck, Andrew wrote: See this article at the Internet Storm Center: http://isc.sans.org/diary.html?storyid=3012 Pump and dump scams now in PDF Published: 2007-06-20, Last Updated: 2007-06-20 21:33:39 UTC by Maarten Van Horenbeeck (Version: 1) Apparently the groups behind what we know as pump and dump spam have found a new way to bypass spam filters. As of yesterday, we've been observing e-mails with bogus text, often in german, each with a PDF in attachment... Andrew. # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Greeting Malware Spike Graph
Pete, The first one of these hit our system at 8:45 a.m. ET yesterday (linked Malware). FYI, this is the same guy that is sending the PDF stock spam, and was responsible for the 3 other large virus seedings in the last 6 months. Matt Pete McNeil wrote: Hello Sniffer Folks, Vertical Wall Of Spam # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: re subscriptions to list
All auto-responders should be burnt in hell Have a nice day :) Matt Pete McNeil wrote: Regarding this thread and to nobody in particular: I would like to say a word or two before this gets out of hand. Our policy on this list is to provide the answers needed no matter how obvious or well posted those answers may be. Emotionally negative responses are discouraged and generally not useful. RTFM type answers should be emotionally neutral, should summarize a quick answer, and should provide a link to TFM. For whatever reason, these kinds of requests are made and these kinds of questions are asked. The folks who make these requests or ask these questions - no matter how obvious - need help. The best thing we can do is provide that help. Keep in mind also that these messages are archived so that they remain searchable on the 'web. This means that any solutions we post here, including references to obvious or well posted answers, serve to make those answers easier to find. Please: Be kind and helpful, or stay away from the send button. I can't remember the number of times something simple and obvious baffled me when I needed it least -- and I'm sure many of us have had similar moments.* A simple answer to an obvious question can go a long way in a positive direction. Please help us keep this forum active, positive, and informative. Thanks, _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: Excessive amounts of Foreign language spam
I recommend not doing this. There are enough possibilities of legitimate discussion and data being transmitted by E-mail that could contain this string that you would be best off being more specific to the encoding of the actual message. Target the header charset tag and you should be good to go if you want to ban most all of it, but keep in mind that legitimate E-mails from Cyrillic based systems can still send messages written in English using this and other charactersets, so make sure there is nothing legitimate at all coming from countries that would use this before using it. Matt Kim W. Premuda wrote: We use this in Declude to filter messages containing Cyrillic (Russian) characters... ANYWHERE 10 CONTAINSkoi8-r Kim W. Premuda FastWave Internet Services, Inc. 858-487-1414 [EMAIL PROTECTED] -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Rick Hogue Sent: Wednesday, December 26, 2007 7:20 AM To: Message Sniffer Community Subject: [sniffer] Excessive amounts of Foreign language spam I have been getting a large amount of foreign language email appears to be Russian but not sure as I do not speak it. What is being done about this kind of spam? I am on Imail 8.21 and I am not using the latest beta of Sniffer. I do not use other methods other than a couple from my Declude engine including my own blacklist, but there are so many different IP addresses on this it is very hard to keep up with. Help! Simplified Association Management Systems Rick Hogue P.O. Box 18304 Louisville, KY 40261 1-502-459-3100 office http://www.samprogram.com # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> --- [This E-mail scanned for viruses by Declude Virus] --- [This E-mail scanned for viruses by Declude Virus] # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]> # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: XYNTService -- Any Problems?
SRVANY works perfectly and is free with Windows. Why not use that? Matt Pete McNeil wrote: Hello Sniffer Folks, We are working on an installer for the command-line version of SNF V3.0. We are considering re-distributing XYNTService to setup the SNFServer.exe as part of the installation. There are some rumblings here and there about XYNTService not working the same way on some versions of Windows. If any of you have experience with XYNTService then we would sure like to hear about it. If anyone knows of an alternative that we can bundle with our installer then please let us know that too... (Why don't you just write something?? -- ) We'd prefer not to reinvent that particular wheel right now -- not that it's hard, just that it's not necessary and we'd rather do other important stuff. Thanks! _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: XYNTService -- Any Problems?
I'm sure that I don't speak for everyone, but I would tend to avoid third-party service systems, and this would also expose Sniffer to the potential pitfalls of that software. You could provide directions on how to install SRVANY, and then have a script that completes the process once the executables are on the system. That would be my short-term recommendation. In the long-term, I would do your own service as opposed to use someone else's container. I would not recommend distributing XYNTService until you have trialled that for several months with a range of systems. The work of properly testing this is possibly more work than creating your own service. All IMO of course. Matt Pete McNeil wrote: Hello Matt, Friday, May 9, 2008, 3:57:42 PM, you wrote: SRVANY works perfectly and is free with Windows. Why not use that? We can't redistribute SRVANY with our installer and we can't be sure it will be present on all systems. I've been using SRVANY every time I do an install for somebody... but we want something that we can deliver with the installer so it can be a (more or less) one click process. _M
[sniffer] Re: It's official. SNF Version 3.0 is Ready!
Pete, Now that you got that taken care of, can you give us an idea when you expect 4.0 to be released? Matt Pete McNeil wrote: Hello Sniffer folks, Back in Q1 we were sure we'd be ready with the new SNF after nearly a year of testing on both large and small systems. What a surprise! After publishing the first release candidate we went from version 1-5 to version 2-27 at a breathtaking pace! Thank you to everyone who has tested, poked, prodded, and twisted the new SNF -- not to mention keeping up with all of those updates during the final phase of testing. I can't imagine getting to this point without your patience, trust, attention to detail, and persistence! Bravo! Without further fanfare: Today the latest release candidate becomes the official production release of Message Sniffer (SNF) Version 3.0. The changes: -- Minor updates to readme files. -- Changed the build / version information and recompiled. -- Removed redundant comments from the configuration file. We have been bug free for more than 2 months with several hundred systems using the new engine. You can download the latest distributions from this page: http://www.armresearch.com/products/index.jsp You may also notice that we've published our new web site! There are a few bits of documentation still under construction here and there, but we're well on our way to filling those in along with a stream of continues improvements and additions based on our work with you! Once again, Thanks to everyone for a fantastic job! Thanks for all of your support, comments, and efforts! As always we're hear to help. Now, onward to the next upgrade... always work to do ;-) Cheers! _M # This message is sent to you because you are subscribed to the mailing list . To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
[sniffer] Re: It's official. SNF Version 3.0 is Ready!
Pete, Glad you got the joke. I'll allow you a a little time to take your mind off of the future :) Thanks, Matt Pete McNeil wrote: Hello Matt, Thursday, June 26, 2008, 4:21:42 PM, you wrote: Pete, Now that you got that taken care of, can you give us an idea when you expect 4.0 to be released? Hehehe. We're not close enough to that to be remotely accurate, but I will tell you that I'd like it to be within something approaching a year. There are a lot of continuous upgrades to do on the back-end that will unleash and enhance V3's power and provide additional tools and products along the way -- plus plenty of work helping new third party products get off the ground w/ SNF inside... So we'll be plenty busy and we'll keep you posted. _M
[sniffer] Re: Sniffer Helper App?
Steve, Since this hasn't yet been mentioned, try Alligate (www.alligate.com). It does selective greylisting (only greylists things that look spammy), and also will validate your users' addresses and do things like country blocking/tarpitting/greylisting. Only one zombie spammer survives greylisting, and after you dump all of that plus validate addresses, you will reduce your traffic down to a point where it is only 1/3 spam. If you only reject bad addresses and clear abuse (many bad addresses in one connection for instance), you can do this with 99.% accuracy. I'm not lying about that either. The only things that fail selective greylisting will be black boxes that don't spool E-mail, and if you give a wide retry time, you will likely allow future attempts from a black box that happens to get greylisted. Selective greylisting is far superior to regular greylisting since it is rarely triggered against legitimate E-mail. I dump around 93% of all connections to my servers and I don't need to falsely trust a single source of data such as SpamCop to achieve those results. I then leave the heavy lifting to a secondary filtering system where the heavy lifting is performed. Alligate requires almost no resources, though you should dedicate a box to it so that other things don't step on it's feet. Matt Steve Guluk wrote: Hello, I run iMail 9.0 and would like a program that can do GeoIP to screen foreign countries before they even get to iMail. I used to use MXGuard (still have an active license) but my server could not handle the CPU draw. I moved to eWall which really has some great potential as it is a nice light gateway client that works with Sniffer but it also crashes and has a few other problems (this program also introduced me to GeoIP). Any other suggestions as I am beat after trying to get some decent spam relief as well as relief from an aging server. My server is an AMD 2.0 with Raid and 2 gigs of Ram It's faired well over the last couple years but the spam levels ramping up are starting to take their toll and I don't want to move to a new server just yet. eWalls got me spoiled on the GeoIP feature where it polls a DB for country info based on the incoming IP and can delete emails before they reach iMail. Any suggestions on what I should consider to help with spam and also use Sniffer. Is Declude worth while? Some other light gateway like eWall ? Thanks in advance for any suggestions, *Steve Guluk* SGDesign (949) 661-9333 ICQ: 7230769
[sniffer] Slow processing times, errors
Pete, I've had many recent incidences where, as it turns out, SNFclient.exe takes 30 to 90 seconds to respond to every message with a result code (normally less than a second), and as a result backs up processing. Restarting the Sniffer service seems to do the trick, but I only tested that for the first time today after figuring this out. I believe the events are triggered by updates, but I'm not sure as of yet. Updates subsequent to the slow down do not appear to fix the situation, so it seems to be resident in the service. When this happens, my SNFclient.exe.err log fill up with lines like this: 20130627155608, arg1=F:\\proc\work\D6063018a2550.smd : Could Not Connect! At the same time, my Sniffer logs start showing frequent "ERROR_MSG_FILE" results on about 1/8th of the messages. I'm currently using the service version 3.0.2-E3.0.17. It's not entirely clear to me what the most current one is. Any suggestions as to the cause or solution? Thanks, Matt # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Re: Slow processing times, errors
Darin, I'm not seeing that sort of thing. With 3.x, there doesn't appear to be any extraneous file creation in the Sniffer program directory, and never any TMP files in my spool. I do not have Sniffer modifying headers, so that may be different on our systems. Matt On 6/27/2013 5:25 PM, Darin Cox wrote: When we had sluggish performance similar that yours, resulting in numerous sniffer .tmp files in the spool, the cause was eventually traced to a proliferation of files in the sniffer directory. Clearing them out brought performance back up to normal. Darin. *From:* e...@protologic.com <mailto:e...@protologic.com> *Sent:* Thursday, June 27, 2013 5:17 PM *To:* Message Sniffer Community <mailto:sniffer@sortmonster.com> *Subject:* [sniffer] Re: Slow processing times, errors We were experiencing this several days ago and couldn't find a fix that worked or worked for long. We uninstalled SNF and reinstalled and have not detected a problem since. I will check the logs and report back if I see anything intermittent. Sent using SmarterSync Over-The-Air sync for iPad, iPhone, BlackBerry and other SmartPhones. May use speech to text. If something seems odd please don't hesitate to ask for clarification. E.&O.E. On 2013-06-27, at 2:06 PM, Matt wrote: > Pete, > > I've had many recent incidences where, as it turns out, SNFclient.exe takes 30 to 90 seconds to respond to every message with a result code (normally less than a second), and as a result backs up processing. Restarting the Sniffer service seems to do the trick, but I only tested that for the first time today after figuring this out. > > I believe the events are triggered by updates, but I'm not sure as of yet. Updates subsequent to the slow down do not appear to fix the situation, so it seems to be resident in the service. When this happens, my SNFclient.exe.err log fill up with lines like this: > > 20130627155608, arg1=F:\\proc\work\D6063018a2550.smd : Could Not Connect! > > At the same time, my Sniffer logs start showing frequent "ERROR_MSG_FILE" results on about 1/8th of the messages. > > I'm currently using the service version 3.0.2-E3.0.17. It's not entirely clear to me what the most current one is. > > Any suggestions as to the cause or solution? > > Thanks, > > Matt > > > # > This message is sent to you because you are subscribed to > the mailing list . > This list is for discussing Message Sniffer, > Anti-spam, Anti-Malware, and related email topics. > For More information see http://www.armresearch.com > To unsubscribe, E-mail to: > To switch to the DIGEST mode, E-mail to > To switch to the INDEX mode, E-mail to > Send administrative queries to >
[sniffer] Re: Slow processing times, errors
This is an automated message from the mailing list software. In order to unsubscribe you must issue the command "please unsubscribe". # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Re: Slow processing times, errors
d : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D8638016d42db.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D862b00e64292.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D8638017342e0.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D865d01804393.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D8631017742b1.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
20130627183820, arg1=F:\\proc\work\D85b100e33f7f.smd : XCI
Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek
I am looking to retool presently just because it's time. So if you are
convinced that this is due to low resources, don't concern yourself with it.
Matt
On 6/28/2013 10:36 AM, Pete McNeil wrote:
On 2013-06-27 20:01, Matt wrote:
I'm attaching a snippet of my log. About 100 lines past the start,
where you see a smattering of error messages, you then see a large
block of them while the Sniffer service is restarting, and then after
that no errors at all. There have in fact been no errors at all in
several hours since this restart of Sniffer.
I can promise you that the error you're reporting comes directly from
a problem with the file system. Here is the code where that error is
generated. Put simply the code tries to open the file and determine
it's size. If that doesn't work it throws the ERROR_MSG_FILE exception
in one of two forms -- that's what ends up in the log.
| *try* {/// Try opening the message file.
/|/||/| MessageFile.open(MessageFilePath.c_str(), ios::in |
ios::binary);/// Open the file, binary mode.
/|/||/| MessageFile.seekg(0, ios::end);/// Find the end of the file,
/|/||/| MessageFileSize = MessageFile.tellg();/// read that position as
the size,
/|/||/| MessageFile.seekg(0, ios::beg);/// then go back to the
beginning.
/|/||/| MyScanData.ScanSize = MessageFileSize;/// Capture the message
file size.
/|/||/| }
|| *catch*(...) {/// Trouble? Throw FileError.
/|/||/| MyRulebase->MyLOGmgr.logThisError(/// Log the error.
/|/||/| MyScanData,*"scanMessageFile().open"*,
|| snf_ERROR_MSG_FILE,*"ERROR_MSG_FILE"*
|| );
|| *throw* FileError(*"snf_EngineHandler::scanMessageFile()
Open/Seek"*);
|| }
||
|| *if*(0 >= MessageFileSize) {/// Handle zero length files.
/|/||/| MessageFile.close();/// No need to keep this open.
/|/||/| MyRulebase->MyLOGmgr.logThisError(/// Log the error.
/|/||/| MyScanData,*"scanMessageFile().isFileEmpty?"*,
|| snf_ERROR_MSG_FILE,*"ERROR_MSG_FILE"*
|| );
|| *throw* FileError(*"snf_EngineHandler::scanMessageFile()
FileEmpty!"*);
|| }
|
Another clue is that in the log snippet you provide, there are hints
of a problem brewing when there are sporadic instances of this error.
Then, when there is a large block -- virtually all requests to open
the files for scan are rejected by the OS. Either something made those
files unavailable, or the OS was unable to handle the request. I find
it interesting also that the time required to report the error started
at about 172 milliseconds and continued to climb to 406, 578, and then
656 before the restart.
SNF does not make log entries in the classic log during a restart, by
the way.
Note also the timestamps associated with these events and you can see
that the event was precipitated by a dramatic rise in message rates.
The first part of your log seems to indicate about 7-10 messages per
second. During the large block of errors, the message rate appears to
have been in excess of 120 (I counted approximately 126 at timestamp
20130627183819). That's an increase at least an order of magnitude
higher than the rate that was causing sporadic errors.
I suspect based on the data you have provided that something on your
system generated a very large spike of activity that your IO subsystem
was unable to manage and this caused snf scans to fail because snf was
unable to open the files it was asked to scan.
Your restart of SNF apparently coincided with the event, but since all
of the SMD file names are unique during the event, and since SNF has
no way to generate scan requests on it's own, SNF does not appear to
have been the cause of the event in any way. It was able to record the
event, none the less.
So the question in my mind now is:
* Is there a way to improve your IO subsystem so that it can gain some
headroom above 10 msg/sec?
* What caused the sudden dramatic spike th
[sniffer] Re: Slow processing times, errors
Pete, Just after the restart of the Sniffer service, times dropped back down into the ms from 30+ seconds before, so what I am saying is that if I/O was the issue, it was merely the trigger for something that put the service in a bad state when it started. I/O issues are not persistent, but could happen from time to time I'm sure. Restarting Sniffer with a backlog of 2,500 messages and normal peak traffic will not re-trigger the condition, and I press Declude to run up to 300 messages at a time in situations like that, and the CPU's are pegged until the backlog clears. In the past, I restarted the whole system, not knowing why it worked. During normal peak times (without bursts), the Declude is processing about 125 messages at a time which take an average of 6 seconds to fully process, and therefore Sniffer is probably handling only about 10 messages at a time (at peak). Since 5/22 I have seen 4 or 5 different events like this, and I confirmed that they are all present in the SNFclient.exe.err log. Matt On 6/28/2013 12:41 PM, Pete McNeil wrote: On 2013-06-28 12:10, Matt wrote: I am looking to retool presently just because it's time. So if you are convinced that this is due to low resources, don't concern yourself with it. Ok. It makes sense that the ~200 messages all at once could have happend at the restart. SNFClient will keep trying for 30-90 seconds before it gives up and spits out it's error file. That's where your delays are coming from. SNF itself was clocking only about 100-800ms for all of the scans. The error result you report is exactly the one sent by SNF -- that it was unable to open the file. I am very sure this is resource related -- your scans should not be taking the amount of time they are and I suspect most of that time is eaten up trying to get to the files. The occasional errors of the same time are a good hint that IO is to blame. The new spam that we've seen often includes large messages -- so that's going to put a higher load on IO resources -- I'll bet that the increased volume and large message sizes are pushing IO over the edge or at least very close to it. Best, _M # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Re: Slow processing times, errors
I'll certainly look more closely next time. Hopefully I'll be migrated before this happens again :) Matt On 6/28/2013 1:44 PM, Darin Cox wrote: How about running performance monitor to watch disk I/O, mem, cpu, page file, etc. over time in the hopes of catching one of the events? Darin. *From:* Matt <mailto:for...@mailpure.com> *Sent:* Friday, June 28, 2013 12:10 PM *To:* Message Sniffer Community <mailto:sniffer@sortmonster.com> *Subject:* [sniffer] Re: Slow processing times, errors Pete, I'm near positive that it's not system resources that are causing Sniffer to not be able to _access the files_. I believe these errors are a symptom and not the cause. You have to keep in mind that on the messages that don't throw errors, they were taking 30-90 seconds to scan, but immediately after a restart it was under 1 second. The system stayed the same, it was just the state of the service that was off in a bad way. I did add a larger client about a month ago around the time that this started, which did inch up load by between 1% and 5% I figure, but I can't say for sure that the two things are connected. I've seen much bigger changes however in spam volumes from single spammers. I have looked at my SNFclient.exe.err log and found that the previous slowdowns were all represented in this file, and nothing else really since a smattering in 2012 of other stuff. I believe that I/O could be the trigger, or general system load, but the error in the service that misses opening some files, and is otherwise slower than normal by 100 times, will persist when everything else is fine again. I figure that this is all triggered by a short-term lack of resources or a killer message type of issue that does something like run away with memory. Certainly there were no recent changes on the server prior to this starting to happen, including Sniffer itself which has been perfectly solid up until 5/22. Regarding the ERROR_MSG_FILE batch that I sent you in that log, it did happen exactly when I restarted Sniffer, and in fact the SNFclient.exe.err log showed a different error while this was happening, and maybe this will point you to something else? That log says "Could Not Connect!" when the regular Sniffer log shows "ERROR_MSG_FILE" about 1/8th of the time while in a bad state. When I restarted the Sniffer service, the regular log showed a bunch of "ERROR_MSG_FILE" in a row, but the SNFclient.exe.err log below shows "XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek". You can match the message ID's with the other log that I provided. I believe that block of messages was already called to SNFclient.exe, but the Sniffer service haddn't yet responded, and so they were dumped as a batch into both logs during shut down of the service. 20130627183807, arg1=F:\\proc\work\D862600e64269.smd : Could Not Connect! 20130627183808, arg1=F:\\proc\work\D86440177431f.smd : Could Not Connect! 20130627183808, arg1=F:\\proc\work\D861200ce41ce.smd : Could Not Connect! 20130627183809, arg1=F:\\proc\work\D864401734321.smd : Could Not Connect! 20130627183809, arg1=F:\\proc\work\D861400da41e3.smd : Could Not Connect! 20130627183810, arg1=F:\\proc\work\D862600d7425f.smd : Could Not Connect! 20130627183811, arg1=F:\\proc\work\D864a00e94346.smd : Could Not Connect! 20130627183811, arg1=F:\\proc\work\D8615019b41f4.smd : Could Not Connect! 20130627183813, arg1=F:\\proc\work\D862900e94282.smd : Could Not Connect! 20130627183815, arg1=F:\\proc\work\D863d01584306.smd : Could Not Connect! 20130627183817, arg1=F:\\proc\work\D86030158416f.smd : Could Not Connect! 20130627183818, arg1=F:\\proc\work\D862300e94255.smd : Could Not Connect! 20130627183819, arg1=F:\\proc\work\D862900e64281.smd : Could Not Connect! 20130627183819, arg1=F:\\proc\work\D864b00d74357.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D864800d7433c.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D861901734205.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D861d01774230.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D8641016d4310.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D865000e64363.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() Open/Seek 20130627183819, arg1=F:\\proc\work\D865000e14361.smd : XCI Error!: FileError snf_EngineHandler::scanMessageFile() O
[sniffer] Re: Slow processing times, errors
Eric, I'm guessing based on what you were seeing, that it was unrelated to what I was seeing. Sniffer never actually died, it just got over 100 times slower, and 1/8th of the time it timed out. This never happened before 5/22, and this same server has been there for years, and the same installation of Sniffer for 2 years or so. I would think that if the issue was I/O (under normal conditions), it would have happened before 5/22 as there were clearly bursty periods often enough that my own traffic didn't change dramatically enough so that it happened 4 to 5 times in one month. The server itself could have some issues that could be causing this. Maybe the file system is screwy, or Windows itself, or memory errors, or whatever. Matt On 6/28/2013 2:12 PM, E. H. (Eric) Fletcher wrote: Matt: I mentioned in a previous post that we had experienced something similar at about that time and resolved it a day or so later by re-installing sniffer when service restarts, reboots and some basic troubleshooting did not give us the results we needed. At this point that still seems to have been effective (about 5 days now). At the time, we did move things around to see whether it was related to the number of items in the queue or anywhere else within the structure of the mail system and found it made no difference. A single item arriving in an empty Queue was still not processed. CPU utilization was modest (single digit across 4 cores) and disk I/O was lighter than usual as it took place over a weekend. Memory utilization was a little higher than I'd like to see, we are addressing that now. Following a suggestion from another ISP, we moved the spool folders onto a RAM drive a couple of months ago. That has worked well for us, we did rule it out as the source of the problem by moving back onto the conventional hard disk during the last part of the troubleshooting and for the first hour or two following the reload. We are processing on the Ramdisk now and have been for over 4 days again. For what it's worth . . . Eric -Original Message- From: Message Sniffer Community [mailto:sniffer@sortmonster.com] On Behalf Of Matt Sent: Friday, June 28, 2013 10:32 AM To: Message Sniffer Community Subject: [sniffer] Re: Slow processing times, errors Pete, Just after the restart of the Sniffer service, times dropped back down into the ms from 30+ seconds before, so what I am saying is that if I/O was the issue, it was merely the trigger for something that put the service in a bad state when it started. I/O issues are not persistent, but could happen from time to time I'm sure. Restarting Sniffer with a backlog of 2,500 messages and normal peak traffic will not re-trigger the condition, and I press Declude to run up to 300 messages at a time in situations like that, and the CPU's are pegged until the backlog clears. In the past, I restarted the whole system, not knowing why it worked. During normal peak times (without bursts), the Declude is processing about 125 messages at a time which take an average of 6 seconds to fully process, and therefore Sniffer is probably handling only about 10 messages at a time (at peak). Since 5/22 I have seen 4 or 5 different events like this, and I confirmed that they are all present in the SNFclient.exe.err log. Matt On 6/28/2013 12:41 PM, Pete McNeil wrote: On 2013-06-28 12:10, Matt wrote: I am looking to retool presently just because it's time. So if you are convinced that this is due to low resources, don't concern yourself with it. Ok. It makes sense that the ~200 messages all at once could have happend at the restart. SNFClient will keep trying for 30-90 seconds before it gives up and spits out it's error file. That's where your delays are coming from. SNF itself was clocking only about 100-800ms for all of the scans. The error result you report is exactly the one sent by SNF -- that it was unable to open the file. I am very sure this is resource related -- your scans should not be taking the amount of time they are and I suspect most of that time is eaten up trying to get to the files. The occasional errors of the same time are a good hint that IO is to blame. The new spam that we've seen often includes large messages -- so that's going to put a higher load on IO resources -- I'll bet that the increased volume and large message sizes are pushing IO over the edge or at least very close to it. Best, _M # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Re: Slow processing times, errors
I just looked through my history of the SNFclient.exe.err log and I was a bit off. There were two small occurrences before 5/22, and both were so small they were without a doubt unnoticeable. Here's my history since installation on 4/10/2012 with the number of occurrences of "Could Not Connect!" noted: 11/19/2012 (949 occurrences) - * No backup in excess of 2 minutes occurred 4/26/2012 (181 occurrences) - * No backup in excess of 2 minutes occurred 5/22 (3,568 occurrences) - * No backup in excess of 2 minutes occurred 5/31 (13,745 occurrences) 6/3 (2,630 occurrences) - * No backup in excess of 2 minutes occurred 6/11 (26,194 occurrences) 6/13 (28,633 occurrences) 6/14 (83,342 occurrences) 6/17 (5,952 occurrences) - * No backup in excess of 2 minutes occurred 6/21 (10,894 occurrences) 6/27 (34,959 occurrences) At peak times, we are probably doing between 30,000 and 40,000 messages per hour. When this hit at peak hour yesterday, the server was backed up 2 minutes within 10 minutes of this starting. If it happens outside of business hours, it's likely not seen at all. The backup was resolved within 1 hour using different Declude settings, but Sniffer wasn't restarted for 2 hours and 45 minutes after the problem started. So those ~35,000 occurrences were from 165 minutes of email. After looking through my logs, it doesn't seem that I rebooted hardly any of these times. I tend to not want to take the server down outside of rush hour. I do tweak Declude for rush processing when backed up, so Declude is restarted which will stop sending files to Sniffer for a period of time. It seems that Sniffer is mostly healing itself without restarting it's service, though maybe the updates will cause a service restart/reset? I also checked and found that we added that larger client on 6/1, and outside of that, customer counts have been fairly consistent. Matt On 6/28/2013 2:56 PM, e...@protologic.com wrote: Matt: Coincidentally (I hope) this happened to us on the 22nd also. It did not stop working completely although we didn't get the throughput you did. We also saw the messages indicating it was not able to open the file. Pretty much the same message as in your first post and not one I've seen before. Eric Sent using SmarterSync Over-The-Air sync for iPad, iPhone, BlackBerry and other SmartPhones. May use speech to text. If something seems odd please don't hesitate to ask for clarification. E.&O.E. On 2013-06-28, at 11:39 AM, Matt wrote: > Eric, > > I'm guessing based on what you were seeing, that it was unrelated to what I was seeing. Sniffer never actually died, it just got over 100 times slower, and 1/8th of the time it timed out. This never happened before 5/22, and this same server has been there for years, and the same installation of Sniffer for 2 years or so. I would think that if the issue was I/O (under normal conditions), it would have happened before 5/22 as there were clearly bursty periods often enough that my own traffic didn't change dramatically enough so that it happened 4 to 5 times in one month. > > The server itself could have some issues that could be causing this. Maybe the file system is screwy, or Windows itself, or memory errors, or whatever. > > Matt > > > On 6/28/2013 2:12 PM, E. H. (Eric) Fletcher wrote: >> Matt: >> >> I mentioned in a previous post that we had experienced something similar at >> about that time and resolved it a day or so later by re-installing sniffer >> when service restarts, reboots and some basic troubleshooting did not give >> us the results we needed. At this point that still seems to have been >> effective (about 5 days now). >> >> At the time, we did move things around to see whether it was related to the >> number of items in the queue or anywhere else within the structure of the >> mail system and found it made no difference. A single item arriving in an >> empty Queue was still not processed. CPU utilization was modest (single >> digit across 4 cores) and disk I/O was lighter than usual as it took place >> over a weekend. Memory utilization was a little higher than I'd like to >> see, we are addressing that now. >> >> Following a suggestion from another ISP, we moved the spool folders onto a >> RAM drive a couple of months ago. That has worked well for us, we did rule >> it out as the source of the problem by moving back onto the conventional >> hard disk during the last part of the troubleshooting and for the first hour >> or two following the reload. We are processing on the Ramdisk now and have >> been for over 4 days again. >> >> For what it's worth . . . >> >> Eric >> >> >&g
[sniffer] Re: What is your oldest production CPU?
Intel 5400 series Xeon here. But don't forget virtualization. I'm not sure what CPU virtualization does to targeting your code. Matt On 12/27/2013 9:43 AM, Pete McNeil wrote: Hello Sniffer Folks, We would like to know what your oldest production CPU is. When building new binaries of SNF or it's utilities we would like to select the newest CPU we can without leaving anybody behind. We're also evaluating whether we should split binaries into a "compatible" version base on Intel i686 (or equivalent AMD), and a "current" version based on Intel Core2 (or equivalent AMD). Please respond here. Thanks for your time!! _M # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Re: What is your oldest production CPU?
On a VMware ESXi 5.x box with a virtual machine version 8, and physical E5-2689 CPU's I see the following: On a Windows 2003 32-bit host, Device Manager shows that it is x86 family 6 model 45. On a Windows 2008 R2 64-bit host, Device Manager shows that it is Intel64 family 6 model 45. Windows does know the processor version as well, though that's just meta information I believe and may not be reliable. There are of course several other popular flavors of virtualization that I am not familiar with. Matt On 12/27/2013 3:58 PM, Pete McNeil wrote: On 2013-12-27 15:45, Matt wrote: Intel 5400 series Xeon here. But don't forget virtualization. I'm not sure what CPU virtualization does to targeting your code. That's a good point The processor should be specified in the VM profile and if I recall correctly it is typically defaulted to the processor of the VM host. I should look closer at this -- but would like some feedback. Thanks, _M # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
Re: [sniffer] rule idea
Please don't, my Grandmother probably couldn't get through then :) The more solid the basis for the rules, the higher the score I can give to the test. Most spammers nowadays will have a time that is only off by a few hours when they hard code the headers for a zombie attack, however once you start getting out several days, or even months or years, the likelihood that this is not spam increases. There's no good rule of thumb IMO. Scott from Declude has been testing this idea out for several months now without releasing the functionality to the public, probably because it's unreliable I'm thinking. It it was to be scored, I would much rather it be separate from other tests. Matt Herb Guenther wrote: At one time we had floated the idea of a rule that would mark any email that was more than 24-48 hrs ahead or behind the actual current time and date as spam. I just got two "You've been invited to a blind date" messages that were dated last summer. 99.9% of these off date messages are spam, and anyone real who has there date that far off should fix it. Would it be hard to add such a rule to sniffer? Herb -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] F-Prot and netsky
F-Prot prompted me for an upgrade of the program when I logged in this morning (full download and install process). I assume that this is meant to take care of the issue in not catching Mydoom.F. Before that I was showing definitions that were last updated on the 18th. Everyone should log onto their servers and run the updater if they aren't immediately prompted. If I'm correct, it also took several days for them to fix an issue with Mimail and that required a version upgrade. Matt Madscientist wrote: At 09:32 AM 2/24/2004, you wrote: I was wondering if anyone else is using F-prot for their virus engine in declude, and what they now think about it. Netsky was discovered on the 18th, and F-Prot actually had it posted on their website as being discovered by them on the 19th. But they didn't update their definition files to actually catch it until early this morning. This meant that netsky ran rampant under F-Prots nose for 6 days. I feel this is completely unacceptable, and I am going to change my virus engine this week unless someone can tell me that there is a good reason why I shouldn't. Any ideas or feedback from someone using F-Prot? Thanks We recently abandoned McAfee for F-Prot. F-Prot is much more efficient and stable on our NT test platform. Though I am not pleased with the delays you mention, I'm also not willing to throw them out given the alternatives and at this point I consider the delay an aberration rather than a systemic flaw. A better strategy to dropping F-Prot, in my opinion, is to also include alternatives - since diversity is better protection anyway and the costs are well within reason. My $0.02. _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation. Chief SortMonster, www.SortMonster.com. Vox 703-406-2016, Fax 703-406-2017 -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
[sniffer] Efficiency request for Declude installations and maybe others.
Pete, Would it be possible to add the ability to tell Sniffer not to run when Declude passes it a weight? I understand that we tend to weight differently, so this would require two switches to pull off, but it would help with efficiency. The command line in a Declude config would be something like the following: "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25" Declude will replace %WEIGHT% with the weight before calling Sniffer, and the next entry, 25, would tell Sniffer to not run if the the current weight was equal to 25 or above. Some people also do some whitelisting with external programs like Sniffer, in which case it might be a good idea to add a weight under which you would also skip processing, i.e. "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25 -10" On my system, the addition of the weight variable would cause Sniffer to not be run on between 50% and 75% of the E-mail depending on the day of the week, so this could be a very big deal. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Efficiency request for Declude installations andmaybeothers.
Pete, That is of course an option. He did make this variable available for such uses, but that's clearly not the only way to skin this cat. I think the trick here is determining which external tests to run according to which skip weight setting. Different external tests would have potentially different trigger levels instead of one main trigger. Certainly Declude could provide this, but it would require a setting for each individual test call. I would imagine that he could add new columns to the external test call like so: - Current Format - SNIFFER-GENERALexternal063 "C:\IMail\Declude\Sniffer\programname.exe mycode"60 - Potential New Format - SNIFFER-GENERALexternal063 "C:\IMail\Declude\Sniffer\programname.exe mycode"60 25 -10 The first extra column would be the skip-if-equal-to-or-above-weight setting and the second extra column would be the skip-if-equal-to-or-below-weight setting. Scott seems extra busy right now, but he obviously monitors this list so I'll leave it here for now. Declude has of course made huge strides in terms of efficiency. It would be nice if possible for Sniffer + Declude to make use of the rulebase loading method mentioned earlier as well. Thanks, Matt Madscientist wrote: I think the best place to handle this would be within Declude since Declude has all of the information and has control over which tests get called. I have seen some discussions on not executing certain tests based on weights that are reached. Probably the most reasonable tests to modulate in this way would be external tests - but Scott's the best one to answer that question. Hope this helps, _M PS: If you are working to mitigate a load problem that may be handled shortly. We will be working on a feature to "nail up" a Message Sniffer instance so that all other instances will run as clients (thus not loading the rulebase). This should significantly reduce system loads and may make complex modulation mechanisms unnecessary. At 01:01 PM 3/5/2004, you wrote: Pete, Would it be possible to add the ability to tell Sniffer not to run when Declude passes it a weight? I understand that we tend to weight differently, so this would require two switches to pull off, but it would help with efficiency. The command line in a Declude config would be something like the following: "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25" Declude will replace %WEIGHT% with the weight before calling Sniffer, and the next entry, 25, would tell Sniffer to not run if the the current weight was equal to 25 or above. Some people also do some whitelisting with external programs like Sniffer, in which case it might be a good idea to add a weight under which you would also skip processing, i.e. "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25 -10" On my system, the addition of the weight variable would cause Sniffer to not be run on between 50% and 75% of the E-mail depending on the day of the week, so this could be a very big deal. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Efficiency request for Declude installations andmaybeothers.
Pete, Forgive me for pressing this, but I have a short-term need to reduce the processor utilization on my server because it also serves Web sites and I need to stay away from pegging my processor until I separate the spam blocking to a new machine. This is one of the things that I am looking at. I'm thinking that unless Scott was willing to work skip functionality into Declude in the short-term (and I don't assume his schedule revolves around my needs), it might be possible to write a simple handler script that calls Sniffer only if the %WEIGHT% variable is in a certain range. It seems that the only tricky part for a novice programmer like myself would be capturing the result code and passing it back to Declude, but I could find someone to help me with that. If you know of any other potential pitfalls please let me know (like message file/data handling), and if you could tell me what the full command line arguments were that are passed by Declude to Sniffer, and the format of the result code that is passed back, that would save me some research time (I'm thinking there is a file name in there unless the data is streamed). Also, if this was done, would it be possible to use such a script in a way that would allow Sniffer to run pre-loaded with the rulebase as a service on Windows? I'll share whatever I come up with if going the script route is necessary. Thanks, Matt Matt wrote: Pete, That is of course an option. He did make this variable available for such uses, but that's clearly not the only way to skin this cat. I think the trick here is determining which external tests to run according to which skip weight setting. Different external tests would have potentially different trigger levels instead of one main trigger. Certainly Declude could provide this, but it would require a setting for each individual test call. I would imagine that he could add new columns to the external test call like so: - Current Format - SNIFFER-GENERALexternal063 "C:\IMail\Declude\Sniffer\programname.exe mycode"60 - Potential New Format - SNIFFER-GENERALexternal063 "C:\IMail\Declude\Sniffer\programname.exe mycode"60 25 -10 The first extra column would be the skip-if-equal-to-or-above-weight setting and the second extra column would be the skip-if-equal-to-or-below-weight setting. Scott seems extra busy right now, but he obviously monitors this list so I'll leave it here for now. Declude has of course made huge strides in terms of efficiency. It would be nice if possible for Sniffer + Declude to make use of the rulebase loading method mentioned earlier as well. Thanks, Matt Madscientist wrote: I think the best place to handle this would be within Declude since Declude has all of the information and has control over which tests get called. I have seen some discussions on not executing certain tests based on weights that are reached. Probably the most reasonable tests to modulate in this way would be external tests - but Scott's the best one to answer that question. Hope this helps, _M PS: If you are working to mitigate a load problem that may be handled shortly. We will be working on a feature to "nail up" a Message Sniffer instance so that all other instances will run as clients (thus not loading the rulebase). This should significantly reduce system loads and may make complex modulation mechanisms unnecessary. At 01:01 PM 3/5/2004, you wrote: Pete, Would it be possible to add the ability to tell Sniffer not to run when Declude passes it a weight? I understand that we tend to weight differently, so this would require two switches to pull off, but it would help with efficiency. The command line in a Declude config would be something like the following: "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25" Declude will replace %WEIGHT% with the weight before calling Sniffer, and the next entry, 25, would tell Sniffer to not run if the the current weight was equal to 25 or above. Some people also do some whitelisting with external programs like Sniffer, in which case it might be a good idea to add a weight under which you would also skip processing, i.e. "C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25 -10" On my system, the addition of the weight variable would cause Sniffer to not be run on between 50% and 75% of the E-mail depending on the day of the week, so this could be a very big deal. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/H
Re: [sniffer] Efficiency request for Declude installationsandmaybeothers.
Pete, Right this second I'm not in a bind, but I've been growing business fast and I need to reserve enough processing power to scale up on the same box for the next couple of months, and then for a period of time, this might remain my backup MX. I throw a ton of tests at my setup, and while I do try to optimize within the boundaries, this is one place that has potential. I'm pretty certain that it will end up not calling Sniffer in excess of 50% of the time because the RBL's will have already reached a comfortable weight for me to stop processing on, and although Sniffer is very lean, this should help with the peaks. FYI, here are two screen captures of Task Manager showing what is going on with Sniffer and without Sniffer from a few moments ago. http://www.mailpure.com/with_sniffer.gif http://www.mailpure.com/without_sniffer.gif It's not bad, but during business rush hours I hit 80% to 100% every few seconds, and I need to protect my response times for my Web server for the time being. Also keep in mind that legitimate traffic causes the biggest hits because of large file attachments, running every filter in Declude, and often having in excess of 32 KB of content to scan. This is a dual PIII 1 GHz machine with RAID 5. Before going hog wild with Declude's custom filters and a second virus scanner, it would bounce around between 0% and 5%. Most of the processing used is the result of all of my custom filters in Declude. I'm interested in nailing-up Sniffer as you suggested, but I would prefer to be a beta tester as opposed to an alpha tester for obvious reasons. I'll trust your judgment on this, just tell me when. I think I watch my system close enough and have redundancies now that would protect me from a hiccup in beta software. I'm going to give the handler script a shot though because it looks simple and should at least save some loading times. No need to rush on my account, I'm just trying to stay one step ahead of things. If this works well enough, I might even code up a handler to disable virus checking on extra large files of certain types (not very common, but they produce long and large spikes). Thanks, Matt Pete McNeil wrote: Matt, I'm sorry you're in such a bind. We all get there from time to time. First, as far as I know you should be able to call a script from declude and pass it parameters. You may need to do this by calling cmd.exe - but you need to know that this isn't likely to save you processing on balance since the majority of Message Sniffer instances are going to load as client instances and will normally use very few resources - probably fewer than any scripting engine. None the less, if you want to try it you can find details on the command line parameters for Message Sniffer at this link: http://www.sortmonster.com/MessageSniffer/Help/TechnicalDetailsHelp.html#CmdLine One thing you might consider in the short term is to reduce the strength of your Message Sniffer rulebase. This is done by increasing the rule strength threshold. A small amount of additional spam will get through - but the rulebase will be much smaller and therefore faster to load. (The majority of Message Sniffer's resource use is in loading the rulebase - not in scanning the messages. Cellular peer-server technology helps mitigate this by allowing one instance to process messages for many before the rulebase is reloaded.) If you are looking for an immediate fix then I recommend this approach. Let me know off list (support@) if this is what you would like to do. Over the next couple of days (possibly beginning tonight) I will be working on a "Persistent" option for Message Sniffer which will allow you to "nail up" a server instance. In theory this shouldn't be hard to do but the implications can be tricky. If you are willing to work with potentially buggy code then I'll be happy to share early betas with you. Hope this helps, _M At 03:05 PM 3/8/2004, you wrote: Pete, Forgive me for pressing this, but I have a short-term need to reduce the processor utilization on my server because it also serves Web sites and I need to stay away from pegging my processor until I separate the spam blocking to a new machine. This is one of the things that I am looking at. I'm thinking that unless Scott was willing to work skip functionality into Declude in the short-term (and I don't assume his schedule revolves around my needs), it might be possible to write a simple handler script that calls Sniffer only if the %WEIGHT% variable is in a certain range. It seems that the only tricky part for a novice programmer like myself would be capturing the result code and passing it back to Declude, but I could find someone to help me with that. If you know of any other potential pitfalls please let me know (like message file/data handli
[sniffer] Question about the command line output
Pete, I just started testing things and I've run Sniffer on a test file containing two matches in it. When I run it from the command prompt it just simply logs the results and returns no value to the window. So my question is...am I supposed to see the exit code in the command window? I found the following in Declude's manual, but I just want to make sure everything is ok before I proceed. " After your program is finished, it needs to return an exit code to Declude JunkMail (in C/C++, this is done simply with "return code;" at the end of the main function; you can (carefully!) use the Windows ExitProcess( ) function in other languages)." Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Config When Using Sniffer With Declude...
I'm with Scott on this and I score roughly the same way. One should definitely break out the different result codes because the categories have different levels of reliability. I personally don't score the GRAY stuff because I have MAILPOLICE-BULK scored fairly high and an extra point or two was causing a ton of false positives. Unfortunately some sources are more gray than others, but there's no good way to differentiate that on its own, though some GRAY rules will get hit in other categories when. On a hold weight of 10, I use the following: SNIFFER-GENERAL external 063 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-EXPERIMENTAL external 062 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 4 0 SNIFFER-OBFUSCATION external 061 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-GRAY external 060 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 0 0 SNIFFER-CASINO external 059 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-DEBT external 058 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-GETRICH external 057 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-INK external 056 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-MALWARE external 055 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-PORN external 054 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-PHISHING external 053 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-PHARMACY external 052 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-SPAMWARE external 051 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-MEDIA external 050 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-AVSOFT external 049 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-INSURANCE external 048 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 SNIFFER-TRAVEL external 047 "C:\IMail\Declude\Sniffer\sniffer2.exe code" 6 0 Matt Scott Fisher wrote: Here are my sniffer settings. I consider SPAM to be weight 20. So it generally takes sniffer and one or two other failure to become SPAM. The return code 60 of "greymail", I've lowered the weight on to lower false positives. SNIFFER-NOTFOUND external 000 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 0 0 SNIFFER-TRAVEL external 047 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 8 0 SNIFFER-INSURANCE external 048 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 11 0 SNIFFER-AV-PUSH external 049 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 12 0 SNIFFER-WAREZ external 050 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 10 0 SNIFFER-SPAMWARE external 051 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 10 0 SNIFFER-SNAKEOIL external 052 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 SNIFFER-SCAMS external 053 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 10 0 SNIFFER-PORN external 054 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 SNIFFER-MALWARE external 055 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 12 0 SNIFFER-ADVERTISING external 056 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 10 0 SNIFFER-SCHEMES external 057 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 SNIFFER-CREDIT external 058 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 SNIFFER-GAMBLING external 059 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 10 0 SNIFFER-GREYMAIL external 060 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 2 0 SNIFFER-OBFUSCATION external 061 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 SNIFFER-SPAM external 062 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 12 0 SNIFFER-GENERAL external 063 "D:\IMail\Declude\Sniffer\sniffer2.exe code" 13 0 Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 03/09/04 03:37PM >>> Hello All, I am running Sniffer with Declude and was wanting to get some ideas on how everyone has Declude setup. Currently I just have the basic setup as follows. SNIFFER external nonzero "d:\imail\declude\sniffer2_2\winx\snifferprog.exe sniffer auth" 10 0 I hold anything with a weight of 10m therefore anything failing sniffer gets held and reviewed. I was thinking that sniffer had a way to check and see why it failed, but I have not found much on that. I guess I am just not looking in the right place... Anyone give me some hints? Thanks! Sincerely, Gr
Re: [sniffer] Question about the command line output
Pete, It does help. I've been researching things and came up with the following VB samples: http://www.mentalis.org/apilist/ExitProcess.shtml My programming buddy is looking at these and figuring out a way to capture the result codes from a unique process. Returning them to Declude once they are captured should be easy. He's going to pick back up on this tomorrow. If anyone has some sample code that would work to capture the result codes returned by a unique process when called with wscript.run, please don't be bashful :) Thanks, Matt Pete McNeil wrote: The value that is returned is an integer "result code" so you won't see it directly. In a batch/cmd file you would detect this result code as an ERRORLEVEL but no text is produced unless there is an error that Message Sniffer expects you to see - such as a bad command line configuration. If your log file looks correct then things should be working fine. Hope this helps, _M At 04:29 PM 3/9/2004, you wrote: Pete, I just started testing things and I've run Sniffer on a test file containing two matches in it. When I run it from the command prompt it just simply logs the results and returns no value to the window. So my question is...am I supposed to see the exit code in the command window? I found the following in Declude's manual, but I just want to make sure everything is ok before I proceed. " After your program is finished, it needs to return an exit code to Declude JunkMail (in C/C++, this is done simply with "return code;" at the end of the main function; you can (carefully!) use the Windows ExitProcess( ) function in other languages)." Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Resident Sniffer?
This all sounds like great stuff. I'm looking forward to the so-called "solid" beta :) Matt Pete McNeil wrote: We are still in the process of moving our offices so development efforts have been paused based on a recent comment that this issue was not urgent. However, primary engineering decisions have been completed for this update and it is the next project in line. Barring unforeseen events I will begin coding this today. In theory it should not take long to complete the work since "hooks" for this mechanism were stubbed into the peer-server code when it was designed. We will implement the change by allowing the filename to be overloaded as a switch/code. We will also use this mechanism for other options in the future. The command line will look something like this: snfrv2r2.exe authenticationxx persistent -- normally the file to be scanned would go where the word "persistent" is. When the file to be scanned is "persistent" then the instance goes in to a persistent server mode. -- A later option will be "stdio" which will place the program in a mode to accept the message on stdin and produce a modified version on stdout - thus allowing it to be piped in front of spamc and other similar programs. -- A later option will be "communigate" which will place the program in a mode to act as an external filter using communigate pro's filter protocols. The details of these options will probably change slightly from the above description and each will be handled in it's own 2-x release process. Additional filtering mechanisms will also be implemented during these releases such as "blinder" rules, fuzzy rules, network learning, etc... I will be sure to keep the list up to date on this work and as soon as I have a solid beta I will ask for beta testers. Once we have completely finished our move we will be able to move forward more quickly with development efforts. Thanks! _M PS: MicroNeil's new contact info is up to date on the MicroNeil web site. At 08:57 AM 3/16/2004, you wrote: Pete, Could you inform us on the progress on the issue below? Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOS Small Office Solutions / Reject / Wannepad 27 - 1066 HW -Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy - Installation - Maintenance Network Security - Internet - E-mail Software Development - Project Management -- -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: woensdag 3 maart 2004 13:14 To: [EMAIL PROTECTED] Subject: Re: [sniffer] Resident Sniffer? At 03:54 AM 3/3/2004, you wrote: >Hi, > >Would it be possible to have a 'bogus' sniffer in memory at all time, >that provides other sniffer instances with the rulebase etc, the way it >is working now if you have concurrent sniffer sessions? > >It would be great to have one instance of sniffer 'out there' (not >scanning >messages) which is just waiting for others to share the DB with. This >speeds up message sniffing ror Mdaemon considerably, because MD can >only have 1 instance of the content filter at all times (and therefore >1 instance of Sniffer). That's very interesting... I didn't know that MDaemon was limited in this way. We are planning to make an update that will allow a sniffer instance to run in a script with long wait times so that it can be set up to do this. We are also planning to help integrate sniffer with MDaemon directly - I've been told we will begin working on that this month. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message has been scanned for spam and viruses by www.reject.nl This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] RunExeSvc for Persistent sniffer.
I'm going to give this one a try right now since I have the Resource Kit installed already. Just one question...do I need to change the arguments in my Declude config, or will the service definition take care of the 'persistence'? Thanks, Matt Bill Boebel wrote: We've been using svrany for years with several custom applications and it works great. This utility has been around since the NT4 Resource Kit... http://www.pyeung.com/pages/win2k/userdefinedservice.html Bill -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Pete McNeil Sent: Friday, March 19, 2004 12:25 AM To: [EMAIL PROTECTED] Subject: [sniffer] RunExeSvc for Persistent sniffer. Hello folks, We've been continuing to test the new persistence enabled sniffer engine and some utilities that will allow it to run as a service. We found a free utility that seems to be very solid, and very simple. http://www.judoscript.com/goodies/RunExeSvc/runexesvc.html One of the scripts we used is: debug=false cmdline=c:\Projects\sniffer2-3\TestBed\snfrv2r2.exe xnk05x5vmipeaof7 persistent home=c:\Projects\sniffer2-3\TestBed (Note: The mismatch between the sniffer2-3 directory and the snfrv2r2.exe is not a type-o. We re-branded the 2-3 to use the snfrv2r2 license in our example - it was easier that than creating a new license. Note also that the cmdline parameter includes the full path to the executable - you will need to do this also. We could not get the service to start on our NT test bed without including the full path to the .exe) We've tested this on our XP based Toshiba laptop, and on our NT4 based IMail test bed. Both seem to setup and work fine. Auto-start works fine, so does logging out and logging in. Once you've set up a persistent sniffer instance as a service, go into your services control panel (usually via administrative tools), set the service to start automatically, and start it. A window will appear for the program - do not close the window! Minimize it. When you log out sniffer will continue to run in the background. When you log in the window will be visible again - it's harmless. If you close it though you will have ended the sniffer.exe out from under the service. This won't cause you any trouble, but you won't get the benefit of the persistent server until you stop and start the service again to relaunch the program. Using RunExeSvc, the actual service is the RunExeSvc program. That program launches sniffer as a client and stands in as a service stub for your OS. You can use this to run all sorts of things... The developer uses it to run Java based web servers, for example. Eventually we will build a win32 service version of Message Sniffer, but for now this is the fastest way we can bring you the features you need. Please give this a try and let us know how it works for you. If you find a different utility that you like better then please let us know. Thanks! _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] RunExeSvc for Persistent sniffer.
Ok, I think I did it. Only took a minute (thanks Bill). Here are some more precise directions, but consider them to be "beta" directions (please correct them if you find a problem): 1) Install the Windows 2000 Resource Kit, or download and install the INSTSRV.exe and SRVANY.exe files in a permanent location, preferably within your path. The individual files can be found at the following location: http://www.pyeung.com/pages/win2k/userdefinedservice.html 2) Open a command prompt (Click on the Start Button, Select Run, and type CMD) 3) Enter the following command (customize for the paths of the executables) C:\Progra~1\Resour~1\INSTSRV Sniffer C:\Progra~1\Resour~1\SRVANY.exe 4) Open up the Registry Editor (Click on the Start Button, select Run, and type REGEDIT) 5) Locate the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer 6) From the Edit menu, select New, select Key, and name the new key Parameters 7) Highlight the Parameters key 8) From the Edit menu, select New, select String Value, and name the new value Application 9) From the Edit menu, select Modify, and type in the full path name and application name, including the drive letter and file extension (don't use quotes, customize path, executable name and authentication code) Example: C:\IMail\Declude\Sniffer\[yourlicx].exe [authenticationxx] persistent [yourlicx] = your license ID [authenticationxx] = your authentication string 10) Open the Services MMC 11) Start the Sniffer service 12) Set the Sniffer service to Automatic Matt Matt wrote: I'm going to give this one a try right now since I have the Resource Kit installed already. Just one question...do I need to change the arguments in my Declude config, or will the service definition take care of the 'persistence'? Thanks, Matt Bill Boebel wrote: We've been using svrany for years with several custom applications and it works great. This utility has been around since the NT4 Resource Kit... http://www.pyeung.com/pages/win2k/userdefinedservice.html Bill -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Pete McNeil Sent: Friday, March 19, 2004 12:25 AM To: [EMAIL PROTECTED] Subject: [sniffer] RunExeSvc for Persistent sniffer. Hello folks, We've been continuing to test the new persistence enabled sniffer engine and some utilities that will allow it to run as a service. We found a free utility that seems to be very solid, and very simple. http://www.judoscript.com/goodies/RunExeSvc/runexesvc.html One of the scripts we used is: debug=false cmdline=c:\Projects\sniffer2-3\TestBed\snfrv2r2.exe xnk05x5vmipeaof7 persistent home=c:\Projects\sniffer2-3\TestBed (Note: The mismatch between the sniffer2-3 directory and the snfrv2r2.exe is not a type-o. We re-branded the 2-3 to use the snfrv2r2 license in our example - it was easier that than creating a new license. Note also that the cmdline parameter includes the full path to the executable - you will need to do this also. We could not get the service to start on our NT test bed without including the full path to the .exe) We've tested this on our XP based Toshiba laptop, and on our NT4 based IMail test bed. Both seem to setup and work fine. Auto-start works fine, so does logging out and logging in. Once you've set up a persistent sniffer instance as a service, go into your services control panel (usually via administrative tools), set the service to start automatically, and start it. A window will appear for the program - do not close the window! Minimize it. When you log out sniffer will continue to run in the background. When you log in the window will be visible again - it's harmless. If you close it though you will have ended the sniffer.exe out from under the service. This won't cause you any trouble, but you won't get the benefit of the persistent server until you stop and start the service again to relaunch the program. Using RunExeSvc, the actual service is the RunExeSvc program. That program launches sniffer as a client and stands in as a service stub for your OS. You can use this to run all sorts of things... The developer uses it to run Java based web servers, for example. Eventually we will build a win32 service version of Message Sniffer, but for now this is the fastest way we can bring you the features you need. Please give this a try and let us know how it works for you. If you find a different utility that you like better then please let us kno
Re: [sniffer] RunExeSvc for Persistent sniffer.
Pete, Although inconclusive, some screen caps of Task Manager seems to show a dramatic reduction in many of the peaks with the service turned on. It's hard to tell the exact impact due to the virus scanners not always being called, and SKIPIFWEIGHT settings disabling a mountain of custom Declude filters which both are processor hogs, but the smaller peaks. I believe the following before and after screen caps are representative of the impact (I looked for similar E-mail hit frequencies): Before http://www.mailpure.com/no_service.gif After (with service) http://www.mailpure.com/service.gif The real test will have to wait for rush hour though. Thanks, Matt Pete McNeil wrote: The service definition takes care of the persistence. Your Declude config should not be changed. _M At 01:05 AM 3/19/2004, you wrote: I'm going to give this one a try right now since I have the Resource Kit installed already. Just one question...do I need to change the arguments in my Declude config, or will the service definition take care of the 'persistence'? Thanks, Matt Bill Boebel wrote: We've been using svrany for years with several custom applications and it works great. This utility has been around since the NT4 Resource Kit... http://www.pyeung.com/pages/win2k/userdefinedservice.html Bill -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Pete McNeil Sent: Friday, March 19, 2004 12:25 AM To: [EMAIL PROTECTED] Subject: [sniffer] RunExeSvc for Persistent sniffer. Hello folks, We've been continuing to test the new persistence enabled sniffer engine and some utilities that will allow it to run as a service. We found a free utility that seems to be very solid, and very simple. http://www.judoscript.com/goodies/RunExeSvc/runexesvc.html One of the scripts we used is: debug=false cmdline=c:\Projects\sniffer2-3\TestBed\snfrv2r2.exe xnk05x5vmipeaof7 persistent home=c:\Projects\sniffer2-3\TestBed (Note: The mismatch between the sniffer2-3 directory and the snfrv2r2.exe is not a type-o. We re-branded the 2-3 to use the snfrv2r2 license in our example - it was easier that than creating a new license. Note also that the cmdline parameter includes the full path to the executable - you will need to do this also. We could not get the service to start on our NT test bed without including the full path to the .exe) We've tested this on our XP based Toshiba laptop, and on our NT4 based IMail test bed. Both seem to setup and work fine. Auto-start works fine, so does logging out and logging in. Once you've set up a persistent sniffer instance as a service, go into your services control panel (usually via administrative tools), set the service to start automatically, and start it. A window will appear for the program - do not close the window! Minimize it. When you log out sniffer will continue to run in the background. When you log in the window will be visible again - it's harmless. If you close it though you will have ended the sniffer.exe out from under the service. This won't cause you any trouble, but you won't get the benefit of the persistent server until you stop and start the service again to relaunch the program. Using RunExeSvc, the actual service is the RunExeSvc program. That program launches sniffer as a client and stands in as a service stub for your OS. You can use this to run all sorts of things... The developer uses it to run Java based web servers, for example. Eventually we will build a win32 service version of Message Sniffer, but for now this is the fastest way we can bring you the features you need. Please give this a try and let us know how it works for you. If you find a different utility that you like better then please let us know. Thanks! _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sni
Re: [sniffer] Help
Have you tried a reboot? Checked your error logs? Made sure that DNS and all of your E-mail services are running? Is there even a chance that you will be able to receive this message? Matt Richard Farris wrote: I just did an Windows NT update and now I cant get any email...when I turn sniffer off I at least can send mail to myself but still cant get from outside..any ideas., Richard Farris Ethixs Online 1.270.247. Office 1.800.548.3877 Tech Support - Original Message - From: "Pete McNeil" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Wednesday, March 24, 2004 2:01 PM Subject: Re: [sniffer] Possible Bad Rule? We had a badly coded rule that matched yahoo. The rule has been removed. About 30 rulebases went out before it was caught. These are being recompiled with the correction right now. I will see if I can push yours to the top. _M At 02:02 PM 3/24/2004, you wrote: I am getting a lot of complaints today from Yahoo users... Sheldon - Original Message - From: "Darrell LaRock" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: "'SnifferSupport'" <[EMAIL PROTECTED]> Sent: Wednesday, March 24, 2004 10:33 AM Subject: [sniffer] Possible Bad Rule? Pete, I am seeing a ton of false positives for RULE 100543. I sent a few in to you to check out ([EMAIL PROTECTED]). I wanted to post this here as well since it seems to take approx. 24 hours to process false positives. Darrell This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Error_Bad_Matrix
Pete, FYI, I was trying to set up log uploads yesterday night and it took me a while to figure out that the FTP connection was unreliable from my server. Packets were being dropped/munged somewhere. I also noted a much lower hit rate on SNIFFER-PHARMACY yesterday, but no indication of matrix problems in the logs today (yesterday's were deleted). Every once in a while my colocator's border router goes on the fritz and starts dropping packets. A reboot usually fixes that issue. If your router checks out fine, you might want to take a look at the routes going from your server to the customers that have indicated a problem and those that have indicated that there is none, that might identify something not so obvious if you run out of ideas. I know how these things go and the worst part is not knowing the source while others expect an quick fix. No big deal on my end in the mean time though. Matt Pete McNeil wrote: snf2check.exe will catch a partial download but it will not catch corruption in the middle of the file. _M At 03:57 PM 3/25/2004, you wrote: I run snf2check.exe against every .snf file downloaded. I just checked it again manually, and no errors were reported. I now have almost 3500 Error_Bad_Matrix entries in today's log. Bill -Original Message- From: Vivek Khera [mailto:[EMAIL PROTECTED] Sent: Thursday, March 25, 2004 12:52 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Error_Bad_Matrix On Mar 25, 2004, at 3:39 PM, Paul Lushinsky wrote: > I decided to look in my log files for the past several days because of > number of Error_Bad_Matrix related messages. I can't find this message > in any of my log files until today starting with the update I auto > downloaded at 8:15 this morning, and went until the update at noon. > While I was look at the log file, another update notice came, so an > update was done and the Error_Bad_Matrix message is back. > I'm curious if the people who are seeing these messages are running snf2check.exe before making the rule files live. I do so, and have not seen a single instance of this error. Can you run snf2check.exe on the current bad matrix you have and see if it reports an error? This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- This message and any included attachments are from Siemens Medical Solutions USA, Inc. and are intended only for the addressee(s). The information contained herein may include trade secrets or privileged or otherwise confidential information. Unauthorized review, forwarding, printing, copying, distributing, or using such information is strictly prohibited and may be unlawful. If you received this message in error, or have reason to believe you are not authorized to receive it, please promptly delete this message and notify the sender by e-mail with a copy to [EMAIL PROTECTED] Thank you This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
How about a byte length compare or checksum of some sort? Matt Pete McNeil wrote: At 06:25 PM 3/25/2004, you wrote: We also saw many BAD_MATRIX errors last night. If the problem was 'wget', shouldn't the snf2check utility detect a corrupt file? Also, we did a manual update yesterday afternoon and there were no 'wget' error messages. The problem got corrected sometime between last night and this morning. Perhaps though some have had trouble throughout the day. At the very least the verification on snf2check should be improved to catch this issue. Updating with a bad ruleset creates many problems. Agreed. I'm looking for some simple ways to do that without changing the rulebase file format. There aren't any simple mechanisms that come to mind. Perhaps there will be no choice but to change the format in order to prevent this possibility. _M -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Thursday, March 25, 2004 7:06 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Spam storm? This helps narrow things down. Specifically we know that the rulebase files are not corrupted on the server but during the download. That explains why I haven't been able to recreate a problem in the lab. I have a suspicion that wget may be failing intermittently. Another customer recently had unexplainable, intermittent issues with wget. They replaced wget with code of their own and have had no further problems. Can we narrow this down to wget under heavy traffic conditions perhaps? _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spam storm?
FYI, I'm on Sprint also and saw issues. Matt Pete McNeil wrote: At 09:31 AM 3/26/2004, you wrote: On Mar 26, 2004, at 7:42 AM, Russ Uhte (Lists) wrote: downloads are coming from. However, I too have noticed really slow download speeds. I use wget, and I've never had a single problem, other than occasionally it is extremely slow sometimes. Once it does actually download, it's always a "clean" download. I haven't seen a single instance of the error_bad_matrix. I haven't been monitoring my d/l speeds, but the last few weeks or so I get about 3 to 4 check failures from snf2check. My pipe is a quite underutilized 100Mbit at a uunet co-lo (Pete, right near ya in Ashburn -- you should think about co-lo there :-) ) I don't recall getting those errors before the big network switch at microneil earlier this year. I've not seen a single bad matrix. But then I'm not on windows... so perhaps it is related to windows. It's starting to come together now. Wget on windows + errors on the Sprint line since the move = corrupted downloads for folks who end up routing through sprint along the way? Could be. _M PS: When we do move again we'll be moving into the local Equinix facility... Sounds like it might be the same place. Right now our hands are tied a bit so we can't move yet. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Final beta (b2) for snfrv2r3
Pete, I haven't been following this thread closely but latest generation SCSI drives can be below 4 ms seek times as rated by their manufacturers. FYI, I haven't seen any issues with the persistent Sniffer beta run as a resource kit service besides some expected brief delays according to the way that processes when traffic is less heavy. Matt Pete McNeil wrote: I must be getting punchy... but this just occurred to me... Anybody else remember when a high performance hard drive had a seek time just under 30ms ?? _M At 06:01 PM 4/7/2004, you wrote: If thats all that happens during the first setup timer than you do have some performance issue on a production machine. My production mail server is not too beefy and does somewhere around 120k+ a day. Heres a snipplet from my logs (with persistent sniffer) for comparison fde2jqoe 20040407041105 D7f587132019a8525.SMD 0 31 Final fde2jqoe 20040407041105 D7f577130019a80fe.SMD 0 15 Final fde2jqoe 20040407041106 D7f5973740202893b.SMD 0 16 Clean fde2jqoe 20040407041109 D7f58737302028553.SMD 0 16 Final fde2jqoe 20040407041109 D7f53712e019a73bf.SMD 0 15 Final fde2jqoe 20040407041120 D7f6490fe0072b647.SMD 0 0 Final fde2jqoe 20040407041120 D7f6590ff0072b721.SMD 15 0 Final fde2jqoe 20040407041120 D7f659172b84a.SMD 0 32 Final fde2jqoe 20040407041120 D7f6591010072ba3e.SMD 0 15 Final fde2jqoe 20040407041120 D7f6691020072bbe4.SMD 0 31 Final fde2jqoe 20040407041121 D7f6691030072bdc9.SMD 0 16 Final fde2jqoe 20040407041123 D7f6991050072c932.SMD 0 16 Clean fde2jqoe 20040407041123 D7f6a91060072cbf2.SMD 0 15 Final fde2jqoe 20040407041123 D7f6a73760202cc6f.SMD 0 16 Final From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Pete McNeil Sent: Wednesday, April 07, 2004 4:36 PM To: [EMAIL PROTECTED] Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 At 04:06 PM 4/7/2004, you wrote: So, making sure I'm following your analysis: I'm looking at my log file and I'm seeing lines similar to snf2beta 20040407020014 D60a4134.SMD 181 30 Match 101576 58 20 38 68 And that 181 figure seems to hold pretty stable. 181 is substantially lower than the values I was seeing prior to the current beta [and I have a production machine similar in content and power to your test machine], but I'm seeing that you achieve numbers 2-6 times faster than I am. Yes... that seems about right. When a persistent server is running the rulebase is almost never reloaded. Only two significant things happen during the setup time as measured by Sniffer: 1) Loading the rulebase, 2) locating a job to process (directory scan + locking). The drop seems to indicate that the rulebase reload has stopped as it should. That only leaves the directory scan and a couple of rename/create operations. _M -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
[sniffer] Possible blip?
Pete, I noted late last night that my rulebase grew by 700 KB over the size of the previous one that was archived on my machine, and also the hits for some of the tests were noticeably lower and I had a definite increase in the number of messages that scored in my Hold range (instead of scoring higher and landing in Drop). This morning though the size of my rulebase again dropped by about 450 KB. I was just wondering if this might have been a hiccup with a bad compilation or maybe you were testing something out? Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Possible blip?
Pete, I was judging based on the size of our Hold range which scores from 10-24. On Monday that was 1.86% of total traffic, but on Tuesday that was 2.83%. Message volume was hardly different. Other notables were that on Monday, Sniffer hit 77.27% of all E-mail but on Tuesday it hit 74.53% (both exclude Gray hits). Our overall spam percentage is about 82% on Monday and 81% on Tuesday. I did also see a drop in XBL hits which are primarily zombies from 38.14% to 34.93%. I've always found static spammers to be much more problematic because they lack many spammy patterns, and it could be that there was a wave of them that came online yesterday which could account for the difference. I don't want to make a huge deal out of this, but I noted the drop in size from one rulebase to another and thought that might be significant, and I like to be aware of what is going on. In reality though the difference in percentages in our Hold file meant manually reviewing 50% more E-mails, or about 500 extra messages. With everything else consistent, I figured it was worth a post just to check. I do recall an old posting where you indicated that you were going to drop the expiration down to 5 days under a certain number of hits. My thought there is that while it does present some savings in processing, it might make more sense to do a 7-8 day expiration in order to help catch spammers that are on weekly schedules, primarily lower volume niche spammers. Unfortunately I can't compare my current results accurately to the pre-change data because the makeup of my traffic has changed significantly over that time frame. Another possibility is that our Chinese language spam might have been extra heavy. I've brought in much more of that recently from a couple different clients and it regularly scores low, probably because it's difficult to determine if most of it is spam. I do know that Sniffer doesn't do nearly as well with this stuff. I've noticed that these guys are spamming mostly during Chinese business hours, and they might have been extra light on Monday due to the lag in hours coming from a weekend. If you are interested in getting these caught messages forwarded to you in an automated fashion for study or for potential inclusion, just let me know. I also have a filter set up for Russian language E-mail, but it is not nearly as high in volume (now). Regarding when I saw the changes in the rule base, I was pulling an all-nighter for server administration and noticed this around 5 a.m. when I ran the stats program on my Declude logs. The renamed 'old' rulebase was just over 4 MB while the active one was 4.7 MB, then at about noon I noticed it was about 4.3 MB, and now it's back up over 4.7 MB (1,000 KB = 1 MB in these stats if that matters). I haven't yet upgraded to the most recent release, I'm still on the prior beta. I'll probably do that this evening. I tend to wait on upgrades until there has been enough time for bugs to surface unless I am already looking for a fix. I'm sure that the extra verification of the rulebase will help prevent the potential of problems, and I guess this has the possibility of being caused by a bit of corrupted data, though that's probably reaching. Again, regardless if there was a blip, Sniffer still does a wonderful job of tagging lots and lots of E-mail, just not quite as much as the day before. Thanks, Matt Pete McNeil wrote: At 12:57 PM 5/19/2004, you wrote: Pete, I noted late last night that my rulebase grew by 700 KB over the size of the previous one that was archived on my machine, and also the hits for some of the tests were noticeably lower and I had a definite increase in the number of messages that scored in my Hold range (instead of scoring higher and landing in Drop). This morning though the size of my rulebase again dropped by about 450 KB. I was just wondering if this might have been a hiccup with a bad compilation or maybe you were testing something out? We didn't have anything under test that would alter the rulebases. I'm going to dig through the logs and see if there's anything I can identify. If the rulebase was corrupted in any way you would have been able to detect that with the latest snf2check utility. It's not unusual for ruelbase sizes to change by as much as 20%. The system is constantly activating and deactivating rules based on new log files that are reported. Currently a significant change might occur once per day - though we are working on new analysis engines that will permit more frequent rule strength adjustments. For example, we might add 300-900 rules over the course of a day - then have that many (or more) removed when the new rule strength numbers are calculated. Another factor that impacts rulebase size is the content of the rules. The folding process is not deterministi
Re: [sniffer] Possible blip?
Pete, Our Hold range has returned to more normal territory on Thursday. Here's the stats from the week as a whole on what has been very consistent traffic. Out of all E-mail processed, both good and bad, the %Hold represents what scored between 10-24 points on our system and needed review, the %Sniffer represents all Sniffer hits except for Gray, the %Spam is what we scanned and didn't deliver (generally about 99.8% of spam is caught at a score of 10 which this is based on), and the Sniffer/Spam is the percentage of Sniffer hits as a portion of messages scoring 10 or more. Day %Hold %Sniffer %Spam Sniffer/Spam Mon: 1.86% 77.27% 80.37% 96.14% Tue: 2.83% 74.53% 79.37% 93.39% Wed: 2.13% 77.60% 79.66% 97.41% Thur: 1.95% 76.50% 80.66% 94.84% The only change that we made to our system was to add two smaller domains later in the week, and we introduced filters for Cyrillic and Chinese languages on Wednesday morning which have cut our hold file down by 0.38 percentage points on Thursday, which explains how our %Hold is lower on than on Wednesday with a lower Sniffer hit rate on spam. I did note two high volume untagged static spammers on Tuesday that we blacklisted locally, and that combined with the increase in Sniffer change rates (spam storm) might account for the changes that I saw. I am wondering though about the recommendations that you have made for possibly fine tuning our rule base. Again though, please keep in mind that I still feel that performance is overall very, very good. One of my thoughts regarding minimum rule strengths and grace periods is that all groups aren't necessarily the same. For instance Nigerian scams are low volume and sporadic, and my system performs the worst on these things. Maybe lower rule strengths and longer grace periods makes much more sense for the Phishing category than it does for many other categories for instance. Is that possible? I also looked up the rule strengths on your site and found that about 50%, or maybe more, have a strength below 1, and maybe lowering that is worth testing out so long as I don't massively increase the number of records. I do think though that I would like to test out extending the grace period. Most of my false positives are not on things that this would affect, and that might give niche sources a little extra coverage if I understand things correctly. I'll follow your directions and contact you directly regarding any affirmative changes, but I thought it might be beneficial to keep this discussion public since some other stats hounds might find this information to be of use :) If you can glean anything from the numbers that I gave you, please add your thoughts. Thanks, Matt Pete McNeil wrote: At 05:00 PM 5/19/2004, you wrote: I haven't yet upgraded to the most recent release, I'm still on the prior beta. I'll probably do that this evening. I tend to wait on upgrades until there has been enough time for bugs to surface unless I am already looking for a fix. I'm sure that the extra verification of the rulebase will help prevent the potential of problems, and I guess this has the possibility of being caused by a bit of corrupted data, though that's probably reaching. There were no substantive changes from the beta to the production version. Largely just a removal of monitoring code. Again, regardless if there was a blip, Sniffer still does a wonderful job of tagging lots and lots of E-mail, just not quite as much as the day before. Last night I was able to adjust the rule strength analysis window back to it's original settings. About 5 days of data were lost - but those days will be recovered quickly. Please let me know if this adjustment improved your conditions. I've noted that on a number of other lists there seem to be posts about a sudden increase in spam over the past few days. We are definitely seeing this also - approximately a 25% or more increase in new rule additions in the past 4 days: http://www.sortmonster.com/MessageSniffer/Performance/ChangeRates.jsp Specifically note from about 4 days ago... Days Ago Adjustments --- 0 356 1 508 2 391 3 410 4 410 5 326 6 309 7 371 8 292 9 347 10 309 ( 5-10 : 1954/6 -> 325.67, 0-5 : 2075/5 -> 415, 325.67/415 -> 78.47 ) Note that day 0 is not complete. So applying a "fudge factor" 78.4 _looks like_ 75%. Besides, 92% of statistics are made up on the spot anyway %^b I think a number of things are combined here... I just want to get a good handle on them and make sure we are doing the best we can. I've noted, Matt, that your rulebase tuning parameters are set at the defaults. If you would like to adjust these to be more aggressive then please let
Re: [sniffer] Possible blip?
Scott, Regarding my Cyrillic and Chinese filters, I did a review of a full week's held spam, looking for foreign languages and patterns to tag. I found from other research that the primary Chinese characterset, GB2312, contains the Western Latin characterset, and so someone could send an E-mail with this characterset defined and still have English as the message. Because of this I do more than just look for the offending characterset, I've built a combo filter that looks for both high bit characters such as ¥ as well as body or header hits for encoding of GB2312 (Chinese/Korean) or Windows-1251 (Cyrillic). I also have Declude END statements for appearances of US-ASCII and ISO-8859-1, so messages like this one that are referencing such patterns won't trip the filter. It seems to be stopping about 80% to 90% of the stuff, but I'm guessing that the stuff that is getting through didn't hit one of the high bit characters in my filter and I might need to simply expand my list a bit. Unfortunately I have no idea what characters are most common, so I'm just eyeballing it from sources. I had one false positive on a Yahoo Groups posting that referenced 163.com, a Chinese free Web mail provider that inserts Chinese language footers. The message was in English, but encoded in GB2312 and didn't indicate any sign of English besides the actual text. Because of this, I might throw in an exception for the word "the " (followed by a space) just as a test to see if text in English is present, but I have to review that. This message was also BASE64 encoded and that might be an appropriate exception??? The last pattern that I might look at is using the new MailPolice test for identifying Web-mail providers, and excepting them from the filter because they have issues with encoding languages I've found. Hope this helps. Matt Scott Fisher wrote: 2 thoughts from me: 1. Right on on the Nigerian scams, possible keeping these rules longer. As I was forwarding out a Nigerian scam to the spam mailbox, I too wondered how long the Nigerian rules were kept in play. I might also add Nigeria's twin sister the International Lottery spam and Stock Spams might also be kept longer. I noticed an increase in the Stock spams this week. 2. I've been tracking different character sets for a couple of weeks, the Chinese, Cyrillic and Korean look promising. I get false hits on Greek, Thai, and Vietnamese Headers. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 12:42PM >>> Pete, Our Hold range has returned to more normal territory on Thursday. Here's the stats from the week as a whole on what has been very consistent traffic. Out of all E-mail processed, both good and bad, the %Hold represents what scored between 10-24 points on our system and needed review, the %Sniffer represents all Sniffer hits except for Gray, the %Spam is what we scanned and didn't deliver (generally about 99.8% of spam is caught at a score of 10 which this is based on), and the Sniffer/Spam is the percentage of Sniffer hits as a portion of messages scoring 10 or more. Day %Hold%Sniffer%SpamSniffer/Spam Mon: 1.86% 77.27% 80.37% 96.14% Tue: 2.83% 74.53% 79.37% 93.39% Wed: 2.13% 77.60% 79.66% 97.41% Thur:1.95% 76.50% 80.66% 94.84% The only change that we made to our system was to add two smaller domains later in the week, and we introduced filters for Cyrillic and Chinese languages on Wednesday morning which have cut our hold file down by 0.38 percentage points on Thursday, which explains how our %Hold is lower on than on Wednesday with a lower Sniffer hit rate on spam. I did note two high volume untagged static spammers on Tuesday that we blacklisted locally, and that combined with the increase in Sniffer change rates (spam storm) might account for the changes that I saw. I am wondering though about the recommendations that you have made for possibly fine tuning our rule base. Again though, please keep in mind that I still feel that performance is overall very, very good. One of my thoughts regarding minimum rule strengths and grace periods is that all groups aren't necessarily the same. For instance Nigerian scams are low volume and sporadic, and my system performs the worst on these things. Maybe lower rule strengths and longer grace periods makes much more sense for the Phishing category than it does for many other categories for instance. Is that possible? I also looked up the rule strengths on your site and found that about 50%, or maybe more, have a strength below 1, and maybe lowering that is worth testing out so long as I don't massively increase the number of records. I do think though that I would like to test out extending the grac
Re: [sniffer] OT: Language filtering in Declude, was Possible blip?
No, just one, but it won't score unless there is a header or body indication of the GB2312 or Windows-1251 charactersets. I'm using a combo filter in Declude where the HIGHBIT filter is non-scoring, and the CHINESE and CYRILLIC filters contain a line that says: TESTSFAILED END NOTCONTAINS HIGHBIT I'm pretty sure that the CHINESE and CYRILLIC filters will always hit where appropriate unless the HIGHBIT test doesn't hit. I have about 65 different high bit characters in that filter presently, all copied from spam. If Scott was around, I would ask him how the NONENGLISH test is tripped because that might accomplish the same goals, however I'm not sure if it also scores the definition of a characterset, in which case it would have false positives in this scenario. Matt Scott Fisher wrote: Interesting. Are you searching for 2 character pairs with GB2312? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 01:46PM >>> Scott, Regarding my Cyrillic and Chinese filters, I did a review of a full week's held spam, looking for foreign languages and patterns to tag. I found from other research that the primary Chinese characterset, GB2312, contains the Western Latin characterset, and so someone could send an E-mail with this characterset defined and still have English as the message. Because of this I do more than just look for the offending characterset, I've built a combo filter that looks for both high bit characters such as ¥ as well as body or header hits for encoding of GB2312 (Chinese/Korean) or Windows-1251 (Cyrillic). I also have Declude END statements for appearances of US-ASCII and ISO-8859-1, so messages like this one that are referencing such patterns won't trip the filter. It seems to be stopping about 80% to 90% of the stuff, but I'm guessing that the stuff that is getting through didn't hit one of the high bit characters in my filter and I might need to simply expand my list a bit. Unfortunately I have no idea what characters are most common, so I'm just eyeballing it from sources. I had one false positive on a Yahoo Groups posting that referenced 163.com, a Chinese free Web mail provider that inserts Chinese language footers. The message was in English, but encoded in GB2312 and didn't indicate any sign of English besides the actual text. Because of this, I might throw in an exception for the word "the " (followed by a space) just as a test to see if text in English is present, but I have to review that. This message was also BASE64 encoded and that might be an appropriate exception??? The last pattern that I might look at is using the new MailPolice test for identifying Web-mail providers, and excepting them from the filter because they have issues with encoding languages I've found. Hope this helps. Matt Scott Fisher wrote: 2 thoughts from me: 1. Right on on the Nigerian scams, possible keeping these rules longer. As I was forwarding out a Nigerian scam to the spam mailbox, I too wondered how long the Nigerian rules were kept in play. I might also add Nigeria's twin sister the International Lottery spam and Stock Spams might also be kept longer. I noticed an increase in the Stock spams this week. 2. I've been tracking different character sets for a couple of weeks, the Chinese, Cyrillic and Korean look promising. I get false hits on Greek, Thai, and Vietnamese Headers. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 12:42PM >>> Pete, Our Hold range has returned to more normal territory on Thursday. Here's the stats from the week as a whole on what has been very consistent traffic. Out of all E-mail processed, both good and bad, the %Hold represents what scored between 10-24 points on our system and needed review, the %Sniffer represents all Sniffer hits except for Gray, the %Spam is what we scanned and didn't deliver (generally about 99.8% of spam is caught at a score of 10 which this is based on), and the Sniffer/Spam is the percentage of Sniffer hits as a portion of messages scoring 10 or more. Day %Hold%Sniffer%SpamSniffer/Spam Mon: 1.86% 77.27% 80.37% 96.14% Tue: 2.83% 74.53% 79.37% 93.39% Wed: 2.13% 77.60% 79.66% 97.41% Thur:1.95% 76.50% 80.66% 94.84% The only change that we made to our system was to add two smaller domains later in the week, and we introduced filters for Cyrillic and Chinese languages on Wednesday morning which have cut our hold file down by 0.38 percentage points on Thursday, which explains how our %Hold is lower on than on Wednesday with a lower Sniffer hit rate on spam
Re: [sniffer] OT: Language filtering in Declude, was Possibleblip?
I think you might have possibly identified the group of required characters. I'll give that a try. I'm not sure if any Cyrillic stuff has been passing through but this bears watching as well and I might have to change my list there as well. I am also tagging BIG5, however almost all spam comes in GB2312. Here's what I'm searching for in the CHINESE filter: # CHINESE v1.0.0 SKIPIFWEIGHT 25 MAXWEIGHT 10 TESTSFAILED END NOTCONTAINS HIGHBIT SUBJECT END CONTAINS charset=gb2312 SUBJECT END CONTAINS charset="gb2312" SUBJECT END CONTAINS charset=big5 SUBJECT END CONTAINS charset="big5" HEADERS 10 CONTAINS =?gb2312?b? HEADERS 10 CONTAINS =?big5?b? HEADERS 10 CONTAINS charset=gb2312 HEADERS 10 CONTAINS charset="gb2312" HEADERS 10 CONTAINS charset=big5 HEADERS 10 CONTAINS charset="big5" BODY 10 CONTAINS charset=gb2312" BODY 10 CONTAINS charset=3dgb2312" BODY 10 CONTAINS charset=big5" BODY 10 CONTAINS charset=3dbig5" BODY 10 CONTAINS content=zh-cn" BODY 10 CONTAINS content=3dzh-cn" The END statements for the subject are meant as a precaution, although it's probably not necessary with the HIGHBIT filter ending on US-ASCII and ISO-8859-1 (plus a language definition hit for 'content="en-us"'). I do believe that you can apply a similar technique to spam in Spanish, but since the characterset is the same as English, you would be searching for those 'content=' markers in combination with special characters (a short list in this case). We hardly see any Spanish spam, or at least held Spanish spam so I'm doing nothing about it. Spanish is of course a lot more common in US E-mail. It may be that some Spanish spam isn't identified as Spanish since that's not necessary for proper display in most E-mail clients, but I have seen no proof of that. Matt Scott Fisher wrote: Interesting. I generally just punish people if GB2312 ?BIG5 or such are in the headers. This is overwhelmingly SPAM, but like you siad there are English in some of those messages. It looks like the GB2312 Chinese characters will have A B0 to F7 as it's highbyte. and an A0 to FF as it's lowbyte. If the GB2312 Chinese is present, I would think most every character should be one of these: °±²³ µ¶· *º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷ Checking some of my e-mails confirms that. The bad news is that requires another body filter. It's too bad there wasn't a BODY256 filter type where only the first 256 bytes would be checked. That would certainly be enough to score up these, and wouldn't be a CPU hog. I'm not certain that I'd want to throw another body filter at my few Chinese spams. How often do you get a body indication of GB2312 / Cyrillic charactersets with no header indication? It's an interesting subject because I those few Chinese spams that get through to three of my accounts frustrate me. Got any tips for Spanish spam? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 03:17PM >>> No, just one, but it won't score unless there is a header or body indication of the GB2312 or Windows-1251 charactersets. I'm using a combo filter in Declude where the HIGHBIT filter is non-scoring, and the CHINESE and CYRILLIC filters contain a line that says: TESTSFAILED END NOTCONTAINS HIGHBIT I'm pretty sure that the CHINESE and CYRILLIC filters will always hit where appropriate unless the HIGHBIT test doesn't hit. I have about 65 different high bit characters in that filter presently, all copied from spam. If Scott was around, I would ask him how the NONENGLISH test is tripped because that might accomplish the same goals, however I'm not sure if it also scores the definition of a characterset, in which case it would have false positives in this scenario. Matt Scott Fisher wrote: Interesting. Are you searching for 2 character pairs with GB2312? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 01:46PM >>> Scott, Regarding my Cyrillic and Chinese filters, I did a review of a full week's held spam, looking for foreign languages and patterns to tag. I found from other research that the primary Chinese characterset, GB2312, contains the Western Latin characterset, and so someone could send an E-mail with this characterset defined and still have English as the message. Because of
Re: [sniffer] OT: Language filtering in Declude, wasPossibleblip?
Not if you want to exclude certain domains from hitting individual languages. The test for HIGHBIT pre-qualifies the message, and then there are separate tests for CHINESE and CYRILLIC that can each be independently excluded. I would have to exclude both languages/charactersets if I reversed the order. There are currently 35 characters to be searched after updating for your list and trimming somewhat randomly for size. Considering the END statements for US-ASCII and ISO-8859-1, and SKIPIFWEIGHT settings, the full filter should rarely ever run. You could also do a TESTSFAILED END NOTCONTAINS NONENGLISH potentially, but I'm still not positive how that hits. Like I said before, I'm not sure if NONENGLISH hits on characters or the definition of charactersets, or both, and whether or not it has any limitations (like hitting on common extended English characters). If it only hits on what we consider high bit characters, the HIGHBIT test could be abandoned. BTW, I just reviewed in detail why some of this stuff was still appearing in my Hold file and the reason is two-fold. For one, I needed to add more points as all of the Chinese and Cyrillic charactersets that I was tagging did in fact hit these filters and scored 10 points. I was being somewhat cautious since this was a new filter and I did find that one false positive so I'll wait a bit longer to increase the score. The second reason why I was still seeing this some funky stuff was that some of the Cyrillic messages were encoded in KOI8-R and I wasn't filtering for that...but I am now :) I also added KOI8-U (the Ukrainian version) just for good measure. Matt Scott Fisher wrote: Wouldn't it be better to reverse the order? Run the subject and header tests on the majority of the mail. Then run the body with a TESTSFAILED END NOTCONTAINS CHINESE. You should end up with less body searches this way. Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 04:28PM >>> I think you might have possibly identified the group of required characters. I'll give that a try. I'm not sure if any Cyrillic stuff has been passing through but this bears watching as well and I might have to change my list there as well. I am also tagging BIG5, however almost all spam comes in GB2312. Here's what I'm searching for in the CHINESE filter: # CHINESE v1.0.0 SKIPIFWEIGHT25 MAXWEIGHT10 TESTSFAILEDENDNOTCONTAINSHIGHBIT SUBJECTENDCONTAINScharset=gb2312 SUBJECTENDCONTAINScharset="gb2312" SUBJECTENDCONTAINScharset=big5 SUBJECTENDCONTAINScharset="big5" HEADERS10CONTAINS=?gb2312?b? HEADERS10CONTAINS=?big5?b? HEADERS10CONTAINScharset=gb2312 HEADERS10CONTAINScharset="gb2312" HEADERS10CONTAINScharset=big5 HEADERS10CONTAINScharset="big5" BODY10CONTAINScharset=gb2312" BODY10CONTAINScharset=3dgb2312" BODY10CONTAINScharset=big5" BODY10CONTAINScharset=3dbig5" BODY10CONTAINScontent=zh-cn" BODY10CONTAINScontent=3dzh-cn" The END statements for the subject are meant as a precaution, although it's probably not necessary with the HIGHBIT filter ending on US-ASCII and ISO-8859-1 (plus a language definition hit for 'content="en-us"'). I do believe that you can apply a similar technique to spam in Spanish, but since the characterset is the same as English, you would be searching for those 'content=' markers in combination with special characters (a short list in this case). We hardly see any Spanish spam, or at least held Spanish spam so I'm doing nothing about it. Spanish is of course a lot more common in US E-mail. It may be that some Spanish spam isn't identified as Spanish since that's not necessary for proper display in most E-mail clients, but I have seen no proof of that. Matt Scott Fisher wrote: Interesting. I generally just punish people if GB2312 ?BIG5 or such are in the headers. This is overwhelmingly SPAM, but like you siad there are English in some of those messages. It looks like the GB2312 Chinese characters will have A B0 to F7 as it's highbyte. and an A0 to FF as it's lowbyte. If the GB2312 Chinese is present, I would think most every character should be one of these: °±²³ µ¶· *º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷ Checking some of my e-mails confirms that. The bad news is that requires another body filter. It's too bad there wasn't a BODY256 filter type where only the first 256 b
Re: [sniffer] OT: Language filtering in Declude, wasPossibleblip?
Scott, All of those patterns came directly from the source code of E-mails. The charset=3dgb2312" pattern is quite common in the body of E-mail, in fact I think that it is FrontPage or another Microsoft product that encodes it this way when it appears in a META tag. Share and share alike :) Matt Scott Fisher wrote: I used this reference on names character sets: http://msdn.microsoft.com/library/default.asp?url=""> Would you really encounter a =3d ? Should it be charset3dgb2312? Enjoyed your thoughts, glad you didn't post this on a Monday... Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 04:28PM >>> I think you might have possibly identified the group of required characters. I'll give that a try. I'm not sure if any Cyrillic stuff has been passing through but this bears watching as well and I might have to change my list there as well. I am also tagging BIG5, however almost all spam comes in GB2312. Here's what I'm searching for in the CHINESE filter: # CHINESE v1.0.0 SKIPIFWEIGHT25 MAXWEIGHT10 TESTSFAILEDENDNOTCONTAINSHIGHBIT SUBJECTENDCONTAINScharset=gb2312 SUBJECTENDCONTAINScharset="gb2312" SUBJECTENDCONTAINScharset=big5 SUBJECTENDCONTAINScharset="big5" HEADERS10CONTAINS=?gb2312?b? HEADERS10CONTAINS=?big5?b? HEADERS10CONTAINScharset=gb2312 HEADERS10CONTAINScharset="gb2312" HEADERS10CONTAINScharset=big5 HEADERS10CONTAINScharset="big5" BODY10CONTAINScharset=gb2312" BODY10CONTAINScharset=3dgb2312" BODY10CONTAINScharset=big5" BODY10CONTAINScharset=3dbig5" BODY10CONTAINScontent=zh-cn" BODY10CONTAINScontent=3dzh-cn" The END statements for the subject are meant as a precaution, although it's probably not necessary with the HIGHBIT filter ending on US-ASCII and ISO-8859-1 (plus a language definition hit for 'content="en-us"'). I do believe that you can apply a similar technique to spam in Spanish, but since the characterset is the same as English, you would be searching for those 'content=' markers in combination with special characters (a short list in this case). We hardly see any Spanish spam, or at least held Spanish spam so I'm doing nothing about it. Spanish is of course a lot more common in US E-mail. It may be that some Spanish spam isn't identified as Spanish since that's not necessary for proper display in most E-mail clients, but I have seen no proof of that. Matt Scott Fisher wrote: Interesting. I generally just punish people if GB2312 ?BIG5 or such are in the headers. This is overwhelmingly SPAM, but like you siad there are English in some of those messages. It looks like the GB2312 Chinese characters will have A B0 to F7 as it's highbyte. and an A0 to FF as it's lowbyte. If the GB2312 Chinese is present, I would think most every character should be one of these: °±²³ µ¶· *º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷ Checking some of my e-mails confirms that. The bad news is that requires another body filter. It's too bad there wasn't a BODY256 filter type where only the first 256 bytes would be checked. That would certainly be enough to score up these, and wouldn't be a CPU hog. I'm not certain that I'd want to throw another body filter at my few Chinese spams. How often do you get a body indication of GB2312 / Cyrillic charactersets with no header indication? It's an interesting subject because I those few Chinese spams that get through to three of my accounts frustrate me. Got any tips for Spanish spam? Scott Fisher Director of IT Farm Progress Companies [EMAIL PROTECTED] 05/21/04 03:17PM >>> No, just one, but it won't score unless there is a header or body indication of the GB2312 or Windows-1251 charactersets. I'm using a combo filter in Declude where the HIGHBIT filter is non-scoring, and the CHINESE and CYRILLIC filters contain a line that says: TESTSFAILED END NOTCONTAINS HIGHBIT I'm pretty sure that the CHINESE and CYRILLIC filters will always hit where appropriate unless the HIGHBIT test doesn't hit. I have about 65 different high bit characters in that filter presently, all copied from spam. If Scott was around, I would ask him how the NONENGLISH test is tripped because that might accomplish the same goals, however I&
[sniffer] Spammer pollution
Pete, I'm guessing that you have seen this already, but check out all of the domains that are listed in this zombie spam: http://groups.google.com/groups?q=elder.com+group:*abuse*&hl=en&lr=&ie=UTF-8&scoring=d&selm=200404022023.i32KNhR00959%40localhost.localdomain&rnum=3 So where's Waldo :) I found it while researching a new client (wanting to make sure that they don't spam). This is the first time that I noticed this pattern though. Thankfully in large part to Sniffer and some combinations of patterns with other tests, this stuff doesn't even get held very often. Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Spammer pollution
Pete McNeil wrote: M> So where's Waldo :) When reviewing a message like that we always troll the actual message for the link that was intended - this helps us discard those that are in there for "fluff". The porn guys do a lot of this too - and their twist is to add hidden links to competitors so that if/when the domains/links get picked up there is collateral damage. (So the theory goes). You didn't answer my question... http://synapse.subneural.net/~denon/waldo/ Matt :) -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Experimental hits on bounce messages
Pete, I've been seeing a good deal of Experimental hits on bounce messages, primarily with the Nazi spam that has been forging recipients, but I've seen these on other messages that seemingly don't have any spam content, though most of course do involve spam. I was wondering if this is due to IP's being logged for spamtrap hits from these bounces, or if it was due to something else. If these are IP related, it would likely be problematic. And another related question regarding the Nazi spam. Would it be possible to move these rules from Experimental to another category such as Malware or General? Personally I have grown accustomed to Experimental being a category for rules that are not as reliable and more likely to hit on personal E-mail so I weight it low, on the other hand this Nazi stuff is certainly problematic and I'm hoping that your rules are tight enough to place in another category. I do have a filter that deals with bounce messages from Joe-Jobs by testing for both a Sniffer hit or other content related filter along with indications of a null sender or other sign of a bounce, but I was excluding Sniffer-Experimental from this. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Experimental hits on bounce messages
Pete, So would the Message-ID produce a hit if it was in the body of a message? The reason why I ask is because I'm concerned about the possibility of legitimate servers getting tagged with Experimental and how that plays into my system. Am I also to assume that you have some protections in place to protect from bounce messages from Joe-Jobs getting a server listed in Experimental? I have definitely seen some of these where legitimate bounces were tagged with Experimental, and I'm guessing this was the result of spam being relayed or bounced. Matt Pete McNeil wrote: On Sunday, June 13, 2004, 11:49:21 PM, Matt wrote: M> Pete, M> I've been seeing a good deal of Experimental hits on bounce messages, M> primarily with the Nazi spam that has been forging recipients, but I've M> seen these on other messages that seemingly don't have any spam content, M> though most of course do involve spam. M> I was wondering if this is due to IP's being logged for spamtrap hits M> from these bounces, or if it was due to something else. If these are IP M> related, it would likely be problematic. There may be a few IPs involved but I suspect it's a new message-id rule we have in place. It seems that the Nazi spam is delivered with a forged qmail header that qmail would never produce. The rule is somewhat broad so we've left it in the experimental group - just in case something about it turns out to be problematic. Most of the Nazi spam rules are actually coded for known subjects and generalizations of these. M> And another related question regarding the Nazi spam. Would it be M> possible to move these rules from Experimental to another category such M> as Malware or General? Personally I have grown accustomed to M> Experimental being a category for rules that are not as reliable and M> more likely to hit on personal E-mail so I weight it low, on the other M> hand this Nazi stuff is certainly problematic and I'm hoping that your M> rules are tight enough to place in another category. I do have a filter M> that deals with bounce messages from Joe-Jobs by testing for both a M> Sniffer hit or other content related filter along with indications of a M> null sender or other sign of a bounce, but I was excluding M> Sniffer-Experimental from this. It's a close call but I think the rules we have are in the right places - at least for now. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Experimental hits on bounce messages
Pete, I've seen a moderate number of these bounces in the last few days tagged by Experimental that didn't have any obvious indications of spam. You are of course correct that a majority of bounces are the result of viruses or Joe-Jobs. Fortunately the Joe-Jobs are mostly returned to non-existant addresses so they aren't a huge issue although the volume is remarkable and growing, especially from the likes of AOL. I'm used to seeing these fail tests besides Experimental however when they hit on a Subject or content, and using a filter that scores bounces with spam content very high takes care of this issue. I'm worried however because we do unfortunately end up blocking some legitimate bounces, especially for customers that operate automated mailers and have lots of customers due to the bouncing servers failing multiple technical tests. So there is a good reason to want to make sure that we aren't improperly tagging legitimate servers by the IP, and that's why we have steered clear of weighting Experimental at more than 30% of the typical hold weight on our system and have left it out of our combo filters until the Nazi issue and the subsequent tagging of these messages in Experimental. If these rules were in a different category, it would make me feel a lot better about it. I'm guessing maybe from my standpoint, Spamware would be the most appropriate category for tagging forged message ID's of this type. This is such a large issue presently that it does matter somewhat and I would feel much more comfortable not having to use combo filters with the Experimental result code. BTW, here's an example of a bounce that I blocked after including Experimental in our BOUNCER combo filter. I honestly can't tell if this is legitimate or not, and I can't see any signs of spam content which makes me think it was the IP that triggered the hit, but I'm posting it here so that you might be able to tell me what rule hit. The log entry is above the source. I'm just trying to figure out how to tune my system appropriately and wondering if I need to be on the lookout for legitimate servers tagged by Experimental which due to their inclusion now in combo filters, may cause legitimate E-mail to bounce. Thanks, Matt hb064pkq 20040614012029 Dfd5a0e3101a49a3c.SMD 16 15 Match 84800 62 1 51 38 hb064pkq 20040614012029 Dfd5a0e3101a49a3c.SMD 16 15 Final 84800 62 0 2674 38 Received: from oxygen.singnet.com.sg [165.21.74.85] by mx1.mailpure.com with ESMTP (SMTPD32-8.05) id AD5AE3101A4; Sun, 13 Jun 2004 21:20:26 -0400 Received: from localhost (localhost) by oxygen.singnet.com.sg (8.12.11/8.12.9) id i5E13liD016172; Mon, 14 Jun 2004 09:20:24 +0800 Date: Mon, 14 Jun 2004 09:20:24 +0800 From: Mail Delivery Subsystem <[EMAIL PROTECTED]> Message-Id: <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> MIME-Version: 1.0 Content-Type: multipart/report; report-type=delivery-status; boundary="i5E13liD016172.1087176025/oxygen.singnet.com.sg" Subject: [20] Returned mail: see transcript for details Auto-Submitted: auto-generated (failure) X-MailPure: X-MailPure: SNIFFER-EXPERIMENTAL: Failed, listed in the Experimental category (weight 4). X-MailPure: LEGITCONTENT: Passed, legitimate content detected (weight -1). X-MailPure: NULLSENDER: Message failed NULLSENDER test (line 3, weight 4) (weight capped at 4). X-MailPure: FOREIGN: Message failed FOREIGN test (line 589, weight 3) (weight capped at 3). X-MailPure: TLD-ASIAN: Message failed TLD-ASIAN test (line 132, weight 1). X-MailPure: BOUNCER: Message failed BOUNCER test (line 13, weight 10). X-MailPure: RECIPIENTS: [snip].com> X-MailPure: X-MailPure: Spam Score: 21 X-MailPure: Scan Time: 06/13/2004 at 21:20:31 -0400 X-MailPure: Spool File: Dfd5a0e3101a49a3c.SMD X-MailPure: Server Name: oxygen.singnet.com.sg X-MailPure: SMTP Sender: <> X-MailPure: Received From: oxygen.singnet.com.sg [165.21.74.85] X-MailPure: Country Chain: SINGAPORE->destination X-MailPure: X-MailPure: Spam and virus blocking services provided by MailPure.com X-MailPure: X-RCPT-TO: [snip].com> Status: R X-UIDL: 385994059 This is a MIME-encapsulated message --i5E13liD016172.1087176025/oxygen.singnet.com.sg The original message was received at Mon, 14 Jun 2004 09:00:06 +0800 from mx12.singnet.com.sg [165.21.74.122] - The following addresses had permanent fatal errors - <[EMAIL PROTECTED]> (reason: 550 5.1.1 User unknown) - Transcript of session follows - 550 5.1.1 <[EMAIL PROTECTED]>... User unknown --i5E13liD016172.1087176025/oxygen.singnet
Re: [sniffer] Gray Hosting Change Of Status - Request For Comments
Pete, From my own perspective, here are some comments. 1) "unwanted commercial advertising" isn't necessarily spam. I often find rules in the General group from others nominating sources that are generally perceived as being of low value, including by myself, but aren't technically spam by the measures that we use. We consider any first-party relationship with the recipient to be ok so long as they provide an easy method of unsubscribing and they honor unsubscriptions. It would be a good idea to establish a general rule for the service and in handling things that are nominated by customers. You can't be all things to all people, and while advertising is mostly unwanted it is sometimes missed. 2) I don't weight the Gray category on my system. There are too many newsletters and first-party advertising messages that come through for me to give it any weight when such things are based on the provider and not the sender it represents. We do however use hits on this category to defeat some filters on our system (Declude based) because it can be a good identifier of a bulk-mailer and all that comes with it. 3) Moving clean Gray rules over to General will mostly be Ok, but it will definitely cause some false positives with our own more conservative definition of spam. I would actually prefer that you kill the rules with false positives and re-purpose this category as something like "Dirty Bulk" to identify rules associated with specifically dirty lists on bulk mailers. Some are 95% junk, some are maybe 10% junk, and IMO, some like CheetahMail are less than that. 4) I think that 50% legitimate for Gray hits is a big underestimate. Possibly part of that figure is explained by more general rules that target the bulk-mailer and not the source it represents. If care was taken to tag information unique to the sender of the spam newsletter or spam advertising sent through bulk-mailers that are mixed, that will help to segregate the good from the bad. Tagging the From line or the Reply-To line is often a very accurate way to identify these sources, especially when they primarily use tracking links in the body. Some also make use of unique campaign ID's that can be tagged with certain providers. I am not a fan of tagging the provider's own domains because that doesn't allow for differentiation between good and bad, and you must whitelist after each false positive. I would also propose that customer submissions still primarily go to the General category and spam trap hits go into a Dirty Bulk category. I trust the General category less than every other category except Experimental primarily due to the fact that it contains submissions from users with a more liberal definition of spam than I have (I found Amazon.com in there once for instance). I would hope that people wouldn't report this stuff because it can affect everyone on the system. 5) Just an FYI, deciphering whether or not bulk-mail is legitimate, partially legitimate or totally spam is the hardest part of my job, so whatever you do, please don't make it any harder :) Thanks, Matt Pete McNeil wrote: Hello Sniffer Folks, We are reviewing a number of statistics with an eye toward reducing false positives. We have already changed a number of our rule coding policies where our highest false positive rates are found. One of the proposed changes is controversial and I would very much like your input about this. The Gray hosting rule group currently has a Block-First, White-Rule-Later policy. Rules coded into this group are for the likes of Constant Contact. Some time ago when this policy was drafted the overwhelming consensus was that most content arriving from these services was unwanted advertisement spam - therefore it was reasonable to white-rule legitimate publications as they were identified, especially since a single white rule would be shared by all subscribers (thus reducing the work and FP load). A recent analysis has shown that the situation has changed somewhat significantly. In general the following seem true - * The gray hosting group typically tags just less than 2% of messages. * Of this 2%, approximately half of the hits would be false positives. * If this is true then any benefit generated by the group is negated by the risk. * Also, if a given system does find benefit from the group then that benefit would likely be very small. If these points stand up to your comments then the proposal is as follows: - Existing gray-hosting rules with any reported false positives will be removed from the system. - The remaining gray-hosting rules will be moved to the "ungrouped" group (result 63). - No special treatment will exist for future rules that might have been placed in the gray-hosting group and no special status will be maintained for previous members of the gray-hosting group. - Result cod
Re: [sniffer] Reporting - was: spam leakage up
Pete, If the data is normalized, tab delimited seems like the most widely available choice. I've never played with XML, and although it might be more useful in many places, in others it presents overhead, especially as far as a learning curve goes. It may also be that real-time reporting isn't that widely sought after, especially on systems where Sniffer is just one part of an overall system. For me there would be no real value to this except for a rare occasion that I'm researching a problem or my interest is peaked (which isn't a good justification for work). Those that desire real-time functionality may well be more experienced DB admins or programmers and may be able to handle whatever format that you throw at them. Matt Pete McNeil wrote: We are working on specs for real-time reporting out of Sniffer and haven't had a lot of feedback on the XML based format. We were looking at this format because, in theory anyway, it's easy to port into a database or even directly into a web page or other format. Am I guessing right that the reason we didn't get a lot of feedback is because not many folks can really use XML data in practice? Should we adopt a different format for a "real-time scoreboard" output file? Tab delimited? CSV? --- perhaps directly to HTML? (if HTML then I will continue with the XML concept and use DOM to read the XML as a data island and format the output - anybody have experience with this - it seems harder in practice than the examples let on.) Any thoughts would be appreciated. Thanks, _M (The idea of a "scoreboard" was to create some useful indicators that could be read in near real-time - without a lot of heavy lifting. At the time it seemed there was a pressing need for this kind of functionality. I'm beginning to wonder - I don't want to spend effort on something that nobody really cares about. There are plenty of other features planned that we could focus on. I need some feedback. Thanks!) On Thursday, June 24, 2004, 12:02:06 PM, Aaron wrote: AC> Thanks Herb but we don't have Coldfusion. AC> Looks great tho! AC> Aaron AC> www.vantech.net AC> On Jun 24, 2004, at 8:55 AM, Herb Guenther wrote: I wrote a coldfusion page that parses the logs into a sql database every night, and then the display page you saw. If you have a coldfusion server I would be happy to give you the code. Herb Aaron J.Caviglia wrote: Herb, How did you generate that SPAM report? Thanks, Aaron Caviglia www.vantech.net On Jun 24, 2004, at 8:46 AM, Herb Guenther wrote: wow, that is even worse than we are seeing, we are at about 80%, but should really be at about 85% if all were tagged. Here is our last weeks stats, we did not see an increase in volume, so much as the amount gettig thru in the last couple days and continuing today. Herb SPAM Report Statistics are based on the last 6,150,612 email messages received. You are viewing Server 1 Stats View Server 2 stats Statistic 06/17 06/18 06/19 06/20 06/21 06/22 06/23 Weekly Total Daily Avg. Delivered Messages 34,291 30,762 22,331 22,484 31,245 33,588 33,582 208,283 25,311 Good Messages 6,493 5,101 1,595 1,721 6,209 6,772 6,170 34,061 5,221 Spam Messages 27,798 25,661 20,736 20,763 25,036 26,816 27,412 174,222 20,090 Spam Percent 81% 83% 92% 92% 80% 79% 81% 84% 79% Mal Formed Headers 3,845 4,277 3,193 3,555 4,094 4,286 4,459 27,709 4,949 Spam Headers 4,544 4,081 3,665 3,367 4,800 5,712 6,129 32,298 3,308 Spam Routing 6,351 5,697 5,200 5,613 5,718 6,072 5,616 40,267 3,375 No Reverse DNS 6,864 7,787 6,529 6,729 7,742 6,783 5,023 47,457 2,446 White Listed 1,157 968 116 162 1,237 1,245 1,229 6,114 785 General Spam 1,021 958 736 851 1,012 1,045 1,122 6,745 1,490 Experimental 1,543 1,190 951 970 1,284 1,342 1,472 8,752 900 Obfuscation 240 183 158 189 196 336 151 1,453 352 Grey Hosts 355 196 29 33 213 343 315 1,484 166 Gambling 272 202 263 261 215 303 161 1,677 124 Refinancing/Loans 2,293 2,216 1,809 1,659 2,167 2,013 1,975 14,132 1,765 Business opportunities 1,989 1,991 1,546 1,547 1,990 2,089 2,163 13,315 1,464 Ink and toner cartridges 159 124 41 91 100 89 63 667 121 Pornography 2,296 1,874 2,189 1,798 2,120 2,224 2,333 14,834 1,731 Send money scams 57 63 66 57 85 84 82 494 65 Online pharmacies 6,792 6,098 5,419 4,907 5,766 5,526 5,767 40,275 5,684 Cable/Satellite descramblers 1,250 1,340 1,190 1,384 1,277 1,710 1,554 9,705 867 Norton/McAfee offers 17 61 4 7 11 19 25 144 68 Insurance quotes, etc. 706 493 374 354 526 552 547 3,552 649 Travel/vacation offers 216 135 82 61 87 160 121 862 238 Viruses Detected 649 440 223 201 537 498 493 3,0
Re: [sniffer] IP Rules moving to Group 60
A big thumbs up from me :) Matt Pete McNeil wrote: Hello Sniffer Folks, We are planning to split the Experimental rule group into two parts. Experimental (Abstract), (62), will contain abstract heuristics. Experimental (Received [IP]), (60), will contain all IP rules. The change is designed to allow the IP rules to be isolated so they can be weighted differently. IP rules are more dynamic and therefore more likely to cause false positives. If there are no objections then this change will take place this afternoon at approximately 1800 EDT. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation Chief SortMonster (www.sortmonster.com) This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] zipping log files
Pete, Due to growing log file sizes and a ridiculous amount of fragmentation caused by all of the associated products, I was forced to move to a system that does hourly rotation of my logs, moving them to a different drive, and then zipping them up after midnight. I'm thinking that you would save a significant amount of bandwidth on your end by allowing us to zip and upload our logs. There was a brief exchange on this list two months ago about you thinking about enabling this and I thought I would chime in on this and suggest it again. This would actually simplify my scripting since I could do it all within a single script without a multiple minute wait for it to upload a +15 MB uncompressed log file. I can however certainly make it work the other way if required. Please note that I'm using WinZip 9.0's Command Line Option which is free for registered users of WinZip. I prefer this over gzip because I use WinZip anyway and like the right click menu zipping that it provides for, and the commands are quite simple and seemingly complete. If you move to support zip files, please don't discriminate against the non-open source stuff :) Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] zipping log files
Pete McNeil wrote: That's a bit confusing. (use Winzip, but don't discriminate against open source (gzip?)) "non-open source", not just "open source". Should make sense. Anyway, I haven't had time to dive into that again. My responsibilities have been expanding at an astonishing rate. We will probably pursue this at some point, and when we do we will probably accept log files gzipped with a .gz extension since they are going to end up on a linux box for processing and I need a way to identify them without going through a lot of hoops. gzip should be able to handle standard zip files. You mentioned a while back about possibly changing everything to a .gz extension first, and I think that makes sense. It shouldn't be more than a single line of code for you to support .zip files in addition to the .gz files once that is in place. I only asked about this because I am about to re-enable the uploads and would have waited a little while if this functionality was on the horizon. I'll do it the old way for now. In the long term I hope to have Sniffer take care of it's own UL & DL ops ... but that will be a while coming. That would be nice for many, but not really a universal solution. It might be a lot of work on your end whereas a group of well documented scripts could net similar results while allowing for custom configurations like my own. I wonder how many logs you really need in order to net the stats that you are after anyway. I'll bet that a standard-type of log naming/placement/rotation mechanism would be of more use to many of us. I've worked around that for the moment, so don't consider that to be a request on my part. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] zipping log files
Pete McNeil wrote: On Friday, July 16, 2004, 11:04:42 PM, Matt wrote: M> gzip should be able to handle standard zip files. You mentioned a while Key word being _should_ ;-) Well, there's always Info-ZIP's UnZip for Linux if it doesn't: http://www.info-zip.org/pub/infozip/UnZip.html . I thought all that stuff was built into most packages, but clearly I know very little about that environment presently. Too many flavors and not enough taste buds. This will be a while coming though - we need to grow a bit more to free up the necessary development resources. Translation: Time to hire some poor schmoe to do the dirty work so that I can spend more time programming :) I hear ya loud and clear. I'm quite happy to see the progress in the areas that you have been focusing on. Changes to the rules are good improvements, and running as a service is as well, and I know that you are working on some other things also. So when is the multi-processor 64-bit executable coming? Just kidding, no need to answer that one, although I'm sure that you've thought about that also. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Surprising missed spam
Corby, Personally, I'm a fan of leaving the generic stuff out due to the potential of false positives. Those of us that are using Sniffer in addition to other spam blocking mechanisms can afford to lose some Sniffer hits on such phrases because they will be picked up by other means almost all of the time. Including such phrases however would increase our false positive rate without a measurable benefit in spam capture rates. I have even asked Pete to remove some phrase hits from my own rulebase for exactly this reason. Matt Agid, Corby wrote: Surprising missed spam Hello, I was surprised recently by some spam that got through without getting caught by the sniffer. We've been getting some plain text messages that have obvious spam words in the subject line. For example, a plain text message with "horny teenagers" came through. The content was also very spammy, but all plain text. I tried sending myself a few messages with standard spam phrases and none of them tripped any sniffer rules. Am I missing something? Corby -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Surprising missed spam
Actually, we scan for many businesses as well as home users, and have clients with mail boxes on every continent except Antarctica. To me it's really a matter of what classifies spam, and while these phrases are spammy, they are not accurate enough to use in my rulebase. Pete knows what he is doing however, and you will note that most of his rules are based on 'payload' hits, which are generally links. Without a payload, the message is merely a statement, and while that has happened (Nazi spamdemic), it is not the norm. These guys do change their payloads around regularly, but the ones that use these sorts of phrases in spam are highly likely to also get tagged by other obfuscation techniques in Sniffer. Of course there are also many blacklists that are good at tagging both zombie and static spam sources. My point was really that I prefer to tag spam based on a positive hit instead of a suggestive one, and for the most part, Sniffer does this. It is especially effective in combination with other spam blocking techniques. If for instance you have 3 hits on perfectly unassociated patterns, and each one is 99% accurate, or rather 1% inaccurate, the net result is that the combination of hits would produce a false positive rate 0.0001%. A good example of this would be a message that is tagged by Sniffer for a link in the body, tagged by SpamCop for leaking spam by the IP, and forges the Mail From domain. Unfortunately I do see false positives frequently enough when Sniffer hits in combination with some other less accurate test giving it enough points to be held on my system, many of which might fall into a gray category or results from a more generic/suggestive hit in combination with some technical shortcoming. Spam bothers me a whole bunch, that's why I'm in the business, but false positives bother me even more. I do wish that over time Pete could further separate his rules into more positive and more suggestive ones so that things like known URL's would be examples of more positive ones and things like "horny teenagers" would be an example of a suggestive one. Given that, I could weight accordingly. Matt Agid, Corby wrote: I suppose everyone's userbases have differenent requirements. An ISP or private enterprise might worry about false postives on "horny teenagers" and "penis enlargement", but for our local government agency, it causes problems. Corby From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Monday, September 13, 2004 5:25 PM To: [EMAIL PROTECTED] Subject: Re: [sniffer] Surprising missed spam Corby, Personally, I'm a fan of leaving the generic stuff out due to the potential of false positives. Those of us that are using Sniffer in addition to other spam blocking mechanisms can afford to lose some Sniffer hits on such phrases because they will be picked up by other means almost all of the time. Including such phrases however would increase our false positive rate without a measurable benefit in spam capture rates. I have even asked Pete to remove some phrase hits from my own rulebase for exactly this reason. Matt Agid, Corby wrote: Hello, I was surprised recently by some spam that got through without getting caught by the sniffer. We've been getting some plain text messages that have obvious spam words in the subject line. For example, a plain text message with "horny teenagers" came through. The content was also very spammy, but all plain text. I tried sending myself a few messages with standard spam phrases and none of them tripped any sniffer rules. Am I missing something? Corby -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
[sniffer] Test ordering/precedence
Pete, Given some of the recent changes in the result codes for Sniffer, I thought I would inquire about the precedence of the result codes and how these can affect systems. On my system I have weighted the result codes differently and overall, I would consider the following order to be suggestive of the order of reliability from the most reliable to the least reliable. Note that this is not scientific, but instead based on doing review and tests that hit less often could appear higher in terms of stated reliability though I have considered this in making the list: 1. SNIFFER-INK(56) SNIFFER-CASINO(59) SNIFFER-INSURANCE(48) SNIFFER-MEDIA(50) SNIFFER-GETRICH(57) SNIFFER-DEBT(58) SNIFFER-PHARMACY(52) 2. SNIFFER-AVSOFT(49) SNIFFER-PHISHING(53) 3. SNIFFER-TRAVEL(47) SNIFFER-PORN(54) 4. SNIFFER-SPAMWARE(51) SNIFFER-OBFUSCATION(61) SNIFFER-MALWARE(55) 5. SNIFFER-EXPERIMENTAL(62) 6. SNIFFER-GENERAL(63) 7. SNIFFER-IP(60) I'm not sure exactly how Sniffer orders the precedence of the result code, but I would like to recommend that you give some consideration to reviewing such things in light of recent changes and also maybe consider allowing us to customize the precedence as a part of our rulebase. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Test ordering/precedence
John, If you read this more carefully, I was not suggesting that action be taken that would affect everyone's system in such a way that it would require modifications. The 60 result code was recently changed from Gray rules to IP rules, and that change may or may not suggest a modification to the standard way that Sniffer operates (considering that the environment will only return one result code). Sniffer may or may not follow the numerical ordering of the result codes at present, but then again, it might. Regardless, it wouldn't be a bad idea to review the precedence as a part of ongoing due diligence. I also recommended one potential solution for customization by controlling the precedence from within the rule base and I would also imagine that the new config file could also be used to control this. So if a change was made, I'm sure it wouldn't be done unless it was measurable and would be to everyone's benefit, and it if Pete felt the need, it could be done in such a way so that only those that would want to change it would need to take action. I try to make it a practice to consider the needs of others before I give suggestions or ask for new capabilities, and I did do that in this case. I don't doubt that others have slightly different ordering in terms of what they feel is more and less accurate, and of course results can vary widely across systems. Pete is especially sensitive to these needs and has done a wonderful job of customizing rule bases without placing the burden on his customers to do so. Matt John Tolmachoff (Lists) wrote: Matt Matt Matt. Then everyone would have to make sure they made the relevant changes on their systems. As we have seen on the Declude Junkmail list, there will always be those who set up their systems and then forget about them. Making a change like that would cause problems. John Tolmachoff Engineer/Consultant/Owner eServices For You -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Matt Sent: Saturday, September 18, 2004 5:28 PM To: [EMAIL PROTECTED] Subject: [sniffer] Test ordering/precedence Pete, Given some of the recent changes in the result codes for Sniffer, I thought I would inquire about the precedence of the result codes and how these can affect systems. On my system I have weighted the result codes differently and overall, I would consider the following order to be suggestive of the order of reliability from the most reliable to the least reliable. Note that this is not scientific, but instead based on doing review and tests that hit less often could appear higher in terms of stated reliability though I have considered this in making the list: 1. SNIFFER-INK(56) SNIFFER-CASINO(59) SNIFFER-INSURANCE(48) SNIFFER-MEDIA(50) SNIFFER-GETRICH(57) SNIFFER-DEBT(58) SNIFFER-PHARMACY(52) 2. SNIFFER-AVSOFT(49) SNIFFER-PHISHING(53) 3. SNIFFER-TRAVEL(47) SNIFFER-PORN(54) 4. SNIFFER-SPAMWARE(51) SNIFFER-OBFUSCATION(61) SNIFFER-MALWARE(55) 5. SNIFFER-EXPERIMENTAL(62) 6. SNIFFER-GENERAL(63) 7. SNIFFER-IP(60) I'm not sure exactly how Sniffer orders the precedence of the result code, but I would like to recommend that you give some consideration to reviewing such things in light of recent changes and also maybe consider allowing us to customize the precedence as a part of our rulebase. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Test ordering/precedence
Thanks Pete, but let me just stress the largest issue that I see and I think you already are aware of it. The new IP classification is the most likely to produce false positives and it's result code of 60 places precedence of that over General, Experimental and Obfuscation hits. There is a large difference in accuracy on my system between IP rules and the other three tests. I hinted at this when you first made the change from that category being Gray (which I didn't score) to IP but got no response :) I score IP at 4 but the other three are all scored at a 6. The false positives with things like General tend to drop significantly over time as you report false positives, and I believe it to be over 98% accurate on my system while the IP hits have a much higher false positive rate based on open relay mail servers and message bounces to forged addresses that correspond to your spamtraps (I get a lot of IP hits on the bounce messages that we block, many of these from legitimate servers). I would have desired the IP hits to have been added as a result code of 64 instead of replacing the result code of 60 for this reason. I'm sure that you can run some stats to figure out how often IP hits might override General, Experimental and Obfuscation hits, and get a better idea as to the potential impact of having a generally higher scoring test hit. I know it would have an effect on weighted systems, though I'm not sure how large that effect might be. As things stand on my system, IP is the #3 test and I fear that it is stealing hits from more accurate tests, especially the #2 test, Experimental which happens to be very good at tagging zombies and hitting new sources of spam that aren't as widely blacklisted due to the types of rules that are present. Here are some recent numbers from my system: SNIFFER-EXPERIMENTAL...23.32% SNIFFER-IP...9.70% SNIFFER-OBFUSCATION...2.02% SNIFFER-GENERAL.1.64% So now might not be the time for this due to the potential of having to modify configs, but please minimally consider it at the next opportunity where a change such as the Gray to IP rules are done. Thanks, Matt Pete McNeil wrote: On Saturday, September 18, 2004, 9:07:55 PM, Matt wrote: M> John, M> If you read this more carefully, I was not suggesting that M> action betaken that would affect everyone's system in such a way M> that it wouldrequire modifications. The 60 result code was M> recently changed fromGray rules to IP rules, and that change may or M> may not suggest amodification to the standard way that Sniffer M> operates (consideringthat the environment will only return one M> result code). Sniffer may ormay not follow the numerical ordering M> of the result codes at present,but then again, it might. M> Regardless, it wouldn't be a bad idea toreview the precedence as a M> part of ongoing due diligence. I alsorecommended one potential I agree it's not a bad idea to review these things from time to time, and in fact we do quite frequently - though not publicly. I also agree that making any sweeping changes would probably be a mistake at this time. Well guys, here is how it goes. When more than one rule matches, the one with the lowest symbol # wins. If there is more than one match within that symbol then the one that is earliest in the message wins. This is why we code white rules to symbol 0, or symbol 1 in some cases; and also why we generally reserve the lower numbered symbols for any specific user requests. As much as possible we've ordered the rule groups so that the least specific rules are found in the higher numbers and the more specific rules are in the lower numbers. We even have some rules (work in progress) that are "above band" in the 65-255 range which have special meanings and functions. These will become more important later as these features are further developed. There are a lot of schemes out there that can be used, and in fact we can use an entirely different scheme for each user if we wish - though that might be a lot of work (so we might have to charge extra for the consulting time to develop and maintain such a thing). The scheme that we have is a little bit out of date*, but it still seems to work for most folks, so we'll probably keep it around for a while. We've had a number of alternate schemes suggested, some that might even be practical to implement - but none that wouldn't cause quite a bit of upheaval if we suddenly decided to rework everything for our current users. In fact, there are only a hand full of people who ever even mention it. Since your list shows 60, 63, 62, and 61 all at the bottom of your list I'm guessing that the current voting scheme is probably in line with your priorities at this point. That is, more specific rules (by symbol #) seem to line
Re: [sniffer] Test ordering/precedence
Pete McNeil wrote: M> SNIFFER-EXPERIMENTAL...23.32% M> SNIFFER-IP...9.70% M> SNIFFER-OBFUSCATION...2.02% M> SNIFFER-GENERAL.1.64% I must be tired, but I don't understand these numbers in this context. What are the percentages? These are hit rates on total messages with a spam percentage of about 87% that weekday, leaving 13% as ham of course. M> So now might not be the time for this due to the potential of having to M> modify configs, but please minimally consider it at the next opportunity M> where a change such as the Gray to IP rules are done. I've actually been thinking very strongly of reorganizing the rule group IDs recently. Especially in light of the new changes we've made with robots et al. The accuracy of the Experimental IP group has gone up considerably - and most of the false positives you've discussed should be eliminated over time (bounces especially). All that said, I think the first step to reordering the groups might be to change the sequence of the 4 highest numbers as follows: 63: Experimental Received [IP] 62: Obfuscation 61: Experimental Abstract 60: General I've had a few more false positives recently on the Experimental Abstract group, however I haven't yet come to terms with what that means in the face of the increase in hit rates for that group. If this group is populated by automated means, I would consider it to be maybe more wise to have it above the Obfuscation category which I have found to be much more accurate and not specific to a particular host, but instead the content. The General rules are are also very unlikely to hit on personal E-mail, which makes false positives with these two groups much more tolerable because most such content isn't missed that much. When it comes to the IP rules and the Experimental Abstract rules however, many of the false positives are on personal E-mail. If I was to weight the accuracy of the tests considering the difference in how they might hit personal E-mail, I would prefer an order as follows: 63: Experimental Received [IP] 62: Experimental Abstract 61: General 60: Obfuscation I weight the Experimental Abstract and General rules the same on my system, so reversing them isn't such a big deal, but the primary change from your recommendation would be that I would suggest putting Obfuscation rules the lowest result code of the four. I think the likelihood that a rule can hit on a personal E-mail is key in this decision making process. Also consider the hit rate of the Experimental Abstract rules being so high and a more exact non-automated method might make other tests more reliable in the long-term so they should get predominance. According to a recent spam test quality analysis the accuracy and coverage for these groups in this order follows like this: 63: Experimental Received [IP]SA = 0.81 Coverage = 7.63% 62: Obfuscation SA = 1.00 Coverage = 2.58% 61: Experimental Abstract SA = 0.92 Coverage = 25.82% 60: General SA = 0.81 Coverage = 1.82% Please take note that Markus' stats are generated from European traffic and I have found at times that there can be a measurable difference between what he is seeing on his system and what I am seeing on my system where about 95% of the legitimate traffic is from North American hosts and to local domains all ending with .com, .net and .org. His FP rate on the General group for instance is many times higher than my own. I would be happy to share my Declude logs with you if you wish to process stats that come from about 200 domains, mostly based in the US. If it doesn't take too long, I would be willing to set up the beta of the stats package for comparative purposes. As far as making changes go, I wouldn't rush into anything, especially since this change would likely mean readjusting all of your clients at once instead of having them download a new release and following instructions. I think it would be a good idea to alert your customers further and more frequently in advance than was done with the Gray to IP rule change. Of course the change wouldn't have a large effect on many systems, but this is definitely something that would affect my own. It is of course easily remedied with some very quick changes in my Declude config file. Thanks, Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Integrating Sniffer with new Imail Collaboration Suite
Andrew, You would also need to lock the file so that the queue didn't steal it. You might be able to just simply rename the Q* file to get the same effect. Pete's being a good friend of Declude by not suggesting a modification, but also a good friend of the user to recommend a much better solution that also brings with it the benefit of weighting, RBL's and other scoreable hits. If you already own Declude and are sticking with IMail, there is absolutely no reason to abandon it. IMO, Ipswitch is about to change their tune on the matter of bundling after signs that they recognize the miscalculation in the response. A miscalculation of this magnitude however points to a more systematic problem, and their claim that service agreements are money losers for them is preposterous, and then to place the burden on the customer when your product is already premium priced is an indication of an organization in total disarray. Matt Colbeck, Andrew wrote: Well, to play devil's advocate ... A poor man's way to run IMail and Message Sniffer without Declude could certainly be done without a massive re-write. I'm not going to claim that it would be *reliable* or *flexible* but you could certainly mimic what Declude does and change one registry key to have IMail call a batch file. Then have your batch file call Message Sniffer, run through multiple "if errorlevel" statements, and take whatever action you deem appropriate, like moving the two message files to a quarantine or just deleting them. If a message passes, you call IMail's original executable (from the original registry entry) to deliver the message. Sure, saying it is easier than doing it, but it is do-able. As for me, I prefer to use Declude & Sniffer. A weighted system rocks. Andrew 8) p.s. Now, if SpamAssassin has a way to shell out to call Sniffer ... hm ... -Original Message- From: Pete McNeil [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 27, 2004 8:52 AM To: Jim Matuska Subject: Re: [sniffer] Integrating Sniffer with new Imail Collaboration Suite On Wednesday, October 27, 2004, 11:30:27 AM, Jim wrote: JM> Is there a way to integrate message sniffer directly with the new JM> Imail Collaboration Suite. We are currently using it with Imail via JM> declude, but that may change soon due to this latest Imail fiasco. JM> If we decide to migrate to the new Collaboration suite I need to JM> know if we can use message sniffer directly or if we would need to JM> use a 3rd party add in still such as declude (if a version is JM> released that will work with the collaboration suite). Any JM> thoughts? Sniffer won't be making a direct connection to IMail because it won't be necessary. It is my understanding the Declude and mxGuard will continue to function with ICS just as they have. There is no good reason for us to duplicate that effort when such a fine job has already been done. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Your Sniffer Setup
Andy, Bill, et al. When the persistent Sniffer was first offered, I typed up the attached directions that I cribbed from the KB when alerted to it by Bill. I am forwarding this as a message attachment since the archives are down currently. I haven't yet upgraded to the latest version, but at least on previous versions it has been running fine. I'm still waiting to figure out what the issues might be relating to this thread. An export of my registry relating to the Sniffer service is as follows: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer] "Type"=dword:0010 "Start"=dword:0002 "ErrorControl"=dword:0001 "ImagePath"=(removed: hex encoded path to srvany.exe) "DisplayName"="Sniffer" "ObjectName"="LocalSystem" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer\Parameters] "Application"="C:\\IMail\\Declude\\Sniffer\\MyExecutableName.exe MyIDNumber persistent" [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer\Security] "Security"=(removed: hex encoded value) [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer\Enum] "0"="Root\\LEGACY_SNIFFER\\" "Count"=dword:0001 "NextInstance"=dword:0001 Sorry to keep this going, but I would like to figure out what the best practices would be, and also help Andy and/or others figure out the same. Matt -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = --- Begin Message --- Ok, I think I did it. Only took a minute (thanks Bill). Here are some more precise directions, but consider them to be "beta" directions (please correct them if you find a problem): 1) Install the Windows 2000 Resource Kit, or download and install the INSTSRV.exe and SRVANY.exe files in a permanent location, preferably within your path. The individual files can be found at the following location: http://www.pyeung.com/pages/win2k/userdefinedservice.html 2) Open a command prompt (Click on the Start Button, Select Run, and type CMD) 3) Enter the following command (customize for the paths of the executables) C:\Progra~1\Resour~1\INSTSRV Sniffer C:\Progra~1\Resour~1\SRVANY.exe 4) Open up the Registry Editor (Click on the Start Button, select Run, and type REGEDIT) 5) Locate the following key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sniffer 6) From the Edit menu, select New, select Key, and name the new key Parameters 7) Highlight the Parameters key 8) From the Edit menu, select New, select String Value, and name the new value Application 9) From the Edit menu, select Modify, and type in the full path name and application name, including the drive letter and file extension (don't use quotes, customize path, executable name and authentication code) Example: C:\IMail\Declude\Sniffer\[yourlicx].exe [authenticationxx] persistent [yourlicx] = your license ID [authenticationxx] = your authentication string 10) Open the Services MMC 11) Start the Sniffer service 12) Set the Sniffer service to Automatic Matt Matt wrote: I'm going to give this one a try right now since I have the Resource Kit installed already. Just one question...do I need to change the arguments in my Declude config, or will the service definition take care of the 'persistence'? Thanks, Matt Bill Boebel wrote: We've been using svrany for years with several custom applications and it works great. This utility has been around since the NT4 Resource Kit... http://www.pyeung.com/pages/win2k/userdefinedservice.html Bill -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Pete McNeil Sent: Friday, March 19, 2004 12:25 AM To: [EMAIL PROTECTED] Subject: [sniffer] RunExeSvc for Persistent sniffer. Hello folks, We've been continuing to test the new persistence enabled sniffer engine and some utilities that will allow it to run as a service. We found a free utility that seems to be very solid, and very simple. http://www.judoscript.com/goodies/RunExeSvc/runexesvc.html One of the scripts we used is: debug=false cmdline=c:\Projects\sniffer2-3\TestBed\snfrv2r2.exe xnk05x5vmipeaof7 persistent home=c:\Projects\sniffer2-3\TestBed (Note: The mismatch between the sniffer2-3 directory and the snfrv2r2.exe is not a type-o. We re-branded the 2-3 to use the snfrv2r2 license in our example - it was easier that than creating a new license. Note also that the cmdline parameter i
Re: [sniffer] Your Sniffer Setup
This might be there in the event that you need to quote certain arguments or handle special characters??? I've found some different requirements for command line arguments and special characters such as "&" which require either quoting them or using an octal encoded value (I'm no expert on this stuff). Maybe the alternate field helps in this instance. Anyway, it looks like it is unnecessary although functional in this instance. Considering that there are many places where you enter both path and arguments in the same registry value, I would assume that there is no problem with doing it that way for the service. Matt Andy Schmidt wrote: Yes, I too suspect that SRVANY actually allows the specifying of the entire command line in the "Appliation" string, even though both the Knowledgebase article and the full documentation implies otherwise. (The KB article and the documentation are very precise in what the "Application" string should be: just the path, name and extension of the executable.) The question is whether Microsoft ever intended it to work that way or if that possibly "accidental" capability may cease working at a later time. Best Regards Andy Schmidt H&M Systems Software, Inc. 600 East Crescent Avenue, Suite 203 Upper Saddle River, NJ 07458-1846 Phone: +1 201 934-3414 x20 (Business) Fax:+1 201 934-9206 http://www.HM-Software.com/ -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Mark E. Smith Sent: Monday, November 01, 2004 02:27 PM To: [EMAIL PROTECTED] Subject: RE: [sniffer] Your Sniffer Setup Looks like both work. If you examine the difference you'll probably see why. One (just with the Application setting specifies all of the parameters in the SZ string. The other specifies the .exe in the App string and the Auth Code and "persistent" parameter in the parameters string. I'm also guessing that Sniffer really doesn't care about the app path so it's probably working in this case. The "proper" way is probably the way where multiple SZ values are specified although both will work with Sniffer. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ = This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
