[sniffer] Upgrades and changing IPs
Hi Sniffer Folks, We are upgrading many of our systems this month. While most of that work will be invisible to you, some IP addresses will change and short disruptions might occur. No reason to be alarmed, SNF itself is designed to operate for long periods even if our systems are down. BUT, if you have made specific rules in your firewalls to allow communications with our systems -- maybe for SYNC or for getting rulebases etc, then you will need to update those rules. The vast majority of you don't have such rules and so will probably not even notice the updates as they occur. I just wanted to put this out there to explain any changes that are observed. Best, _M -- Pete McNeil (MadScientist) ARM Research Labs, LLC # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
RE: [sniffer] Call for beta testers... snfrv2r3b1
I am still working a problem at our hosting facility (a t1 is down) so it will be a while before I can get back to the list, however I wanted to clear this one up to minimize confusion. A persistent server instance uses a dynamic poll timing algorithm to minimize system loads while maximizing response times. It is probably not appropriate to use a fixed time interval for polling since this can cause unnecessary system loading and since the dynamic approach we're using has proven in our labs to offer improved all-around performance. When the server has processed a job for a client it polls again immediately without waiting. If no job is found then it will wait a short time before polling again. If there are no available clients on a poll then the wait time between polls will increase in a natural spiral based on a Fibonacci sequence. This wait time will continue to expand until either a new job is found or the limit is reached. The limit is currently set to 1/2 the maximum client base wait time - which amounts to 4 seconds. It's worth noting that in order for a server instance to get to a given wait time (such as 4 seconds) there must have been no messages to process for that amount of time. It's also worth noting that some folks using spamassassin regularly report message processing times on the order of 5 to 9+ full seconds for each message (I just read this on the sa list). Based on these two factors I've considered that waiting a maximum of 4 seconds to process a message after a 4 second lull in activity is probably not an issue - especially considering that once the message is processed it will likely take only an additional 30ms or so on average for a total of 4.030 sec (ymmv). This also represents the worst case given the current tuning parameters... Once a job is found then the wait time is reset to the minimum. Once again, the first poll after a job has been processed has no wait time... so if there is a burst of message activity after a 4+ second lull, the first message waits a maximum of 4 seconds and the rest wait only a few tens of milliseconds. The monitor messages you are seeing are only a debugging/tuning aid and they will be removed for the production version. The timing message is only emitted when the server instance has found no messages to process during the previous poll. Hope this helps, _M PS: In a situation where peer-server instances become mixed it is possible for more than one server instance to become active for a period of time. The Fibonacci timing spiral helps to ensure a distributed scattering of lock requests when multiple instances are active - thus reducing collisions. At 03:04 PM 3/17/2004, you wrote: Pete, After my previous message, I noticed that 'polling' really means that Sniffer is waiting that many milliseconds before it processes another e-mail. If I'm seeing this correctly, I'd like to request another option available when spawning the persistent exe: /polling:x (where x = a fixed amount of milliseconds between polling) Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOS Small Office Solutions / Reject / Wannepad 27 - 1066 HW -Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy - Installation - Maintenance Network Security - Internet - E-mail Software Development - Project Management -- -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: woensdag 17 maart 2004 20:05 To: [EMAIL PROTECTED] Subject: [sniffer] Call for beta testers... snfrv2r3b1 Hello folks, I know folks are anxious to get their hands on this version so I'm going to play this beta round a little looser than usual. Version 2-3b1 implements a persistent mode feature for our cellular peer-server technology. Launching a persistent instance of Message Sniffer has the effect of creating a daemon so that all other instances will elect to be clients. We observed a DRAMATIC improvement in system performance on our NT4/Imail/Declude test bed. In static tests on my Toshiba 6100 we saw no memory leaks and consistent performance over the past 18+ hours of testing. This included several tests with more than 100+ concurrent client instances - all without failure and without making the system unresponsive (though the WinXP file system did start to show signs of strain). This beta is for the windows platform only... once we're happy with this version will will make the source and *nix versions available as always. Windows platform users who are interested in testing the new beta should download the following file: http://www.sortmonster.com/MessageSniffer/Betas/snfrv2r3b1.zip The file contains an executable and a short readme file. We are going to be extremely busy for the next few hours so we won't be able to provide support on this until later
Re: [sniffer] SLM files
At 03:30 PM 3/17/2004, you wrote: I have Imail 7.07 running on Win2000, with Declude Junkmail I come up with errors scanning .SLM files. Does sniffer use SLM files to process the messages. Attached a snip from my log files Sniffer scans whatever file is passed to it with the expectation that it is an SMTP message. It doesn't make any special allowances for the type of file that is passed. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] updater script for Linux
I'm not sure - but I think there are user submitted perl based update scripts on the help page that probably do all of this: http://www.sortmonster.com/MessageSniffer/Help/AutomatingUpdatesHelp.html Hope this helps, _M At 11:05 PM 3/5/2004, you wrote: Has anyone written a good Sniffer updater script for Linux which has the error checking like the one for Windows has? Bill This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Bagle J others
At 01:33 PM 3/3/2004, you wrote: On Mar 3, 2004, at 12:44 PM, Madscientist wrote: We have adopted the current policy at least for the short term: 1 ) We block all potentially hazardous extensions including .zip. Can these virus rules be bypassed? We have real virus checking and don't want our spam checker doing any virus blocking. Thanks. Yes. Any rule can be blocked from any rulebase. I made a mistake when I posted my original message. It is confusing. The Malware rules we are coding into the system only block messages that match known virii/worm patterns, and of those, we are focusing only on those that have .zip file attached. We are not focusing on other .exe types. Just to be clear, the malware rules we are putting in place are very much the same as malware rules we have coded in the past. We are not creating any rules that block attachments. I apologize again for the confusion. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Rules Question
At 04:55 PM 3/3/2004, you wrote: I am using Declude and have indiv. Sniffer Tests and lets say the following gets tripped in an email SNIFFER-WHTLIST result code 000 SNIFFER-PORNresult code 054 Which would take precedence over the other, as far as which would be the final code passed to Declude? There is some confusion about this. A zero result from Message Sniffer as seen by Declude could mean that a white rule has fired, or it could mean that no rules matched at all. In the first case - where an actual white rule has fired, the Message Sniffer log will show a White entry and the Final result will reflect that white rule. In this case, the white rule takes precedence. Declude will see a 0 result code. In the second case - where no rules matched, the Message Sniffer log will show a Clean entry and Declude will see a zero result. So, from Declude's perspective it will see a zero result in both the Clean and the White case. As a result, your SNIFFER-WHTLIST result code 000 test will fire. In a case where a white rule is present and a black rule is present the white rule will always win. So, if Sniffer saw both rules match a message it would return a zero result. SNIFFER-WHTLIST is a misnomer. It's probably not a good idea to name the zero result test this way because most of the time a zero result doesn't mean White but instead means Clean. If you wish to have the white rules in your rulebase separated out then we could code those to a 1 result and then you would be able to legitimately create a SNIFFER-WHTLIST test checking for a result of 1. I will point out here that this has been tried once or twice and in both cases the user switched back almost immediately because the results were confusing. In Sniffer we use white rules to force a non result more than we ever use them to indicate a true white result. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Rules Question
White rules are entered either upon request or in response to a false positive report with your permission. In some cases we will enter a white rule based on our own research or in response to a false positive report if we feel a core white rule would be more appropriate. We add core white rules without permission. We add local rules of any type only with permission or by request. Hope this helps, _M At 06:43 PM 3/3/2004, you wrote: Thanks for the aid. One last question, you mentioned: In a case where a white rule is present and a black rule is present the white rule will always win So if the White Rule fired 000, it would override a Porn Rule of 54? If so, how are these White Rules entered? Thanks, Keith -Original Message- From: [EMAIL PROTECTED] on behalf of Madscientist Sent: Wed 3/3/2004 6:01 PM To: [EMAIL PROTECTED] Cc: Subject: Re: [sniffer] Rules Question At 04:55 PM 3/3/2004, you wrote: I am using Declude and have indiv. Sniffer Tests and lets say the following gets tripped in an email SNIFFER-WHTLIST result code 000 SNIFFER-PORNresult code 054 Which would take precedence over the other, as far as which would be the final code passed to Declude? There is some confusion about this. A zero result from Message Sniffer as seen by Declude could mean that a white rule has fired, or it could mean that no rules matched at all. In the first case - where an actual white rule has fired, the Message Sniffer log will show a White entry and the Final result will reflect that white rule. In this case, the white rule takes precedence. Declude will see a 0 result code. In the second case - where no rules matched, the Message Sniffer log will show a Clean entry and Declude will see a zero result. So, from Declude's perspective it will see a zero result in both the Clean and the White case. As a result, your SNIFFER-WHTLIST result code 000 test will fire. In a case where a white rule is present and a black rule is present the white rule will always win. So, if Sniffer saw both rules match a message it would return a zero result. SNIFFER-WHTLIST is a misnomer. It's probably not a good idea to name the zero result test this way because most of the time a zero result doesn't mean White but instead means Clean. If you wish to have the white rules in your rulebase separated out then we could code those to a 1 result and then you would be able to legitimately create a SNIFFER-WHTLIST test checking for a result of 1. I will point out here that this has been tried once or twice and in both cases the user switched back almost immediately because the results were confusing. In Sniffer we use white rules to force a non result more than we ever use them to indicate a true white result. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html p/ This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Moving follow up...
Hello Sniffer Folks. The critical portions of our move have been completed. We had very few outages. We are not expecting any more. False and Spam processing schedules will stabilize over the next day or so. Thanks for your support! _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Tanx / Bagle.b
Hello folks, The new worm Tanx / Bagle.b seems to be spreading quickly. We have added a rule to Sniffer for this and we are currently pushing it out to all rulebases. Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation. Chief SortMonster, www.SortMonster.com. Vox 703-406-2016, Fax 703-406-2017 This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Referrals page.
Now I understand. Certainly - we will add the referral link. Thanks! _M At 02:56 PM 2/17/2004, you wrote: In that case, I should rephrase my request: In addition to our software product for IMail, we also offer email services to individuals and businesses. http://www.mxguard.com/individual http://www.mxguard.com/organization We currently describe how the service works at: http://www.mxguard.com/individual/how_it_works.asp There is a blurb about Sniffer at the bottom of the page (that I just noticed needs an image and a link to you). Maybe you can link to these pages? You guys are already linked through our Installation pages - you have a page to your selves in fact :-) http://www.sortmonster.com/MessageSniffer/Installation/IMail-mxGuard.html The referrals page is for links to service/product providers who use and reference Sniffer. Hope this helps, _M At 12:30 PM 2/17/2004, you wrote: Pete, We interface with your product very well. Please consider adding our *mxGuard for IMail* website to your list: http://www.mxguard.com/postmaster Regards, David Gregg dgSoft Internet Services +1.949.584-1514 --- mxGuard for IMail Server based spam and virus protection for under $100 Request a free trial at http://www.mxGuard.com/postmaster --- - Original Message - From: Madscientist [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Tuesday, February 17, 2004 9:11 AM Subject: [sniffer] Referrals page. Our referrals page is up and running. http://www.sortmonster.com/MessageSniffer/Referrals.html Thanks, _M Pete McNeil (Madscientist) President, MicroNeil Research Corporation. Chief SortMonster, www.SortMonster.com. Vox 703-406-2016, Fax 703-406-2017 This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html Pete McNeil (Madscientist) President, MicroNeil Research Corporation. Chief SortMonster, www.SortMonster.com. Vox 703-406-2016, Fax 703-406-2017 This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html Pete McNeil (Madscientist) President, MicroNeil Research Corporation. Chief SortMonster, www.SortMonster.com. Vox 703-406-2016, Fax 703-406-2017 This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Autoupdating rule file
At 10:49 AM 2/12/2004, you wrote: On Feb 12, 2004, at 8:58 AM, Timothy C. Bohen wrote: Anyone willing to send me a script that I can use? Sure, here's mine written in Perl. It knows enough to check the timestamps so it doesn't fetch files when unecessary, keeps a backup copy, and does everything in a safe manner such as to not leave your system in an unusable state at any time. It relies on the fact that the rename() function is atomic. I don't make that guarantee on non-unix systems. Slightly off topic - Be careful with that assumption. rename() is NOT atomic on windows systems. You script should work since it won't be competing with multiple instances of itself and is not coordinating with other threads, but it's good to keep in mind for other projects. Similarly, writes to files in append mode are also not atomic in windows. Watch out! _M This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
[sniffer] Recent hotmail false positives and click atdmt
Hello folks, Rule 11075 in the gray hosting group has been temporarily suspended. This is one of our strongest rules which has been in place for more than 500 days. Microsoft recently began using this service to post an advertising link at the bottom of all of their messages. We have been trying to compensate for this by creating white rules, however the combinations are growing without bounds - particularly where forwarding is concerned - so we are abandoning this rule for the time being. Due to the rule's strength ( 4.0) there will likely be an increase in spam for a short period while we develop additional black rules to compensate for specific spam associated with this service. Faced with the choice of creating false positives for all hotmail, or dealing with increased spam as a result of dropping the rule our policy is always to avoid the false positives wherever practical. I wanted to let everyone know about this since there may be a sudden noticeable change in filtering effectiveness, however short lived we can make it. Thanks, _M This E-Mail came from the [EMAIL PROTECTED] mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Rule Strength Analysis Window Change.
I found myself wondering why the message suddenly got through so I did some digging. Turns out the message that got through was sent via 65.32.5.133 which was another Experimental IP rule that had just been pulled. I'm guessing the rule was in place when your previous notes were sent. The false@ address handles filtering differently than our normal addresses (for obvious reasons). An explanation about our Experimental IP rule program: A few months ago when DNSBLs started to be heavily attacked and defeated by spammers, we implemented a policy of capturing source IPs to verified spam that reaches our spamtraps. This is in addition to our standard practices of capturing domains, links, structural features, obfuscation mechanisms, etc... Recently we have had a higher than normal rate of false positives on experimental IP rules - probably due to the increase in worm activity. Our policy on Experimental IP rules is very conservative and has just been made more so: 1. We only add single IP sources as part of this program, not blocks. (blocks may be added through other research). 2. We only add source IPs when we have no doubt about the message we are reviewing and the source is through one of our spamtraps - user submissions are not used for sourcing IP rules. 3. IP source rules are permanently removed on the first legitimate false positive report. Once an IP rule is removed, it cannot be added back to the core rulebase. It can be added to specific rulebases by request only. The intent of the Experimental IP rule program is two fold: 1. Incrementally build and maintain an IP map of sources where there is unanimous agreement that the source is not legitimate (as defined by our user base). That means, if anybody finds an FP on an IP it is no longer eligible for this program. 2. Call attention to compromised equipment quickly wherever it is appropriate and assist in correcting the problem if possible. For example, we recently worked with a local military base to identify and correct a source on their network that was being used to relay porn (and other) spam. As is always the case, our registered users can block this rule group or any specific rules if they wish. If after seeing this explanation you wish to block this rule group from your rulebase please send a note to support@ (off list). I don't advise this since this program is very effective, but I don't wish to discourage it either. In the end the rulebase must be compatible with your specific policies. Hope this helps, Thanks! _M At 06:00 PM 2/10/2004, you wrote: List Folks! The Sniffer guys are awesome and responded immediately with a phone call when my previous post today finally went thru! I have been sending support e-mails with header info, snippets from my logs, etc. to support@ and the list - but they were not getting thru. Unfortunately, I was not sending to the correct address even though I read it many times to o so. The reason I did not, is as I was concerned that my rule base would have been updated allowing e-mail from those domains we host to be wide open. I learned that this would not have been the case and I would have been contacted prior to any such changes. The cause was due to our e-mails failing Code 84701 Symbol 62 which was catching a rule base filtering on IP 65.32.5.132 which is Road Runner in Tampa Bay. This was causing our own e-mail domains we host to fail. Once identified on phone it was immediately corrected and all back to normal. Unfortunately, I did not submit my e-mails to [EMAIL PROTECTED] as instructed... (see http://www.sortmonster.com/MessageSniffer/Help/FalsePositivesHelp.html) ...which would have avoided all my frustrations. Also, I found out that they do have a phone number on the Micro Neil site. Pete informed me that they are going to look into another contact or reporting e-mail address / procedure when someone gets to the point of panic mode, which I was nearing. I want to reiterate that Micro Neil, once they got my message responded immediately and professionally and I was really at fault by not submitting my info to the false@ address. Thanks. -Don -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Madscientist Sent: Tuesday, February 10, 2004 5:26 PM To: [EMAIL PROTECTED] Subject: RE: [sniffer] Rule Strength Analysis Window Change. We didn't get your notes. I'll call you right away. _M At 05:11 PM 2/10/2004, you wrote: I have sent email several times to this list and support and even Pete's email addy which I picked up from a post and both from my personal email and our special registered email address [EMAIL PROTECTED] I am again trying today. I know of no other way to contact someone there and if I could secure a phone number would call. It seems none of our emails are getting through. We are having a major problem whereas any e-mail sent from any domain hosted to another domain hosted are getting caught by Sniffer. Can someone
