[sniffer] Re: Our IP got listed on GBUdb Truncate
On 11/2/18 11:52, Daniel Bayerdorffer wrote: > > Is there anyway for us to see what the offending email was that got us > on the list? Or some other data point to help us clean up our system? SNF doesn't leak message info -- With the exception of auto-sampling of spam (truncated messages, and only if you have it enabled) we don't see message content. What we do get are anonymous statistics and training data. The good news is that you are running SNF, so you can scan your messages and identify any content that might have triggered SNF. Truncate is trained by counting good and bad events -- bad events are when a message matches spam/malware patterns. ... so you can actually check with your own scanner. Truncate is completely automated... so we can't change the list data. It actually doesn't come from a database but rather by skimming the telemetry for these events. In effect the reputation for any given IP resides in each SNF instance around the globe and the truncate list works by eves-dropping on the conversations between those nodes as they "discuss" IP reputations. If the IP is still listed and you send a note to support with the IP requesting a trace then we can collect some events with timestamps. That may help you track things down -- but since you're an SNF user you would probably do better with your own scanner. Hope this helps. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list . This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: To switch to the DIGEST mode, E-mail to To switch to the INDEX mode, E-mail to Send administrative queries to
[sniffer] Happy Holidays!
This is just a quick note to let you all know that we're thinking of you. On behalf of the whole team: We wish you a Merry Christmas and a happy, prosperous New Year. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Rulebase refactoring
Hi Message Sniffer folks, Over the past few days we've refactored the databases we use to manage our rulebase. As of about 1600e you should notice that all of the rule IDs in your system are significantly smaller and completely different. Unfortunately, during the transition there were several unforeseen problems that introduced delays and other disruptions. We apologize for the inconvenience. All is well now. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Reminder - the Rule Panic feature
Hello Sniffer Folks, In light of today's bad rule event I've discovered that many of you are not aware of the rule-panic feature. The rule panic feature has been built in to the Message Sniffer engine for many years now, and I suppose is used so rarely that folks have forgotten about it. The feature allows you to render any single rule inert immediately without disrupting anything else in the system. So, it could have been used to mitigate this event without taking more drastic measures. Here is a link to the QA article about the rule panic feature: http://www.armresearch.com/Documentation/QA/ltrulepanicsgt-628138610.jsp Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Bad Rule Alert 2654821
Hello Message Sniffer folks, This morning a dormant rule from 2009 was reactivated when new messages reached our spamtraps this morning matching the rule. Unfortunately rule 2654821 causes a high rate of false positives in our current year that it apparently did not cause back in 2009. Since the rule was not recently coded and had been in the system for so many years our monitoring systems did not immediately detect the rule as a false positive case. However, the team did discover the problem after a few hours and removed the rule. This is the only time an old, reactivated rule has caused significant false positive cases -- so it is an exceedingly rare event. None the less we are in the process of reviewing our tools and processes to improve our sensitivity should any similar event occur in the future. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: rule panic not working
On 12/29/2016 08:55 AM, Daniel Ivey wrote: Thanks, but it appears that my server is failing multiple 54- rules. For example from Google, it is failing 54-8064853-304-318-m and 54-8064853-0-2423-f while from Yahoo it is failing 54-8064853-2063-2077-m and 54-8064853-0-3703-f. That is in fact a single rule hitting in multiple places. http://www.armresearch.com/Documentation/QA/ltmatchesgt-1193870513.jsp The rule ID is 8064853. The rule has been removed. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Error Code 69
On 12/15/2016 07:04 AM, Don Winsauer wrote: I have had 419 occurrences of this error since the 1st of the month. I don't even run a virus scanner on this Windows mail server. We are running IMail, Declude with Sniffer. This could be an indication of a file system problem? The only reason that occurs is when the file system / OS prevents SNF from removing the original file. Are the files still there? What changed since the 1st? (did the problem begin then precisely?) _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Error Code 69
On 12/14/2016 06:12 PM, John Tolmachoff wrote: When SNF is configured to inject headers it does so safely--- First, it reads the entire original message into a buffer, then scans it,... Then it writes a new copy of the message to a .tmp file with the headers injected. When that completes without errors, it deletes the original and renames the .tmp file in it's place. That way, if anything goes wrong, the original is always there as a backup until the last moment. The error above indicates that SNF was trying to delete the original message file so that it could move the new version over. Something went wrong and it wasn't able to do that. On windows systems this is most likely because something removed the file first -- perhaps a virus scanner or some other process. On linux systems it's usually a permissions issue (but this does sometimes happen on windows in rare cases). So, if you can figure out what is preventing SNF from deleting the original file you will solve the problem. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: DEB Packages
On 12/01/2016 02:07 PM, Daniel Bayerdorffer wrote: I see that the DEB packages for Message Sniffer are for Ubuntu 14.04. Will these work with 16.04? They should -- there haven't been any significant changes in SNF nor in the parts of Ubuntu that SNF cares about. Still, the packages are considered experimental (mostly due to a lack of exhaustive testing) so be ready to roll back just in case; and do share your results with us. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: .smd.tmp files being left in proc\work folders
On 08/09/2016 05:11 PM, Don Winsauer wrote: These file are being left and not being delivered. They are usually over 20mg. Something is preventing SNF from renaming the file. Find out what it is and then prevent it from blocking SNF. Perhaps a virus scanner has the file open when SNF comes by to rename it?? SNF tries to do everything safely, so when injecting it's headers it writes the new message file with a .tmp extension and makes sure that succeeded before it removes the original and then renames the .tmp. My guess is that the new .tmp file catches the attention of some scanner or other program and that when SNF goes to rename the .tmp file to replace the original it is unable to do it. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] SNF Engine Update to 3.2.1 / Short Buffer Bug Fix
Hi Sniffer folks, Today we have released a new SNF engine with a minor bug fix. Please update your SNF installation at your convenience. Chances are that you've not seen any problems from this bug. If you have experienced problems they most likely presented as very rare, random errors possibly causing a crash. As with most SNF engine updates the simplest process is to replace your binary with the latest. For windows users here are some links to the latest engine: http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox32-3.2.1.exe http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox64-3.2.1.exe Simply stop your SNFServer, swap in the new .exe (renamed of course) and restart SNFServer. For folks running linux platforms the packages and source tarballs on our web site have all been updated. OEMs using the windows SDK should upgrade to the latest DLL which should be a swap-in replacement. http://www.armresearch.com/Downloads/index.jsp --- Technical details: The bug fix is for a short buffer allocation in the codedweller/configuration.cpp module. The bug fix also solves problems unrelated to SNF where applications using the CodeDweller/configuration engine to parse unusually large XML attributes could cause a stack overflow. The solution allocates the buffer for attributes from the heap instead of the stack and eliminates a short-by-one allocation error. Those curious about the source code can see the important diff here: Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller
[sniffer] Re: [Alligate]Alligate and Sniffer again (NL)
On 01/18/2016 07:26 AM, Bonno Bloksma wrote: Hi, Ok, downloaded Alligate trial, installed in on a 2012 R2 server. Made a local dns “server” (resolver) on the machine but I am not sure if I need it now that we can use the Google dns server by default. A local resolver will speed up your bl lookups quite a bit since they don't have to traverse the network. How do I hook up Sniffer? I used to have Declude (and IMail) and had Sniffer connected that way, I now need to connect sniffer into Alligate. I cannot find anything in the Alligate Docs I downloaded. As far as I know SNF4Alligate still works. https://www.armresearch.com/Documentation/Papers/InstallGuides/SNF4Alligate.jsp Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: New Version -- SNFMulti 3.2.0 -- Strangers
On 2016-01-04 11:44, Daniel Bayerdorffer wrote: > Are there any other gotcha's I should be aware of? I took a quick look through the tarball and was reminded -- all of the configuration elements are provided as samples after make-install. The instructions say to copy the samples to their correct names and then modify them appropriately-- so that part of it is a manual process. In that case it should be safe to do make install and just skip those steps since your configuration is already happy. All that said; again -- you're really only interested in updating your SNFServer binary. The rest isn't changed. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: New Version -- SNFMulti 3.2.0 -- Strangers
On 2016-01-04 11:44, Daniel Bayerdorffer wrote: > Hi Pete, > > I have a couple of questions about upgrading. We will be upgrading SNF4SA > running on Ubuntu 14.04 with Zimbra email server. > > I previously compiled the source code to install SNF4SA. Can I compile the > latest version and run the make-install to overwrite the existing version? If > so, do I need to re-apply our license information to the configuration files, > etc.? I think that should work. The only thing that's really different is SNFServer. You could stop after make and then replace your SNFServer binary leaving everything else in place. I don't think there are any other gotchas. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: [BULKMAILER] [sniffer] Windows SDK with SNFMulti 3.2.0 -- coming soon.
On 2015-12-28 11:28, Michael Murdoch wrote: > I AM. The latest Windows SDK is posted. It's exactly like the previous one except we changed the version number and the DLLs have been updated. They should be a drop-in replacement for the previous DLLs. http://www.armresearch.com/message-sniffer/download/updates/SNFMultiSDK_Windows_3.3.zip Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Windows SDK with SNFMulti 3.2.0 -- coming soon.
Hi Sniffer Foiks, If you're curious about the Windows SDK (DLLs) ... they should be posted in the next few days, but not yet. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] New Version -- SNFMulti 3.2.0 -- Strangers
Hello Sniffer Folks, A new version of Message Sniffer is available. The most exciting new feature for this version is: Strangers. The "Strangers" algorithm replaces the previous White-Guard algorithm. Strangers prevents high-intensity pre-tested spam from poisoning IP reputations in GBUdb and enhances SNF's sensitivity to these kinds of attacks. Once pattern rules begin to match the pre-tested attack the IP reputations quickly climb into the black enhancing all of SNF's learning systems. Normal, but new, IP sources are held to low-confidence reputations for several hours, but after that are allowed to develop normally. Short summary: Strangers lets SNF close the door more quickly on pre-tested spam while enhancing SNF's learning sensitivity to those events and without interfering with normal IP reputation processing. Here are some links: Packages from the LabRats... http://www.armresearch.com/message-sniffer/download/packages/ SNFMilter tarball... http://www.armresearch.com/message-sniffer/download/updates/snf-milter-1.2.0.tar.gz SNFServer tarball... http://www.armresearch.com/message-sniffer/download/updates/snf-server-3.2.0.tar.gz SNFServer 32bit Windows exe... http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox32-3.2.0.exe Not better, but if you _really_ want it ... SNFServer 64bit Windows exe... http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox64-3.2.0.exe Thanks! and Happy Holidays! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] ShortMatch Resolved - Update your SNF software to remain immune.
Hi Sniffer Folks, According to our latest data, the Short-Match FP problem has subsided - most likely due to rule sequestration. We have not seen any significant events in our detection software since 2100e last evening. In the mean time we have updated the SNF software to check for short-match events and treat them as rule-panic events. This renders them inert so that if this kind of rulebase corruption occurs again the SNF engine will be immune. Please update your SNF software to this latest version using the links below. NOTE: The Windows installer is in the process of being redesigned and does not have the latest software. This will take some time. If you are using SNF on Windows and use(d) the installer then use this procedure to update your software: * Stop your SNF service (usually XYNT Service based). * Copy your SNFServer.exe file to SNFServer.old * Download SNFServer-windows-7-prox32-3.1.0.exe (32 bit) or SNFServer-windows-7-prox64-3.1.0.exe (64 bit) and rename it to SNFServer.exe to replace your previous SNFServer.exe. * Start your SNF service. If you were using the 32 bit version (very likely) then replace it with the 32 bit version. There really isn't any difference, but just in case it's simpler to keep things the same. There is no benefit to running the 64 bit version -- It is not faster and is in fact less efficient due to the use of extra large (64 bit) pointers that aren't necessary ;-) Some folks really want a 64 bit version, so we have one. Here are some links to updated versions: http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox32-3.1.0.exe http://www.armresearch.com/message-sniffer/download/updates/SNFServer-windows-7-prox64-3.1.0.exe http://www.armresearch.com/message-sniffer/download/updates/snf-server-3.1.0.tar.gz http://www.armresearch.com/message-sniffer/download/updates/snf-milter-1.1.1.tar.gz http://www.armresearch.com/message-sniffer/download/updates/SNFMultiSDK_Windows_3.2.zip And for the really adventurous: http://www.armresearch.com/message-sniffer/download/packages/ In the packages link you will find all of the latest snapshots and some old ones from our LabRats. The LabRats compile and test SNF for all of the different platforms. You will find RPM and DEB packages as well as tarballs and even the windows stuff that's posted in the updates links above. Be sure to pick the latest version in all cases. It will take a bit of time before all of the ordinary links on our web site are updated with the latest software, so please use the above links instead if you're going to update right now. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: ShortMatch Resolved - Update your SNF software to remain immune.
On 2015-12-03 21:24, Daniel Bayerdorffer wrote: > Just so I understand correctly, can we use the packages to install over a > current installation that was compiled from source? Probably not -- the deployment might not be exactly the same. If you originally compiled from source then your easiest solution will be to use the tarball and compile from source again. Then you can simply replace the executable you have with the new one you make -- everything is compatible and nothing will need to move. If you use the packages you are essentially starting over. The packages are deployed differently than the source instructions. For example, to do the generic postfix integration with SNF Server you would need to install two packages: the snf-server_ package and then the snf-server-postfix_ integration package. If you wanted to roll your own integration you might just install the snf-server_ package and then build your own scripts and other software on top of that. It's a different paradigm. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Short Match FPs.
On 2015-12-01 18:12, Darin Cox wrote: > Thanks for the info, Pete. Appreciate your proactiveness on this. > > Hope you had a good Thanksgiving! Thanks! I did. I'd also like to report that some of our experiments might be showing results. It is possible that the trouble has been mitigated based on the latest data I'm seeing. I will know better how good this data is after about an hour. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Short Match FPs.
Hi folks, Good News! After much research and experimentation we have determined that some time on Nov 28 a corrupted rule entered the rulebase and caused the intermittend short-match problem. We have removed a group of rules surrounding that timeframe and have observed a 3 sigma drop in the rate of short-match events. This indicates that the problem is solved and not likely to return. Now that we know this kind of event is possible (it's not supposed to be mathematically) we will be building a detection and mitigation strategy into the engine... just in case it does happen again. Once in two decades makes that seem unlikely. We will also be continuing our research on the sequestered rules to identify the one(s) that caused the problem and identify a way to prevent that recurring. In the mean time the detection mechanisms we used to monitor our experiments will remain in place so that if we do see any future events we will be able to identify them much more quickly. Sorry for the trouble, Thanks for your patience and continued support! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Question, changing from SNF4SA to Milter, using freebsd
On 2015-09-08 04:04, P Pruett wrote: > > Interesting, yes, the spamassassin SNF4SA does seem to be able to use > snf-milter instead of snf-server. That's probably not a good way to go. This will cause each message to be scanned twice. Once by the milter and again by the engine via SNF4SA. If you want to use SNF4SA then you should turn off the milter and use SNFServer instead. > On freebsd 9.3 with Sendmail, I did add the milter and restarted sendmail > and its seems to be playing okay. > > Now I turned it on, I am not sure what the snf milter is doing. That will depend on how you have it configured. The milter interface only provides a few options. Your SNF log should tell you what was found in the scan and the snfmilter configuration will tell you what SNF told the milter to do. > Can you point me to some more documentation with details about what the > milter is doing? > From what I saw in the setup file it can Allow, Accept, Retry, Reject That is defined by the milter interface. Milter.org was shut down permanently just recently. That page says this is where to find documentation on milters: http://www.sendmail.com/sm/open_source/download/ > > I was think it might insert information in the header SNFMilter should inject the usual SNF headers if they are configured (they are by default). > > Would be nice if the milter could be somehow be used to promote IP > addresses into a pf table > for the pf firewall to redirect with? That's an entirely different software project. If you want that kind of functionality then you'd do better to use SNFServer/SNFClient in a postfix filter. The filter script could then be modified to look at the results and respond in any way you can code. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Re: Question, changing from SNF4SA to Milter, using freebsd
On 2015-09-06 13:11, P Pruett wrote: > So what "gotchas" do you know that I need to be aware of if I already > have snf-server > setup and I am going to try snf-milter? The two are not designed to work together. It turns out that SNFMilter has the full SNF engine in it so if you have SNFMilter running you should also be able to use SNFClient and things that act like SNFClient such as SNF4SA. This is not something we test heavily though because almost nobody tries to do this. Most folks who run SNFMilter either build their own software to manage messages (Milter API is highly restrictive) or have SNFMilter inject headers that are later consumed by SpamAssassin and other ubiquitous tools so that they can customize their system easily. If you are using SA after SNFMilter, consider simply adding rules that recognize headers injected by SNFMilter and add appropriate weights for SNF's results. This is a common and successful configuration which allows you to reject some messages during SMTP with SNFMilter and then score the remaining messages using SNF's scan results with SA and other tools that are usually bundled with SA. You shouldn't try to run SNFMilter and SNFServer on the same system at the same time. If you have SNFMilter running, the SNFServer "back-end" should already be provided in that service. (Check that XCI is on, it should be by default). In that case running SNFServer would be redundant. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: <sniffer-...@sortmonster.com> To switch to the DIGEST mode, E-mail to <sniffer-dig...@sortmonster.com> To switch to the INDEX mode, E-mail to <sniffer-in...@sortmonster.com> Send administrative queries to <sniffer-requ...@sortmonster.com>
[sniffer] Bad Rule Alert: 6948148
Rule 6948148 was coded as an abstraction to a fake header and was rapidly removed by QC checks. Most systems are automatically removing this rule. The rule coding has been added to our problematic group so that it cannot be reinvented. Due to our auto-panic feature it is likely this rule will not affect most systems. We apologize for any inconvenience. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Adding Message Sniffer to Zimbra
On 2015-02-10 01:20, Daniel Bayerdorffer wrote: But there are no headers in the messages showing snf's results. I can see that the snf4sa.cf has it set to add them though. # Header line containing the results from SNFServer. add_header all SNF-Result _SNFRESULTTAG_ add_header all MessageSniffer-Scan-Result _SNFMESSAGESNIFFERSCANRESULT_ add_header all MessageSniffer-Rules _SNFMESSAGESNIFFERRULES_ add_header all GBUdb-Analysis _SNFGBUDBANALYSIS_ Do you have any more suggestions? Unfortunately, some implementations of SA are hiding these headers. We've seen this a few times recently. There doesn't seem to be a way around it outside of hacking SA itself. (A few people have done that,... but it was ugly). If you want to be able to more easily associate SNF logs with messages you might consider changing SNF's message identifier to use the Message ID. http://www.armresearch.com/Documentation/QA/ltidentifiergt-2021367617.jsp _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: milter and smtp auth
On 2015-02-10 11:23, Thomas Klaube wrote: Sometimes we see false positives from some of the users although they have been authenticated correctly. Is there a way to tell SNFMilter to whitelist authenticated users? There is no such mechanism in Message Sniffer at this time. I might also point out that white-listing mechanisms generally lead to abuse. For example, much of the worst malware these days infects a machine, gain's authentication through email and other systems, and then uses the authenticated accounts to spread itself further -- this vector takes advantage of social hacking (trust of known friends/peers) and hard security hacking (by gaining access to secured accounts the old fashioned way, by stealing the keys). We don't get many requests for this kind of thing -- I'm pretty sure this is the first time I've heard this one. SNFMilter is distributed as source code so you certainly could code this modification yourself if you need it for your system, or you might use a different milter to force acceptance of messages that you've whitelisted either by list or by behavior. Please if you do find a false positive do report it to us so that we can adjust the filters appropriately... much better to get the filtering right than to make holes in it. For reference: http://www.armresearch.com/Support/falsePositives.jsp Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: milter and smtp auth
On 2015-02-10 14:53, Thomas Klaube wrote: I might also point out that white-listing mechanisms generally lead to abuse. I tend to agree that white-listing is usually not the best solution But please consider this case: one of our users tries to relay mail through our servers and is originating from a Dial-up IP address with very bad reputation (maybe within truncate) but is correctly authenticated. Would you agree that such mails should not be marked as spam or even discarded (at least not based on IP address reputation)? My answer in this case is - it depends. Some systems I know of would consider this too high a risk as you've described it. Others would completely agree that any authenticated system should automatically be white-listed. Unfortunately for the latter group this often costs them a lot in clean-up consulting fees when customers get infected. (we see that a lot lately). Since this is a policy based decision, you could take advantage of the GBUdb drilldown feature and teach your SNF to trust the IPs that this customer might use. What would happen then is that SNF would not be able to identify the source IP and so only the pattern matching engine would apply. http://www.armresearch.com/Documentation/QA/ltdrilldowngt--468945561.jsp Effectively you'd be telling SNF not to worry about the IP address for this customer (or for that matter any of the IPs used for dialup by the customer's provider)... only pay attention to pattern matches. That's still making a hole,... but it's your hole and you know why you made it. It's also a pretty small one because if some known spam or malware comes from there it will still get tagged -- maybe not as efficiently -- but it will still get tagged. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Adding Message Sniffer to Zimbra
On 2015-02-09 16:23, Daniel Bayerdorffer wrote: libpthread package they have listed for 14.04. But the config script still can't find that library. Can you offer any advice? apt-get install build-essential seems to be the equivalent of CentOS yum groupinstall Development Tools which usually solves this problem for redhat variants. Give that a shot and see if it fills in the holes. Usually by the time I've got g++ up and running on ubuntu it just works -- hopefully that's not broken in 14. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Adding Message Sniffer to Zimbra
On 2015-02-02 19:53, Daniel Bayerdorffer wrote: Does anyone have any advice or tips for adding Message Sniffer to Zimbra 8.6? Specifically with Zimbra's implementation of spam assassin? The SNF4SA plugin included with the Linux source code distribution should do the trick. SNF4SA looks to SpamAssassin like any other SA plugin. It creates a temp file of the message, calls SNFServer to scan the message, and then processes the results in a way SA expects so it can be scored. It _should_ be as easy as that. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Report one off spams
On 2014-12-16 13:59, John Tolmachoff wrote: When sending occasional one off spam not caught to spam@ would it help to attach the original headers and source of the body as text files to the forwarded email? Not usually -- that would complicate things. If we can get the original message in it's original form (like redirecting (not forwarding) from Thunderbird) that would help. However, if simply forwarding from some other client, the less extra done, the better it will be. On our end we have to plow through a lot of different formats and sources, so the simpler everything is the more we are able to decipher what we're looking at and locate useful artifacts structures. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Bad rule report 6237276
Hi Sniffer Folks, A short time ago Rule 6237276 was detected on our conflict instruments and removed from the core rulebase. The rule was in place from approximately 1130 to approximately 1400. We recommend that if you have the ability to release messages matching this rule from your quarantines and rescan them then please do so. The rule was coded to catch a variant of the /goo.gl/ link spam and was coded too broadly. The rule was removed when we identified it on our IP/Rule conflict instruments and reexamined it. It's signature on the conflict instrument is already falling dramatically and we expect that most systems auto-panicked the rule making it inert automatically. We are very sorry for any trouble. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Saccades anyone?
On 2014-02-18 17:02, Daniel Bayerdorffer wrote: Any plans to modify the milter code to this in the future? Yes. All platforms will be updated shortly. In fact, if you wish, you can download the snfmulti source from our SVN server and then recompile your milter with the new code. Here is a link: Examine it here with websvn https://svn.microneil.com/websvn/listing.php?repname=SNFMulti Get the source here via svn https://svn.microneil.com/svn/SNFMulti/trunk/ Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Saccades anyone?
Hello Sniffer Folks, We are preparing to release a new version of the Message Sniffer engine that includes an exciting new technology. The saccades engine allows SNF to intelligently skip large portions of most messages without missing any important content. The engine borrows from MicroNeil's synthetic intelligence research relating to visual systems processing and essentially gives SNF a behavior similar to what we all do with our eyes: http://en.wikipedia.org/wiki/Saccade The engine learns where matches are most likely to occur and then applies what it is learning in real-time. This allows SNF to rapidly identify messages of a type it has already seen without having to scan the entire contents. This has the potential to improve scanning efficiency by 90% or more. That is, scanning typical messages can happen with 1/10th the work for a 10x improvement in efficiency. Not kidding, we're actually seeing these results on some of our testbed servers! You may have seen me tweet about it: https://twitter.com/codedweller/status/434020178352148480 If you'd like to get in on the fun early and you are using SNFServer.exe then you can find a copy of the new engine at the following link: http://www.armresearch.com/message-sniffer/download/SNFServerV3.0.2-E3.1.0.zip To swap it in, * Download and unzip the new engine. * Stop your Message Sniffer. * Rename your SNFServer.exe to something like SNFServer.exe.bakup (always a good idea to keep a backup). * Rename the new engine to SNFServer.exe * Restart your Message Sniffer. Please let us know how this works for you. Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: increase in missed spam
On 2014-02-05 13:56, Herb Guenther wrote: For the last week or 10 days I have seen an increase in missed spam in Sniffer, Declude seems to be picking it up but I require more than a single hit to filter. Anyone else seeing this? This is what we are seeing. The trend has been toward very high volume spikes. To be clear, the graph shows new spam not yet filtered, so the higher numbers mean higher numbers of new campaigns with higher diversity. Hope this helps. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller
[sniffer] Re: large log.xml files
On 2014-01-22 10:33, Daniel Ivey wrote: I was checking out our Imail servers this morning and noticed that under the imail\declude\SNF folder I have a lot of .log.xml files from Sniffer. Is there a way to turn off these files in Sniffer or at least to have it only store about 3 days worth? If you're not using them you can turn them off: http://www.armresearch.com/Documentation/QA/ltxmlgt--999318835.jsp I also noticed that the size of these files has grown from about 60 megs a day to over 500 megs the past couple of days. Does anyone have any ideas as to why the file sizes would increase so much, I haven't seen an increase in messages. We have seen a very large increase in the number of messages... that might explain it. Still, that's an order of magnitude there so you should take a look at the large files and see if something else is happening. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Bulk / Noisy Rule Group
Hi Sniffer Folks, Some of you have been experimenting with our Bulk / Noisy rule group which is currently tagged with code 65. This above band rule group matches anything that might be bulk mail, list mail, etc... similar to a popular feature of Postini in the past. As an above band rule group it does not train GBUdb, and can only be reliably detected by systems that are set up to look for it in SNF's result data. If you are using it -- then you know you are because you've had to tweak your systems to expect it. This note is to let you know that we will be changing this result code to 100 next Friday. The change is to avoid any conflicts with some existing error result codes before we make this feature available more broadly. If you are curious about this feature let us know and we will be happy to answer any questions you have. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Happy New Year!!
Happy New Year!! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: What is your oldest production CPU?
On 2013-12-27 15:45, Matt wrote: Intel 5400 series Xeon here. But don't forget virtualization. I'm not sure what CPU virtualization does to targeting your code. That's a good point The processor should be specified in the VM profile and if I recall correctly it is typically defaulted to the processor of the VM host. I should look closer at this -- but would like some feedback. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Whitelist HOW?
On 2013-11-28 19:55, A wrote: I want to set an ignore list in my MessageSniffer installation, so I can receive FBL complains from major ISP. The directives you've set up will adjust GBUdb training, but SNF pattern rules will still tag messages if they match. Normally if you want to globally white-list a particular sender or a particular mailbox on your system you would do that within other systems that you use to process the messages -- so, for example, you might add a rule to your SpamAssassin configuration to white-list all messages going to a particular address; or, if you're using SNF with a postfix filtering script then you would create some scripting to ignore SNF scan results when that mailbox is the recipient; or if you are using one of the many Windows based email platforms then you would enter an appropriate processing rule in the MTA to bypass all filtering rules etc... If you want your SNF rulebase to have a custom white-rule then we can code that for you -- send a note to support@ describing the custom rule you want coded. Be aware, however, that custom white-rules often have unwanted side-effects including that they can be discovered and abused by attackers. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Milter Version
On 2013-10-31 15:01, Daniel Bayerdorffer wrote: Ive been reading the Install notes, but one thing that is not clear is that the Milter version is up to date. Is it current and if not will it be in the near future? We have several folks using SNFMilter with postfix et al and no problems. As far as I know it's up to date :-) SNF in general is built to be stable and highly available, so most of the changes over time happen in the rulebase and not in the engine. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: snf plugin regions question
On 2013-09-10 11:15, Peer-to-Peer
(Spam-Filter.com) wrote:
Regarding the regions section in the
SNF Plugin, were currently using the defaults (see below).
Could you give us a refresher: Why are there
2 entrys for each (white / caution / black)?
For example:
edge
probability='-1.0' confidence='0.4'/
edge
probability='-0.8' confidence='1.0'/
They define corner points on a parallelogram that maps the region in
the x,y space:
http://www.armresearch.com/support/articles/software/snfServer/config/node/gbudb/regions/
The points you listed above define the white region with two points.
These points essentially define a line in the graph.
Everything to the left of that line (in the white region) is
considered to be "inside" the region.
Please let us know if there is more we can do.
Best,
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: snf plugin regions question
On 2013-09-10 17:02, Peer-to-Peer (Spam-Filter.com) wrote: Is that the right direction? That would open up the black range a bit. Use caution :-), but have fun. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] White-Guard
Hi Sniffer Folks, We've been experimenting with a new machine learning behavior. White-Guard is improving early capture rates for new spam and with it overall accuracy and throughput. For example, one thing we've seen since implementing White-Guard is higher truncate numbers across the network-- meaning that more messages are blocked for having bad IP reputations than before we implemented White-Guard. Here is a new blog post that explains what White-Guard is and how it works: http://www.lifeatwarp9.com/2013/08/lies-machine-learning-and-blackhatzes/ You DO NOT need to install or change anything to take advantage of this. White-Guard is implemented in the bigger brain back here in the lab. Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Slow processing times, errors
On 2013-06-27 20:01, Matt wrote:
I'm
attaching a snippet of my log. About 100 lines past the start,
where you see a smattering of error messages, you then see a large
block of them while the Sniffer service is restarting, and then
after that no errors at all. There have in fact been no errors at
all in several hours since this restart of Sniffer.
I can promise you that the error you're reporting comes directly
from a problem with the file system. Here is the code where that
error is generated. Put simply the code tries to open the file and
determine it's size. If that doesn't work it throws the
ERROR_MSG_FILE exception in one of two forms -- that's what ends up
in the log.
try { // Try opening the message file.
MessageFile.open(MessageFilePath.c_str(), ios::in | ios::binary); // Open the file, binary mode.
MessageFile.seekg(0, ios::end); // Find the end of the file,
MessageFileSize = MessageFile.tellg(); // read that position as the size,
MessageFile.seekg(0, ios::beg); // then go back to the beginning.
MyScanData.ScanSize = MessageFileSize; // Capture the message file size.
}
catch(...) {// Trouble? Throw FileError.
MyRulebase-MyLOGmgr.logThisError( // Log the error.
MyScanData, "scanMessageFile().open",
snf_ERROR_MSG_FILE, "ERROR_MSG_FILE"
);
throw FileError("snf_EngineHandler::scanMessageFile() Open/Seek");
}
if(0 = MessageFileSize) { // Handle zero length files.
MessageFile.close();// No need to keep this open.
MyRulebase-MyLOGmgr.logThisError( // Log the error.
MyScanData, "scanMessageFile().isFileEmpty?",
snf_ERROR_MSG_FILE, "ERROR_MSG_FILE"
);
throw FileError("snf_EngineHandler::scanMessageFile() FileEmpty!");
}
Another clue is that in the log snippet you provide, there are hints
of a problem brewing when there are sporadic instances of this
error. Then, when there is a large block -- virtually all requests
to open the files for scan are rejected by the OS. Either something
made those files unavailable, or the OS was unable to handle the
request. I find it interesting also that the time required to report
the error started at about 172 milliseconds and continued to climb
to 406, 578, and then 656 before the restart.
SNF does not make log entries in the classic log during a restart,
by the way.
Note also the timestamps associated with these events and you can
see that the event was precipitated by a dramatic rise in message
rates. The first part of your log seems to indicate about 7-10
messages per second. During the large block of errors, the message
rate appears to have been in excess of 120 (I counted approximately
126 at timestamp 20130627183819). That's an increase at least an
order of magnitude higher than the rate that was causing sporadic
errors.
I suspect based on the data you have provided that something on your
system generated a very large spike of activity that your IO
subsystem was unable to manage and this caused snf scans to fail
because snf was unable to open the files it was asked to scan.
Your restart of SNF apparently coincided with the event, but since
all of the SMD file names are unique during the event, and since SNF
has no way to generate scan requests on it's own, SNF does not
appear to have been the cause of the event in any way. It was able
to record the event, none the less.
So the question in my mind now is:
* Is there a way to improve your IO subsystem so that it can gain
some headroom above 10 msg/sec?
* What caused the sudden dramatic spike that led to this episode?
On most tiny systems I monitor, scan times are generally 100
ms. On your system they are frequently in excess of 400 ms which
leads me to believe your system is a bit underpowered for it's
current load.
Hope this helps,
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
This list is for discussing Message
[sniffer] Re: Slow processing times, errors
On 2013-06-28 16:49, Matt wrote: maybe the updates will cause a service restart/reset? Rulebase updates (all updates in fact) happen without restarting anything. SNF simply loads the new configuration, validates it, uses it for new scans, and when all of the old scans are complete it drops the old data. All of this happens without impacting scan operations. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Slow processing times, errors
On 2013-06-27 17:25, Darin Cox wrote: When we had sluggish performance similar that yours, resulting in numerous sniffer .tmp files in the spool, the cause was eventually traced to a proliferation of files in the sniffer directory. Clearing them out brought performance back up to normal. This is usually the problem. NTFS performs very badly when there are a lot of files in a directory -- and that slows everything down. If SNF takes 30 seconds or more to process a message then SNFClient will give up and let the message through (fail safe). _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Slow processing times, errors
On 2013-06-27 16:04, Matt wrote: like this: 20130627155608, arg1=F:\\proc\work\D6063018a2550.smd : Could Not Connect! That is SNFClient giving up after waiting for SNF to process the message for too long. At the same time, my Sniffer logs start showing frequent ERROR_MSG_FILE results on about 1/8th of the messages. This is SNFServer giving up after trying to open the message file and read it. What's happening is that the OS is not delivering the file to SNF, SNF is waiting for this (it has no choice, it's a call to the OS's open() command), and then eventually it fails so SNF produces the ERROR_MSG_FILE result because it was not able to open the file it was supposed to scan. This is often caused by fragmentation, or it can be that there are too many files in the directory that contains the message file. Ultimately it is an IO problem. This shouldn't be associated with updates -- but if it is, I might guess that's because the file system is ready to fall over and saving a new rulebase file to disk, or reading afterward is enough to push it over the edge. Seeing ERROR_MSG_FILE on 1/8th of the scans means that SNF is being asked to scan a message that the file system can't or won't give it. That is a strong indication that the system is IO bound. SNF can't really do anything different in that case -- it's simply asking to open the file so it can read it. If the IO system says No then it spits out that error. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: 2nd level IP scanning
On 2013-06-07 18:16, Peer-to-Peer
(Spam-Filter.com) wrote:
Hey Pete and all,
Is there an option to have SNF scan second or
third deep header IPs? Im trying to block an originating IP
(66.83.88.42), however they are hopping thru Comcast and
Verizon.
Yes! You can use drilldown directives to teach SNF to "trust"
intermediate servers and find the originator:
http://www.armresearch.com/support/articles/software/snfServer/config/node/gbudb/training/drilldown.jsp
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044 x7010
twitter/codedweller
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: 2nd level IP scanning
On 2013-06-07 19:18, Peer-to-Peer (Spam-Filter.com) wrote: Im seeing one spammer whos IP address is (x.x.x.x). Or at least thats the originating IP in the headers. Im seeing thousands of messages originating from this IP, however they are passing thru hundreds of different Verizon and Comcast servers. Literally. I cant block (or I dont want to block) the Verizon or Comcast IPs, but I need to block the originating IP (x.x.x.x). I think you have misunderstood what drilldown does. If you teach drilldown to recognize the versizon and comcast servers then it will learn to ignore them and pinpoint this specific IP. It will also learn to find any other IPs that are doing the same kind of thing. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-24 08:38, Richard Stupek wrote: Pete I thought the local gbudb got updates from the service or was that a future enhancement? That's true right now. GBUdb is part of a distributed machine learning system. There is a conversation going on between all SNF nodes where they share their point of view on IP reputations. This happens approximately once per minute, out of band. Each node alerts the system that they have new activity on a given IP. Then, via the SYNC server(s), each node receives a reflection of the consensus on that IP. So, when an IP is new to a node it will be updated within about a minute with the consensus reputation from the other nodes. As there are more interactions, the consensus matters less and the local experiences matter more -- but the conversation continues so the each node is always influencing the other nodes about any active IPs. The conversation protocols are intelligent so that there is just enough traffic to accomplish the learning goals and so that a hostile / compromised node cannot poison the system; and so that each node can maintain it's own point of view about each IP. For example: Say node A regularly corresponds with an ISP in blackhatistan. So, node A sees a mixture of good and bad messages. Node B only gets bad messages from the same ISP. Node A will have a local reputation for the ISP that is good enough to let messages through on that system, but node B will have a local reputation for the ISP that blocks most messages. The consensus of all GBUdb nodes will be somewhere in between. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 15:22, Richard Stupek wrote: Looks like I have this issue again (pegging 4 core cpu) and resetting the process doesn't make a difference. Not sure what is causing it but it does slow down spam detection to 40-50 seconds for many emails. Any ideas what I can look at or do to resolve this? Check the message sizes. As part of the newest spam storms we've noticed that a lot of the messages are huge (65536++). I suspect this might impact throughput as large buffers are allocated and moved around to handle these messages. This kind of thing has also been known to cause NTFS to crawl. Please let us know what you find. If you are not already doing it -- you should consider blocking connections using the truncate blacklist. No sense taking on some of these messages if they can be eliminated up front. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 16:41, Richard Stupek wrote: Can you point me at the documentation for the truncate blacklist and its usage? http://gbudb.com/truncate/index.jsp It's an ordinary ip4 dnsbl. Most email systems have some mechanism for blocking connections based on this kind of blacklist. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-05-23 17:21, Richard Stupek wrote: Would this: http://armresearch.com/support/articles/software/snfServer/xci/gbudb.jsp yield the same results as using the ip4 blocklist? No. Asking your local GBUdb about an IP will only give you a local perspective. The truncate blacklist contains the currently active worst-of-the-worst as seen by all SNF nodes working together. Also -- getting your MTA to pay attention to your local GBUdb is nontrivial since no MTA software (that I know of) can speak XCI yet. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Rulebase Compiler Improvements
Hello Sniffer Folks, We have improved our rulebase compiler scheduling and efficiency. This has allowed us to increase the pace of rulebase updates by approximately 20%. You should see a further reduction in leakage rates and slightly more frequent rulebase updates. Also: Did you know that Message Sniffer is designed not only to filter spam, but also to filter any email containing malware or viruses? We recently performed a spot check by throwing a corpus of 200K known virus emails at SNF. Only 76 leaked. According to that and other similar tests SNF typically captures 99.962% of infected email. This means that for most systems using SNF there is no need to also scan for viruses on the mail server -- SNF does it all at once. Anything that does get by SNF should be captured by desktop virus protection -- something nobody should do without since email is not the only way viruses can get into your systems. It's important to note that infected messages may not be marked as malware -- SNF is designed learn and look for all kinds of unusual artifacts that help to identify unwanted messages and many of those are used not only in malware but also in other kinds of spam. So, a lot of the time infected messages are captured by patterns that were learned while looking at ordinary spam. Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Volume
On 2013-04-26 13:12, Peer-to-Peer (Spam-Filter.com) wrote: Anyone else seeing the same? The spamtrap pre-filter volume is about 4x typical. It's really quite something. The stock-push stuff has a lot to do with it -- since that's become popular again we've seen striking volumes associated with it. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Upgrading Stand-Alone Sniffer (for Declude)
On 2013-04-18 02:52, Andy Schmidt wrote: SNFMulti Engine Version 3.0.11 Build: Aug 21 2009 18:42:53 SNF Server Version 3.0.2 Build: Jul 28 2009 14:48:00 That is currently the latest official release. There is a slightly newer SNFServer.exe that is an interim (snapshot) release: http://www.armresearch.com/message-sniffer/download/SNFServerV3.0.2-E3.0.23.zip Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Reputation Lookup DNSBL?
On 2013-04-18 03:06, Andy Schmidt wrote: Is there a GBUdb IP based lookup that is recommended to get the benefit of all Sniffer customers' experiences? There is the truncate blacklist http://www.gbudb.com/truncate/index.jsp Other than that SNF will automatically learn what the other SNF nodes know about an IP within about a minute of the first encounter. Then as your SNF node has more experience with the IP it will begin to trust it's own data more than that of the other nodes. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IPScan results
On 2013-04-16 17:02, Peer-to-Peer (Spam-Filter.com) wrote: Hey all, I noticed a couple of my MDaemon mailservers are not performing the “SNF IPScan”. Check that your MDaemon versions are the same -- some versions implement the plugin API differently. Then make sure that your Plugins.Dat file configures the SNF plugin correctly. If you've got one server working correctly and other's not, then that gives you a good way to compare. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IPScan results
On 2013-04-16 20:40, Peer-to-Peer (Spam-Filter.com) wrote: Once I installed a new Security Plus license, Outbreak Protection engine started and so did SNF IPScan. Around 20120905 I asked about this and Arvel put in a patch to remove the restriction. That is, since Md 13.0, you do not need the security plus license to use the full API on plugins, so the original SNF4MDaemon plugin design will work. (that's what you have configured). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Convert your Declude OEM license now and get full credit!
On 2013-04-11 08:22, e...@insight.rr.com wrote: Because of this entire issue with declude. It might be nice if you contacted smarterTools and offered to work with them on them integrating message sniffer directly into smarterMail. :) We have attempted this on several occasions and it hasn't worked out. We remain optimistic and ready to work with them if they decide to change their minds about it. I know that SNF's features and speed, if fully integrated, would take SM spam and malware filtering to the next level. In the mean time, it is possible to integrate SNF directly with Smarter Mail by calling it as a command line scanner. Then the injected headers can be used in filtering rules or to add weight to the built-in SpamAssassin scores. http://www.armresearch.com/support/qa/integration/smarterMail.jsp Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Convert your Declude OEM license now and get full credit!
Hi Sniffer Folks, It appears that Declude (the company) is failing. After many rumors of problems and some first hand experience, today the Declude web site has gone dark. We have a long standing relationship with the Declude community, and we want to make sure we do what we can to support them even if Declude itself goes away. Place a new order for Message Sniffer (SNF) now and we will give you credit for any time you have left on your Declude OEM license. Tell us your OEM expiration date with Declude and we will add the time you have left to your new SNF license. For the best pricing we recommend you purchase through one of our resellers: https://www.armresearch.com/products/resellers.jsp Please be sure to pass this information on to any interested folks that might not be on this list! There is bound to be a lot of turmoil right now and we don't want anybody to miss it. Please let us know if there is more we can do! Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-29 12:59, Richard Stupek wrote: well when all else fails restarting snf seems to have corrected the issue for now. In that case, it is likely that RAM fragmentation was involved. Dropping the process allowed the fragmentation to be cleared. (theory). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-28 12:10, Richard Stupek wrote: Ok looking at the log I see quite a few messages taking over a second to process (samples below): s u='20130328155503' m=\temp\1332407477322.msg' s='0' r='0' p s='1172' t='1109' l='72697' d='127'/ g o='0' i='12.130.136.172' t='u' c='0.486243' p='-0.625' r='Normal'/ /s Great! Now we're getting somewhere. It seems likely that your system is bound in 2 ways: * Allocating RAM * Reading / writing from the hard drive. The setup time s='1172' indicates that it took more than a second to allocate a 72K buffer and read the message. That is the only work done during setup. The scan time t='1109' indicates the amount of time it took to complete the rest of the process. I'm guessing that since the setup took more than a second that writing the message back with injected headers probably took a while too. A scan depth of 127 is nominal. The data you sent indicates this is a problem when the messages are fairly large. There is likely a boundary condition between smaller messages and larger ones that allow the smaller messages to be handled more efficiently. Since you are indicating that CPU utilization is high during these events and since you're not mentioning other performance issues, it seems likely that RAM fragmentation or RAM starvation might be the problem here. During the setup, SNF allocates a buffer large enough for the entire message and then reads it all at once. This is most efficient because there is a single system call for each of these events - so the OS has complete control over the entire step and it only happens once. After the scanning process is done, SNF will typically allocate another buffer large enough to include the message and all of it's headers. This new version of the message is constructed in the buffer and then written all at once to the file system. If the problem is RAM fragmentation or starvation then it could be taking as much as a second for the OS to allocate a 72K buffer --- that's the kind of thing that should be nearly undetectable, but in these cases where high CPU loads are reported with this kind of log data I have seen that happen. If the problem were simply IO then it is likely that CPU utilization would be low while the CPUs wait for the IO operations to happen. Of course it could be a combination of things -- it's hard to tell what's happening in the internals of the OS. Hope this helps, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 14:38, Darin Cox wrote: Probably unrelated... and due to a significant increase in spam over the past few days. I agree with that -- our inbound spamtrap pre-processor has seen 4x increase over the past few days so that's likely to be related. Also, Richard, I took a quick look at your telemetry and verified that your rulebase file(s) are up to date. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 16:49, Richard Stupek wrote: Its odd because the number of messags snf is processing isn't more than usual and the % of spam being detected through snf is actually lower than typical yet is is routinely maxing out 4 processors at 100%. You're saying that SNF is maxing out 4 processors? ... or is the combination of operations on your server maxing out 4 processors? We're using the same engine and ruelbase in our CGP server and humming along nicely at between 2000 - 8000 msg/minute with nominal CPU loads. I don't see anything unusual in your telemetry and I haven't heard any other complaints, so I can't explain why SNF would act differently on your system. I hate a mystery though -- so I would love to get to the bottom of it. Do you see anything else that might be causing the CPU load? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: IP Change on rulebase delivery system
On 2013-03-27 17:16, Richard Stupek wrote: The spikes aren't as prolonged at the present. Interesting. A short spike like that might be expected if the message was longer than usual, but on average SNF should be very light-weight. One thing you can check is the performance data in your logs. That will show how much time in cpu milleseconds it is taking for each scan and how long the scans are in bytes. This might shed some light. http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Look for something like p s='10' t='8' l='3294' d='84'/ in each scan. From the documentation: sp//s - Scan Performance Monitoring (performance='yes') p:s = Setup time in milliseconds p:t = Scan time in milliseconds p:l = Scan length in bytes p:d = Scan depth (peak evaluator count) Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] IP Change on rulebase delivery system
Hi Sniffer Folks, We are about to change the IP of the rulebase delivery system. This change should be completely transparent and you should not need to take any action; however if you do notice anything unusual please let us know. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] GBUdb Tool
Hello Sniffer Folks, We have been playing with a new utility that some of you may enjoy. http://www.armresearch.com/message-sniffer/download/GBUDBTool-V0.1.zip GBUDB Tool allows you to create a list of IP addresses from your GBUdb snapshots (.gbx files). You can select IPs that are blacker or whiter than a provided probability figure and confidence figure. It outputs one IP per line, optionally with details about the statistics for the IP. This can be useful for feeding-forward blacklists to block at your firewall or for other research purposes. Run GBUDBTool without any parameters and it will tell you about it's command line options. Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] SNFServer Interim Release E3.0.23
Hello Sniffer Folks, Here is a link to an interim release of SNFServer for Win* boxen: http://www.armresearch.com/message-sniffer/download/SNFServerV3.0.2-E3.0.23.zip This interim release fixes a bug in the previous E3.0.19 interim where large messages might be corrupted during message header injection. This new version has been tested thoroughly against large messages. If you don't recall, the E3.0.19 interim and above allows for up to 8 messages to be scanned simultaneously when sufficient CPU cores are available. If you are running *nix and would like to try the interim version then feel free to pull down the updated SNFMulti source code from the SVN server: https://svn.microneil.com/websvn/filedetails.php?repname=SNFMultipath=%2Ftrunk%2FSNFMulti.cpp It is not necessary to upgrade your SNF installation if you are not running one of the interim releases. If you are running a production release then you're good to go as you are. Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] High Throughput Windows version of SNFServer available
Hi Sniffer Folks, If you have a high volume Windows based mail server and you would like to beta test some performance features then you can find a new version SNFServer here: http://www.armresearch.com/message-sniffer/download/SNFServerV3.0.2-E3.0.19.zip There are two key features we're testing on this version: * If header injection is turned off then the message file will only be read up to a maximum of 32K (or the current scan horizon). * There are now 8 processing channels in the XCI server so that 8 simultaneous scans can occur if sufficient cores are available. If you decide to test this then please let us know. Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Creeping higher on those rule numbers
On 6/26/2012 9:41 PM, Colbeck, Andrew wrote: Rule number 5 million rolled on by this week. Yes indeed! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Creeping higher on those rule numbers
On 6/26/2012 9:41 PM, Colbeck, Andrew wrote: Rule number 5 million rolled on by this week. Message Sniffer Rule # 500 was coded by Andy (Worm Thunder) 20120626.1408 SortMonsters Rock! I wonder who won the pool? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: FPs on Sniffer-Schemes
On 3/13/2012 11:19 AM, Scott Fosseen [Prairie Lakes AEA] wrote: Can you check to see if all looks ok with my copy as well. Sure. I'll respond off-list _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: FPs on Sniffer-Schemes
On 3/12/2012 5:17 PM, Darin Cox wrote: Hi Pete, We're seeing a ton of FPs on a Sniffer-Schemes rule # 4764784. That rule was detected as an error and removed almost immediately after it was created. You should not be seeing any additional hits on that rule. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: FPs on Sniffer-Schemes
On 3/12/2012 5:41 PM, Darin Cox wrote: Started getting hits at 4:30pm EST up to 15 minutes ago (5:25pm EST). Not sure if the rule has been pulled or corrected yet. It was corrected nearly as soon as it was created. It did escape into some rulebases - we saw that on our conflict instrument. Most systems auto-panicked the rule right away. It no longer appears on our conflict instruments - so there is no reason you should see any hits from it. I'm chasing things down to see what I can see -- based on your message. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: FPs on Sniffer-Schemes
On 3/12/2012 5:41 PM, Darin Cox wrote: Started getting hits at 4:30pm EST up to 15 minutes ago (5:25pm EST). I think I can see part of the problem (possibly). I do not have telemetry from your system (based on looking up your Id from your domain). I suspect this means that you are running an older version of SNF. By extension, that would mean a couple of things: * Your rulebase update would not come as quickly as for most systems. * Your SNF engine won't match on many of the newer rules. * Your SNF engine will not have GBUdb and also will not be able to auto-panic new rules that conflict with IP reputation data. Am I right about these assumptions? If not, then we should figure out why I don't see your telemetry. Thanks, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 twitter/codedweller # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Bad rule event
Hi Sniffer Folks, We had a short bad rule event this morning. The following rule IDs were matching unintended text, they were discovered quickly on our conflict instrumentation and removed after approximately 30 minutes. Most systems were rejecting the rules (that's how the conflict instrument gets it's data). 4711347 4711362 4711345 4711346 4711360 4711361 We have identified how the rules were coded and adjusted our practices to make this less likely in future. Also, our system will remember these rules automatically so that we cannot make the same mistake again. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] System Upgrades
Hello Sniffer Folks, Just a quick note to let you know: * We have boosted our rulebase production system. New rulebase updates will arrive about 25% faster on average. * We have optimize Rulebot productivity to respond to a wider range of spam / malware variants automatically. * We have augmented our QC processes to seek out more potential false positive cases and stop them before they occur. * We have added additional research channels to help identify more threats more quickly. --- Note that over the next few weeks we will be making additional changes to our technical infrastructure. During service windows occurring at times of low-activity there may be short disruptions in SYNC server connections and/or rulebase delivery. We will do our best to avoid these, and those that do occur should go unnoticed. Your Message Sniffer software installation is designed for high performance and high availability. It will continue to function normally even if we have a disruption during our upgrades, and it will automatically recover from any such disruption without any assistance. Please let us know if there is more we can do. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Ok, I'm the 3rd person to ever report the Bad Matrix error on this mailing list
On 1/9/2012 4:15 PM, Colbeck, Andrew wrote:
snip/
If there's here for the
SortMonsters, it's to make sure that a "bad matrix"
error doesn't interfere with downloading a fresh
rulebase so that SNFserver.exe can get itself out of
that jam.
SNFServer will reject a bad rulebase and keep running with the old
one. So, if somehow a bad rulebase shows up then SNFServer won't
crash... it will keep trying to get a new rulebase with the
getRulebase script until it is successful. By default it will try
about once every 3 minutes if it's not initially successful.
However, once SNFServer is down for any reason, it will refuse to
start without a good rulebase file to work with.
The SNFClient utility can only attempt to ask SNFServer to scan a
message -- it can't do it by itself. If SNFClient is not able to
contact SNFServer and get a good answer then it will create
SNFClient.exe.err to explain the problem and will return 0 to the
calling program -- It returns 0 as a fail-safe so that the messages
will go through. Better to allow all messages through than to block
any good messages by mistake.
On a well functioning system there should [almost] never be an
SNFClient.exe.err file. I say [almost] because things happen from
time to time on every system.
However, if you see a .err message, check it out. If they persist -
something is wrong.
If you try to start SNFServer and it is unhappy, then download a
fresh rulebase first. It's usually a good quick-fix.
Best,
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044
x7010
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Training GBUdb on the client IP for telus.net
On 10/24/2011 2:46 PM, Colbeck, Andrew wrote: would this snippet in snf_engine.xml I don't see the snippet from snf_engine.xml? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Training GBUdb on the client IP for aol.com
On 10/24/2011 3:21 PM, Colbeck, Andrew wrote: header name='X-Originating-IP:' received='.aol.com [' ordinal='0' / As far as I know that one still works. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Training GBUdb on the client IP for telus.net
On 10/24/2011 3:20 PM, Colbeck, Andrew wrote: header name='X-Telus-Outbound-IP:' received='.telus.net [' ordinal='0' / Which is in the GBUDB/Training/Source section as per: http://www.armresearch.com/support/articles/software/snfServer/config/no de/gbudb/training/source-header.jsp That appears to be correct and appears to have worked correctly. Top Received header would have been picked as source IP (unless you already have it ignored). It appears that you have successfully told SNF to find the source IP in the X-Telus-Outbound-IP: header in this case. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Training GBUdb on the client IP for telus.net
On 10/24/2011 3:20 PM, Colbeck, Andrew wrote: header name='X-Telus-Outbound-IP: Hrmm... Do you want the source to be the outbound IP? _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] SNF Server / Client for *NIX updated - IMPORTANT bug fix included
Hello Sniffer Folks, Tarball snf-server-3.0.13.tar.gz has been posted to the Products Page: http://www.armresearch.com/products/index.jsp Or you can download it directly here: http://www.armresearch.com/message-sniffer/download/snf-server-3.0.13.tar.gz This distribution contains some minor bug fixes and code improvements bringing the SNFMulti Engine up to 3.0.17. IMPORTANT: This distribution also contains a clean SNFServer/main.cpp that fixes a random crash bug! The previous distribution snf-server-3.0.12 contained testing code that would intentionally force a crash (seg fault) under some specific load conditions. The testing code would make it appear that SNFServer was crashing at random with crashes being more likely under higher load conditions. This testing code should not have escaped the lab and was not intended for use in production. We have reviewed and revised our publishing procedures to ensure this does not happen again. This new distribution snf-server-3.0.13 does not contain the testing code. This bug was not included in Win* distributions - only snf-server-3.0.11.tar.gz and snf-server-3.0.12.tar.gz included the errant testing code. Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Bug Report: SNFServer for *nix
Hello Sniffer folks, We have discovered that some testing code escaped into the latest tarball: snf-server-3.0.12.tar.gz This testing code intentionally causes SNFServer to crash (seg fault) under special conditions. This was done so that we could examine the resulting core dump. You may experience this as random crashes, especially during spikes in traffic. We are looking at our procedures to see how this happened. When we have resolve that issue we will publish a new tarball. In the mean time you can correct this problem immediately by replacing your SNFServer/main.cpp file with the one found on our SVN server here: https://svn.microneil.com/websvn/dl.php?repname=SNFServerpath=%2Ftrunk%2FSNFServer%2Fmain.cpprev=16peg=16; Download the file, rebuild SNFServer, and replace your existing SNFServer.exe. Sorry for the trouble. We will get a new tarball out shortly. Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Bad Matrix errors
On 8/22/2011 4:04 PM, Peer-to-Peer (Support) wrote:
HelloSNF,
I think something broke.I'm
seeing a lot of "Bad Matrix!" warnings in my logs.Likely startedabout an
hour ago.
Running MDaemon
mailserver.
I note in your telemetry that you have a new rulebase since then.
Have the errors stopped?
_M
--
Pete McNeil
Chief Scientist
ARM Research Labs, LLC
www.armresearch.com
866-770-1044
x7010
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
This list is for discussing Message Sniffer,
Anti-spam, Anti-Malware, and related email topics.
For More information see http://www.armresearch.com
To unsubscribe, E-mail to: sniffer-...@sortmonster.com
To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com
To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com
Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Change in default settings
Hello Message Sniffer Folks, We're recommending a change in the default settings for message sniffer in order to improve our response times for new campaigns. The change is small and enhances our virtual spamtrap technology so that we see new spams sooner and with greater sampling coverage. If you locate this block of code in your snf_engine.xml file: black on-off='on' symbol='63' edge probability='0.8' confidence='0.2'/ edge probability='0.8' confidence='1.0'/ truncate on-off='on' probability='0.9' peek-one-in='3' symbol='20'/ sample on-off='on' probability='0.8' grab-one-in='3' passthrough='no' passthrough-symbol='0'/ /black You will notice that your settings are probably slightly different. The changes we would like you to make are: peek-one-in='3' grab-one-in='3' Your current settings most likely use higher numbers for these settings. Once you make the change and save your file then Message Sniffer should pick up the changes right away - you do not need to restart Message Sniffer when making adjustments to your configuration. Please let us know if you have any questions. Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Change in default settings
On 5/9/2011 3:43 PM, Peer-to-Peer (Support) wrote: Hi Pete, Just double checking: My snf_engine.xml file does not have any 'single quotes' around any numbers or characters. See attached as an example. What you have there in that png is your configuration log -- it is SNF's interpretation of your configuration file. The actual configuration file does use single quotes (unless you changed it). _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Change in default settings
On 5/9/2011 4:53 PM, Colbeck, Andrew wrote: Pete, for sample on-off='on' I wrote myself this note... !-- We can sample during a peek if passthrough = yes -- ... Is it still valid? Your sample and my own configuration have: passthrough=no On the balance of it, I suspect my own note is wrong, so it would be nice if you could verify it one way or the other. The passthrough option is for local sampling. We have used it occasionally on our spamtrap processors, but not for some time. Passthrough takes any messages that would have been samples and instead of sending them to the virtual spamtrap network it lets them go through with a specific result code. Presumably the local system would see the special result code and treat the message differently. Please leave passthrough='no' Thanks! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] IMail mail1.exe removed
Hi Sniffer Folks, Many of you who use Imail also use the imail1.exe command line utility to send yourself updates and reports from your system. Be advised (if you don't already know) that this program has been removed from IMail and there does not appear to be a replacement at this time. Here is a discussion: http://forums.ipswitch.com/Topic58862-10-1.aspx#bm58935 One customer who recently upgraded IMail discovered this problem when their update reports stopped working. The work-around was as follows: 1. Locate a backup of your previous IMail installation. 2. Locate the imail1.exe and imailsec.dll files from your backup. 3. Create a sub-directory inside your SNF directory (suggest IMAIL1). 4. Copy the imail1.exe and imailsec.dll files into the new sub-directory. DO NOT copy them into your IMail directory. 5. Modify your scripts to call imail1.exe in the new location using the full path, or if the script runs inside of your snf directory you can use a relative path (like imail1/imail1.exe). This should restore the original functionality. There are also command line utilities around for sending messages, but for long-time IMail users this is probably the easiest solution. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: So, another botnet bites the dust.
On 3/18/2011 4:17 PM, Colbeck, Andrew wrote: Pete, now that Microsoft has taken down the Rustock botnet, what's your telemetry say about spam volumes? Any significant change? I don't see a dent. -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010
[sniffer] Re: IPv6
On 3/11/2011 9:44 AM, Bonno Bloksma wrote: Hi, I remember reading somewhere research was being done about ipv6 block lists using the fact that the same /64 net would probably be the same machine or very near it. Prety much what we now Block when we list an ipv4 NATted gateway to a private network which houses an infected PC. I had some heated conversations about this at HostingCon in Texas last year. The fellow was pushing IPv6 preparedness and had not considered some of the issues I raised about it. As I see it the real trouble is going to show up where IPv6 and virtualization meet. Consider that to keep costs and manageability reasonable a /64 (or some largeish sub block) will be allocated to a chunk of hardware hosting virtualized machines. Consider also that virtualization is being highly optimized. Therefore an attacker (spammer etc) will be able to map themselves a virtually inexhaustible supply of IP addresses and hosts and move them around at will. Even if they violate TOS on a legitimate system they will be able to do it for such a short period of time with such a small data footprint that they will be able to remain undetected for long periods-- perhaps even indefinitely. By the time anybody looks to see what is going on the offending VPS will long since have been destroyed and it's cousin will be living in a completely different data-center picking up where it left off. The good news is that this behavior is still dramatically different than that of legitimate senders so it leaves a statistically important footprint. What might be predicted from this? Off the top of my head I think maybe the following... * Attacks from anonymously controlled virtual bots will rise dramatically and will be very difficult to defend. Consider that we currently have sufficient RAM available to completely bitmap every IP4 address if we choose to do so -- and that could be done at wire speed on even the fastest routers (Still can't get any love for the concept, but it's been possible for a long while now). This will not be possible with large scale IPv6 deployment. * Black-listing will become softer and much more difficult. Due to the convergence of virtualization and IPv6 deployment, IPs of legitimate systems will necessarily merge with the IPs of illegitimate systems. Only specific, long-lived IPs from legitimate systems will be worth tracking. * Legitimate systems that do bulk mailing will make increasing use of virtualization to keep costs down -- standing up large bot-nets of their own to deliver a campaign and then evaporating those bot-nets when delivery is complete. This will make such systems virtually indistinguishable from illegitimate senders since the IP blocks will have significant overlap and the usage statistics will be very similar. * White-listing mechanisms will become more important. * Content analysis will become more important. SNF is good at that :-) * Systems that delay delivery of messages from unknown and untrusted systems will be more important -- especially those that allow for delivery after re-scanning content rather than conventional gray-listing. Conventional gray-listing mechanisms will become more difficult to use because all kinds of legitimate bulk mailing systems simply will not be there to re-send undelivered messages due to the systems being shut-down after the first volley in order to contain costs. I expect some significant increases in the complexity of such systems to compensate for this. * On the way to IPv6 there will be a lot of fragmentation and confusion of all types. For a long time to come some folks will be unable to deploy IPv6. Others will be unavoidably required to do so. Bridges between these networks will be necessary, difficult to regulate, and unpredictable. Best practices for IPv6 and mixed networks will be difficult to define and constantly evolving -- this churn in general will slow adoption and increase fragmentation. What other things might we expect? Anybody think one or more of these predictions are unrealistic? If so, why? After all, it's just conjecture at this point ;-) In any case, SNF is evolving to become ever more intelligent and adaptive. We will concentrate not only on more sophisticated content analysis, but also behavioral analysis and an increasingly cognitive approach to blending data from all of these subsystems and responding in realtime. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig
[sniffer] CommuniGate Pro Plugin for MS Windows Updated
Hi Sniffer Folks, Just a quick note to let you know we've updated the MS Windows version of our Anti Spam / Anti Malware plugin for CommunigGate Pro. http://www.armresearch.com/products/SNF4CGP.jsp http://www.armresearch.com/message-sniffer/download/CGPSNF-Win32-Intel.zip We have rewritten the documentation and distribution files to make the installation process simpler and clearer. We've also updated the main configuration file with CSS and XSL so that you can view a clear, human friendly version of your snf_engine.xml file simply by opening it in your web browser. Enjoy! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
On 1/7/2011 10:19 AM, Darin Cox wrote: Hi guys, We're seeing a lot of FPs on 3741490 this morning. I've added a RulePanic for it in our systems. The rule was created at 0539 and removed at 0853 when it was detected by our early warning system. It codes for a binary segment found in some image files. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
On 1/7/2011 12:33 PM, Darin Cox wrote: Hmmm... so 70 minutes after the rule was released we were notified of the rule update for auto-update of rulebase, but at 10:11ET we still hadn't gotten the update for the 8:53am removal. Anything we can do to speed up the rulebase update notifications? Update notifications happen as soon as the rulebase compilers have created a new rulebase. We are in the process of reworking our compiler cluster to improve it's performance and further shorten update times. Also, for rules identified as problematic and removed, what about an automated email so we can remove it immediately via RulePanic. For peak times like beginning of the business day, that would be very helpful. An hour could save a lot of headaches for both us and our customers. Or are there so many of those that we would be swamped with notifications? We have features on the short list to automatically render removed rules inert in near real-time (within seconds). Just trying to figure out a way to avoid this as much as possible in the future. It cost me a half hour this morning, and, more importantly, delayed over 150 legitimate messages to our customers. We are constantly improving our process to minimize these cases, increase the speed with which we can detect and correct these, and add features to automate and expedite the process. Thanks in advance for anything you can do. Thanks very much for your feedback! _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: RulePanic on 3741490
On 1/7/2011 2:00 PM, Darin Cox wrote: H "Update notifications happen as soon as the rulebase compilers have created a new rulebase." I don't know what your internal processes are, butif I understand this correctlythe rule was created at 5:39am ET, and was compiled into the rulebase somewhere just before 8:53am ET, at which point update notifications were sent. From the customer point of view, when the rule was created or removed doesn't really matter,and those times are meaningless to us. What matters is when the rulebases that include them are published/updated, as that is what we key off of for updates. The rulebase compiler that was responsible for your update would have performed it's queries to collect rules for folding. For some time after that the compilers would crunch on that data in order to create the folded token matrix. Both the query and folding operations take time. There are many compilers and many, many rulebase files. The system is adaptive so the times required to perform these operations changes constantly. All that by way of saying that you are essentially correct, but also that there is no single answer we can give that would describe the case for every customer. The rule was removed at about 0853. Some customers rulebase compilation began immediately after that event, others before. All took some time to complete. Part of our rework is to reduce the time required for both phases of this operation so that on average less time is required for a change to appear in active SNF systems. The goal is ultimately to make that as close to instantaneous as possible without incurring performance, availability, scalability, and reliability penalties. "We have features on the short list to automatically render removed rules inert in near real-time (within seconds)" Sounds good. That would definitely be better than notifications for us to be able to put in RulePanics, assuming there's no negative effect to overall performance from checking each rule for active/inactive state. I assume some sort of push mechanism to all subscribers, to notify their systems that a rule is no longer valid,is what you're planning here. The implementation that is currently planned would operate something like our system making temporary entries into your rule-panic system. Those entries would survive long enough to ensure that your active rulebase contains those changes before the entries expire. Wherever possible we like to leverage tried and true mechanisms (like the rule-panic entry system) when producing new functionality. Also, wherever possible we like to engineer facilities that can be leveraged in multiple ways in future. It's a planning heavy process, but one that pays off in better reliability and greater overall flexibility. (IMO). Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Bad Rule Event
Hello Sniffer Folks, We have had a bad rule event. The bad rules were created near 0830E, and removed by 1030E. The bad rules were discovered by our IP/Rule conflict instrument indicating that most were automatically rejected by Auto-Panic features. The rules were part of a rule family designed to capture highly obfuscated porn subjects. Unfortunately the extensive abstraction of the rules matched other subjects containing similar combinations of characters. The Rule IDS are in the range 3694383 - 3694428. We are very sorry for any trouble and have already taken measures to prevent this error in future. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Bad Rule Event
On 12/16/2010 11:07 AM, Bonno Bloksma wrote: Hi Pete, Hello Sniffer Folks, We have had a bad rule event. The bad rules were created near 0830E, and removed by 1030E. [...] Regarding this event A while ago we talked about sniffer installations exchanging rule-panic info via the GUBdb sync info as that is happening every (few) minute(s) in stead of every few hours. Any idea when a new version of Sniffer with that feature will be launched? Actually -- rule-panics are triggered instantaneously based on local GBUdb data. Auto-Panic: When a relatively new rule conflicts with a known good IP on your system that rule is made inert until the next rulebase update. The next full release will include features for near-real-time rule additions and removals. We plan to begin releasing interim updates of the SNF engine with some of these features early next year. We plan to complete the next full release by Q3. _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Hello again
Hello Sniffer Folks, This is just a quick note to touch base. It's been a while since we've had an update, and the list is so quiet! * Over the past few days we've finished a major re-tuning of our rulebase compiler system. The improved rulebase compiler bots are just a bit smarter and as a result many systems are receiving their updated rulebase files sooner than ever before. This means capturing more spam early on more systems and as a result more accurate data in GBUdb for new bot-nets. A win for everyone. * We're growing again: Please welcome our newest SortMonster: Brian (The Fall Guy). We look forward to Brian's help as we expand our research capacity and improve our flexibility so that we can rapidly respond to new spam storms and preemptively defeat those we can predict. * If you don't already know, a few months ago we launched truncate.gbudb. Truncate is an rfc5782 compliant DNSBL that contains IPs from the dark edge of Message Sniffer's GBUdb system. So far the comments we've received have been very positive. Be sure to give it a look when you get a chance and see if it can help you safely block connections at the edge of your systems. http://www.gbudb.com/truncate/index.jsp * Over the next few months we will be upgrading our servers to increase our capacity and support more sophisticated analysis and research engines. All of this translates to improved rulebase quality and response times. Expect this round of upgrades to be finished in the first half of the new year when we will have some exciting surprises. * Quietly in the back of the lab we have been building a sophisticated proactive false positive prevention system and we've been seeing some impressive results. Working closely with several of our largest partners we have developed a system which allows us to identify and eliminate many false positives before they can occur! This has allowed us to take our already low false positive rate and reduce it by nearly 30% over the past few months - especially focusing on low-level false positives that might otherwise never be reported or detected. As a result we have seen a measurable reduction in false positive related support traffic across all of the systems we monitor. We expect further improvements as we develop more sophisticated tools to help us with our research. = We would love to hear from you! Please send us a note from time to time and let us know what we're doing right and how we can improve. Best, _M -- Pete McNeil Chief Scientist ARM Research Labs, LLC www.armresearch.com 866-770-1044 x7010 # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: Testing SM direct intergration
On 9/22/2010 6:58 PM, Keith Dovale wrote: Hi Guys, I would be interested in testing this .. Here is a reminder of the link. http://www.armresearch.com/support/qa/integration/smarterMail.jsp Currently this is the only way to integrate SNF directly with SmarterMail. There are a number of 3rd party products which also allow SNF to integrate with SmarterMail while providing additional functionality. Perhaps in the future there will be a tighter integration with SmarterMail that leverages SNF during the SMTP conversation in order to take maximum advantage of what SNF can provide-- we hope so, but it's not happening yet. If this sounds like a good idea to you be sure to let the folks at SmarterMail know you think so. Best, _M -- Chief Scientist ARM Research Labs, LLC www.armresearch.com # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. This list is for discussing Message Sniffer, Anti-spam, Anti-Malware, and related email topics. For More information see http://www.armresearch.com To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
