[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 12:50:23 AM, you wrote: PS: And, for bonus points, to correctly support your sub-directory feature in your sample script, you would do that with the P parameter, e.g.: wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf-N -P %RULEBASE_PATH%--header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m snip/ Thanks for your help. We have to cover a lot of ground so we often get solutions from our customers and others that just want to help out. We do our best to test and edit. I will see that your suggestions / corrections are reviewed and included in our updates. In any case they will be in the mailing list archives ;-) THANKS! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 1:13:50 AM, you wrote: Hi Pete, Thanks for giving it your consideration. If you decide to revise these parameteres, then it will require an extra command in your script (because the WGET command will output the compressed file as .SNF). There is actually a bit more to it than that -- The existing script generally works even though it doesn't preserve the servers's timestamp because: 1. It is usually triggered from the SNFServer when SNF detects a newer rulebase file. 2. Any rulebase fill recently downloaded is guaranteed to be newer provided the local server's clock is correct (or close to it). Also -- are you saying that with the parameters you've provided WGET would decompress the file on it's own so that we wouldn't need to do that in our script? If so, how does it know for sure where to find GZIP? If not then it would be a little dangerous to have a .snf file around that looked correct but was in fact not yet decompressed. Another consideration is that if the file name is going to collide with the existing rulebase file we would want to move that into another location so that we don't stomp the existing rulebase file until we've tested the new one. It would be preferable to use WGET since there's nothing wrong with it and we've been using it long enough that most SNF folks already have it. That doesn't mean you shouldn't provide an alternate script that works with CURL just in case someone has a preference. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping
Hello Andy, Wednesday, October 8, 2008, 1:35:41 AM, you wrote: snip/ Also -- are you saying that with the parameters you've provided WGET would decompress the file on it's own so that we wouldn't need to do that in our script? If so, how does it know for sure where to find GZIP? snip/ Sorry--- the above was just me hitting the button before my brain caught up with my fingers. You did say that the compressed file would be output as .SNF -- _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Path apparently doesn't tolerate embadded blanks
Hello Andy, Sunday, October 5, 2008, 11:25:37 PM, you wrote: Hi Pete, Found a bug (I think): update-script on-off='on' call='D:/Program Files/SNF/getRulebase.cmd' guard-time='180'/ With THIS configuration, the script apparently gets never launched. Whats particularly disturbing is, that I didnt find any place where the service reports/logs any errors about the update process? http://www.armresearch.com/support/articles/software/snfServer/config/node/network/update-script.jsp When the update launcher detects that a new rulebase file is ready it passes the contents of call= to the system() function. The system() function _SHOULD_ exist on every platform, but it's implementation can be different on every one. More importantly for this question - the result code returned by system() is implementation specific, so there is no way to know what that result code will be (what it means) without actually testing every case. Worse than that an apparently successful call to system() might only indicate that the OS supports syste(), or that the command interpreter didn't crash or didn't have a problem with the command string even though the command string didn't necessarily do anything. When testing a new update script you need to install it and see if SNFServer can launch it successfully by testing the result of the script -- did it do what it was supposed to do? We are considering some features for the next minor upgrade to help test this feature such as the ability to trigger it and a log entry that returns whatever system() returned (that will have to be interpreted by the user according to their platform's documentation for that call). For now you can trigger this feature by changing the timestamp of your .snf file. If the .snf file's timestamp is earlier than the timestamp of the file available from our system then the update-script feature will fire. After spending some time, trying to narrow down the problem but not finding any diagnostic info, I eventually decided to eliminate spaces from the path: update-script on-off='on' call='D:/Progra~1/SNF/getRulebase.cmd' guard-time='180'/ and now it DOES seem to work. Everything in call= is passed to system() as a string. No additional parsing takes place. In theory that means if you can type the contents of call= at the command line and get your desired result then SNFServer will be able to do the same. A couple of things can get in the way of that -- * Does SNFServer have the correct permissions to do what you have put into call= ? * Does the system() function on your platform interpret this string correctly ? On Windows systems spaces can be a problem whenever one program is calling another. Wherever possible it is usually best to use tilde to eliminate spaces in file or directory names -- otherwise the spaces are likely to be interpreted as breaks between command line parameters. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Path apparently doesn't tolerate embadded blanks
Hello Andy, Monday, October 6, 2008, 10:20:05 AM, you wrote: Hi Pete: http://www.armresearch.com/support/articles/software/snfServer/config/node/network/update-script.jsp Yep, had read that but that page just instructs me to use the full path, which I did. Does NOT tell me that the Windows platform requires to quote the string or use DOS style paths It will shortly. I wanted to get the link out there for those following along. snip/ As you know, there are a few different ways to launch programs in Windows some of them accept a full path as a separate parameter (spaces allowed). I dont think its safe to assume that your customers would know which function call you had chosen, and to predict what its limitations might be. Some of us are Win32 API programmers many probably arent. Indeed. Since SNF is cross-platform software I could not chose a Windows specific call to launch the external script. I also had to chose something that would be thread safe. The system() call is a posix standard -- posix having been around a very long time. I really dont have a problem with either including the string in quotes or using DOS style paths, etc. its just that there was nothing helpful in the product or documentation that would have lead me to conclude that either before or after it didnt work. There will be shortly. Thanks for your detailed posts! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Update Script - Path apparently doesn't tolerate embadded blanks
Hello Andrew, Monday, October 6, 2008, 12:40:04 PM, you wrote: Pete, We aren't windows-greeking the call='xyz/getrulebase.cmd' call on the installer. Do we want to do that on this next release? The XYNTService double quotes the calls, so I'm not worried about that but if it failed for him, why wouldn't it fail for others? You're right. We will need to tweak that in the installer if it isn't already. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Testers wanted...
Hello Sniffer Folks, If you are interested in working with us to test SNF on the following platforms please let us know: SNF4Alligate -- SNF external filter on Alligate. SNF4ASSP - SNF plugin for ASSP 2.0. New SNF control script for any of: OpenBSD, FreeBSD, CentOS, RedHat, SUSE, Ubuntu Please respond off list to support@ Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: .xml Error
Hello Greg, Sunday, October 5, 2008, 1:34:42 PM, you wrote: When I try to view the log files in the SNF directory, I get XML Parsing Error: junk after document element through Firefox. I get The XML page cannot be displayed Cannot view XML input using XSL style sheet. Please correct the error and then click the Refresh button, or try again later when sent to IE. I assume these logs have some valuable info, how do I view them? Since the logs must be continually appended, SNF does not put the opening and ending document tags on them. However, that is something you can do. If you wish to convert an SNF XML log file to a proper XML document then you can add the top level element to the top of the file, and append the closure of that element. Then you can open the file without errors in most XML editors... You could even provide a style-sheet in the top of the file to format the log as you would like to see it. That said--- The XML logs have been designed to be relatively easy for people to read directly with a text editor. If you open the file as text with your favorite programmer's editor or even notepad then you should be able to interpret the log entries as easily as you would the entries in any ordinary text type log file. This link will show you how to interpret the log file: http://www.armresearch.com/support/articles/software/snfServer/logFiles/activityLogs.jsp Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer 3.0 Installed
needed when this script is run REM from SNF's update-script/ feature since only one copy will run REM at a time. However, if you are going to run a version of this REM script as a scheduled task you will want to uncomment the next REM line to make sure only one copy runs at a time-- just be sure to REM clean out any stale .lck files after a restart. REM if exist %WORKSPACE_PATH%\UpdateReady.lck GOTO DONE :DOWNLOAD COPY %WORKSPACE_PATH%\UpdateReady.txt %WORKSPACE_PATH%\UpdateReady.lck wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -O %RULEBASE_PATH%\%LICENSE_ID%.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m if exist %RULEBASE_PATH%\%LICENSE_ID%.new.gz gzip -d -f %RULEBASE_PATH%\%LICENSE_ID%.new.gz snf2check.exe %RULEBASE_PATH%\%LICENSE_ID%.new %AUTHENTICATION% if errorlevel 1 goto CLEANUP if exist %RULEBASE_PATH%\%LICENSE_ID%.old del %RULEBASE_PATH%\%LICENSE_ID%.old rename %RULEBASE_PATH%\%LICENSE_ID%.snf %LICENSE_ID%.old rename %RULEBASE_PATH%\%LICENSE_ID%.new %LICENSE_ID%.snf if exist %WORKSPACE_PATH%\UpdateReady.txt del %WORKSPACE_PATH%\UpdateReady.txt if exist %WORKSPACE_PATH%\UpdateReady.lck del %WORKSPACE_PATH%\UpdateReady.lck :CLEANUP if exist %RULEBASE_PATH%\%LICENSE_ID%.new del %RULEBASE_PATH%\%LICENSE_ID%.new if exist %WORKSPACE_PATH%\UpdateReady.lck del %WORKSPACE_PATH%\UpdateReady.lck :DONE ENDLOCAL -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer 3.0 Installed
Hello Andy, Saturday, October 4, 2008, 12:28:44 PM, you wrote: HI Pete, Thanks for your feedback. I had to create the UpdateReady.txt file before I was able to test my update script from the command line but I didnt realize that I would be created in the Workspace folder. Without that information, one cannot adapt the update script to ones needs. Since the server always creates UpdateReady file in the Workspace folder and always expects the .SNF file in the Rulebase folder, its pretty safe to say that anyone using the getRuleBase.cmd would absolutely have to add the Workspace and Rulebase paths otherwise they cant possibly find the UpdateReady file and the script will just exit OR it will not place the SNF File where the server will find it. Anyone who has their own update mechanism clearly doesnt fall under this discussion at all. My conclusion is, that the current getRuleBase.cmd only handles the case when there are no separate directories but the with the changes I made, the getRuleBase.cmd would allow a user to define separate directories at the top of the script (if thats how they configured things) and thus correctly handle a SINGLE as well as separate directories. In my opinion, that is the more correct behavior. I'm still trying to think of a way to describe this modification so that it makes sense without causing lots of confusion. Since we're trying to reach a larger audience these days we've created a generalized approach and built an installer that configures SNF, Declude, mxGuard, and MINIMI on IMail, SmarterMail, and even a "generic" (roll your own) configuration. The installer performs upgrades from the previous version as well. The idea is that more folks won't have to do any tweaking at all. Your proposed getRulebase works great for your structure-- and I agree it's a nice idea to have a WORKSPACE_PATH and RULEBASE_PATH variable.. BUT I'm having a hard time figuring out a way to include those and their various options without adding a lot of confusion and complexity... The existing getRulebase script works perfectly when used with the installer and nobody has to touch it. My best thinking at the moment is to perhaps do something like this: REM - Edit This Section - SET LICENSE_ID=licenseid SET AUTHENTICATION=authenticationxx SET SNIFFER_PATH=D:\IMail\declude\SNF REM Modify the next two lines if you modify SNF's directory structure. SET RULEBASE_PATH=%SNIFFER_PATH% SET WORKSPACE_PATH=%SNIFFER_PATH% REM - Of course doing that would mean rewriting our installer too (Since it needs to modify/generate the getRulebase script. For the immediate future this discussion is archived and searchable and I will add a task to the web site project to describe some of these getRulebase.cmd scenarios. How does that sound? _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: FW: [sniffer] Re: Sniffer 3.0 Froze Mail Server
Hello Andy, Saturday, October 4, 2008, 10:21:31 PM, you wrote: Hi Pete, Well, I eliminated WeightGate for the time being, just to do my due diligence. Also, since there is a fix sized buffer, I assume actually LOWERING the 3rdnumber (the allocation for each non-interactive process) would allow for MORE parallel processes to run (as long as the value is still large enough to support each of the applications that rely on it.) Of course, I assume the heap issue in reality is actually a SECONDARY problem ( a symptom of too many non-interactive tasks being launched and not completing). Since the heap space is finite, there is a hard limit as to how many processes can be in a wait state at the same time. The problem to focus on is not the known, limited heap, but rather the reason why these processes were unable to complete and thus eventually too many processes being active. Indeed. Eliminating WeightGate might impact this because it will represent one less process per message. I just did a search of errors in the SNF logs and didn't find anything unusual. I was unable to pinpoint the time of the problem -- that will require a harder analysis of the data. Indications are that SNFServer didn't see any significant issues during the period covered by the two logs you sent. When client's talked to it they were served (according to the logs). You're showing about 40 msg/minute on average. According to a spot check of log entries SNFServer is finished processing these in an unmeasurable amount of time (0 indicates 15 ms for both setup, read, scan, and response). Most of the logs performance metrics p/ indicate s='0' and t='0' -- setup time in ms, and scan time in ms. On occasion I see some nonzero t values - but nothing unusual (16, 47, 63, etc). You probably don't need a lot of threads active on your system. If you have provided for a high number then you might consider reducing that number... Processing 1 message per second would exceed your average handily and doesn't take a lot of threads. If for some reason you were hit with a large number of messages and put them in work in parallel then that might have exhausted the heap. The new SNF is much more efficient than the old one and so it would have more easily allowed this... Sometimes introducing a more efficient component into a system exposes problems that were hidden by the previous less efficient component -- the less efficient component may have masked the problem by artificially reducing or shaping throughput. When we see this kind of thing we call it a "lens effect" -- the newer component reshapes the dynamics of the system and brings previously unknown problems "into focus". It's possible the heap problem you experienced was caused by a "lens effect" since the new SNF engine is more efficient and would naturally allow for more messages to be handled concurrently in a burst than the previous version would have allowed. A theory -- the previous version would naturally be constrained by I/O contention since it would need to create, scan, modify, and remove job control files. This would naturally couple performance to other I/O intensive operations such as writing new messages to the spool etc. The new version does not have any of this overhead and so would allow for an unconstrained ramp-up of new instances -- that might lead to a higher number of concurrent tasks and cause heap exhaustion--- after heap exhaustion is achieved additional tasks build up in a failed and partially initialized state. This typically continues until the failed tasks are manually removed -- since none of them is ever properly initialized none of the tasks can time out, fail, or shut down on their own. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer Version 3 Install for FreeBSD?
Hello Harry, Sunday, September 28, 2008, 10:39:42 PM, you wrote: I have been using Sniffer for several years with Declude and SmarterMail on Windows. I would like to move Sniffer to my IMGate Mail Gateway (Postfix / FreeBSD). Has anyone installed Version 3 of Sniffer on FreeBSD? The *nix download of Sniffer v 3 doesnt contain a FreeBSD pkg and port like most FreeBSD software. We have just completed work on a new set of control scripts for SNF that cover several pacakages including FreeBSD. We have to rework the distribution and documentation a bit before posting. The *nix distributions we have covered now are: RedHat, SUSE, Ubuntu, FreeBSD, and OpenBSD. I will be happy to send you what we have ready now off list along with some instructions to make sense of it. If all goes according to plan the new *nix distribution will be posted to our site some time this week. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer Version 3 Install for FreeBSD?
Hello Harry, Monday, September 29, 2008, 8:11:09 AM, you wrote: Hi Pete, Please do send the new FreeBSD control script and doc at your convenience. Our email are crossing in the ether. Before posting the new distribution prototype I created a README-SETUP file to help pull the process together. If I understand correctly, IMGate uses postfix and postfix allows for more than one filter. The provided filter scripts should get you started. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: ASCII art spam
Hello Paul, Thursday, September 25, 2008, 10:06:17 PM, you wrote: Have others also been getting pummeled recently by ASCII art spam? A lot seem to be slipping through the sniffer gate. Any good ways to tighten up the fight on them? Please zip up a few messages and send them to me off line. I will research them with the team and we'll code up some rules for whatever is getting through. At the moment things are pretty quiet (99.7% nominal capture) and we're not seeing these in the traps. If they are on their way we would like to get them early (of course). Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Alt-n Security Gateway
Hello Daniel, I'm looking into this. _M Tuesday, September 9, 2008, 5:15:47 PM, you wrote: Hi Everyone, Can SNF be used with Alt-N's Security Gateway product? http://www.altn.com/Products/SecurityGateway-Email-Firewall/ I know the plug-in works great with Mdaemon itself, but I might be switching to Exchange. And want to use this product with it. -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: What's in a name - or - objects in mirror.exe are bigger than they appear
Hello Sanford, Friday, September 5, 2008, 2:21:38 PM, you wrote: I say, yes, remove the extensions. For when one deals with multi-boot or VM environments, seeing a Windows-style name can make you think it's just showing through from another filesystem or somethin' or was accidentally dumped there. Obvs. direct SNF admins wouldn't be likely to have this confusion, but other people on the box could. Feel you should leave the names in mixed case; that isn't non-*nix IMO. Forcing lowercase on case-preserving + case-sensitive systems is like pretending they're not cp/cs (easier to remember, sure, but not using the power, etc.). Thanks! This is what we've decided to do. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] What's in a name - or - objects in mirror.exe are bigger than they appear
Hello Sniffer Folks, We are in the process of updating the control script in our source distribution. The goal is to provide a drop-in script that can be used on most distirbutions. The current control script works fine on many - but we want to do better than that :-) Along the way the question of naming conventions was raised (again). I'll explain: SNF runs on many platforms and currently most SNF installations are on Win* of some type. So, we elected the names SNFServer.exe and SNFClient.exe since that would make sense to most folks currently using SNF. On *nix platforms folks don't use the .exe extension, and in fact most of the time don't use mixed case names. In order to keep things simple we decided to use the same names across all platforms. This helps make documentation more consistent. The thinking was that being consistent in our support materials was more important than platform specific naming conventions. None the less, on occasion someone will point out that using a .exe extension on a *nix platform is, well, simply not done. It makes me chuckle a little bit, after all it's ok to have .sh at the end of a bash script for example-- but it is true that more than once now I've been asked about the .exe at the end... usually something like Do you really want to keep that .exe there?? implying there is something wrong and unspoken about it. We aim to please so if it makes *nix folk more comfortable when there is no .exe on the end we're happy to change that. My question is: Do any of our *nix users consider this to be an issue? Should we change the names of SNFServer.exe and SNFClient.exe in the *nix distribution to drop the .exe ? Feel free to email me directly if you wish. I look forward to your insights. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Stampede - amazing!
Hello Sniffer Folks, I had been wondering why the blackhats had been pushing so hard for new bots these last few weeks. Then the other day I saw something very strange in the SNF telemetry. A storm came in that seemed to stop all other traffic. For more than an hour I really thought something was broken -- but I wasn't sure I'd really seen it. Just a short time ago our SortMonster on duty (Mitchell Skull) called all-hands for a new spam storm. This was another of the new penis spams. We coded the rules quickly and as they went out I saw it again: T rates fell to zero on many systems and close to that on all of the others. This means that virtually all of the IPs were brand-new. At the same time traffic spiked on all systems and capture rates went off-scale high as the new rules tagged virtually every message. This is not an entirely new tactic by the blackhats-- I've talked about it before. It is essentially a high-amplitude burst - where a new campaign is pre-tested against all known filters and then launched on a large number of new bots that are unknown to IP reputation systems. What is new is the purity of these recent events. When we've seen them before they were mixed in with a lot of other traffic from other bot nets and even other campaigns from the same bot net. While there was still a trickle of this activity, the purity of this burst was astounding. This was a stampede where essentially all visible bots started running in a single new direction. T rates have recovered now by and large -- so the new bots are already largely recognized by GBUdb, but the wild swing in telemetry across the network was amazing to watch -- as is the new telemetry showing dramatically increased traffic and capture rates indicating a nearly pure stream of spam from this new herd. Theories, comments, and observations welcome. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] GBUdb Memory Requirements
Hello Sniffer Folks, In light of a recent discussion here about SNF memory usage I thought I would post that I've completed the documentation for the size-trigger mechanism in GBUdb. This feature allows you to limit the amount of RAM that GBUdb will use by triggering condensation at a specific size limit. Here is a link: http://www.armresearch.com/support/articles/software/snfServer/config/node/gbudb/database/condense/size-trigger.jsp Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: MDaemon latest
Hello Grant, Thursday, August 7, 2008, 11:37:46 AM, you wrote: Greetings, What is the most up to date version of Sniffer that is recommended for MDaemon? I see info on the other versions for V3 but none for MDaemon. The MDaemon plugin is also at V3 now. You can find download links here: http://www.armresearch.com/products/index.jsp Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Am I ready to upgrade to version 3?
Hello Stefan, Tuesday, August 5, 2008, 6:35:14 AM, you wrote: Andy, Pete, I did install V3 this morning using the current Windows installer. Everything seems to work. Can you please check if you get good telemetry data? I see good telemetry. Andy and I will discuss the hiccup. snip/ - Question: I installed using Windows Remote Desktop and all looked good. But on the real screen of that server I now see a command line window related to Sniffer. The title contains the WGet command. Some action seems to take place (a simulated rolling wheel in the last line). Must this process run? Why was it started on that other screen? That's interesting. I don't think that happened in testing. If I understand correctly, we're using XYNTService to run SNFServer.exe. In theory it should run silently. It sounds like you are able to see it's screen. Your message rate is not high-- but you do usually have a message or two running through. When that happens, the msg/minute number on that line should change. If it does then that is the live version of SNFServer so you'll want to leave it running. Or, pheraps if you restart the service from the console it will be invisible as it should be? I'll let Andy work on that with you to understand it and come up with a working solution. Thanks for the detail! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Am I ready to upgrade to version 3?
Hello Stefan, I've been chatting with Andy on this. He is the engineer who developed the installer. He should be responding shortly about that piece. I will answer some of the other questions... Monday, August 4, 2008, 8:46:30 AM, you wrote: snip/ 2.) Anything I MUST read and understand before upgrading? You probably should (not necessarily must) read this so you will understand the differences: http://www.armresearch.com/support/articles/installation/upgradeHelp.jsp 3.) What about network traffic generated by V3 compared to V2? Our (small) company has a DSL-line which is used by the mailserver exclusively. But it is not a full flat, so at some point we have to pay traffic. V3 will SYNC every minute with our servers. The session size depends upon how much traffic you are seeing, however in all cases it is fairly small. Our spam trap processor handling about 4000 msg/min at the moment produced about 35K down and 49K up in it's last session. A customer node handling about 430 msg/min at the moment produced about 18.3K down and 24.5K up in it's last session. A customer node handling about 60 msg/min at the moment produced about 3.4K down and 5.8K up during it's last session. That should give you an idea about the extra traffic. If you have been uploading log files then you will not need to do that with the new version. 4.) What about processor load? Do we need more power to get V3 running smoothly. V3 typically uses less CPU than V2... Sometimes significantly less depending upon what it has learned. In general the engine is slightly more efficient due to optimizations, and in addition to that it is able to truncate the scanning process on between 10% and 50% of messages based on learned IP statistics. The new version also uses less CPU and I/O because jobs are coordinated via a local TCP connection instead of job files on disk. The new version is also fully multi-threaded so it's work can be spread among the physical virtual CPUs in your system. 5.) Anything important I forgot to ask? I'd like to know the answer to that, too ;-) I don't think so. If you have gateways or other message processing systems in front of SNF you will want to be sure to tell GBUdb about them so that they can be skipped when SNF is determining the source IP for the message. The rest I think you covered. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: FW: Memory Usage of MessageSniffer 3
Hello Peer-to-Peer, Thursday, July 31, 2008, 10:05:15 PM, you wrote: Would it be correct to say the higher we can increase the size-trigger 'megabytes' value, the better filtering results (accuracy) we will achieve? In other words, would it be beneficial for us to purchase more memory on our server (say an additional 2GB), then increase the 'megabytes' value to 400 or 800? Several of our servers are hitting the upper limit (159,383,552) 150 MB I don't think so. A quick look at your telemetry indicates that your systems are typically rebooted once per day. This is actually preempting your daily condensation. One result of this is that many of your GBUdb nodes only condense when they reach their size limit. From what I can see, when this happens a significant portion of your GBUdb data is dropped. For example, several of the systems I looked at have not condensed in months. Here is some data from one of them: timers run started=20080801081753 elapsed=19637/ sync latest=20080801134415 elapsed=55/ save latest=20080801131823 elapsed=1607/ condense latest=20080406160144 elapsed=10100606/ /timers gbudb size bytes=50331648/ records count=214313/ utilization percent=91.1357/ /gbudb This one has not condensed since 200804 most likely due to restarts that prevented the daily condensation timer from expiring. If this is the case with your other systems as well, it is likely that they are occasionally condensing when they reach their size threshold, but if they were allowed to condense daily they would never reach that limit. In that case, adding additional memory for GBUdb would probably not improve performance significantly. The default settings are conservative even for very large message loads. for example our spamtrap processing systems typically handle 3000-4000 msg/minute continuously and typically have timer GBUdb telemetry like this: timers run started=20080717205939 elapsed=1270156/ sync latest=20080801134844 elapsed=11/ save latest=20080801134721 elapsed=94/ condense latest=20080801132958 elapsed=1137/ /timers gbudb size bytes=117440512/ records count=568867/ utilization percent=99.6626/ /gbudb Note that this SNF node has not been restarted since 20080717 and that it's last condensation was in the early hours today-- most likely due to it's daily timer. Note also that it's GBUdb size is only 117 MBytes. It is unlikely that this system will reach 150Mbytes before the day is finished. Since most systems we see are handling traffic rates significantly smaller than 4.75M/day it is safe to assume that most systems would also be unlikely to reach their default GBUdb size limit during any single day... So, the default of 150 MBytes is likely more than sufficient for most production systems. --- All that said, if you want to intentionally run larger GBUdb data sets on your systems there is no harm in that. Your system will be more aware of habitual bot IPs etc at the expense of memory. Since all GBUdb nodes receive reflections on IP encounters within one minute, it is likely that the benefit would be the ability to reject the first message from a bad IP more frequently... Subsequent messages from bad IPs would likely be rejected by all GBUdb nodes based on reflected data. It is likely that increasing the amount of RAM you assign to your GBUdb nodes will have diminishing returns past the defaults currently set... but it might be fun to try it and see :-) --- If you are looking for better capture rates you may be able to achieve those more readily by adjusting your GBUdb envelopes. The default envelopes are set to avoid false positives on large filtering systems with a diverse client base. It is likely that more restricted systems could afford to use more aggressive envelopes without creating false positives because their traffic would be more specific to their systems. In a hypothetical case: If your system generally never receives legitimate messages from Russian or Chinese ISPs, then it is likely that your system would begin to learn very negative statistics for IPs belonging to those ISPs. A slight adjustment to your black-range GBUdb envelope might be just enough to capture those IPs without creating false positives for other ISPs where you do receive legitimate messages. In any case, since the default ranges are extremely conservative and tuned for large scale filtering systems it is worth experimenting with them to boost your capture rates on nodes that have a more restricted client base. If you have a larger system and you use a clustering deployment methodology then you might still take advantage of these statistics by grouping similar clients on the same node(s) based on where they get their messages. Even if you don't adjust your envelopes this clustering will have the effect of increasing the signal to noise ratio for GBUdb as it learns which IPs to trust and which ones to suspect. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC
[sniffer] Re: FW: Memory Usage of MessageSniffer 3
Hello Peer-to-Peer, Friday, August 1, 2008, 10:49:52 AM, you wrote: snip/ I also have a scheduled reboot every night since we did confirm w/ Arvel at MDaemon there is a memory leak in MDaemon.exe (if heavily utilizing their Gateway feature). Have yet to hear anything from AltN regarding a fix on the MDaemon.exe leak. In any case, do you think lowering the upper limit will help the St9bad_alloc error, or am I fishing in the wrong area. That will help your memory leak issue because it will leave more room for the leak to expand before causing allocation failures. You shouldn't see a significant drop-off in GBUdb performance after you reduce your upper RAM limit because your message rates are low enough that GBUdb should be able to function quite well with fewer entries-- Also there is a shared memory effect that emerges from the interaction of GBUdb nodes and the cloud... When records are condensed they are more likely to be bounced off the cloud and get new data so what you might loose in fewer records you will gain in more frequent reflections. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] A few news items...
Hello Sniffer Folks, * We have reached our 2 Millionth Rule! -- Our rule bots now have more than 2 Million heuristics available for activation at any moment. When new spam is spotted that matches an old rule, that rule is reactivated automatically. The vast majority of our rules have been coded by hand over the years by our amazing Rule-Techs (The SortMonsters). These highly trained professionals work around the clock (24x7x365) and consistently produce the most accurate rules available anywhere. They are really a fantastic team and a great bunch of folks to boot. :-) At present about 122315 rules are typically active at one time. Our most active rule at the moment was coded some 2062 days ago (has it been that long? Wow!). Here's to the next 2 million ! * We have re-posted our Windows Installer for the new Client/Server version of SNF 3.0. This installer will help you upgrade from previous versions of SNF if you are using any of: -- IMail + Declude -- IMail + mxGuard -- SmarterMail + Declude Other combinations are also supported and often detected automatically including raw client/server installations for use on systems we don't know about yet ;-) This installer is relatively new (but well tested in our lab). Please keep us posted on how it works for you. http://www.armresearch.com/message-sniffer/download/SNFWinClientServerInstaller.3.0.exe If you prefer to get the new SNF and install it yourself: http://www.armresearch.com/message-sniffer/download/SNFWinClientServer3.0.zip ** MDaemon users-- don't forget that we nave an installer for the new MDaemon plugin also! http://www.armresearch.com/message-sniffer/download/SNFMDaemonPluginInstaller.3.0.exe If you prefer to install the new SNF MDaemon plugin manually: http://www.armresearch.com/message-sniffer/download/SNFMDPlugin.3.0.zip ** Linux/BSD/OSX users-- If you haven't heard, the new *nix distribution of SNF has been updated with improved V3 specific instructions and example control update scripts. Also - if you had trouble compiling SNF before on your PowerPC or 64bit box the latest version includes big/little endian detection and bug fixes. At this time there are no known problems on all of these platforms. As always: Keep us posted please :-) http://www.armresearch.com/message-sniffer/download/SNFSourceClientServer.3.0.1.zip * Everyone should upgrade to the newest version as soon as practical. While we will continue to support version 2 for a time, version 2 of SNF is deprecated. Of course, that's not the only reason to upgrade. SNF Version 3.0 has many improvements that reduce leakage, reduce the chance for false positives, reduce administration costs, and reduce system loads. -- More efficient, fully multi-threaded scanning engine. -- Realtime collaborative IP reputation system. -- Realtime rulebase checking and telemetry (no need to upload logs) -- Realtime system status information in XML format. -- Integrated, customizable rulebase update mechanism. You can find links to our latest distribution files on our products page: http://www.armresearch.com/products/index.jsp You can find installation guides and upgrade notes here: http://www.armresearch.com/support/articles/installation/index.jsp Some of our site is still under construction (it is large). If you find something under construction that would help you please let us know and we will reschedule that work to get it done more quickly. In the mean time we'll be happy to answer your questions directly. * If you are using SNF on a platform we don't talk about on our web site please let us know. We'd love to work with you to publish an install guide that works for your favorite platform! * A number of software developers are integrating SNF into their platforms directly to improve performance, take advantage of advanced features, and simplify installation. MX UpTime (MXScan) and Server Side Solutions (eWall) have both recently released versions with tight integration to SNF. http://www.mxuptime.com/ http://www.armresearch.com/support/articles/installation/eWall.jsp If you are a developer and you would like to integrate SNF support in your solution please let us know. SDKs are in the works - pre-release versions and support are available. Plus the new XCI protocol makes access to SNF services as easy as a local TCP connection! Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: FW: Memory Usage of MessageSniffer 3
Hello Ian, The new (V3) SNF does use more ram than the old SNF (V2). GBUdb adds records over time as it learns new IP data. The amount of RAM that will be used by GBUdb depends on how quickly it is learning new IPs and how frequently the database is condensed. You can set an upper limit on the size of GBUdb in the configuration file: condense minimum-seconds-between='600' time-trigger on-off='on' seconds='86400'/ posts-trigger on-off='off' posts='120'/ records-trigger on-off='off' records='60'/ size-trigger on-off='on' megabytes='150'/ /condense By default GBUdb will condense once per day or when it reaches 150 MBytes. Roughly twice as much RAM is needed for the condensing process since the GBUdb data must be copied to a new location. Condensing the GBUdb data is relatively expensive, so if sufficient RAM is not released by the first pass GBUdb will condense again every 10 minutes (600 seconds above) until GBUdb is below the size limit you have set. I recommend you determine how much ram you want to make available for SNF and then set your size-trigger/ to 40% of that size. This should leave room for GBUdb to condense and for the rest of SNF to fit inside your memory limit. You can monitor your GBUdb status in your status.minute or status.second reports. Here is some sample data from one of our spamtrap processors. It has been stable for months so this should be indicative of what you would see on a busy machine that's been up for a while: gbudb size bytes=142606336/ records count=650314/ utilization percent=95.8431/ /gbudb For information on reading your status reports: http://www.armresearch.com/support/articles/software/snfServer/logFiles/statusLogs.jsp Hope this helps, _M Tuesday, July 29, 2008, 10:31:23 PM, you wrote: This is from one of our engineers. Anybody else had this sort of issue? Ian -Original Message- Does the new sniffer stuff have a higher memory requirement than the old? Sebastian pointed out to me today that a number of our gate servers were using a ton of swap space. Restarting snfctrl frees up a few hundred megs. Our newer gate servers (all with 2 or more GB of RAM) seem to be doing alright, but we have 16 gates at IAD with 1 GB of RAM that are being affected by this. It looks like the memory usage increases progressively over the course of a couple days, so I don't know if it's a memory leak or what. Is there anything we should do or add a snfctrl restart to our nightly cron jobs and just live with it for now? # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Problem with Sniffer-Porn rule this morning
Hello Darin, Friday, July 18, 2008, 9:37:18 AM, you wrote: Pete, There appears to be a problem with rule 1984485 this morning. I'm getting a number of FP hits on it from AOL users. The rule has been pulled already. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Problem with Sniffer-Porn rule this morning
Hello Darin, Friday, July 18, 2008, 11:39:47 AM, you wrote: We had 18 hits on it from ~6:40-9:30am EDT before putting in the rule panic, 5 of which reached our hold weight. We've had 27 more hits since adding the rule panic. When a rule panic is in place the rule should be inert. Please check your snf_engine_cfg.log to see if the rule panic was picked up in your configuration. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Problem with Sniffer-Porn rule this morning
Hello Darin, Friday, July 18, 2008, 1:07:56 PM, you wrote: Yes. The rule is inert. However, according to the logs the rule would have been hit 27 more times had we not added the rule panic. Thanks for clarifying. If it were something else I'd want to get on that right away ;-) _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Problem with Sniffer-Porn rule this morning
Hello Darin, Friday, July 18, 2008, 1:12:39 PM, you wrote: Hmmm... I don't think the rule was already pulled. We update our rulebase upon receipt of the notification of a new rulebase being available, and according to our logs the rule was in until at least 11:24am EDT. The rule bots would have queried the database for rules 20-40 minutes before you you received it. The rule may have still been in place at that time. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: upgraded to 3.0
Hello Bonno, Friday, July 18, 2008, 2:27:59 PM, you wrote: Hi, Well I did it, upgraded to 3.0 as well. The automatic rule panic feature and all the other stuff seemed a good idea. :-) Setting it up turned out to be straight forward, just follow the instructions. Ran into just 2 things and one question. Thanks for the details. 1) Forgot to set correct path to identity file, was set to a nonexisting path. Started server. -start screenshot-- C:\IMail\declude\Sniffer3c:\IMail\declude\Sniffer3\SNFServer3.0.exe c:\IMail\declude\Sniffer3\snf_engine.xml SNF Server Version 3.0 Build: Jun 26 2008 13:25:19 SNFMulti Engine Version 3.0 Build: Jun 26 2008 13:25:06 Launching with c:\IMail\declude\Sniffer3\snf_engine.xml Unhandled Exception: snf_LoadNewRulebase() Zero length SecurityKey Thrown! -end screenshot-- Should have said something like "error in path to identity file" I will consider adding an error case for a missing identity file. As a side note, there are a lot of ways to provide identity information to the SNF engine -- so that's why it throws this exception. An error message for a bad identity file path will be specific to that case. 2) On page http://www.armresearch.com/support/articles/software/snfServer/core.jsp resultcode 63 is still listed as "Received IPs from spamtraps research." in stead of "Black.." Question: Is there still a log file for me to ZIP every night or is all logging now at ARM research? The new version provides us with rule-hit statistics when it checks in to share GBUdb data and check for new rules. Once you are running the new version you do not need to upload log files. p.s. Aren't we at version 3.01? This one I just downloaded still reports 3.0 as it's version. Ot was that just the *nix version? The change to 3.01 in the *nix distribution is not required for Win* systems which are all pre-compiled. The change was to remove "casting" in one line of code concerned with printing out thread status when in debug mode. The effect of the change is: * It prevents 64 bit GNU compilers from complaining about a loss of resolution. Specifically casting a pointer to (int) which on a 64 bit system essentially drops half of the data. (64 bit pointer, 32 bit integer). * It changes the output format of the thread ID slightly when running in debug mode. The updated version displays thread object numbers in HEX while the 3.00 version displays them as decimal (base 10) integers. Since the change does not effect any critical functions we didn't see the need to make a new release for the other (non source) distributions. When the next general revision is produced this change will be rolled in. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: MD - Headers in body
Hello Peer-to-Peer, Thursday, July 17, 2008, 12:37:08 PM, you wrote: Hello, Is there any common cause for the SNF headers to appear in the body of an email? We're running MDaemon. We have a customer using a webform to receive their sales-orders via email. When the message arrives in the customers mailbox the SNF headers appear at the top of the message body, and the webform itself shows-up as webcode in the body. I assume it's the way the webform was created or being sent. Sorry I have limited details, but any suggestions? Per RFC822, SNF finds the end of the existing headers by looking for the first blank line. It then injects the headers ahead of that blank line so that the SNF headers are the last one's present. Most of the time this works ok. Sometimes if the message isn't coded quite right there can be something that looks like an extra blank line in the message. If it's not coded correctly for some reason then SNF will skip it and move on-- but some software in the chain might clean-up or interpret the blank line differently -- even the email client. The best way to debug this is to put the raw message file through a hex editor and see how it is encoded. Each line should end with CRLF and the first blank line should be CRLFCRLF. If you find something else in there then that's likely where the trouble is. Hope this helps, Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Upgraded Rulebase Delivery System
Hello Sniffer Folks, Early this morning we completed significant upgrades to our rulebase delivery system yielding a 10 fold increase in available bandwidth and a 5 fold increase in delivery transaction rates. Please let us know if you observe any negative or positive effects. From observations and theory rulebases should be delivered more quickly and more frequently. I will continue to monitor the system closely for any aberrations. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: [Fwd: FW: [sniffer] Re: Upgraded Rulebase Delivery System]
Hello Steven, Saturday, July 12, 2008, 11:04:57 AM, you wrote: I am having the same problem. My existing production wget script is not able to download the new .snf file, but I can do it manually with a browser. From my log I see that wget makes the connection to the server, and can see the .snf file, but the connection is closed immediately when the download starts. I'm working on this. I'm not sure what's causing it-- Apparently some option in wget. I have verified that some older scripts don't work. It appears to be related to whether gzip is accepted. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Upgrades termporarily off-line.
Hello Sniffer Folks, Due to some unexplained problems w/ apache and tmpfs, I have taken the upgrades to our delivery system off-line. It appears that when rulebase files were retrieved using compression (gzip) the downloads would work correctly. When compression was turned off apache would complain that the file did not exist. I have tested non-compressed downloads and they appear to be working correctly again. Sorry for the trouble. I will keep you posted on our progress. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Upgrades restored!
Hello Sniffer Folks, The rulebase delivery system upgrades have been restored. I have tested the system with the new settings and the older scripts successfully. Some rulebases will be absent from the server for a short time while the rule-bots recompile them. After that there should be no problems. Please let us know if you have any trouble. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. PS: If you are still using the old version of SNF, or the old way of downloading rulebase files please upgrade as soon as you can. Thanks! # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] SNF Client/Server Source (*nix) Update, Now 3.0.1.
Hello Sniffer Folks, We have published an update to the SNF Client/Server *nix distribution with the following features: * New V3-Fresh-Install-Readme.txt * Fixed minor error in SNFServer main.cpp when compiling on 64 bit. * Updates Tweaks to sample scripts. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: It's official. SNF Version 3.0 is Ready!
Hello Daniel, Friday, June 27, 2008, 10:00:57 AM, you wrote: Hi Pete, I just installed Mdaemon Plugin version. It seems to be working properly, but I can't download updates. When I do, snf2chk says it is not a valid gz file. Do you have any suggestions on what I can try to fix it? I responded to this last night on list. I'm guessing you didn't get that response so I'm responding to this new one directly (off list). _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: It's official. SNF Version 3.0 is Ready!
Hello Steve, Friday, June 27, 2008, 3:25:06 PM, you wrote: Pete are there new log files as I do not see them in my working sniffer dir...? I'm using the integrated model with eWall. There are two kinds of log files. Classic (like your used to) or the new XML based log files. Check your configuration file -- they may be turned off by default in that configuration. Here's some documentation on configuring SNF log files: http://www.armresearch.com/support/articles/software/snfServer/config/node/logs/index.jsp Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] It's official. SNF Version 3.0 is Ready!
Hello Sniffer folks, Back in Q1 we were sure we'd be ready with the new SNF after nearly a year of testing on both large and small systems. What a surprise! After publishing the first release candidate we went from version 1-5 to version 2-27 at a breathtaking pace! Thank you to everyone who has tested, poked, prodded, and twisted the new SNF -- not to mention keeping up with all of those updates during the final phase of testing. I can't imagine getting to this point without your patience, trust, attention to detail, and persistence! Bravo! Without further fanfare: Today the latest release candidate becomes the official production release of Message Sniffer (SNF) Version 3.0. The changes: -- Minor updates to readme files. -- Changed the build / version information and recompiled. -- Removed redundant comments from the configuration file. We have been bug free for more than 2 months with several hundred systems using the new engine. You can download the latest distributions from this page: http://www.armresearch.com/products/index.jsp You may also notice that we've published our new web site! There are a few bits of documentation still under construction here and there, but we're well on our way to filling those in along with a stream of continues improvements and additions based on our work with you! Once again, Thanks to everyone for a fantastic job! Thanks for all of your support, comments, and efforts! As always we're hear to help. Now, onward to the next upgrade... always work to do ;-) Cheers! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: It's official. SNF Version 3.0 is Ready!
Hello Matt, Thursday, June 26, 2008, 4:21:42 PM, you wrote: Pete, Now that you got that taken care of, can you give us an idea when you expect 4.0 to be released? Hehehe. We're not close enough to that to be remotely accurate, but I will tell you that I'd like it to be within something approaching a year. There are a lot of continuous upgrades to do on the back-end that will unleash and enhance V3's power and provide additional tools and products along the way -- plus plenty of work helping new third party products get off the ground w/ SNF inside... So we'll be plenty busy and we'll keep you posted. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Bad rule alert: 1940812
Hello Message, Rule 1940812 has already been removed from the core rulebase. You can render the rule inert immediately by adding it to your rule panics list. Rule was coded at 13:03:17 EDT The rule was coded for an obfuscated version of the word Tuesday and was coded with a bad abstraction character. We sincerely apologize for the inconvenience. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Bad rule alert: 1940812
Hello, --- following up. Intended to make the original post with a high priority flag. Also - the rule was removed at approximately 15:10:00 EDT Hope this helps, _M Tuesday, June 17, 2008, 3:35:47 PM, you wrote: Hello Message, Rule 1940812 has already been removed from the core rulebase. You can render the rule inert immediately by adding it to your rule panics list. Rule was coded at 13:03:17 EDT The rule was coded for an obfuscated version of the word Tuesday and was coded with a bad abstraction character. We sincerely apologize for the inconvenience. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Bad rule alert: 1940812
Hello Andrew, Tuesday, June 17, 2008, 4:41:41 PM, you wrote: Thanks, Pete. I had very few actual hits; I have lots of lines that indicate the rule panic in place, but the number of actual hits is quite small. How I found my hits: cd /d C:\MessageSniffer gawk ($6 == \Final\) ($7 == 1940812) *.20080617.log I haven't checked telemetry yet -- still very busy here battling the stock-push spam other storms. However, you were likely protected by the Auto-Panic feature in the new SNF. The first time the bad rule hit a message with an IP source in the white range it would have been automatically added to your node's internal panic list rendering it inert. That probably explains why you have very few hits. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: GBUdb dump
Hello Michael, Tuesday, June 17, 2008, 4:48:54 PM, you wrote: Pete, How soon should we expect to see a new gbx file after a dump? If you are using the default settings then it should appear after about an hour. By default GBUdb creates a snapshot of it's database every 3600 seconds. gbudb database ... checkpoint on-off='on' secs='3600'/ _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Bad rule alert: 1940812
Hello Andrew, Tuesday, June 17, 2008, 5:03:25 PM, you wrote: Thanks, Pete. I had four actual false positives on one server, versus 324 unique hits for the bad rule. So yes, I'd say that the autopanic feature worked quite well. It's a little odd to say this under the circumstances, but congratulations on the success of the first live test of auto-panic. (all previous tests were in the lab) :-) _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Spam no using CAPTCHA!
Hello Daniel, Wednesday, June 11, 2008, 9:19:47 AM, you wrote: Hi Everyone, I just sent a spam sample to Message Sniffer, that was using CAPTCHA, it said CIALIS in the CAPTCHA. I'm curios to see what Pete thinks of this new tactic? On first look it is simply another way to use an obfuscated image to deliver their message and should be handled the same way. Use of CAPTCHA software to create this obfuscated image is an interesting choice -- it means people making good OCR resistant CAPTCHA generators are now unintentionally helping the blackhats defeat OCR based spam filtering. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Spam no using CAPTCHA!
Hello Andrew, Wednesday, June 11, 2008, 11:48:55 AM, you wrote: ... and it also means that OCR based spam filtering is succesful enough for the spammers to adopt CAPTCHA-style text-obfuscation-in-images as an evasion method. Possibly, but I wouldn't put too fine a point on it. It's very easy for spammers to adopt this new technique--- it may have happened just on a whim. They often try things at random just because they think it might work, or because they get an idea and start tinkering with it. In that context this is a kind of random mutation that may result in a kind of spam more fit for survival. Or it might not. Another reason for them to try this is that their current methods for modulating their images are getting old - and the artifacts associated with those methods are themselves fairly easy to detect-- so rather than invent a new way a quick easy choice is to coopt CAPTCHA and let somebody else do the work. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Final RC before Version 3 (fingers crossed)
Hello Sniffer Folks, The latest SNF distributions have just been posted: http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions#NEW_SNF_V2-9_Wide_Beta SNFMulti engine 2-9rc 25 SNFClient 2-9rc 7 This release is a performance update, no new bugs in many weeks now. Here is a snip from the change log: 20080524 - Version V2-9rc2.25.7 Optimized networking library for additional speed stability by moving receive buffer allocation from heap to stack (automatic). Optimized timing parameters in SNFClient for improved speed. Polling dealys are now reduced to 10ms from 30ms. Removed speed-bug in SNFClient, 100ms guard time between retries was always executed after an attempt (even a successful attempt). The guard time is now condition and only fires on unsuccessful attempts. Updated XCI server logic to ensure non-blocking sockets for clients in all socket implementations. PS: ** We expect to begin wide testing of two new pieces of software soon: Windows Installers for the MDaemon plugin and Command Line versions of the new SNF. Stay tuned! Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] XYNTService -- Any Problems?
Hello Sniffer Folks, We are working on an installer for the command-line version of SNF V3.0. We are considering re-distributing XYNTService to setup the SNFServer.exe as part of the installation. There are some rumblings here and there about XYNTService not working the same way on some versions of Windows. If any of you have experience with XYNTService then we would sure like to hear about it. If anyone knows of an alternative that we can bundle with our installer then please let us know that too... (Why don't you just write something?? -- ) We'd prefer not to reinvent that particular wheel right now -- not that it's hard, just that it's not necessary and we'd rather do other important stuff. Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XYNTService -- Any Problems?
Hello Matt, Friday, May 9, 2008, 3:57:42 PM, you wrote: SRVANY works perfectly and is free with Windows. Why not use that? We can't redistribute SRVANY with our installer and we can't be sure it will be present on all systems. I've been using SRVANY every time I do an install for somebody... but we want something that we can deliver with the installer so it can be a (more or less) one click process. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XYNTService -- Any Problems?
Hello Matt, Friday, May 9, 2008, 5:10:29 PM, you wrote: I'm sure that I don't speak for everyone, but I would tend to avoid third-party service systems, and this would also expose Sniffer to the potential pitfalls of that software. I thought of that -- but given that the requirement is simple, and the source code for the "business end" of XYNTService is close to what I would write it seems like a logical candidate. You could provide directions on how to install SRVANY, and then have a script that completes the process once the executables are on the system. That would be my short-term recommendation. In the long-term, I would do your own service as opposed to use someone else's container. Scripts and instructions are subject to interpretation and tend to be troublesome. Those will still exist for those who want to customize their installation and want to know where everything is and how it got there... but for those who don't we'll set up the installer to do things in a "standard / predictable" way-- that should save everybody some time and trouble. In the long term we will create a number of gadgets that will take care of running SNFServer (or it's core anyway) ... but those are different projects and we want to get V3.0 out there ;-) One of the big inhibitors right now is the complexity. SNF V3.0 does so much more than V2.x that it is unavoidably more complex. Not everyone who can use V3.0 is ready (or willing) to do the technical bits of the install. One of the more complicated bits of the installation has been setting up SNFServer to run as a service. Folks seem to have trouble with this part of it even if the rest of the install goes without a hitch. I'm hoping that a simple third party device that's been out there a while and can be integrated with our installer will solve that problem nicely. I would not recommend distributing XYNTService until you have trialled that for several months with a range of systems. The work of properly testing this is possibly more work than creating your own service. I've heard about XYNTService for quite some time and found plenty of positive references to it. I'm hoping to find some references that are closer to home. The only problems In the mean time I'm researching the few problem reports I can find. BTW: If we were to develop one in-house it would require at least the same level of testing. All IMO of course. And well appreciated! :-) _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XYNTService -- Any Problems?
Hello Andrew, Friday, May 9, 2008, 5:40:14 PM, you wrote: snip/ Since you, as a the developer, start with XNTService with the source code, http://www.codeproject.com/KB/system/xyntservice.aspx then you can modify it and deploy it any way you want snip/ So it does seem less bad that at first, but if you're going to be supporting XNTService because you built it, and you're going to be supporting your own SNFServer.exe because you built it... you'd aim higher and write SNFServer.exe as a Windows Service anyway. Actually -- the way things look moving forward we will probably keep the SNFServer executable as it is and then keep any service stub separate. There are a lot of advantages to this approach. I understand your point though. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] New version: Engine 24, MDPlugin 6
Hello Sniffer Folks, This release is an upgrade more than a bug fix. Replace your SNFServer.exe or snfmdplugin.dll as appropriate. No changes have been made to the configuration file. This version improves memory management in the SNF Engine for improved performance, improves the header injection mechanism for improved reliability, and improves logging for IP scans done with the MDaemon plugin. As usual you can get the latest distributions here: http://kb.armresearch.com/index.php?title=Message_Sniffer.GettingStarted.Distributions#NEW_SNF_V2-9_Wide_Beta Here is an excerpt from the change log (this time from the MDaemon plugin change log since it contains all changes from the last version): 20080424 - Version V2-9rc6.24.6 Refactored snfScanData.clear() to reduce heap work and fragments. Added mutex to scanMessageFile() entry point just in case some app attempts to put multiple threads through a single engine handler. scanMessage() is already protected and fully wraped by the new scanMessageFile() mutex. Added non-specific runtime exception handling to XHDR injection code. Added 2 retries w/ 300ms delay to remove original message in XHDR inject code. If remove fails after 3 attempts the injector throws. Added 2 retries w/ 300ms delay to rename temp file to msg in XHDR inject code. If rename fails after 3 attempts the injector throws. Added IPTest logging. -- Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Source distribution corrected re: snf2check utility
Hello David, Thursday, April 24, 2008, 2:46:34 PM, you wrote: Sorry - meant this version: SNFv2-9rc5.23.6 A little off topic for this thread. Check that the snfmdplugin.xml is set up correctly - especially, provide full paths. If you still have trouble then send us a note at support@ and include your configuration log and config files. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Source distribution corrected re: snf2check utility
Hello Sniffer Folks, The source distribution of the SNF2-9 beta/rc has been corrected. The previous build of the source distribution was missing a compile script. The new build -- just uploaded -- contains a compile script and some minor modifications to the source code so that it can be built in the SNF2Check directory. NO OTHER MODIFICATIONS WERE MADE ;-) Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] RePost Overview of Upgrade Process from 2.3x SNF to 2-9 (V3) SNF
SNF in C:\SNF then do this: C:\SNF\SNFServer.exe C:\SNF\snf_engine.xml You should either see SNFServer start up and begin showing you real-time status information or you will see an exception that tells you what is broken. The two most common problems are: * The paths in the snf_engine.xml are not correct. * The rulebase file is missing. If you did step 5 then you should have a rulebase file in the right place. 7. With your SNFServer up and running (in a dos window for now) you can run SNFClient to scan messages. 8. In Declude, or mxGuard, or postfix, etc... you will have a line that calls your old sniffer program. Comment that line out and make a new one that calls the new sniffer program instead. YOU DO NOT NEED TO SHUT DOWN YOUR OLD SNF INSTALLATION -- This way you can go back to it immediately if you're having trouble. Here are some examples: ___ Declude: SNIFFER external nonzero c:\imail\declude\sniffer\licensid.exe ... becomes SNIFFER external nonzero c:\imail\declude\SNF\SNFClient.exe ___ mxGuard: PathToEXE=C:\IMail\Sniffer\efdsxfeff3.exe becomes PathToEXE=C:\IMail\SNF\SNFClient.exe ___ postfix: You may want to consider reworking your sniffer script, but if not... SNIFFER_EXE=/var/spool/snfilter/snfrv2r3.exe becomes SNIFFER_EXE=/var/spool/snfilter/SNFClient.exe 9. Once you've updated your scanning software to use the new SNFClient you should be able to see the statistics change on the SNFServer (remember we left that running?). If not, check to see that your scanning software is set up properly -- sometimes you need to restart services for them to see new settings, for example. 10. The new version produces the same result codes that the old version did with a few exceptions. The new version also produces result code 20 (truncate) and result code 40 (caution). You should account for these in your program of choice if you apply separate weights to SNF result codes. 20 should be weighted as high as the highest SNF value and 40 should be weighted at least as high as the lowest SNF value on your system. 11. Once you have all of this working you will probably want to set up the new SNFServer to run as a service. On *nix systems we've provided a handy snfctrl script to use as a template. On win* systems you will want to set up SNFServer the same way you set up your persistent instance (using srvany, XYNTService, FireDaemon, etc...) The only real differences will be the command line and the name (so that it doesn't conflict with your existing SNF installation). If you're used to using srvany then first reference this article: http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.Persistent.Setup#Setup_Instructions_for_running_Persistent_Sniffer_as_a_service_using_the_Windows_2003_Resource_Kit In step 3) in the example... C:\ Instsrv Sniffer C:\Program Files\Windows Resource Kits\Tools\SRVANY.exe becomes C:\ Instsrv SNFServer C:\Program Files\Windows Resource Kits\Tools\SRVANY.exe In step 9) in the example... C:\IMail\Declude\Sniffer\[yourlicx].exe [authenticationxx] persistent becomes C:\IMail\Declude\SNF\SNFServer.exe C:\IMail\Declude\SNF\snf_engine.xml 12. You will need to shut down the SNFServer we started earlier. Before doing that you may want to stop SMTP so that SNFClient instances don't build up -- just in case something goes wrong ;-) Once you're ready, the clean way to shut down SNFServer is with the SNFClient using: SNFClient -shutdown You will see the shutdown progress on the SNFServer screen. 13. Once the old test SNFServer has been shut down you can start your new SNFService. 14. HOW DO I KNOW IT'S WORKING? You can check to see that it's up and running with SNFClient this way: SNFClient -status.second You should get back some XML data showing the current message rates and some other information. If SNFServer is not running you will get nothing back and your errorlevel will not be zero. You can see your errorlevel on win* using: echo %errorlevel% on *nix something like: echo $? 15. You can monitor the activity of your SNFServer by opening the status.second report in your browser and refreshing it periodically. status.minute is usually easier to understand. You can (as shown above) get these status reports using the SNFClient also. 16. You can shut down your old version of SNF at any time once you're satisfied that the new version is working correctly for you. Until then you can switch back quickly if you have trouble by changing your scanning software's setup (see step 8. above). The two versions will not interfere with each other. That's it for the overview. The above is not perfect - you may need to read between the lines a bit, but it should be close enough for most folks and should clarify any questions you have along the way. Please let us know how this goes for you -- we are working on documentation and our new web site right now and your input will make it into our work. Thanks! _M -- Pete McNeil
[sniffer] MXScan for MailEnable
Hello Sniffer Folks, For those of you considering MailEnable, MX Uptime (www.mxuptime.com) has an anti-spam plugin that includes a fully integrated SNFEngine (the new version!). Just put in your login code and authentication string and you're good to go. Here's a screen shot link: http://www.mxuptime.com/screenshots/3b.jpg If you try this out please post a note to let us all know how it works for you. Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Version 2-9rc1.8.2 Release Candidate (Std Test Package), and other plans/announcements!...
range hits. Added Timestamp and Command Line Parameter data to SNFClient.exe.err - When an error occurs with SNFClient that may not appear in the SNFServer logs an entry is appended to the SNFClient.exe.err file. That in itself is not new. The new feature is that the entries added to the SNFClient.exe.err file now include timestamp and command line data to aid in debugging. Added BIG-ENDIAN Conversion - When the SNFServer program is compiled on a system that uses a BIG-ENDIAN processor (such as a power-mac) the rulebase load process now includes a routine to convert the token matrix from it's native LITTLE-ENDIAN format to a BIG-ENDIAN format. This solves a bug where Power-Mac (and presumably other BIG-ENDIAN systems) could compile and run the SNF* software but were unable to capture spam because the token matrix in the rulebase file was misinterpreted. Note: The BIG-ENDIAN Conversion feature is still considered experimental because it has not yet been thoroughly tested. Updated the Configuration Log to include all of the current configuration features and to improve it's readability. 20080207 - SNF2-9b1.7.exe SYNC Timeout now 2x SYNC Schedule SNFServer now produces an UpdateReady.txt file when the UTC timestamp on the SYNC server is newer than the UTC timestamp of the active rulebase. It is presumed that a suitable update script or program will run periodically and download a fresh rulebase file if the UpdateReady.txt file is present. The update script should remove the UpdateReady.txt file when it completes a successful download of the new rulebase file. Added available rulebase UTC in status reports udate utc.../ Added Automatic path fixup for ending / or \ Added option to use local time in log rotation rotation localtime='no'/ The default is still utc. 20071102 - SNF2-9b1.6.exe Increased MAX_EVALS from 1024 to 2048. Adjusted defult range envelopes in snf_engine.xml to be more conservative. -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Gateway solution
Hello Nick, Thursday, March 6, 2008, 10:25:18 AM, you wrote: We use XWall (www.dataenter.com) on some of our gateway servers its very efficient, but not sure about running Sniffer on there however, its no-doubt possible Its described as a product for feeding into Exchange, but in reality it works with any SMTP server. Would you be willing to do some experimenting with this using the new version of SNF? Since you already have experience with XWall that would probably go quickly for you. If you could develop a quick installation guide with some screen shots then we can post XWall as an additional integration option. Thanks for the tip! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Proper way to setup a SNFServer on it's own box
Hello Shawn, Wednesday, February 27, 2008, 2:57:29 PM, you wrote: Hello, I searched the Wiki but could not find a proper answer to this question. If I have my mail server on it's own server, and I want to have Message Sniffer on it's own server, how would I go about setting this up? I will have the SNFServer instance running on it's own server. Does SNFClient.exe still get called on the server that is running the mail server? If so, how do I setup SNFClient so it knows where to find the SNFServer? For the trial I had everything running on 1 server and it was easy to get working. But now I have purchased a copy I want to put SNFServer on it's own box. At the moment SNFServer and SNFClient need to run on the same box. SNFClient calls the SNFServer via localhost TCP. SNFServer processes the message as a file on the local file system. If you want to run SNF on a gateway then that gateway must run some software to process the email going through that gateway - either an SMTP proxy like eWall, or an actual SMTP server connected to SNF such as a postfix on a *nix box. There are many options for gateway solutions - the choice depends upon your needs. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Bad Rule Alert - 1771029
Hello Sniffer Folks, Rule ID 1771029 was coded incorrectly for a URL fragment and matched some common dtd reference code. The rule has already been removed but it was posted to some rulebase files before the error was discovered. The rule was created earlier today (2008-02-26 06:00:18) and destroyed moments ago (2008-02-06 16:10:00). Our sincere apologies, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Ideal config for scaleable solution?
Hello Paul, Thursday, February 21, 2008, 7:52:55 PM, you wrote: Ie, ideal for processing/serving 10+ million emails per day in an imail/declude/snf configuration. SNF seems to generally be the big processor hog (though the new beta has definitely made huge performance improvements over the prior version). One of our test platforms uses a single 2.6G processor, IMail, and SNF and consistently handles 4.6 million messages per day. (Typ 2800 - 4500 msg/minute) Of course, that's a special appliation (pre-screening inbound traps) but it does give a rough idea what is possible in your chosen environment. OK...this is a bit off-topic, but I'm looking for some feedback in how to plan for handling this type of load (current load is between 1.3m and 1.8m/day). Should I just throw more high performance hardware at it? Probably not. Scale out perhaps by dedicating a server to just the junk mail scanning. Then have a relatively wimpy server taking care of normal Imail stuff (recipient of the declude/snf clean and/or tagged emails). Two key problems with the IMail platform is that it never stops taking messages and it doesn't support any kind of dynamic connection blocking. Fixing these problems allows IMail to scale to numbers like that. One way to go there would be to set up proxy gateways using eWall in front of your IMail servers. Put SNF on eWall and use it to reject dictionary harvest attacks and possibly even some traffic based on SNF. Definitely use SNF truncate events to add sources to the eWall blacklist. Generally this approach alone will kill off a LOT of traffic that would otherwise bog down IMail/Declude without introducing false positives. You would have many additional options with eWall in this configuration - but you wouldn't need them necessarily and since eWall is amazingly inexpensive you would be getting a big bang for your buck. Your IMail servers w/ Declude could sit behind a pair of eWall gateways (for redundancy) and provide all of the flexibility and additional testing you're used to -- so you wouldn't need to re-tool your infrastructure very much. You probably don't want to scale up using heavy hardware. Instead, use cheaper - more generic hardware and increase your redundancy. One good reason for this philosophy is that if you have a pair of very high - end boxes handling *anything* and one of them dies then it is unlikely the remaining box will be able to absorb twice as much *anything* as it normally handles. In contrast, when one of three moderate boxes handling *anything* the remaining two boxes are very likely to be able to absorb 50% more traffic each. Along that line of thought, can SNF be configured to work directly with the MS/IIS SMTP server? This combo could work great as a spam-killing gateway. Yes and no. IIRC, ORF uses IIS SMTP and will tie in SNF nicely. Also - if you have some skills you could tie SNF into IIS SMTP using our DLL (not published yet, but available and in use on a number of proprietary systems). Out of the box we don't have an SNF + IIS SMTP solution (yet). Has anyone assembled this sort of configuration in a load balanced/redundant environment? It has been done. Most that I know of who have done this eventually moved away from IMail/Declude as they grew beyond the numbers you're talking about and developed their own proprietary filtering platform (using SNF as it's core) on top of a more robust EMail platform (Communigate for example). Others who are comfortable with mixed environments have deployed SNF in their gateways. The general model for scalability is to isolate inbound gateways, user-centered email servers (pop, imap) and outbound gateways into separate layers with their own redundancies. It is also common for each layer to use it's own hardware and software platforms - each best suited to the specific task. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Ideal config for scaleable solution?
Hello Andrew, Friday, February 22, 2008, 4:37:18 AM, you wrote: snip/ If you were a *nix shop, you would still lean towards having a dedicated gateway server (or many) and your CPU hog would be spamassassin, which you would run in a client/server model to shift the CPU usage to other boxes. Of course, SNF also can plug into SA. However SNF tends to be much leaner than SA with comparable (or even slightly better) capture rates. You may want to run SNF in front of SA to get rid of most of the junk and rapidly inform local blocking lists and gray-listing mechanisms. The combination of SA SNF is superior to either on it's own if you have the technical resources. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Updated - did I forgot anything?
Hello Alexander, Monday, February 4, 2008, 8:40:44 AM, you wrote: Hello, I updated our sniffer on IMail/Declude to 2.9b. I created I new directory, installed sniffer as a service, modified the global.cfg to call the SNFClient, modified the xml files (license, directories). Anything else? Its running and the ID.date.log.xml is growing :) I have good telemetry from your installation (most of the time). Is this list useful or do I only need one call with the new beta? SNIFFER-NOTFOUNDexternal 000 "C:\IMail\Declude\Sniffer3\SNFClient.exe" (removed the weight) SNIFFER-TRAVEL external 047 "C:\IMail\Declude\Sniffer3\SNFClient.exe" snip/ Two additional result codes are used with the new version. Result code 20 is sufficient to delete messages on most systems. Result code 40 should be treated similarly to a reasonably accurate RBL score. I have updated the wiki: http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.ResultCodes#Core_Rule_Group_.26_GBUdb_Result_Codes _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: snfilter - linux - postfix
Hello Pi-Web, Saturday, February 2, 2008, 6:07:19 PM, you wrote: Hi Pete, Just for information, we renamed the msg folder again today, and again SNFClient.exe.err only state: Could Not Connect! /etc/init.d/snfilter stop + /etc/init.d/snfilter start helped. When SNFClient cannot connect the SNFServer is either not accepting new connections, or it is down. If it is down then restarting it helps by starting it again. If it is not down then restarting helps by abandoning all of the existing connections -- many of them will re-try and succeed when the SNFServer is active again. This stalling effect is seen only when you rename the folder that contains the message files -- I wonder if there is a quirk of the environment that causes the SNFServer engine to be hung attempting to open and/or read the files in the changed directory such that additional scan requests queue up and are not serviced. Since SNFServer is not giving any errors (or at least you're not reporting any so it is likely that it is not) then I can only assume the program has not seen any errors that it can report. It is probably not a good idea to rename the folder while there is any possibility of active scans in progress. Instead I would suggest that you create a new folder with the correct name for new scans (perhaps by date) and then abandon the older folder in place. New scans would be done in the new folder and old or existing scans would continue in the old folder until they were complete. By the time you do anything with the old folder it will be several generations behind and safe. I do not completely understand your methodology -- but if I'm correct about it then the above approach should work. I might also recommend a different approach -- Use a single directory for scans and have them always performed there. Then - depending upon the scan result move them into the appropriate directory. This way you could always be assured that the scanner is finished with a file before it is ever moved. This is an efficient process on ext3 and most other modern *nix file systems since it only requires the adjustment of a node and that operation will itself be journalized first. Thanks for keeping us posted. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: snfilter - linux
Hello Pi-Web, Sunday, January 27, 2008, 6:31:15 AM, you wrote: Hi Not sure what we have done - but snfilter has stoped working. The x.200801??.log.xml is not more created. SNFClient.exe.err says: /var/spool/snfilter/msg/20080127122626_4614.msg: Could Not Connect! Messages are put in /var/spool/snfilter/msg/ but not checked. I cant see what I have done wrong, but guess we did something around here: Jan 26 00:43 x37l67rv.20080125.log.xml as this is the last log. These are beeing created: -rw-r--r-- 1 root root 743591 Jan 27 12:29 x37l67rv.status.minute.20080127.log.xml -rw-r--r-- 1 root root 1079 Jan 27 12:30 x37l67rv.status.second.log.xml SNFServer has stopped --- if you had it set up as a service you should be able to restart it and solve the problem. If you were running it in a dos window -- start up a new dos window with it. Please look for any errors in your logs that might indicate why the SNFServer stopped. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: snfilter - linux - postfix
Hello Pi-Web, Sunday, January 27, 2008, 1:16:08 PM, you wrote: Sorry, I might not have been clear. It is on Linux with postfix. I should have picked that out of the path. ;-) Yes stop/start of the service did solve the problem. Before start/stop pstree showed 14*SNFserver.exe SNFClient.exe.err only state: Could Not Connect! Last x.200801??.log.xml ends with: i u='20080125234317' context='--INITIALIZING--' code='0' text='Success'/ Rest seems normal. That also seems normal for a start-up. So I have no clue why it stoped. This is unusual. I've repeatedly had SNFServer run for weeks and months on various platforms -- almost without exception it only stops when I tell it to stop (including earlier test versions). If you come across any new info please let me know. If there is a bug I want it gone ;-) Thanks! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] New reference settings for GBUdb ranges.
Hello Sniffer Folks, We have been researching/refining the default ranges for GBUdb. Here are our latest reference settings. These are conservative for large systems (500/min) and should be even more conservative for smaller systems. Smaller systems that experience lower message rates will tend to have lower confidence numbers in their GBUdb due to fewer message interactions. If you run a system that sees fewer than 500 messages per minute then you may achieve higher capture rates before FPs with lower confidence values in some of your ranges. Another way smaller systems may adjust their GBUdb sensitivity is to adjust the time between condensation from one day to two days (or more) or to eliminate the time based trigger and rely on the memory usage trigger instead (by triggering condensation events only when a specific memory threshold has been reached). The latter method is typically recommended for systems with fewer than 10 messages per minute. All of the above tuning recommendations are somewhat experimental since GBUdb is relatively new and at present sparsely populated (about 300 participating nodes at present). As time goes on we will all learn more about how to optimize GBUdb - please experiment cautiously and scientifically (one change at a time and understand what has happened) and please share your results. Here is the current reference: regions white on-off='on' symbol='0' edge probability='-1.0' confidence='0.4'/ edge probability='-0.8' confidence='1.0'/ panic on-off='on' rule-range='1000'/ /white caution on-off='on' symbol='40' edge probability='0.1' confidence='0.0'/ edge probability='0.8' confidence='0.3'/ /caution black on-off='on' symbol='63' edge probability='0.8' confidence='0.2'/ edge probability='0.8' confidence='1.0'/ truncate on-off='on' probability='0.9' peek-one-in='5' symbol='20'/ sample on-off='on' probability='0.8' grab-one-in='5' passthrough='no' passthrough-symbol='0'/ /black /regions If you are running the new SNF and you haven't checked your GBUdb range settings in a while this might be a good time to make some adjustments ;-) Some of the settings in previous releases were less conservative and some were less aggressive -- all were backed by less experience (of course). The settings shown above are likely to become the default settings for the production release, however we will continue to refine these settings through our research prior to (and following) the production release (planned in Q1). Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: GBUdb question
): SNFClient.exe -bad IP4Address (If on my IP whitelist, it just won't do anything here.) Ok. (C) If my spam filter marks a message as ham, then it will issue the following command (again, regardless of whether or not the message was run through sniffer) SNFClient.exe -good IP4Address That sounds very much like what these tools were designed for. However the effect may not be what you intend. If the IPs you track are not detected as the source IP by GBUdb then it is likely to ignore the data during it's scans. It will evaluate the statistics of the IP it believes to be the source. When it gets that right it will find your data. When it gets that wrong it will find no data (most likely) so GBUdb will be effectively inert in those cases. If your intent is simply to input this data into the GBUdb system so that it is available as a resource then that will work - somewhat. One other thought that I have is that you could use the command line (or the ignore list) to mark the IPs on your internal white-list as Infrastructure (ignore flag). This might effectively train GBUdb to skip those IPs when finding the source of the message - and in any case would render GBUdb inert for those IPs. ** ** I know that this puts more trust on me and my system, but I have also know that the quality of stats you'd receive from my system would vastly improved due to my abilities in this area and this would be a huge contribution to other Sniffer users over the norm. (I run one of the best RBLs and URI blacklists in the world... I know what I'm doing here!) Each SNF node contributes to the overall consensus of the cloud and can learn from that consensus. The mathematics that govern these interactions ensure that malicious or erroneous inputs remain isolated from the other SNF/GBUdb nodes. And, of course, if we discover any serious problems we can always turn off any malicious users. The influence that any single node can exert on GBUdb is limited by a mathematical curve that separates each node from the cloud. This ensures that each node retains it's own perspective and that the consensus is representative of information that GBUdb nodes consistently agree is correct. Erroneous or malicious signals have a tendency to be washed out of the system quickly. Specifically, the strength of any specific input as seen by any other specific node is reduced by log2(log2(input)). Any input sufficiently large to cause a disruption would need to be astronomically large and so it would be detected as highly unusual and would immediately attract our attention. Conversely, small inputs made by large numbers of nodes consistently over time tend to produce large signals that are useful to individual nodes - especially when a new IP source is detected. The first time a node sees a new IP it is more likely to be influenced by the opinion of other nodes that have already seen the IP. Once a node has a sufficient number of it's own experiences it tends to trust it's instincts. Can these things be done? Some can be done now. Some may be done in the future. I look forward to seeing an example of your header. Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: New reference settings for GBUdb ranges.
Hello David, Tuesday, January 22, 2008, 12:43:09 PM, you wrote: Hi, I think I must have missing something or been asleep. I've had a look at the Sniffer site and to be honest I don't fully understand what GBUdb is. I've read the technical details page but I don't see how it fits into the whole scheme of things, if it's useful to me, and if it is, how to implement it. I understand what it's trying to acheive but I can't see beyond that. Think of GBUdb as an enhancement to the SNF scanning engine. GBUdb keeps track of where messages come from and whether those messages are spam or not. If they fail an SNF pattern rule then they are considered to be spam. If they do not fail an SNF pattern rule then the are not considered to be spam. When a new message comes from a source that GBUdb knows about then it SNF work better and faster. Reducing Leakage: If GBUdb knows that messages from a particular source are almost always spam then SNF will detect the message as spam even if there is no pattern rule yet. This helps reduce leakage. That is-- new spam from old bots will generally get killed by GBUdb. Reducing False Positives: On the other side of things; if an SNF pattern rule tags a message that comes from a trusted source then GBUdb will make sure that the message gets through. This reduces false positives. _ GBUdb has Friends: One other thing that is important about GBUdb is that it doesn't work alone -- it has friends. All of the GBUdb systems on the 'net share what they know about message sources. This way when a spam bot starts to send messages to a new system that's never seen it before the other GBUdb systems can tell the new system that the message source (IP) is bad so it doesn't have to start learning that information all on it's own. _ Faster and More Efficient: In addition to reducing leakage and false positives, GBUdb also makes message scanning go faster and take fewer resources. If GBUdb knows that a message source is very, very bad then it will cause SNF to stop scanning the message as soon as it sees the IP address that sent it. This is the truncate feature. The result is that between 15% and 50% of messages going through the SNF scanner will be handled almost instantaneously - without bothering to look at most of the message. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: New reference settings for GBUdb ranges.
Hello David, Ooops, I missed a question... Tuesday, January 22, 2008, 12:43:09 PM, you wrote: snip/ ..., how to implement it. GBUdb is built in to the new version of Message Sniffer. It is turned on by default and the default settings work for just about everybody. If you have any email gateways or an email address where you legitimately receive spam (such as an abuse reporting address) then you will want to tell GBUdb about those so that it doesn't get the wrong idea about them. If you have more questions then please let us know. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: GBUdb question
adjust and even disable the ranges that are defined for the GBUdb. This allows you to develop your own statistical model for classifying messages based on the behavior of their source. The ranges that you select for your GBUdb do not impact what is reported and learned about the IPs - - only how your system responds to those statistics. The new reference settings we have created are such that they are reasonably safe even when large ISPs are involved. A message source must be observed producing more than 95% spam before it will create a truncation event. Sad but true - many major ISPs generate just shy of that amount of spam through various vectors (forwarded mailboxes, being one of them). You may find that the new reference settings produce something very close to your desired result -- especially if you also provide the additional hinting that you propose. If you are able to teach your GBUdb to ignore the correct IPs then you will be able to tighten the range envelopes significantly and dramatically reduce leakage without adding false positives. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Postfix
Hello Pi-Web, ERROR_MSG_FILE means that SNF could not open the file to be scanned. Be sure the you pass the full path of the message file and that permissions are correct so that SNF can open the file. Hope this helps, _M Wednesday, January 16, 2008, 12:31:58 PM, you wrote: No its not the message format. A message the get ERROR_MSG_FILE work fine on our windows SNF installation. Hi We trying to setup snf with postfix. It seems to work - except it does not reject ant messages. The x.20080116.log.xml says: s u='20080116110805' m='20080116120805_22626.msg' code='69' error='ERROR_MSG_FILE'/ This I belive is because the msg file that is send to sniffer has a wrong format. - If true - how do we setup the right format for sniffer? -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Postfix
Hello Pi-Web,
Yep.
The clue was in the log:
m='20080116183528_10882.msg'
Note that the path was missing - only the file name was present.
Now your logs should look more like:
m='/var/spool/snfilter/msg/20080116183528_10882.msg'
Best,
_M
Wednesday, January 16, 2008, 1:23:14 PM, you wrote:
Adding $INSPECT_DIR to the $SNIFFER_EXE $AUTHENTICATION
$INSPECT_DIR$MSGFILE || { command
Now it seems to work.
It seems right - but no go:
In /var/spool/snfilter/msg/
-rw--- 1 snfilter snfilter 2965 Jan 16 18:35 20080116183528_10882.msg
(deleted after process finished)
Result:
s u='20080116173528' m='20080116183528_10882.msg' code='69'
error='ERROR_MSG_FILE'/
sniffer setup:
SNIFFER_EXE=/var/spool/snfilter/SNFClient.exe
AUTHENTICATION=
INSPECT_DIR=/var/spool/snfilter/msg/
SENDMAIL=/usr/sbin/sendmail -i
MSGFILE=`date +%Y%m%d%H%M%S`_$_$RANDOM.msg
Hello Pi-Web,
ERROR_MSG_FILE means that SNF could not open the file to be scanned.
Be sure the you pass the full path of the message file and that
permissions are correct so that SNF can open the file.
Hope this helps,
_M
Wednesday, January 16, 2008, 12:31:58 PM, you wrote:
No its not the message format. A message the get ERROR_MSG_FILE work
fine on our windows SNF
installation.
Hi
We trying to setup snf with postfix.
It seems to work - except it does not reject ant messages.
The x.20080116.log.xml says:
s u='20080116110805' m='20080116120805_22626.msg' code='69'
error='ERROR_MSG_FILE'/
This I belive is because the msg file that is send to sniffer has a
wrong format.
- If true - how do we setup the right format for sniffer?
--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Rule Database copy question
Hello Shawn, Wednesday, January 16, 2008, 2:26:14 PM, you wrote: Hello, I am using the latest beta version of Message Sniffer. I am asking this question because I thought I read this somewhere but I can not find where I read it. If I copy my rule database file to the c:\snf directory while SNFServer.exe is running, does SNFServer automatically load the new updated rule database? Yes. If so, how long does it usually take before SNFServer realizes that there is a new rule database that was copied to that directory? Within about a second of seeing the new file it will load and check the new rulebase. If there is something wrong with the rulebase file it will keep the current rulebase active until a better one shows up. Is there anyway to verify that SNFServer has loaded the latest rule database that was copied? I know I can run a SNF2check.exe on the rule database to check the file before I copy it, but it would be great to know if SNFServer.exe has loaded the latest copy that I have copied to the c:\snf directory. SNFServer will indicate that the new rulebase was loaded in it's log file. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: ERROR_SYNC_FAILED
Hello Shawn, Wednesday, January 16, 2008, 4:53:29 PM, you wrote: Thanks for the quick reply Pete. When SNF connects to your SYNC servers, what information/data is it exchanging? The telemetry we receive is roughly equivalent to what you see in your .status.minute. file. In addition your SNF node sends: * GBUdb alerts - These contain periodic updates on IP information in your GBUdb database so that the information can be shared with the cloud. An example might be: gbu time="20080116220039" ip="190.28.248.159" t="Ugly" b="1" g="0"/ gbu time="20080116220041" ip="74.50.113.233" t="Ugly" b="1" g="0"/ gbu time="20080116220045" ip="201.92.79.22" t="Ugly" b="1" g="0"/ gbu time="20080116220047" ip="74.50.113.233" t="Ugly" b="2" g="0"/ * Spam samples - Messages that would normally be truncated but do not fail pattern rules are randomly sampled by default and sent to our virtual spamtrap system. This feature can be disabled if you wish. Your node then recieves: * Rulebase status - Our system sends back information on the latest rulebase file. * GBUdb reflections - Our system sends back GBUdb reflections (same format as above) corresponding to any alerts that your system sends us. This allows your system to learn from the cloud. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer Win32 command line output
Hello Shawn, Thursday, January 10, 2008, 2:16:24 PM, you wrote: Hello, I am evaluating Message Sniffer beta version but I am totally confused. :-) snip/ But how do I get the result code for the spam message to output back to the command prompt? If I try to call SNFClient.exe from my C# code, I still cannot get a result code returned to me. I can get a result code if I do this: SNFClient.exe -test xx.xx.xx.xx but SNFClient.exe does not return the result code when I am passing a filename to be tested. Can someone point me in the right direction on how to see this result code via my C# software code or command prompt box? I'm not sure how C# behaves when it calls an external program and how it handles that progam's result code -- I'll do some looking. However, most programs that call SNFClient do so explicitly to get the result code so I know it works ;-) One thing that you might try that will improve your performance since you're rolling your own C# code: Check out the XCI interface. The SNFClient uses it to talk to the SNFServer instance. You should be able to write a quick bit of code to use XCI to talk to SNFServer also. The basics are (per scan request): 1. Connect to 9001 on localhost via TCP 2. Transmit your request string (XML using the XCI examples as a guide) 3. Read the response string (XML again) 4. Close the connection Making your own XCI request saves the step of launching yet another program to do it for you. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Sniffer Win32 command line output
Hello Shawn, Following up a bit... Most likely you're using a Process object to call the SNFClient. If I've read the MS docs correctly you will want to get the "exit code" once SNFClient finishes. http://msdn2.microsoft.com/en-us/library/system.diagnostics.process.exitcode(VS.71).aspx Hope this helps, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: I got a strong attack today
Hello Alberto, Friday, January 4, 2008, 4:56:29 PM, you wrote: Hello I got a strong attack today, over thousand messages at the same time!! The usual technique: Impersonate the victim and send to non valid users of one domain of mine!! Changing IP for each message UNBELIEVABLE!! This is very common these days. We call it getting caught in the light. Our spamtrap server is currently experiencing a similar attack and is seeing 1850+ messages per minute. Luckily we've killed this particular campaign a few hours ago so leakage is only 7/min and 890+/min of these messages are being truncated (scan stopped based on IP via GBUdb) The only solution was, to stop all the services and move all the spool files in a temp directory. I won't use the nobody alias because at least the iMail Access Control can stop some bad IPs. My config is: Imail 9.23 Mxguard 3.1 Message Sniffer InvURIBL 3.7 Two questions: 1) There is a way or tool to recycle back good messages from the temp directory into the queue? You should be able to write a cmd script to test the messages in your temp folder against SNF and place the clean messages back into the spool for delivery. This doesn't give you a complete solution, but it is reasonably viable in such cases. I've not heard of it, but you may be able to find or write a similar utility to put the temp messages through the entire scan process at some reasonable pace -- You might ask DG about that - I'm not sure what would be the best way to go about that w/ mxGuard and he may have a solution already or know where it's buried. Side Note: We actually have a technology that we've simulated and not deployed called Gauntlet. Under certain conditions messages are shunted to a waiting area where their scanning and delivery are delayed for a period of time so that filtering systems can catch up... For example, messages that arrive from completely unknown IPs would have to run the gauntlet before being delivered. The sensitivity of the shunting system could be guided by storm data (B and C counts) from GBUdb to reduce the possibility of delaying ordinary messages. What you are describing is a manual version of this process. 2) How can I reduce or block(!) this kind of attacks? The new version of SNF is very good at reducing this kind of attack because the GBUdb component frequently can identify bad IP sources very quickly after a new campaign begins and is able to block many of the messages based on the IP reputation information known by the network. In some cases this might include substantially all of the attack prior to new pattern rules reaching your system -- in all cases at least some fraction of the attack would be identified (based on observations). The system will become more sensitive as more systems begin using the new software -- at this time it is remarkably sensitive even though only a small fraction of SNF users are already using it -- so we expect significant improvements. In this case, for example, many of the messages arriving would be seen by SNF, identified after a very short scan (only the first few hundred bytes), and then most-likely deleted (depending on how you tune your system; also I'm not sure what options are available from mxGuard w/ regard to preempting additional tests and/or test ordering). Given your system's configuration I don't know of any way to block this kind of attack without adding additional components. A couple that come to mind are SPF checking (so that any message pretending to come from your domains must actually be coming from your servers before being accepted), and graylisting which, while sometimes problematic, currently provides some pretty good protection against dumb-bot attacks. (Note that the newer bot softwares out there easily defy gray listing so it's effectiveness is dropping quickly) Hope this helps, Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: I got a strong attack today
Hello Paul, A relatively easy and reliable way to recognize one of these storms is whenever your new SNF engine starts throwing Bs and Cs- That is - you can check the second.stat or minute.stat file for Black and Caution hits: rates c .. m b .. m /rates On most systems Caution and Black events are relatively rare, but during a storm these numbers tend to be high. It is conceivable that you could detect these conditions by checking the stat files and adjust your system's settings during a storm. _M Friday, January 4, 2008, 5:38:38 PM, you wrote: We saw the same thing this morning between 7:00 AM (GMT-0500) and about 8:30 AM. Big chunks were getting through (spam detection rate dropped to about 65-70% (from its normal 97-99%). Sniffer updates seemed to start quelling the attack after about an hour of getting pummeled. Because of the relatively short lifespan of these types of attacks you need to: 1) be aware of attack quickly - e.g. w/in 10-15 mins of seeing average detection rates drop below a certain threshold (maybe 85%?)) and 2) be able to determine if there is an easy way to ID the leaked messages (common source IP(s), From domains (SPF check would help), subject lines, etc) 3) then be able to create a temporary rule to help block messages - must be viable until SNF has an updated ruleset to start clearing out the attack - I don't think declude (what I use w/SNF) has rule expirations (but would be a nice feature) Paul --- -Original Message- From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Alberto Santoni Sent: Friday, January 04, 2008 4:56 PM To: Message Sniffer Community Subject: [sniffer] I got a strong attack today Hello I got a strong attack today, over thousand messages at the same time!! The usual technique: Impersonate the victim and send to non valid users of one domain of mine!! Changing IP for each message UNBELIEVABLE!! The only solution was, to stop all the services and move all the spool files in a temp directory. I won't use the nobody alias because at least the iMail Access Control can stop some bad IPs. My config is: Imail 9.23 Mxguard 3.1 Message Sniffer InvURIBL 3.7 Two questions: 1) There is a way or tool to recycle back good messages from the temp directory into the queue? 2) How can I reduce or block(!) this kind of attacks? With my best regards Alberto # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to sniffer- [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED] -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: I got a strong attack today
Hello Alberto, Friday, January 4, 2008, 6:50:55 PM, you wrote: Pete Thank you very much for your very exhaustive response! It's what we do. ;-) Do you have any other information on this technology called Gauntlet that seems me very very interesting. There really isn't much more to it than what's been said. The concept has been around for several years now -- the details are platform and policy specific. We have it on the drawing board to include it as a feature in some platforms that we support - however that is a complicated piece of engineering since each platform is different and we support _MANY_ platforms. (sideline = put messages through the gauntlet) Consider just a few, for example: MDaemon calls SNF as a plugin and doesn't provide any simple (fool proof) method for message re-injection. Also, it is not clear that there is a friendly and reliable way to sideline the messages on this platform. We could sideline messages in IMail by parking the Q and D files in a special directory and then later re-processing them through SNF back to the spool... -- But, if Declude is present then we might instead wish to re-process the messages through the proc folder, and there are uncertainties about when and how to do this and how to pace it. -- If mxGuard is in place -- how would we re-process the messages at all? -- How could we ensure that virus scanning etc would be enabled (or not if desired?) SmarterMail could be handled (presumably) in a similar way to IMail except that the file structures are different as are a few assumptions about message processing and acceptable loads, etc. In Postfix systems we would need to create our own data structures to capture envelope information before we sidelined the message -- all that in addition to considerations of other processes that might be in place (without notice) and might need to be considered when we re-process the messages. Communigate systems store routing information in the message file itself which would simplify sidelining the messages but complicates the re-processing task - and again there are other processes that might be in place unannounced... All that by way of illustrating that the concept of Gauntlet is powerful and simple to understand, but not so simple to implement. For now we've been describing it to folks and helping them implement versions of Gauntlet in their proprietary systems. With a bit of luck and elbow grease we will hopefully release utilities and/or special versions of SNF to support this on some platforms -- This is particularly attractive since the GBUdb engine produces signals that theoretically allow us to activate and deactivate (or desensitize) Gauntlet under specific conditions very accurately. Specifically, GBUdb can provide a clear signal for the presence of a spam storm by monitoring Black and Caution activity. GBUdb also provides ready statistics on IPs so that we can define which IPs not to sideline (when the IP is reasonably well known and reasonably unlikely to send spam). -- That's about all I can think of to say about it at this time (at least without some more specific questions). But I don't think that Mxguard can manage all of this you are explaining in the message. That's probably true -- but not certain. Consider, for example, that your re-injection script could act just like IMail... * Drop the D file back into the spool * Drop the Q file back into the spool * IMMEDIATELY call mxGuard with the Q file in precisely the same way IMail does. In theory this would work for mxGuard or Declude since both programs would see this activity no differently than if IMail had just dropped a new message in for processing. That's a very big In theory -- because I've not tried it, but based on the available documentation the theory is sound. I will try to write a CDM to solve my queue problems Please keep us posted. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: The new version of SNF
Hello Alberto, Friday, December 28, 2007, 2:32:55 PM, you wrote: Hello The new version of SNF is released? How much is it stable? It's not yet officially released, but the current beta (1.5) has been production stable for quite a while now. The official release will wait for a few extra features we want to add to make it easier to administer and extend. That release will happen Q1. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: re subscriptions to list
Regarding this thread and to nobody in particular: I would like to say a word or two before this gets out of hand. Our policy on this list is to provide the answers needed no matter how obvious or well posted those answers may be. Emotionally negative responses are discouraged and generally not useful. RTFM type answers should be emotionally neutral, should summarize a quick answer, and should provide a link to TFM. For whatever reason, these kinds of requests are made and these kinds of questions are asked. The folks who make these requests or ask these questions - no matter how obvious - need help. The best thing we can do is provide that help. Keep in mind also that these messages are archived so that they remain searchable on the 'web. This means that any solutions we post here, including references to obvious or well posted answers, serve to make those answers easier to find. Please: Be kind and helpful, or stay away from the send button. I can't remember the number of times something simple and obvious baffled me when I needed it least -- and I'm sure many of us have had similar moments.* A simple answer to an obvious question can go a long way in a positive direction. Please help us keep this forum active, positive, and informative. Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. * One of the biggest problems with technology is that as people come up the learning curve they tend to forget what it was like when they didn't know the obvious. All too often, once we have come up the learning curve a bit we go on to punish those who are just starting out. Even the best of us leave unintentional barriers by simply not discussing the obvious. * Those of us who should already know the obvious are still subject to moments from time to time when it escapes us -for whatever reason. Think of how much time we could all save if it were easier to escape those moments. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: FTP access to snf rulebase files is no longer available.
Hello Bill, Friday, November 23, 2007, 4:55:09 AM, you wrote: Hi Pete, I don't think our logs are being compressed even though I'm using the snifferupdatetools. I see where this is mentioned but after checking the scripts I don't think this is happening. How can I do this? It is ok to compress your log files before uploading them. Compress one file at a time -- not groups of logs. .zip or .gz formats are ok. There's nothing else special about it. For example, your licenseid.log file is compressed with gzip or zip and ends up being licenseid.log.gz or licenseid.log.gz -then you upload the compressed version. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Was: Database Compiler Upgrades Now: When will the new version be out of beta?
Hello Serge, Friday, November 16, 2007, 4:40:29 PM, you wrote: Hello Pete Since installing the Beta, I'm having many smtp32 errors. A windoows pops up with the following message: smtp32.exe application error Application could not initialize correctly 0xc142 Cannt it be related ? I suppose it's possible - but not because of what the SNFClient does, rather because of how the new SNF engine is more efficient. The Application failed to initialize problem has to do with a hidden heap being exhausted and it is an old problem. Since the new SNF engine is multi-threaded and more efficient you are likely to run a different number of concurrent messages than before. This will effect how the resources on the machine are used. You might try adjusting the number of threads you allow. See previous discussions on this error for guidelines and fixes. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Database Compiler Upgrades
Hello Robert, Thursday, November 15, 2007, 4:42:25 PM, you wrote: Timing on release to production? We are continuously improving our back-end systems. There is no specific timing for any of the many projects. The current hardware upgrade process will be completed this week. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Database Compiler Upgrades
Hello Sniffer Folks, There have been a few unavoidable disruptions regarding rulebase deliveries and/or update notifications over the past few days. We apologize about that. There should not be any more noticeable disruptions in the foreseeable future. Over the past few days we have upgraded nearly all of our database, bot, and compiler servers. There are a few more systems to upgrade, but at this point the heavy lifting has been done :-) Moving forward you will see more frequent updates of your SNF rulebase files and you will also see improvements in the rulebase quality as we bring new back-end systems online to take advantage of our new hardware. Thanks for your patience and support! _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Was: Database Compiler Upgrades Now: When will the new version be out of beta?
Hello Robert, Sorry, I wasn't really specific enough. Actually interested in the estimated date for Sniffer itself. For most of the year, we enjoy being modestly early adopters. During the holiday quarter, we're late adopters. Ah - you hijacked the thread. I thought something like that was going on. Please don't do that ;-) The projected Production Certification Date for the new SNF engine is a moving target as we collect features and bugs (if any). Currently it looks like 5 - 7 weeks is a good estimate. The current beta is in production on many systems with no current bugs*. All of the items on the to-do list look much more like features: Automatic trailing / for paths Drilldown Header Directive Log Rotate On Local Time Option SNF BIG_ENDIAN* (* Ok, this might be considered a bug if you plan to run SNF on hardware like a G5 Mac.) The biggest thing holding back the beta is documentation -- that takes time, and we are working on it. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: SNF V2-9b1.5 Released - Please Upgrade
Hello Serge, Wednesday, November 7, 2007, 4:57:24 PM, you wrote: That will help Please keep us updated Regards BTW, we have been on Beta for few days now Any problems about our system you see from your side ? (besside long sessions) Your telemetry looks good. I see that your SYNC sessions are long (about 1.2 secs), but there are no apparent retries. I see that between 20 and 30% of your traffic is being truncated by GBUdb, and that most of your traffic comes from new IPs -- so your system is learning most of it's IP reputation data from the cloud. Your system is remembering about 94K IPs. Spam is about 95% of your traffic -- a little on the high side, but still nominal. From what I can see everything is running normally. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: SNF V2-9b1.5 Released - Please Upgrade
Hello Serge, Tuesday, November 6, 2007, 9:56:26 PM, you wrote: Hello what files need to go in the workplace directory ? TIA Normally, all of the distribution files plus your rulebase (.snf) file. Also, it is common to have your update script and utilities in the workspace or a sub directory from there. It is possible with the new version to put some of these files in different locations - but that is more complex. You can see the directory options in the top few lines of the snf_engine.xml file where you can set paths for logs, rulebase files, workspace, and identity. Be sure to include the full path (on winx boxes this includes the drive letter). One common option when setting up the new beta on a system that already has the old version running is to configure the snf_engine.xml so that the rulebase file is located in the old SNF workspace. This way it is easy to switch back if desired, and existing update mechanisms can remain unchanged until you are ready to make a permanent switch. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: SNF V2-9b1.5 Released - Please Upgrade
Hello David,
Tuesday, November 6, 2007, 10:39:46 PM, you wrote:
When do you think the beta version will go to non beta i.e. live.
The short answer is 6-8 weeks. The more comprehensive answer -- read on...
We are slowly building a set of features that we think should be in the production version. All but two of these are minor adjustments.
One that isn't minor is a drilldown/ training directive that will be able you to automatically add IPs to your ignore list for mixed sources based on matching text patterns in headers.
So, for example, if you'd like to drill down to sources coming through yahoo or aol servers without having to identify the IPs for their outbound servers, then drilldown/ will (in theory) do it for you by matching the reverse DNS portion of your trusted (top) received headers and adding the IP to your ignore list. The effect is to allow a system to see down to the actual source of the message before training GBUdb while using only a few entries to train the engine. Theoretically this will provide a more fine grained approach to dealing with forwarded mailboxes ("the other kind of open relay") and large ISPs that don't control the outbound flow from 0wn3d machines very well. There is much study, trial, and error to be done with this feature but it does look promising so we're going to put it in.
Another nontrivial feature will allow the SNF engine to run properly on big-endian systems (such as G5's) by detecting the big-endian processor at compile time and converting the format of the SNF rulebase each time it is loaded. There is some work to do to verify that the GBUdb code will work in a big-endian environment, but code review so far has not spotted any trouble in that part of the code. Snapshots of the GBUdb data will not be portable to other systems, but they are not intended to be portable anyway - so that is not considered an issue.
The less invasive features include things like:
* Extending the MAX_EVALs limit.
* Log rotation file names may use local (not UTC) time.
* Adjusted default settings for GBUdb (see below).
* Additional telemetry for error and special event tracking.
* Improved persistence for life-time statistics (run time, last save, last condense, etc).
* Others TBD.
I expect the list of "must have features" to grow a tiny bit over the next couple of weeks.
We are not seeing any fault reports on the current beta so I doubt there will be bug fixes at this point.
After we implement the new "must have" features list we will continue in beta for another week or two to ensure that we have not introduced any bugs.
During that time we will build additional documentation.
I think based on this back-of-the-envelope analysis that we are 6-8 weeks from a "production" release.
That said, the current version does appear to be stable in all supported production environments.
We are working on refining the default tuning for the GBUdb section. The current thinking uses the following, extremely conservative tuning that will be included in the next minor release (probably this weekend).
We recommend that all new Beta installations adjust their configuration files to use the following settings for GBUdb Caution and Black ranges. These are also appropriate adjustments for any existing beta users who have not otherwise resolved any GBUdb based false positives due to oversensitivity.
black on-off='on' symbol='63'
edge probability='0.7' confidence='0.2'/
edge probability='0.7' confidence='1.0'/
truncate on-off='on' probability='0.9' peek-one-in='5' symbol='20'/
sample on-off='on' probability='0.8' grab-one-in='5' passthrough='no' passthrough-symbol='0'/
/black
caution on-off='on' symbol='40'
edge probability='0.3' confidence='0.0'/
edge probability='0.7' confidence='0.3'/
/caution
Thanks,
_M
--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.
#
This message is sent to you because you are subscribed to
the mailing list sniffer@sortmonster.com.
To unsubscribe, E-mail to: [EMAIL PROTECTED]
To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED]
To switch to the INDEX mode, E-mail to [EMAIL PROTECTED]
Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XCI Error!: snf_EngineHandler::MaxEvals
Hello Pi-Web, Friday, November 2, 2007, 6:46:26 PM, you wrote: On 8438 t today we got a average T=111,1176819 Min=0, Max=7211. (57 scans took above 1000, 6384 scans took less than 101). The server is rather old and serving both web mail, pop3 and smtp. And heavy usage of web mail does slow it down. This might be the case on the slow scans. The long scans is not at the same time, but from time to time during the day. Still this should not lock up snfserver. True. Is the server at least a p3? I have many production servers running now (hundreds) that never lock up -- so it's not likely that I will be able to reproduce this error. I recommend running the SNFServer component from the command line and pumping it's output to a file -- you might even run it in a loop in a .cmd script so that it will be sure to restart after a crash. By sending it's output to a file we should be able to see any errors that it reports on it's way down. This will give us somewhere to look. To call snf we use a dll of own development (pluged in to Merak mail server). The call to snfclient is done using a: WaitforSingleObject with INFINITE wait time. (perhaps we should change this). I think that's correct. However, since you've developed your own DLL, you might consider bypassing the client altogether and connecting to the SNFServer w/ TCP using the XCI protocol. On your somewhat overloaded server, launching an external process may be lending to performance reduction at the very least. When it finish - and it does - we get the snf result using GetExitCodeProcess. This return zero (whitch is good, else all messages would be rejected) when the snfserver is in the Could Not Connect! state. Right. The client will return a fail-safe result when it has a problem getting a real result. I have changed the maximum number of evaluators in the code. I hope to be able to put it up on the server some time today. However, I doubt that has anything to do with what's happening here. The max eval error is handled properly in the code and recovery is very simple and tidy. This particular case has been around for as long as the engine has been in place. Also, the max evals number is 1024 (now 2048) while almost every scan recorded is well below 100. This does cause me to wonder if it's a good idea to change this safety check at all. It seems more likely that the test is doing what it should - and perhaps detecting some corrupt memory (you did say the server is old). The test was originally designed as a sanity check to avoid having the scanner run off in a tight loop allocating itself out of memory due to a corrupt rulebase file. Anyway --- I doubt that the max evals condition is directly connected to the SNF Server shutdowns. SNFServer should tell us why it shuts down when that happens and we should be able to get that info if we run it from the command line and capture it's output. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: SNF V2-9b1.5 Released - Please Upgrade
Hello Serge, Saturday, November 3, 2007, 4:04:32 PM, you wrote: pete Now that i'm sure it is running, I will configure declude in the next few minutes Long sessions time is normal in our cas as we have to go thru 2 satellite conexions would that be a problem ? It is possible that some sessions will fail from time to time when congestion is high, but it should not be a problem overall. The system is designed to survive outages without causing trouble. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XCI Error!: snf_EngineHandler::MaxEvals
Hello Pi-Web, Friday, November 2, 2007, 2:01:30 PM, you wrote: snip/ 31st oct. spam level raised, SNF was not validating the mails, the snfclient.exe.err shows lines like: C:\Program Files\Merak\temp\2007110215013101AF.tmp: Could Not Connect! Could not connect indicates (most likely) that the SNFServer was down. Any time the client produces a .err it is unusual. Normal errors are reported to the SNFengine's log file(s). We restrated the SNFserver (running as a service) and scans run smoothly until today (2nd nov.), where same issue happen: Could Not Connect!. No errors in between. Something is knocking the server offline. The log also show: (first line). C:\Program Files\Merak\temp\200711021416181623.tmp: XCI Error!: snf_EngineHandler::MaxEvals Think this MaxEvals is what cause the error. Is it due to the engine getting to many mails to evaluate? No. MaxEvals is a condition that is theoretically possible but extremely rare. As a message is scanned, little creatures called evaluators are created and re-used during the scan to identify any patterns that might exist in the message. The scan depth metric indicates the peak number of evaluators that were alive during the scan. Normally this number is between 60 and 150 though it changes all the time. In order to detect possible rulebase corruption there is a hard-coded limit to the number of evaluators that are allowed to live for a particular scan. It is possible that this number needs to be adjusted. That hasn't happened in a while - but since you're not getting any other errors (that we know of) that's the most likely scenario. The number of evaluators that are alive at one time for a particular scan depends on the active rules in the rulebase and the data in the message. The number is almost impossible to predict though it does (and should) normally stay in a fairly restricted range. How do we avoid this? First, let's verify that there were no other errors. Please look in your snf log files and check for any e/ elements. These will describe any other errors that occurred. If we find no other errors then I will make an adjustment to the maximum evals metric and we will go from there. While you are in your logs -- look a the p/ (performance) elements and get an idea what the scan depth is typically. That will help us compare your system to others and to determine what the new limit should be. Originally the scan depth limit was designed to help detect possible corruption or unexpected conditions in the scanning engine. It's been there since the first version. It's a kind of sanity check -- Most likely it just needs to be adjusted since spam has changed so much over the years. In the early days scan depths were consistently well below 100 -- even in the 40-60 range. These days there are more abstracts in the rulebase so more creatures are required to get a comprehensive idea of what is in each message. Another thing I will look at is that this exception should be handled gracefully. I will look into this -- it may be that we want the SNFServer to fail under these conditions because it is a clue to something being out of adjustment -- In this case, probably just the limit setting. In the mean time, if you automatically restart your SNFServer after a failure it should be safe and will pick up any waiting clients before they fail in most cases. We also see this error, but this might be while restarting the service: C:\Program Files\Merak\temp\2007103119380319A0.tmp: XCI Error!: snf_EngineHandler::FileError Most likely this is a request coming from an snfclient after the message file has already been handled and moved out of the temp folder. The FileError exception indicates that the SNFServer could not open and/or read the file. Normally this wouldn't appear in a .err file - it would appear in the normal logs. If this error was in a snfclient .err file then I may need to look at the client code again. Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: XCI Error!: snf_EngineHandler::MaxEvals
Hello Pi-Web, Friday, November 2, 2007, 5:04:47 PM, you wrote: The SNFserver.exe is present on the task list, so it will not automatic restart. ERROR in todays log: snip/ e u='20071102100835' context='SNF_NETWORK' code='99' text='ERROR_SYNC_FAILED'/ e u='20071102100956' context='SNF_NETWORK' code='99' text='ERROR_SYNC_FAILED'/ e u='20071102113453' context='SNF_NETWORK' code='99' text='ERROR_SYNC_FAILED'/ snip/ The ERROR_SYNC_FAILED errors are caused by network congestion between your systems and ours. Ping times are well above 120ms at the moment, for example. I note that there are periods of time when there is no trouble making the connection and your current telemetry also looks good so we can ignore that error for the time being. Your latest SYNC took only 290ms and occurred with no retries. Here is my telemetry on that: session-data time=290 standby=15 cycles=6 sent=424 received=1930 comms=Completed Ok success=true s u='20071102131619' m='C:\Program Files\Merak\temp\200711021416181623.tmp' code='72' error='ERROR_MAX_EVALS'/ The above scan s/ failed due to too many evaluators. s u='20071102174238' m='c:\...tmp' code='69' error='ERROR_MSG_FILE'/ ... cut a lot... s u='20071102174358' m='c:\...tmp' code='69' error='ERROR_MSG_FILE'/ ERROR_MSG_FILE indicates that the SNFServer program was unable to open or read the file. Something must have removed it before it could be processed. This error is unrelated to the SYNC and MAX_EVALS errors. I also noted that the SYNC errors do not seem to coincide closely with the MSG_FILE errors. For now we will need to treat all three as separate cases. On some systems we have found cases where the system becomes so busy that scans take too long and are then cancelled before they are complete. This condition might account for some of the MSG_FILE errors. Is there a timeout on the mechanism that calls the SNFClient? If there is, then we might be able to mitigate the ERROR_MSG_FILE condition by extending that timeout. Considering the SYNC errors -- they are not critical because the SNF engine will tolerate them provided it is able to make a connection most of the time. When a connection is made and the SYNC session is successful then all of the data from previously unsuccessful sessions is transferred in the process. p before/after max_evals: - What does that tell me? p s='0' t='411' l='34867' d='72'/ The p/ element always belongs to an s/ element. An s/ element represents a single message scan. The p/ element describes the system's performance during that scan. In the case of the p/ element above, it took 0ms to setup the scan (read the file etc) and then took 411ms to perform the scan. This would usually indicate that your system is CPU bound. Normally an SNF scan will take a very short time. This one took almost half a second. The l indicates the length of the message scan in bytes and the d indicates the scan depth. That is, the maximum number of evaluators that were alive during the scan. ... s u='20071102131619' m='C:\Program Files\Merak\temp\200711021416181623.tmp' code='72' error='ERROR_MAX_EVALS'/ ... p s='0' t='80' l='4140' d='93'/ The p/ element here does not belong to the s/ element. It belongs to a different scan. Once the s/ element closes (with /s) anything after that point belongs to a different event. --- I don't have any other reports of MAX_EVAL errors. That doesn't mean that they are not out there, but it does mean that they are not usually a problem for other folks. I'm not sure what can be causing your SNFServer to crash -- it should not be MAX_EVAL errors. They are handled safely by the code according to what I've seen so far in my search. None the less, I will be increasing the max eval setting in the next release and I will push it out sooner rather than later. Since you have reported this problem I won't wait for the other features before pushing out beta 1.6. If I can get to it tonight I will. In the mean time, do you have any idea what might be causing your CPU to be so heavily loaded that your SNF scans are taking 400+ milliseconds? Do you have many p/ records that show high t values like that? (I do see the 80 that you reported above. That's on the high end of normal). Your telemetry shows about 10 msg/minute on average, 90% capture. This seems a low number for such high scan times. In contrast, I have a generic single CPU server that is currently showing 400-500 msg/minute w/ times in the 20-30ms range consistently. Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Beta
Hello John, Wednesday, October 17, 2007, 1:41:18 AM, you wrote: Our SYNC server software rejects connections by default. If an SNF node follows the expected connection protocols and authenticates properly and consistently then it will be allowed to communicate with the system. If it fails to do any of these things or looks suspicious in any way then it will be automatically black listed for increasingly extended periods and potentially null routed by our fire-walls. The security mechanisms are fully automatic and constantly monitored. If something goes wrong on my server, either by a mistake I make in a configuration file or a bug or whatever, and my server in connecting to the SYNC server should be rejected and subsequently black listed, is there a notification that takes place that some one will review to see if that sniffer license is otherwise valid and otherwise no known problems are seen so that I will then be notified saying hey there is a problem contact us so that the problem can be resolved? Yes. The system is completely automated and reliable. There is nothing to be concerned about. Quite simply, nothing can go wrong, go wrong, go wrong... go.. Seriously though-- In order to be black-listed by our system you would have to be abusing the SNF software or using some alternative software to attempt to gain access or deny access to the SYNC servers. Otherwise the most you could do would be to loose contact for some time. That said, if any system does something to become black-listed then you can be sure it will have our attention. It is basically impossible for you to cause a properly functioning SNF node to become black-listed by altering the configuration file. It is far more likely that your SNF node would simply fail to connect. Chances are that if you were making an adjustment that could cause this you would also be watching to make sure that things were working correctly when you finished. In case you did cause the system to lose it's connection with us, the system is designed so that SNF nodes will remain reliable and effective for extended periods even if they are unable to contact the SYNC server. It is also designed to recover gracefully when the problem is corrected. The GBUdb system is highly effective even when it does not share it's information with the other SNF nodes. Each GBUdb node learns first about it's local traffic. As long as your SNF rulebase file is up to date - or even close to being up to date, your system is likely to be very effective at filtering spam. If your SNF/GBUdb node becomes detached from the main system for an extended period, it will degrade in it's performance. Once the problem is corrected it should recover in a very short time. In the event we detect any IPs being black listed or acting suspiciously we will be watching closely so that we can analyze any potential threats and take appropriate actions. If we can identify a customer involved in such a case we will contact them to investigate and correct the problem. Locally, your status reports indicate when the last sync event occurred. This is one of the ways you can check the status of your system. Consider this example from recent telemetry: timers run started=20070928174736 elapsed=1620714/ sync latest=20071017115919 elapsed=11/ save latest=20071017111334 elapsed=2756/ condense latest=20071017081746 elapsed=13304/ /timers You can see when the last sync event occurred (about 11 seconds ago in this case): sync latest=20071017115919 elapsed=11/ We plan to encourage the development of third party tools for monitoring and analyzing SNF system data. In addition we plan to build monitoring and analysis services of our own to include features that will notify system administrators when something doesn't look quite right. If you (anyone) develop something nice for displaying and/or monitoring SNF status data then please share it with the SNF community. In the mean time - we have done extensive testing and monitoring throughout the development process. High availability is (has always been) a design requirement and we're confident SNF can deliver that. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Beta
Hello Darrell, Tuesday, October 16, 2007, 9:26:47 PM, you wrote: Pete, Can you cover how the communication for the GBUdb system works? Sure. (And documentation is coming along with a web site redesign, so there will be plenty of additional detail on all things new SNF arriving over the next few weeks). Who does it exchange information with and how? Does it need special ports open? I'm going to assume this question is at least in part about security issues so I will frame my response from that perspective: Every minute or so (adjusted dynamically by the system) each new SNF node contacts one of our SYNC servers. The connection is made to port 25 by default. Since most MTAs will regularly make these kinds of connections no special ports will need to be open. If your MTAs are not allowed make outbound connections to port 25 for some reason then you have the option to make the connection to port 80. Our SYNC server software rejects connections by default. If an SNF node follows the expected connection protocols and authenticates properly and consistently then it will be allowed to communicate with the system. If it fails to do any of these things or looks suspicious in any way then it will be automatically black listed for increasingly extended periods and potentially null routed by our fire-walls. The security mechanisms are fully automatic and constantly monitored. The authentication protocol used to identify properly licensed SNF nodes is described in the file AuthenticationProtocol.swf. This file is included in the beta distributions and is also visible in the page linked below. At present there is no mechanism for connections to be made into SNF nodes -- only from SNF nodes to the SYNC servers. Also, there is no mechanism that allows the SYNC server (or any of our systems) to manipulate the SNF nodes except by the protocols described in the GBUdb documentation and by reporting update availability, tuning data, etc. This link helps explain how these interactions work: http://kb.armresearch.com/index.php?title=Message_Sniffer.TechnicalDetails.GBUdb The SYNC system is separate from the rulebase delivery system. All of the data transmitted and received by the SNF nodes is in plain text or base64 encoded. The format of the data is XML. With the exception of GBUdb traffic, the telemetry transmitted to us is available to you directly in the .status. reports made by the SNF engine. Status reports can be found in the same directory as your SNF log files. You can use these XML based posts to create your own real-time monitoring systems. In addition to the GBUdb functions, the telemetry eliminates the need to send in log files by providing near real-time pattern matching statistics; supports virtual spamtraps and other collaborative learning systems; and provides performance analysis, error detection / correction, and system tuning metrics. One other security note -- the virtual spamtrap system can be turned off easily if you wish. Normally the virtual spamtrap system will send us a random sampling of messages that come from the worst known IPs when those messages do not match known pattern rules. In most systems, these are messages that would normally be discarded. Samples are infrequent by design so they should not account for any appreciable bandwidth. Similarly, the GBUdb protocol is designed to share information sparsely so that no appreciable bandwidth or CPU capacity is required. Please let me know if I missed the mark on your questions. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Bad Rule: 1604021
Hello Sniffer Folks, This is an alert about a potentially bad rule 1604021. The rule was an abstract pattern for some of today's image spam. Indications are that the final coding was too broad. The rule was in place for approximately 5 hours ending about 30 minutes ago. Some differences in timing are inevitable since all rulebases are compiled individually. If you have the ability to release and rescan from quarantine based on SNF rule IDs then we recommend executing that process against this rule id: 1604021. Hope this helps, Thanks, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Beta
Hello Phillip, Monday, October 15, 2007, 6:39:22 PM, you wrote: I just tried installing the beta and it appeared to work too well as it was sending all the mail to the spam box. I am not sure if this was due to the bad rule that was just replaced or something I was doing wrong. The bad rule turned out not to be as bad as we suspected -- so I don't think that was the problem. Also the bad rule has been gone for a while so you're not likely to have it in your rulebase. I am currently running in the persistent mode. I set the xml file to point to the correct paths in the 4 places and started up the sniffer server. I then changed the bat file that the agent calls to run snifclient.exe file licenseID %1 is this the correct format? I am still using an old vopmail 5 mail server. The correct way to call SNFClient.exe is either just like you call the older SNF (compatibility mode): SNFClient.exe authenticationxx scanthis.msg I'm guessing from your notes and memory SNFClient.exe authenticationxx %1 --- The other way you can call SNFClient.exe to perform a scan is simply: SNFClient.exe scantthis.msg --- If you run SNFClient.exe without any parameters it will remind you how it can be used. If you called it with: SNFClient.exe file licenseid %1 Then that would be too many parameters so you would probably get an error and possibly a nonzero result. I may simply be misinterpreting your notes - but if you did something like this in your script (batch file) then it's possible the result would be to hold every message. At the moment I switched back to the old version of sniffer after going through 600 emails by hand and sorting out spam and real mail and manually placing them in the correct mailboxes, that was fun. Sorry about that. I tried to be as explicit as possible in the readme files and the help system in the program itself (running SNFClient.exe on the command line by itself should list all of the ways the client can be called.) The new SNF doesn't have a persistent and non-persistent mode like the old version. Instead, it is strictly a client/server model. Run the SNFServer.exe program as described in the read me and then leave it running. Then, with your message processing script you can call the SNFClient.exe program in place of the old SNF program without any modifications if that is easier -- -the SNFClient.exe will accept the same parameters as the old SNF program without a problem. This makes it relatively easy to switch back (as you did). If you start out running the SNFServer from the command line then it's display will help you to know when things are working correctly -- you will be able to see when messages start going through and you should quickly get an idea of what looks correct. Once you're confident in that setup then you can run the SNFServer using srvany or firedaemon or your other favorite utility that runs programs as a service. Hope this helps, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
