Right this second I'm not in a bind, but I've been growing business fast and I need to reserve enough processing power to scale up on the same box for the next couple of months, and then for a period of time, this might remain my backup MX. I throw a ton of tests at my setup, and while I do try to optimize within the boundaries, this is one place that has potential. I'm pretty certain that it will end up not calling Sniffer in excess of 50% of the time because the RBL's will have already reached a comfortable weight for me to stop processing on, and although Sniffer is very lean, this should help with the peaks.
FYI, here are two screen captures of Task Manager showing what is going on with Sniffer and without Sniffer from a few moments ago.
http://www.mailpure.com/with_sniffer.gif http://www.mailpure.com/without_sniffer.gif
It's not bad, but during business rush hours I hit 80% to 100% every few seconds, and I need to protect my response times for my Web server for the time being. Also keep in mind that legitimate traffic causes the biggest hits because of large file attachments, running every filter in Declude, and often having in excess of 32 KB of content to scan. This is a dual PIII 1 GHz machine with RAID 5. Before going hog wild with Declude's custom filters and a second virus scanner, it would bounce around between 0% and 5%. Most of the processing used is the result of all of my custom filters in Declude.
I'm interested in nailing-up Sniffer as you suggested, but I would prefer to be a beta tester as opposed to an alpha tester for obvious reasons. I'll trust your judgment on this, just tell me when. I think I watch my system close enough and have redundancies now that would protect me from a hiccup in beta software. I'm going to give the handler script a shot though because it looks simple and should at least save some loading times. No need to rush on my account, I'm just trying to stay one step ahead of things. If this works well enough, I might even code up a handler to disable virus checking on extra large files of certain types (not very common, but they produce long and large spikes).
Thanks,
Matt
Pete McNeil wrote:
Matt,
I'm sorry you're in such a bind. We all get there from time to time.
First, as far as I know you should be able to call a script from declude and pass it parameters. You may need to do this by calling cmd.exe - but you need to know that this isn't likely to save you processing on balance since the majority of Message Sniffer instances are going to load as client instances and will normally use very few resources - probably fewer than any scripting engine.
None the less, if you want to try it you can find details on the command line parameters for Message Sniffer at this link:
http://www.sortmonster.com/MessageSniffer/Help/TechnicalDetailsHelp.html#CmdLine
One thing you might consider in the short term is to reduce the strength of your Message Sniffer rulebase. This is done by increasing the rule strength threshold. A small amount of additional spam will get through - but the rulebase will be much smaller and therefore faster to load. (The majority of Message Sniffer's resource use is in loading the rulebase - not in scanning the messages. Cellular peer-server technology helps mitigate this by allowing one instance to process messages for many before the rulebase is reloaded.)
If you are looking for an immediate fix then I recommend this approach. Let me know off list (support@) if this is what you would like to do.
Over the next couple of days (possibly beginning tonight) I will be working on a "Persistent" option for Message Sniffer which will allow you to "nail up" a server instance. In theory this shouldn't be hard to do but the implications can be tricky. If you are willing to work with potentially buggy code then I'll be happy to share early betas with you.
Hope this helps, _M
At 03:05 PM 3/8/2004, you wrote:
Pete,
Forgive me for pressing this, but I have a short-term need to reduce the processor utilization on my server because it also serves Web sites and I need to stay away from pegging my processor until I separate the spam blocking to a new machine. This is one of the things that I am looking at.
I'm thinking that unless Scott was willing to work skip functionality into Declude in the short-term (and I don't assume his schedule revolves around my needs), it might be possible to write a simple handler script that calls Sniffer only if the %WEIGHT% variable is in a certain range.
It seems that the only tricky part for a novice programmer like myself would be capturing the result code and passing it back to Declude, but I could find someone to help me with that. If you know of any other potential pitfalls please let me know (like message file/data handling), and if you could tell me what the full command line arguments were that are passed by Declude to Sniffer, and the format of the result code that is passed back, that would save me some research time (I'm thinking there is a file name in there unless the data is streamed).
Also, if this was done, would it be possible to use such a script in a way that would allow Sniffer to run pre-loaded with the rulebase as a service on Windows?
I'll share whatever I come up with if going the script route is necessary.
Thanks,
Matt
Matt wrote:
Pete,
That is of course an option. He did make this variable available for such uses, but that's clearly not the only way to skin this cat. I think the trick here is determining which external tests to run according to which skip weight setting. Different external tests would have potentially different trigger levels instead of one main trigger. Certainly Declude could provide this, but it would require a setting for each individual test call. I would imagine that he could add new columns to the external test call like so:
----- Current Format ----- SNIFFER-GENERAL external 063 "C:\IMail\Declude\Sniffer\programname.exe mycode" 6 0
----- Potential New Format ----- SNIFFER-GENERAL external 063 "C:\IMail\Declude\Sniffer\programname.exe mycode" 6 0 25 -10
The first extra column would be the skip-if-equal-to-or-above-weight setting and the second extra column would be the skip-if-equal-to-or-below-weight
setting.
Scott seems extra busy right now, but he obviously monitors this list so I'll leave it here for now. Declude has of course made huge strides in terms of efficiency. It would be nice if possible for Sniffer + Declude to make use of the rulebase loading method mentioned earlier as well.
Thanks,
Matt
Madscientist wrote:
I think the best place to handle this would be within Declude since Declude has all of the information and has control over which tests get called.
I have seen some discussions on not executing certain tests based on weights that are reached.
Probably the most reasonable tests to modulate in this way would be external tests - but Scott's the best one to answer that question.
Hope this helps, _M
PS: If you are working to mitigate a load problem that may be handled shortly. We will be working on a feature to "nail up" a Message Sniffer instance so that all other instances will run as clients (thus not loading the rulebase). This should significantly reduce system loads and may make complex modulation mechanisms unnecessary.
At 01:01 PM 3/5/2004, you wrote:
Pete,
Would it be possible to add the ability to tell Sniffer not to run when Declude passes it a weight? I understand that we tend to weight differently, so this would require two switches to pull off, but it would help with efficiency. The command line in a Declude config would be something like the following:
"C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25"
Declude will replace %WEIGHT% with the weight before calling Sniffer, and the next entry, 25, would tell Sniffer to not run if the the current weight was equal to 25 or above. Some people also do some whitelisting with external programs like Sniffer, in which case it might be a good idea to add a weight under which you would also skip processing, i.e.
"C:\IMail\Declude\Sniffer\aaabbbccc.exe your-code %WEIGHT% 25 -10"
On my system, the addition of the weight variable would cause Sniffer to not be run on between 50% and 75% of the E-mail depending on the day of the week, so this could be a very big deal.
Thanks,
Matt
-- ===================================================== MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =====================================================
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
-- ===================================================== MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =====================================================
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
-- ===================================================== MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =====================================================
This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
