RE: [sniffer] Possible blip?
At 06:38 PM 5/20/2004, you wrote: Crew, I reposrted this speed issue before, but despite very intensive debugging and testing, we have not found an external cause (meaning: not sniffer) for the following: But, now comes the big mystery: when persistent mode is ON, it takes a lot more time to execute (while max polling is only 50ms!) 0,"2004-05-20 23:48:41",md5845373.msg,827,812,15,0,0,0,3607,1 0,"2004-05-20 23:48:52",md5845374.msg,842,812,0,0,0,0,3833,1 0,"2004-05-20 23:51:15",md5845375.msg,936,874,0,0,0,0,9560,1 0,"2004-05-20 23:51:35",md5845376.msg,889,859,15,0,0,0,26387,0 0,"2004-05-20 23:53:21",md5845377.msg,937,922,0,15,0,15,1922,0 Which averages at 850 ms! While I expected 45 + 25 ms (to compensate for average waiting time) = 70 ms! Pete, could you please check why this is happening (particularly in code OUTSIDE what's measured and logged)? I you can't find anything, I'll ask my collegue to come up with a timing program, which I would like to release on this list so other ppl can check how long it really takes to execute sniffer (measured from 'the outside'). As I recall when this last came up the solution turned out to be an on-access virus scanner that was introducing the extra delays. Turning off and/or adjusting the on-access virus scanner solved the timing problem. The theory goes that the MDaemon CF is single threaded so when Sniffer runs normally there will only be one instance at once, and as a result each instance loads it's own rulebase and scans it's own message... this results in two file reads and no write operations. With the persistent sniffer instance running as a server, there are several additional file creation, write, and access events per message. Each causes the on-demand scnner to intervene and thereby introduce the additional timing delays. The "transparent" way on-access virus scanners interfere with file operations accounts for the odd placement of the additional time. ... as I said, in theory ... Hope this helps, _M
RE: [sniffer] Possible blip?
Crew, I reposrted this speed issue before, but despite very intensive debugging and testing, we have not found an external cause (meaning: not sniffer) for the following: When I use sniffer without the persisten flag, I get this log: h0t861s4 20040520214718 md5845369.msg 125 16 Clean 0 0 0 2844 40h0t861s4 20040520214718 md5845370.msg 110 15 Clean 0 0 0 2747 36h0t861s4 20040520214804 md5845371.msg 109 16 Match 109406 62 43 93 43h0t861s4 20040520214804 md5845371.msg 109 16 Match 115560 58 2286 2307 43h0t861s4 20040520214804 md5845371.msg 109 16 Final 115560 58 0 3580 43h0t861s4 20040520214825 md5845372.msg 110 15 Match 29048 52 2757 2788 46h0t861s4 20040520214825 md5845372.msg 110 15 Match 122523 52 2930 2942 46h0t861s4 20040520214825 md5845372.msg 110 15 Match 122017 52 2968 2977 46h0t861s4 20040520214825 md5845372.msg 110 15 Match 122016 52 3346 3355 46h0t861s4 20040520214825 md5845372.msg 110 15 Final 29048 52 0 5504 46 which looks good (total execution time about 125ms) When I have a persistent version running (max 50 ms polling time), I get: h0t861s4 20040520214841 md5845373.msg 0 16 Clean 0 0 0 3597 53h0t861s4 20040520214852 md5845374.msg 16 31 Match 119377 62 684 741 38h0t861s4 20040520214852 md5845374.msg 16 31 Final 119377 62 0 3810 38h0t861s4 20040520215115 md5845375.msg 0 31 Match 29081 63 2413 2432 44h0t861s4 20040520215115 md5845375.msg 0 31 Final 29081 63 0 9458 44h0t861s4 20040520215134 md5845376.msg 0 94 Clean 0 0 0 24370 42h0t861s4 20040520215320 md5845377.msg 47 15 Clean 0 0 0 1945 35 Which are very good exec times (average 45 ms). We have created our own program that does lots of spam checking for messages. At some point, it fires Sniffer. We log the time it takes for Sniffer to run, for statistical purposes. When sniffer is NOT persistent, I get the following log snippet (same messages as 1st sniffer log above, the second number after the .msg is the time it takes for sniffer to run): 0,"2004-05-20 23:47:18",md5845369.msg,172,157,0,15,15,0,43406,20,"2004-05-20 23:47:18",md5845370.msg,172,156,16,0,0,0,43309,20,"2004-05-20 23:48:04",md5845371.msg,188,172,0,15,0,15,3578,10,"2004-05-20 23:48:25",md5845372.msg,186,156,14,0,0,0,5572,1 Average time to run sniffer is 160 ms (sniffer said 125 ms). That means, sniffer can't report about 35 ms which is normal for application startup and shutdown (also the log is written _after_ the exec time calculation has been made, file operations also take time). But, now comes the big mystery: when persistent mode is ON, it takes a lot more time to execute (while max polling is only 50ms!) 0,"2004-05-20 23:48:41",md5845373.msg,827,812,15,0,0,0,3607,10,"2004-05-20 23:48:52",md5845374.msg,842,812,0,0,0,0,3833,10,"2004-05-20 23:51:15",md5845375.msg,936,874,0,0,0,0,9560,10,"2004-05-20 23:51:35",md5845376.msg,889,859,15,0,0,0,26387,00,"2004-05-20 23:53:21",md5845377.msg,937,922,0,15,0,15,1922,0 Which averages at 850 ms! While I expected 45 + 25 ms (to compensate for average waiting time) = 70 ms! Pete, could you please check why this is happening (particularly in code OUTSIDE what's measured and logged)? I you can't find anything, I'll ask my collegue to come up with a timing program, which I would like to release on this list so other ppl can check how long it really takes to execute sniffer (measured from 'the outside'). Regards, ing. Michiel Prins SOS Small Office Solutions / REJECT Wannepad 27 1066 HW Amsterdam tel. 020-4082627 fax. 020-4082628 [EMAIL PROTECTED] Spamvrije zakelijke e-mail? reject.nl! Consultancy - Installation - Maintenance Network Security - Project Management Software Development - Internet - E-mail
Re: [sniffer] Possible blip?
At 05:00 PM 5/19/2004, you wrote: I haven't yet upgraded to the most recent release, I'm still on the prior beta. I'll probably do that this evening. I tend to wait on upgrades until there has been enough time for bugs to surface unless I am already looking for a fix. I'm sure that the extra verification of the rulebase will help prevent the potential of problems, and I guess this has the possibility of being caused by a bit of corrupted data, though that's probably reaching. There were no substantive changes from the beta to the production version. Largely just a removal of monitoring code. Again, regardless if there was a blip, Sniffer still does a wonderful job of tagging lots and lots of E-mail, just not quite as much as the day before. Last night I was able to adjust the rule strength analysis window back to it's original settings. About 5 days of data were lost - but those days will be recovered quickly. Please let me know if this adjustment improved your conditions. I've noted that on a number of other lists there seem to be posts about a sudden increase in spam over the past few days. We are definitely seeing this also - approximately a 25% or more increase in new rule additions in the past 4 days: http://www.sortmonster.com/MessageSniffer/Performance/ChangeRates.jsp Specifically note from about 4 days ago... Days Ago Adjustments --- 0 356 1 508 2 391 3 410 4 410 5 326 6 309 7 371 8 292 9 347 10 309 ( 5-10 : 1954/6 -> 325.67, 0-5 : 2075/5 -> 415, 325.67/415 -> 78.47 ) Note that day 0 is not complete. So applying a "fudge factor" 78.4 _looks like_ 75%. Besides, 92% of statistics are made up on the spot anyway %^b I think a number of things are combined here... I just want to get a good handle on them and make sure we are doing the best we can. I've noted, Matt, that your rulebase tuning parameters are set at the defaults. If you would like to adjust these to be more aggressive then please let me know off list (support@). More aggressive settings will keep more rules active in your rulebase at lower strengths and will also allow new rules more time to gain strength before being evaluated. Respectively the current defaults are: Minimum Rule Strength: 1.0 Grace Period: 5 days. Adjusting these settings can significantly increase the size of your rulebase file. Best, _M
Re: [sniffer] Verizon fiber cut...
At 10:04 PM 5/19/2004, you wrote: Hello folks, At aproximately 1553 EDT connectivity at our business office was severed due to what is now known to be a cut fiber in the area. We have no ETA for restoration but we are told to expect "soon" to mean within 24 hours or possibly even before sunrise (we are hoping). All is well. Connectivity at our offices was restored at approximately 0356 EDT. We are "digging out" and should be completely up to speed in the next 2 hours. Thanks to everyone who offered their support!! Our hosting operations and rulebase generation functions were not effected during this outage. Thanks, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
