RE: [sniffer] Final beta (b2) for snfrv2r3
Hmmm, log file from sniffer shows significant increase in performance (up to 50% faster, see below). However, according to my own logs, the total time that sniffer takes is way longer. During non-persistent operation about 300 ms on top of what sniffer logs, which could be because of loading times of sniffer itself. When sniffer is persistent, 'loading' time is about 1.5 seconds. My conclusion from this, is that when sniffer is running persistent, cpu usage and rulebase loading times are decreased but total execution time seems to have tripled from about 550 ms to about 1650 ms. To calculate the total execution time, I store system time in ms just before and after ShellExecuteEx() and calculate the difference. That seems like an honest and reliable way to determine execution time for sniffer. sniffer log: h0t861s420040407080330md5581512.msg26532Clean000221432h0t861s420040407080340md5581513.msg26516Clean000150335h0t861s420040407080356md5581514.msg28278Clean0001366440h0t861s420040407080408md5581515.msg265110Clean0002692944h0t861s420040407080412md5581516.msg28132Clean000219935h0t861s420040407080422md5581517.msg28116Final33612540252040h0t861s420040407080426md5581518.msg25031Clean000263635h0t861s420040407080431md5581519.msg26631Clean000591341h0t861s420040407080436md5581520.msg18846Final105667520352241h0t861s420040407080446md5581521.msg10932Clean000215236h0t861s420040407080454md5581522.msg12547Clean000408335h0t861s420040407080506md5581523.msg18747Clean000520532h0t861s420040407080514md5581524.msg18847Clean000563234h0t861s420040407080524md5581525.msg188109Clean0002476343h0t861s420040407080531md5581526.msg18847Final105667520274239h0t861s420040407080538md5581527.msg18816Clean000196735h0t861s420040407080550md5581528.msg187125Clean0002471850h0t861s420040407080557md5581529.msg18732Clean000323634h0t861s420040407080607md5581530.msg12531Clean000291832h0t861s420040407080620md5581531.msg18732Final105073500237444h0t861s420040407080632md5581532.msg18815Clean000361133h0t861s420040407080638md5581533.msg125125Clean0002756845h0t861s420040407080650md5581534.msg18778Clean0001615533 I'm really puzzled about the cause for the extra delays. Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOSSmallOffice Solutions /Reject / Wannepad 27 - 1066 HW - Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy- Installation- Maintenance Network Security -Internet - E-mail SoftwareDevelopment - Project Management -- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeilSent: woensdag 7 april 2004 11:21To: [EMAIL PROTECTED]Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 What does the sniffer log show during this time?_MAt 04:48 AM 4/7/2004, you wrote: Pete,Despite my suggestions with less polling time, I can't seem to get the persistent version to speed up my message processing. I've copied part of my custom log file below. Bold numbers are the amount of ms it takes to execute sniffer (timed by an external program that executes it). Persistent sniffer was turned ON on the blue lines. I've set max polling time to 50ms for this test. However, scanning takes more than a second longer...0,"2004-04-07 10:03:31",md5581512.msg,672,546,78,0,2223,0,0,3,10,"2004-04-07 10:03:40",md5581513.msg,657,531,93,0,1490,0,0,3,10,"2004-04-07 10:03:57",md5581514.msg,734,594,93,0,14601,0,0,3,10,"2004-04-07 10:04:09",md5581515.msg,797,624,93,0,29398,0,0,3,10,"2004-04-07 10:04:13",md5581516.msg,686,562,93,0,42408,2,0,3,10,"2004-04-07 10:04:22",md5581517.msg,749,547,93,0,2611,1,0,3,10,"2004-04-07 10:04:26",md5581518.msg,656,532,93,0,43402,2,0,3,10,"2004-04-07 10:04:32",md5581519.msg,671,547,93,0,6022,0,0,3,10,"2004-04-07 10:04:37",md5581520.msg,1905,1672,92,0,3564,1,0,3,10,"2004-04-07 10:04:47",md5581521.msg,1811,1688,93,0,2152,0,0,3,10,"2004-04-07 10:04:55",md5581522.msg,1811,1688,78,0,4122,0,0,3,10,"2004-04-07 10:05:05",md5581523.msg,1843,1671,93,0,5250,0,0,3,10,"2004-04-07 10:05:13",md5581524.msg,1811,1688,78,0,5677,0,0,3,10,"2004-04-07 10:05:21",md5581525.msg,1797,1671,93,0,273387,0,0,3,10,"2004-04-07 10:05:30",md5581526.msg,1891,1671,93,0,2760,1,0,3,10,"2004-04-07 10:05:37",md5581527.msg,1811,1672,93,0,36384,2,0,3,10,"2004-04-07 10:05:49",md5581528.msg,1796,1656,93,0,27065,0,0,3,10,"2004-04-07 10:05:56",md5581529.msg,1812,1686,79,0,3554,2,0,3,10,"2004-04-07 10:06:06",md5581530.msg,1843,1671,78,0,44939,2,0,3,10,"2004-04-07 10:06:19",md5581531.msg,1874,1655,94,0,2363,1,0,3,10,"2004-04-07 10:06:31",md5581532.msg,1811,1671,94,0,3670,0,0,3,10,"2004-04-07
Re: [sniffer] Final beta (b2) for snfrv2r3
Pete, I haven't been following this thread closely but latest generation SCSI drives can be below 4 ms seek times as rated by their manufacturers. FYI, I haven't seen any issues with the persistent Sniffer beta run as a resource kit service besides some expected brief delays according to the way that processes when traffic is less heavy. Matt Pete McNeil wrote: I must be getting punchy... but this just occurred to me... Anybody else remember when a high performance hard drive had a seek time just under 30ms ?? _M At 06:01 PM 4/7/2004, you wrote: If thats all that happens during the first setup timer than you do have some performance issue on a production machine. My production mail server is not too beefy and does somewhere around 120k+ a day. Heres a snipplet from my logs (with persistent sniffer) for comparison fde2jqoe 20040407041105 D7f587132019a8525.SMD 0 31 Final fde2jqoe 20040407041105 D7f577130019a80fe.SMD 0 15 Final fde2jqoe 20040407041106 D7f5973740202893b.SMD 0 16 Clean fde2jqoe 20040407041109 D7f58737302028553.SMD 0 16 Final fde2jqoe 20040407041109 D7f53712e019a73bf.SMD 0 15 Final fde2jqoe 20040407041120 D7f6490fe0072b647.SMD 0 0 Final fde2jqoe 20040407041120 D7f6590ff0072b721.SMD 15 0 Final fde2jqoe 20040407041120 D7f659172b84a.SMD 0 32 Final fde2jqoe 20040407041120 D7f6591010072ba3e.SMD 0 15 Final fde2jqoe 20040407041120 D7f6691020072bbe4.SMD 0 31 Final fde2jqoe 20040407041121 D7f6691030072bdc9.SMD 0 16 Final fde2jqoe 20040407041123 D7f6991050072c932.SMD 0 16 Clean fde2jqoe 20040407041123 D7f6a91060072cbf2.SMD 0 15 Final fde2jqoe 20040407041123 D7f6a73760202cc6f.SMD 0 16 Final From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Pete McNeil Sent: Wednesday, April 07, 2004 4:36 PM To: [EMAIL PROTECTED] Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 At 04:06 PM 4/7/2004, you wrote: So, making sure I'm following your analysis: I'm looking at my log file and I'm seeing lines similar to snf2beta 20040407020014 D60a4134.SMD 181 30 Match 101576 58 20 38 68 And that 181 figure seems to hold pretty stable. 181 is substantially lower than the values I was seeing prior to the current beta [and I have a production machine similar in content and power to your test machine], but I'm seeing that you achieve numbers 2-6 times faster than I am. Yes... that seems about right. When a persistent server is running the rulebase is almost never reloaded. Only two significant things happen during the setup time as measured by Sniffer: 1) Loading the rulebase, 2) locating a job to process (directory scan + locking). The drop seems to indicate that the rulebase reload has stopped as it should. That only leaves the directory scan and a couple of rename/create operations. _M -- = MailPure custom filters for Declude JunkMail Pro. http://www.mailpure.com/software/ =
Re: [sniffer] Final beta (b2) for snfrv2r3
What is the best and proper way to setup Persistent mode on a windows 2000 computer and run as a service. Fred - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, April 07, 2004 8:30 PM Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Pre-persistant sniffer my times sometimes got high, but never beyond 3 digits. While running the persistant beta, about half of my times are in the thousands. The machine also seems to be far more prone to bogging down under a mail load. This is on a P2/800mhz 1g ram machine. Pre-beta 20040304211333 d9bec001201263026.smd 312 0 Match 89089 20040304211333 d9bec001201263026.smd 312 0 Final 89089 Persistant sniffer 20040407042039 d819316c90154969c.smd 100032 Match 48754 20040407042039 d819316c90154969c.smd 100032 Match 94972 20040407042039 d819316c90154969c.smd 100032 Final 94972 This doesn't make any sense. I have no good theory for this. I am unable to create any scenario where using the persistent engine degrades performance. In all of my tests on three separate platforms the persistent engine produces a significant improvement - even under unreasonably harsh conditions. Aside from rebooting the machine and not starting sniffer in persistant mode, how do I stop sniffer from running persistantly? Sniffer is adaptive. You can turn the persistent instance on and off at will. Simply stop the service - a reboot is not needed. If the persistent instance is turned off then the remaining instances will organize themselves in the usual way. _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Final beta (b2) for snfrv2r3
Sniffer is adaptive. You can turn the persistent instance on and off at will. Simply stop the service - a reboot is not needed. If the persistent instance is turned off then the remaining instances will organize themselves in the usual way. I don't have it running as a service, I started the persistant instance via command line. That's fine... I nearly forgot something important anyway. sniffer.exe stop - will stop the persistent server by sending it a message file. Run 'sniffer.exe stop' at the command line and your persistent instance will exit cleanly on it's own. [ replace sniffer.exe with the name of your executable of course ] If you are running it from the command line then it will stop before the command returns. To restart it simply run your persistent command line again. For those running as a service If you are running it as a service, the persistent instance will stop - possibly under the service stub. If this is the case (as with RunExeSvc) then you will need to stop and start the service when you are ready to bring it back. _M PS: If you do just kill the persistent instance it will leave it's .SVR file behind and will abandon the job it is doing. While this is unkind, it will not be a problem - the normal peer-server instances will quickly clean out the stranded .SVR file and the abandoned job will be handled by the client instance when it gets tired of waiting. This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Final beta (b2) for snfrv2r3
My findings are that persistent is offering great benefits, havnt tried an excessively harsh test yet, but i'm about to do that. Just ran sniffer in both persistent and non-persistent modes with over 1,000 mesages in the overflow and MaxQueProc at 50. This pegs out my CPU between 90% 100% for the duration of delivery. Screenshots sniffer log snipplets at http://staff.netsmith.net/sniffer/Extreme_Load/ I wont waste the mailing lists bandwith for the attachments for those who dont want them. I dont see an obvious different when the system is under heavy load, at least not by skimming the log files. Could do some math on overall performance statistics I guess... # of messages processed in same timeframe, average times, etc. winmail.dat
RE: [sniffer] Final beta (b2) for snfrv2r3
At 09:11 PM 4/7/04 -0400, Pete McNeil wrote: sniffer.exe stop - will stop the persistent server by sending it a message file. Run 'sniffer.exe stop' at the command line and your persistent instance will exit cleanly on it's own. [ replace sniffer.exe with the name of your executable of course ] Tried the above and got an error message. Tried: sniffer.exe xxauthenticationxx stop and it paused a few seconds and returned to command prompt, so I'm guessing that it stopped. -- Kirk Mitchell-General Manager[EMAIL PROTECTED] Keystone Connect Unlock Your World Altoona, PA 814-941-5000 http://www.keyconn.net This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] Final beta (b2) for snfrv2r3
This worked great. Thanks. - Original Message - From: Pete McNeil [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Wednesday, April 07, 2004 8:46 PM Subject: Re: [sniffer] Final beta (b2) for snfrv2r3 At 08:36 PM 4/7/2004, you wrote: What is the best and proper way to setup Persistent mode on a windows 2000 computer and run as a service. Fred * Make a backup copy of your current executable (just in case). * Rename the 2-3b2 executable for your license and replace your current executable. At this point your system will be running in the normal way. Next, you can use a third party utility or the windows toolkit to run your sniffer executable as a service with the persistent switch. Here are two links from previous discussions to help. I prefer RunExeSvc because it seems simpler. http://www.mail-archive.com/[EMAIL PROTECTED]/msg00165.html Here it is done with the toolkit... http://www.mail-archive.com/[EMAIL PROTECTED]/msg00169.html Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Final beta (b2) for snfrv2r3
Tried the above and got an error message. Tried: sniffer.exe xxauthenticationxx stop and it paused a few seconds and returned to command prompt, so I'm guessing that it stopped. That doesn't sound quite right. In the distribution there are some .CMD files that show examples of the commands: stop - Ends the persistent server reload - Reloads the rulebase config file data rotate - Moves the current log file to sniffer.log.mmddhhmmss Note that all commands and configuration options are case sensitive. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
RE: [sniffer] Final beta (b2) for snfrv2r3
Since you're up, sorry to ask, where's the beta? Didn't save the e-mail. Rob -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, April 07, 2004 9:23 PM To: [EMAIL PROTECTED] Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Tried the above and got an error message. Tried: sniffer.exe xxauthenticationxx stop and it paused a few seconds and returned to command prompt, so I'm guessing that it stopped. That doesn't sound quite right. In the distribution there are some .CMD files that show examples of the commands: stop - Ends the persistent server reload - Reloads the rulebase config file data rotate - Moves the current log file to sniffer.log.mmddhhmmss Note that all commands and configuration options are case sensitive. Hope this helps, _M This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html --- [This E-mail scanned for viruses by Declude Virus] --- [This E-mail scanned for viruses by Declude Virus] This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
