RE: [sniffer] Final beta (b2) for snfrv2r3
Pete, The speed problem has been found. McAfee Netshield 4.51 was making our server RIDICULOUSLY slow, despite the fact that we tried excluding the Sniffer folder and even disabling the service from the tray-icon. Upgrading to Virusscan Enterprise 7.x fixed our problem and our performance levels are in the regions that you mentioned. Thanks for thinking along! Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOSSmallOffice Solutions /Reject / Wannepad 27 - 1066 HW - Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy- Installation- Maintenance Network Security -Internet - E-mail SoftwareDevelopment - Project Management -- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michiel PrinsSent: donderdag 8 april 2004 21:11To: [EMAIL PROTECTED]Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Preliminary tests show there's no I/O problem but I'll do some additional benchmarking here and get back to you on this. Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOSSmallOffice Solutions /Reject / Wannepad 27 - 1066 HW - Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy- Installation- Maintenance Network Security -Internet - E-mail SoftwareDevelopment - Project Management -- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeilSent: woensdag 7 april 2004 17:38To: [EMAIL PROTECTED]Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Extraordinary...Compare with a snippet from our IMail/NT4 test platform (severely underpowered)...snf2beta 20040407140913 D0b86122.SMD 30 90 Final 75148 63 0 6891 68snf2beta 20040407140913 D0b8614e.SMD 90 140 Final 103691 57 0 8878 72snf2beta 20040407140914 D0b88122.SMD 40 141 Final 103689 57 0 9003 71snf2beta 20040407140915 D0b880b6.SMD 90 20 Final 106244 52 0 817 65snf2beta 20040407140916 D0b8a0de.SMD 40 210 Final 104044 52 0 8779 76snf2beta 20040407140917 D0b8b122.SMD 30 60 Final 70077 53 0 3727 73snf2beta 20040407140920 D0b8e0b6.SMD 20 40 Clean 0 0 0 2958 54snf2beta 20040407140927 D0b960b6.SMD 30 80 Final 30439 54 0 3885 73snf2beta 20040407140934 D0b930b6.SMD 20 40 Clean 0 0 0 2647 67snf2beta 20040407140935 D0b9e0a8.SMD 20 130 Final 73558 52 0 6242 80snf2beta 20040407140942 D0ba414e.SMD 20 160 Final 105444 52 0 8252 87snf2beta 20040407140942 D0ba40de.SMD 201 60 Final 105825 52 0 3351 68snf2beta 20040407140947 D0baa0b6.SMD 30 121 Final 30439 54 0 3898 72snf2beta 20040407140947 D0baa14e.SMD 40 80 Final 66835 52 0 5358 64snf2beta 20040407140952 D0bad122.SMD 20 110 Final 97422 57 0 6104 79snf2beta 20040407140952 D0bae0d2.SMD 30 81 Final 83761 57 0 4790 72snf2beta 20040407140952 D0bac0b6.SMD 40 90 Final 1686 48 0 5415 80snf2beta 20040407141003 D0bb90b6.SMD 20 40 Final 49992 54 0 2186 69The first thing I notice is that the setup times (first number) on your system are consistently large. According to your log entries it is taking a quarter of a second to scan the working directory for a job... That's a LOT of time for a directory scan to take.The message scan itself doesn't seem to be out of range.The next thing I notice is that your messages arrive several seconds apart consistently. I see 10 sec, 16, 12, 4, 10, etc... In our log we frequently scan several messages in the same second.I see two things going on based on this data:I suspect your system is I/O bound. There is no reason that a directory scan should take more than a few tens of milliseconds except occasionally... That puts your numbers out by nearly an order of magnitude (compare 20s 30s w/ 109, 187, 280+!). Be sure that Sniffer's working directory does not have any extra files in it. Sniffer instances measure their apparent work load by counting the number of files in their working directory... The theory is that aside from a handful of necessary files the rest are jobs waiting to be processed... so if the number of files is large then the load must be high and so a Sniffer instance should be prepared to wait a bit longer for service.Sniffer should be running in it's own directory with no other files present that don't need to be there. Be sure to clean out any dead job files that might have built up with a prior error etc...My thinking on I/O is that if it takes 100-280 msec to scan the directory for job files then it's likely to take quite a while to load any program - including the shell. This can explain the additional time you are seeing in your measurements. Under normal circumstances I would expect that operation to happen almost instantaneously since the Sniffer executable, command shell, and other files that must load should remain consistently in memory due to their being called so
RE: [sniffer] Final beta (b2) for snfrv2r3
That's fantastic news... Another mystery bites the dust! _M At 09:56 AM 4/13/2004, you wrote: Pete, The speed problem has been found. McAfee Netshield 4.51 was making our server RIDICULOUSLY slow, despite the fact that we tried excluding the Sniffer folder and even disabling the service from the tray-icon. Upgrading to Virusscan Enterprise 7.x fixed our problem and our performance levels are in the regions that you mentioned. Thanks for thinking along! Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOS Small Office Solutions / Reject / Wannepad 27 - 1066 HW - Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy - Installation - Maintenance Network Security - Internet - E-mail Software Development - Project Management -- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Michiel Prins Sent: donderdag 8 april 2004 21:11 To: [EMAIL PROTECTED] Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Preliminary tests show there's no I/O problem but I'll do some additional benchmarking here and get back to you on this. Groet, (regards) -- ing. Michiel Prins bsc [EMAIL PROTECTED] SOS Small Office Solutions / Reject / Wannepad 27 - 1066 HW - Amsterdam t.+31(0)20-4082627 - f.+31-(0)20-4082628 -- Consultancy - Installation - Maintenance Network Security - Internet - E-mail Software Development - Project Management -- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Pete McNeil Sent: woensdag 7 april 2004 17:38 To: [EMAIL PROTECTED] Subject: RE: [sniffer] Final beta (b2) for snfrv2r3 Extraordinary... Compare with a snippet from our IMail/NT4 test platform (severely underpowered)... snf2beta 20040407140913 D0b86122.SMD 30 90 Final 75148 63 0 6891 68 snf2beta 20040407140913 D0b8614e.SMD 90 140 Final 103691 57 0 8878 72 snf2beta 20040407140914 D0b88122.SMD 40 141 Final 103689 57 0 9003 71 snf2beta 20040407140915 D0b880b6.SMD 90 20 Final 106244 52 0 817 65 snf2beta 20040407140916 D0b8a0de.SMD 40 210 Final 104044 52 0 8779 76 snf2beta 20040407140917 D0b8b122.SMD 30 60 Final 70077 53 0 3727 73 snf2beta 20040407140920 D0b8e0b6.SMD 20 40 Clean 0 0 0 2958 54 snf2beta 20040407140927 D0b960b6.SMD 30 80 Final 30439 54 0 3885 73 snf2beta 20040407140934 D0b930b6.SMD 20 40 Clean 0 0 0 2647 67 snf2beta 20040407140935 D0b9e0a8.SMD 20 130 Final 73558 52 0 6242 80 snf2beta 20040407140942 D0ba414e.SMD 20 160 Final 105444 52 0 8252 87 snf2beta 20040407140942 D0ba40de.SMD 201 60 Final 105825 52 0 3351 68 snf2beta 20040407140947 D0baa0b6.SMD 30 121 Final 30439 54 0 3898 72 snf2beta 20040407140947 D0baa14e.SMD 40 80 Final 66835 52 0 5358 64 snf2beta 20040407140952 D0bad122.SMD 20 110 Final 97422 57 0 6104 79 snf2beta 20040407140952 D0bae0d2.SMD 30 81 Final 83761 57 0 4790 72 snf2beta 20040407140952 D0bac0b6.SMD 40 90 Final 1686 48 0 5415 80 snf2beta 20040407141003 D0bb90b6.SMD 20 40 Final 49992 54 0 2186 69 The first thing I notice is that the setup times (first number) on your system are consistently large. According to your log entries it is taking a quarter of a second to scan the working directory for a job... That's a LOT of time for a directory scan to take. The message scan itself doesn't seem to be out of range. The next thing I notice is that your messages arrive several seconds apart consistently. I see 10 sec, 16, 12, 4, 10, etc... In our log we frequently scan several messages in the same second. I see two things going on based on this data: I suspect your system is I/O bound. There is no reason that a directory scan should take more than a few tens of milliseconds except occasionally... That puts your numbers out by nearly an order of magnitude (compare 20s 30s w/ 109, 187, 280+!). Be sure that Sniffer's working directory does not have any extra files in it. Sniffer instances measure their apparent work load by counting the number of files in their working directory... The theory is that aside from a handful of necessary files the rest are jobs waiting to be processed... so if the number of files is large then the load must be high and so a Sniffer instance should be prepared to wait a bit longer for service. Sniffer should be running in it's own directory with no other files present that don't need to be there. Be sure to clean out any dead job files that might have built up with a prior error etc... My thinking on I/O is that if it takes 100-280 msec to scan the directory for job files then it's likely to take quite a while to load any program - including the shell. This can explain the additional time you are seeing in your measurements. Under normal circumstances I would expect that operation to happen almost instantaneously since the Sniffer executable, command shell, and other files that must load should
Re: [sniffer] log file growing
Ok, There is a logrotate.cmd that you modified for me. I don't know why it isn't kicking off automatically like it was before, but it isn't. It had been running automatically for months. How do you recommend doing that so that you get the log files when you want them? Thanks, Andy - Original Message - From: Pete McNeil To: [EMAIL PROTECTED] Sent: Monday, April 12, 2004 2:09 PM Subject: Re: [sniffer] log file growing Usually the log rotation is handled in a different .cmd.I guess it could have been cobbled together but I don't recall doing it.You can get the starter scripts here: http://www.sortmonster.net/Sniffer/Updates/WindowsTools.zip ftp://ftp.sortmonster.net/Sniffer/Updates/WindowsTools.zipA number of user submitted scripts are also available at the bottom of the Automated Updates Help page:http://www.sortmonster.com/MessageSniffer/Help/AutomatingUpdatesHelp.htmlHope this helps,_MAt 12:56 PM 4/12/2004, you wrote: Hi,The .snf file is up to date, so the program alias is working.I ran the autosnf.cmd file you help me setup and it is working with noerrors, but it isn't doing anything with rotating the log files, as it wasbefore.I have no idea why.,I do know that you had set it up for me to rotate the logs...can you send methe section of the autosnf.cmd file that is missing that does that?Thanks, andy- Original Message -From: "Pete McNeil" [EMAIL PROTECTED]To: [EMAIL PROTECTED]Sent: Saturday, April 10, 2004 9:12 AMSubject: Re: [sniffer] log file growing H, If we were triggering it - then that would have been our update notification message. If that's stopped working then you might want tolook at your rulebase to see that it's up to date... What you're looking for is a program alias that launches your updatescript. That's the best place to start. You can probably send yourself a message to that address to trigger (or not) the events and see what is broken. Hope this helps, _M At 08:23 AM 4/10/2004, you wrote: Ok, That's what's happening. It was being rotated. You helped me set thatup. I haven't changed/moved anything so it has stopped working... It wasbeing initiated automatically by an email sent by you to the system in Imail. Where do I look? Thanks, andy - Original Message - From: "Pete McNeil" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, April 09, 2004 3:20 PM Subject: Re: [sniffer] log file growing At 12:18 PM 4/9/2004, you wrote: HI, My log file used to write to a new file everyday, now it is writingto the same file... I didn't change anything, how do I fix it? This is confusing. Message Sniffer has always written to a single logfile that does not change. External utilities could be used to rotate thelog file as needed. The only time this has changed is with the new beta which includes a command option for persistent servers: [snflicid.exe] rotate If this command is run and you are running a persistent instance of sniffer then the log file will be rotated to [snflicid].log.mmddhhmmss. This does not happen automatically and never did in the past. If your log file was being rotated then it was handled by anotherutility on your system and that utility has stopped working. Hope this helps, _M PS: snflicid = your specific sniffer license id. mmddhhmmss = date/time stamp in a compressed ISO format. This E-Mail came from the Message Sniffer mailing list. Forinformation and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html This E-Mail came from the Message Sniffer mailing list. For informationand (un)subscription instructions go tohttp://www.sortmonster.com/MessageSniffer/Help/Help.htmlThis E-Mail came from the Message Sniffer mailing list. For information and (un)subscription instructions go to http://www.sortmonster.com/MessageSniffer/Help/Help.html
Re: [sniffer] log file growing
It is working, I tested it from the command line. What time of day do you want it run? - Original Message - From: Pete McNeil To: [EMAIL PROTECTED] Sent: Tuesday, April 13, 2004 7:06 PM Subject: Re: [sniffer] log file growing First, give it a test by launching it manually to make sure it's not broken.If that works then set up a scheduled task to run the .cmd once a day (that's usually enough).That should be it.Thanks!_MAt 05:57 PM 4/13/2004, you wrote: Ok,There is a logrotate.cmd that you modified for me. I don't know why it isn't kicking off automatically like it was before, but it isn't. It had been running automatically for months.How do you recommend doing that so that you get the log files when you want them?Thanks, Andy - Original Message - From: Pete McNeil To: [EMAIL PROTECTED] Sent: Monday, April 12, 2004 2:09 PM Subject: Re: [sniffer] log file growing Usually the log rotation is handled in a different .cmd. I guess it could have been cobbled together but I don't recall doing it. You can get the starter scripts here: http://www.sortmonster.net/Sniffer/Updates/WindowsTools.zip ftp://ftp.sortmonster.net/Sniffer/Updates/WindowsTools.zip A number of user submitted scripts are also available at the bottom of the Automated Updates Help page: http://www.sortmonster.com/MessageSniffer/Help/AutomatingUpdatesHelp.html Hope this helps, _M At 12:56 PM 4/12/2004, you wrote: Hi, The .snf file is up to date, so the program alias is working. I ran the autosnf.cmd file you help me setup and it is working with no errors, but it isn't doing anything with rotating the log files, as it was before.I have no idea why., I do know that you had set it up for me to rotate the logs...can you send me the section of the autosnf.cmd file that is missing that does that? Thanks, andy - Original Message - From: "Pete McNeil" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, April 10, 2004 9:12 AM Subject: Re: [sniffer] log file growing H, If we were triggering it - then that would have been our update notification message. If that's stopped working then you might want to look at your rulebase to see that it's up to date... What you're looking for is a program alias that launches your update script. That's the best place to start. You can probably send yourself a message to that address to trigger (or not) the events and see what is broken. Hope this helps, _M At 08:23 AM 4/10/2004, you wrote: Ok, That's what's happening. It was being rotated. You helped me set that up. I haven't changed/moved anything so it has stopped working... It was being initiated automatically by an email sent by you to the system in Imail. Where do I look? Thanks, andy - Original Message - From: "Pete McNeil" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, April 09, 2004 3:20 PM Subject: Re: [sniffer] log file growing At 12:18 PM 4/9/2004, you wrote: HI, My log file used to write to a new file everyday, now it is writing to the same file... I didn't change anything, how do I fix it? This is confusing. Message Sniffer has always written to a single log file that does not change. External utilities could be used to rotate the log file as needed. The only time this has changed is with the new beta which includes a command option for persistent servers: [snflicid.exe] rotate If this command is run and you are running a persistent instance of sniffer then the log file will be rotated to [snflicid].log.mmddhhmmss. This does not happen automatically and never did in the past. If your log file was being rotated then it was handled by another utility on your system and that utility has stopped working. Hope this helps, _M PS: snflicid = your specific sniffer license id. mmddhhmmss = date/time
