[sniffer] Re: DST update problem - server changes
Pete, I upgraded to the latest getRulebase file and followed the instructions, but now all I see on my windows system (DST) is the following: (I replaced my license ID # with ) snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 over and over again for pages and pages in my console window. Everything worked great until I updated to the latest getRulebase. My license ID and everything are all the same and I re-verified them after I copied the info from the other getRulebase script. What is causing this? Thanks, Shawn On Mon, Mar 9, 2009 at 2:44 PM, Pete McNeil madscient...@armresearch.comwrote: Hello Sniffer Folks, DST Update Problem: A bug in the old getRulebase.cmd script caused Win* systems to discard the server's timestamp on rulebase files and substitute the local timestamp. As a result any system that change to DST (daylight savings time) after our rulebase delivery servers would continuously show a newer rulebase file on our servers. As a result these systems would repeatedly download the rulebase file as quickly as they could. Solutions: 1. Everyone should upgrade their getRulebase.cmd script to the latest version: http://www.armresearch.com/message-sniffer/download/CURL-getRulebase.zip ** Note that most *NIX systems do not have the same problem with wget, but everyone should check. *** Note that going forward a CURL based update script is preferred. Since CURL is available on most *NIX systems by default we do not expect this to be a problem. 2. If not upgrading to the latest version then they should modify their wget based scripts to ensure that the server's timestamp on the rulebase file is preserved. 3. Since many systems will not be upgraded in the short term, we are also taking action on the delivery server to prevent problems with ruelbase updates: From now on a new rulebase will show it's new timestamp for 5 minutes after it is posted. Then the timestamp will be pushed back one hour to limit the amount of time systems with later DST transitions will see the files as new. The results of this change will be: * Systems that have upgraded to the new getRulebase.cmd script or are using an otherwise correct update script will see no difference. By default, SNFSync events occur about once per minute and since the new rulebase file will be shown with it's current timestamp for 5 minutes each correctly configured SNF node will see and download the fresh rulebase file as soon as it is available. * Some systems that have not upgraded may attempt to download a new rulebase file twice, or possibly three times depending upon timing. However after that time (based on a 180 second guard time) these systems should cease to see the rulebase files as new and will stop trying to download the files. Once these systems move to DST they will operate normally. Of course we hope that all systems will upgrade their update scripting before this! * Systems that are using a scheduled task to update their rulebase may sometimes see the newer time stamp and may sometimes see the delayed (one hour old) timestamp. This will cause update lag to shift in time with an average of 30 minutes. At this time this seems to be the best compromise for everyone. We apologize for any inconvenience. Thanks, _M # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: sniffer-...@sortmonster.com To switch to the DIGEST mode, E-mail to sniffer-dig...@sortmonster.com To switch to the INDEX mode, E-mail to sniffer-in...@sortmonster.com Send administrative queries to sniffer-requ...@sortmonster.com
[sniffer] Re: DST update problem - server changes
I to have the same problem I have reverted back to the old script. (We are windows based) Regards David Moore moo...@romtech.com.au J.P. MCP, MCSE, MCSE + INTERNET, CNE. www.adsldirect.com.au for ADSL and Internet www.romtech.com.au for PC sales Office Phone: (+612) 9453 1990 Fax Phone: (+612) 9453 1880 Mobile Phone: +614 18 282 648 Skype Phone: ADSLDIRECT POSTAL ADDRESS: PO BOX 190 BELROSE NSW 2085 AUSTRALIA. - This email message is only intended for the addressee(s) and contains information that may be confidential, legally privileged and/or copyright. If you are not the intended recipient please notify the sender by reply email and immediately delete this email. Use, disclosure or reproduction of this email, or taking any action in reliance on its contents by anyone other than the intended recipient(s) is strictly prohibited. No representation is made that this email or any attachments are free of viruses. Virus scanning is recommended and is the responsibility of the recipient. - Shawn wrote: Pete, I upgraded to the latest getRulebase file and followed the instructions, but now all I see on my windows system (DST) is the following: (I replaced my license ID # with ) snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 over and over again for pages and pages in my console window. Everything worked great until I updated to the latest getRulebase. My license ID and everything are all the same and I re-verified them after I copied the info from the other getRulebase script. What is causing this? Thanks, Shawn On Mon, Mar 9, 2009 at 2:44 PM, Pete McNeil madscient...@armresearch.com mailto:madscient...@armresearch.com wrote: Hello Sniffer Folks, DST Update Problem: A bug in the old getRulebase.cmd script caused Win* systems to discard the server's timestamp on rulebase files and substitute the local timestamp. As a result any system that change to DST (daylight savings time) after our rulebase delivery servers would continuously show a newer rulebase file on our servers. As a result these systems would repeatedly download the rulebase file as quickly as they could. Solutions: 1. Everyone should upgrade their getRulebase.cmd script to the latest version: http://www.armresearch.com/message-sniffer/download/CURL-getRulebase.zip ** Note that most *NIX systems do not have the same problem with wget, but everyone should check. *** Note that going forward a CURL based update script is preferred. Since CURL is available on most *NIX systems by default we do not expect this to be a problem. 2. If not upgrading to the latest version then they should modify their wget based scripts to ensure that the server's timestamp on the rulebase file is preserved. 3. Since many systems will not be upgraded in the short term, we are also taking action on the delivery server to prevent problems with ruelbase updates: From now on a new rulebase will show it's new timestamp for 5 minutes after it is posted. Then the timestamp will be pushed back one hour to limit the amount of time systems with later DST transitions will see the files as new. The results of this change will be: * Systems that have upgraded to the new getRulebase.cmd script or are using an otherwise correct update script will see no difference. By default, SNFSync events occur about once per minute and since the new rulebase file will be shown with it's current timestamp for 5 minutes each correctly configured SNF node will see and download the fresh rulebase file as soon as it is available. * Some systems that have not upgraded may attempt to download a new rulebase file twice, or possibly three times depending upon timing. However after that time (based on a 180 second guard time) these systems should cease to see the rulebase files as new and will stop trying to download the files. Once these systems move to DST they will operate normally. Of course we hope that all systems will upgrade their update scripting before this! * Systems that are using a scheduled task to update their rulebase may sometimes see the newer time stamp and may sometimes see the delayed (one hour old) timestamp. This will cause update lag to shift in time with an average of 30 minutes. At this time this seems to be the best compromise for everyone. We apologize for any inconvenience. Thanks, _M # This message is sent to you
[sniffer] Re: DST update problem - server changes
David Moore wrote: I to have the same problem I have reverted back to the old script. (We are windows based) snip/ Shawn wrote: Pete, I upgraded to the latest getRulebase file and followed the instructions, but now all I see on my windows system (DST) is the following: (I replaced my license ID # with ) snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 This is most likely NOT a problem -- This happens when the update script runs and the file on the server is not newer than the file on your system. CURL will refuse to download the file and then snf2check produces an ERROR_RULE_FILE! error because it cannot find the new rulebase file it is looking for. We are reworking the script to include line to test for this and exit the script instead of producing the error. You can add the following line just before the snf2check.exe line: if not exist %LICENSE_ID%.new goto DONE In this case the ERROR_RULE_FILE error itself is harmless. _M
[sniffer] Re: DST update problem - server changes
Hi, That's why the enhanced version of your script (which properly supports Sniffer's ability to keep the rulebase and the workspace in subfolders!) that I sent you checks for CURL success AND for an existing file. curl http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -s -R -f -o %RULEBASE_PATH%\%LICENSE_ID%.new -z %RULEBASE_PATH%\%LICENSE_ID%.snf --compressed -u sniffer:ki11sp8m -D %SNIFFER_PATH%\curlhdr.txt if %ERRORLEVEL% NEQ 0 goto CLEANUP if not exist %RULEBASE_PATH%\%LICENSE_ID%.new goto CLEANUP snf2check.exe %RULEBASE_PATH%\%LICENSE_ID%.new %AUTHENTICATION% goto DONE I recommend you go to Cleanup - where the .LCK file will be cleaned up if it exists. Best Regards, Andy From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Tuesday, March 10, 2009 7:59 AM To: Message Sniffer Community Subject: [sniffer] Re: DST update problem - server changes David Moore wrote: I to have the same problem I have reverted back to the old script. (We are windows based) snip/ Shawn wrote: Pete, I upgraded to the latest getRulebase file and followed the instructions, but now all I see on my windows system (DST) is the following: (I replaced my license ID # with ) snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 snf2check: .new ERROR_RULE_FILE! 1 file(s) copied.R:2349772 [0/12 - 0] W:0 C:0 B:0 T:0 S:0 This is most likely NOT a problem -- This happens when the update script runs and the file on the server is not newer than the file on your system. CURL will refuse to download the file and then snf2check produces an ERROR_RULE_FILE! error because it cannot find the new rulebase file it is looking for. We are reworking the script to include line to test for this and exit the script instead of producing the error. You can add the following line just before the snf2check.exe line: if not exist %LICENSE_ID%.new goto DONE In this case the ERROR_RULE_FILE error itself is harmless. _M
[sniffer] Re: DST update problem - server changes
Andy Schmidt wrote: Hi, That's why the enhanced version of your script (which properly supports Sniffer's ability to keep the rulebase and the workspace in subfolders!) that I sent you checks for CURL success AND for an existing file. curl http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -s -R -f -o %RULEBASE_PATH%\%LICENSE_ID%.new -z %RULEBASE_PATH%\%LICENSE_ID%.snf --compressed -u sniffer:ki11sp8m -D %SNIFFER_PATH%\curlhdr.txt *if %ERRORLEVEL% NEQ 0 goto CLEANUP* *if not exist %RULEBASE_PATH%\%LICENSE_ID%.new goto CLEANUP* The newest version (just posted before I received this note) checks for a new file from CURL. It also uses the -v (verbose) option with CURL and captures it's output to the getRulebase.txt file for reporting. If there is an error then the details will be seen. curl -v http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf; -o %LICENSE_ID%.new -s -S -R -z %LICENSE_ID%.snf -H Accept-Encoding:gzip --compressed -u sniffer:ki11sp8m 2 getRulebase.txt if not exist %LICENSE_ID%.new echo New rulebase file NOT downloaded getRulebase.txt if not exist %LICENSE_ID%.new goto CLEANUP ** snf2check.exe %RULEBASE_PATH%\%LICENSE_ID%.new %AUTHENTICATION% goto DONE I recommend you go to Cleanup -- where the .LCK file will be cleaned up if it exists... The new script does this and it also reports success explicitly snf2check.exe %LICENSE_ID%.new %AUTHENTICATION% 2 getRulebase.txt if errorlevel 1 goto CLEANUP echo New rulebase file tested OK getRulebase.txt The script we are using does not yet support alternate/sub folders because we have not built that capability into our windows installer. Using alternate/sub folders is a custom modification and is likely to be different on each system so we aren't (yet) making any attempt to have our installer predict or understand this kind of configuration. A later version of the script might include some hooks to do that but they will need to be ignored by the installer for the time being. Trying to keep things simple. Thanks, _M
[sniffer] Re: DST update problem - server changes
1 checks for a new file from CURL. It also uses the -v (verbose) option with CURL and captures it's output to the getRulebase.txt file for reporting. If there is an error then the details will be seen. I'm glad to see that (after a few tries), your script finally looks like the one I had sent last September G The benefit for checking the CURL errorlevel: if %ERRORLEVEL% NEQ 0 goto CLEANUP is that it gracefully handles incomplete .NEW files that might result from interrupted file transfers. There's no reason for snf2check to Hick Up and report on a SNF file, if the download was never successful! 2. Agreed on complexity. Just to keep things in perspective - the added two lines below shoe how complex the support for dedicated rulebase and work subfolders really is G: Nothing changes for clients who don't use subfolders - nothing changes in the installer! REM - Edit This Section SET SNIFFER_PATH=C:\SNF SET RULEBASE_PATH=%SNIFFER_PATH% SET WORKSPACE_PATH=%SNIFFER_PATH% But clients who do choose to set up subfolders, would at least be able to use your standard script: REM - Edited to Support Dedicated Subfolders SET SNIFFER_PATH=D:\IMail\declude\SNF SET RULEBASE_PATH=%SNIFFER_PATH%\Rulebase SET WORKSPACE_PATH=%SNIFFER_PATH%\Workspace I'm a big believer in keep dynamic data separate from static program/config files. Prevents things from being overlooked, things from accidentally being cleaned up that shouldn't be, allows proper execute/change permissions being set - and by avoiding mistakes ultimately improves stability. I'm not complaining. Last year I had pointed out that CURL should be used to maintain file dates and had submitted the simple change to the script (including checking the return code AND checking for the existence of a downloaded file AND cleaning up the Log File.!) - which were finally incorporated. So there's hope that some future event will eventually make the benefits of dedicated data subfolders equally apparent G. Best Regards, Andy From: Message Sniffer Community [mailto:snif...@sortmonster.com] On Behalf Of Pete McNeil Sent: Tuesday, March 10, 2009 10:20 AM To: Message Sniffer Community Subject: [sniffer] Re: DST update problem - server changes Andy Schmidt wrote: Hi, That's why the enhanced version of your script (which properly supports Sniffer's ability to keep the rulebase and the workspace in subfolders!) that I sent you checks for CURL success AND for an existing file. curl http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf http://www.sortmonster.net/Sniffer/Updates/%25LICENSE_ID%25.snf -s -R -f -o %RULEBASE_PATH%\%LICENSE_ID%.new -z %RULEBASE_PATH%\%LICENSE_ID%.snf --compressed -u sniffer:ki11sp8m -D %SNIFFER_PATH%\curlhdr.txt if %ERRORLEVEL% NEQ 0 goto CLEANUP if not exist %RULEBASE_PATH%\%LICENSE_ID%.new goto CLEANUP The newest version (just posted before I received this note) checks for a new file from CURL. It also uses the -v (verbose) option with CURL and captures it's output to the getRulebase.txt file for reporting. If there is an error then the details will be seen. curl -v http://www.sortmonster.net/Sniffer/Updates/%25LICENSE_ID%25.snf http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf; -o %LICENSE_ID%.new -s -S -R -z %LICENSE_ID%.snf -H Accept-Encoding:gzip --compressed -u sniffer:ki11sp8m 2 getRulebase.txt if not exist %LICENSE_ID%.new echo New rulebase file NOT downloaded getRulebase.txt if not exist %LICENSE_ID%.new goto CLEANUP snf2check.exe %RULEBASE_PATH%\%LICENSE_ID%.new %AUTHENTICATION% goto DONE I recommend you go to Cleanup - where the .LCK file will be cleaned up if it exists. The new script does this and it also reports success explicitly snf2check.exe %LICENSE_ID%.new %AUTHENTICATION% 2 getRulebase.txt if errorlevel 1 goto CLEANUP echo New rulebase file tested OK getRulebase.txt The script we are using does not yet support alternate/sub folders because we have not built that capability into our windows installer. Using alternate/sub folders is a custom modification and is likely to be different on each system so we aren't (yet) making any attempt to have our installer predict or understand this kind of configuration. A later version of the script might include some hooks to do that but they will need to be ignored by the installer for the time being. Trying to keep things simple. Thanks, _M
[sniffer] Re: DST update problem - server changes
Andy Schmidt wrote: 1 checks for a new file from CURL. It also uses the -v (verbose) option with CURL and captures it's output to the getRulebase.txt file for reporting. If there is an error then the details will be seen. I'm glad to see that (after a few tries), your script finally looks like the one I had sent last September G The benefit for checking the CURL errorlevel: if %ERRORLEVEL% NEQ 0 goto CLEANUP is that it gracefully handles incomplete .NEW files that might result from interrupted file transfers. There's no reason for snf2check to Hick Up and report on a SNF file, if the download was never successful! I was under the impression that a broken download in curl would not leave behind a fragment. If it will then we should add that line and perhaps some supporting report text. 2. Agreed on complexity. Just to keep things in perspective - the added two lines below shoe how complex the support for dedicated rulebase and work subfolders really is G: Nothing changes for clients who don't use subfolders -- nothing changes in the installer! REM - Edit This Section SET SNIFFER_PATH=C:\SNF *SET RULEBASE_PATH=%SNIFFER_PATH%* *SET WORKSPACE_PATH=%SNIFFER_PATH%* But clients who do choose to set up subfolders, would at least be able to use your standard script: REM - Edited to Support Dedicated Subfolders SET SNIFFER_PATH=D:\IMail\declude\SNF *SET RULEBASE_PATH=%SNIFFER_PATH%**\Rulebase* *SET WORKSPACE_PATH=%SNIFFER_PATH%\**Workspace* I'm thinking along those lines also -- but the installer would need to understand this because the expectation is that the installer can handle upgrades and roll-backs. Also, we have some tasks on the list to make SNFServer more aware of alternate paths -- perhaps including a root path mechanism that allows for some sub paths and some arbitrary paths. We'll want to get that nailed down and then observe some preferred practices before we go too far with the installer and update script. We're also looking at upgrading the update-script mechanism to pass parameters to the update-script directly so that all of the configuration can live in the snf_engine.xml file --- There are a lot of decisions to be made there before that work begins, but once it is done the standard update scripts will likely be built to take all of their important parameters from the command line. --- Thanks for your help! _M
