[sniffer] Re: Update Script - Replace WGET and GZIP with CURL
Hi Pete, Agreed, with WGET it gets quite a bit complicated (because it really doesn't understand the GZ format). That's why you currently have to override the filename, call it GZ, then call GZIP to unzip it. I've come to the conclusion that it's not worth the trouble with WGET (as you surmised, it would make the script more complicated - so forget that approach.) The reason to switch to CURL is that behaves like a true HTTP application with GZIP support. You don't need to ship an extra GZIP on top of WGET in your distribution. CURL requests your file from your server, asks for it to be GZ compressed during transport, receives it, decompresses it before saving it - and sets timestamp from the server. All that in ONE command. Basically, you would replace these TWO lines in your current script: wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -O %RULEBASE_PATH%\%LICENSE_ID%.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m if exist %RULEBASE_PATH%\%LICENSE_ID%.new.gz gzip -d -f %RULEBASE_PATH%\%LICENSE_ID%.new.gz with this single line: curl http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -R -o %RULEBASE_PATH%\%LICENSE_ID%.new -z %RULEBASE_PATH%\%LICENSE_ID%.snf --compressed -u sniffer:ki11sp8m Best Regards, Andy From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, October 08, 2008 1:36 AM To: Message Sniffer Community Subject: [sniffer] Re: Update Script - Choice of WGET Parameter Prevents TimeStamping Hello Andy, Wednesday, October 8, 2008, 1:13:50 AM, you wrote: Hi Pete, Thanks for giving it your consideration. If you decide to revise these parameteres, then it will require an extra command in your script (because the WGET command will output the compressed file as .SNF). There is actually a bit more to it than that -- The existing script generally works even though it doesn't preserve the servers's timestamp because: 1. It is usually triggered from the SNFServer when SNF detects a newer rulebase file. 2. Any rulebase fill recently downloaded is guaranteed to be newer provided the local server's clock is correct (or close to it). Also -- are you saying that with the parameters you've provided WGET would decompress the file on it's own so that we wouldn't need to do that in our script? If so, how does it know for sure where to find GZIP? If not then it would be a little dangerous to have a .snf file around that looked correct but was in fact not yet decompressed. Another consideration is that if the file name is going to collide with the existing rulebase file we would want to move that into another location so that we don't stomp the existing rulebase file until we've tested the new one. It would be preferable to use WGET since there's nothing wrong with it and we've been using it long enough that most SNF folks already have it. That doesn't mean you shouldn't provide an alternate script that works with CURL just in case someone has a preference. Best, _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Rulebase, bogus UTC Timestamps?
Hi Pete, I'm running a Sniffer service on a secondary system so that I can test my rulebase update script. After I changed to curl (to maintain the server timestamps), I'm now seeing the following in the status.minute.log: rulebase utc=20081008183610 / active utc=20081008183610 / update ready=no utc=20081008143610 / The update ready note matches the timestamp of 2:36 PM of actual rulebase SNF file. Which is correct, because when it downloaded from your server at 11:35 AM EDT, your server presented this HTTP header: Date: Wed, 08 Oct 2008 15:33:44 GMT Server: Apache/2.0.46 (Red Hat) Last-Modified: Wed, 08 Oct 2008 14:36:10 GMT ETag: 3ec4df2-cb96c0-458bed6588a80 Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Transfer-Encoding: chunked Content-Type: application/x-sortmonster But, how is the rulebase and active UTC determined? Where is this 18:36:10 coming from. It seems to me as if somehow Sniffer adjusted the (already) GMT time of 14:36 by yet ANOTHER 4 hours, giving it a fantasy timestamp of 18:36. The net effect appears to be that my test machine doesn't get an UpdateReady.txt until 4 hours have passed. My improved getRulebase.cmd works perfectly, but it will only get launched every four hours, at best. Best Regards, Andy
[sniffer] Re: Updated getRuleBase.cmd
Hello Andy, Wednesday, October 8, 2008, 12:52:59 PM, you wrote: Hi, After resolving the issues with UTC vs. local time (apparently the Sniffer service doesnt actually use a version identifier inside the SNF file, but relies on the Windows file date to determine what rulebase version is in place), here the updated getRuleBase.cmd. snip/ 1. Get the latest CURL.EXE for Win 2000 or higher fromhttp://curl.haxx.se/latest.cgi?curl=win32-nossl-sspi(dont use older builds to avoid timezone issues). Does this resolve the timestamp issues you indicated in your previous message? SNF gets the timestamp from the file system the using gmtime() of the modification timestamp on the file. The same call is made in the SYNC server software when the rulebase timestamp is provided to the SNF node for comparison. gmtime() provides the UTC time (used to be known as GMT) for any timestamp. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC. # This message is sent to you because you are subscribed to the mailing list sniffer@sortmonster.com. To unsubscribe, E-mail to: [EMAIL PROTECTED] To switch to the DIGEST mode, E-mail to [EMAIL PROTECTED] To switch to the INDEX mode, E-mail to [EMAIL PROTECTED] Send administrative queries to [EMAIL PROTECTED]
[sniffer] Re: Updated getRuleBase.cmd
Hi, Yes, recent Windows curl builds will convert between UTC and local time. I was just caught off-guard, that Sniffer is using an external datum which is subject for wanted or unwanted manipulation for something as crucial as determining the file version of the rule base? If (due to copying files between servers) a sniffer file has a bogus file date, Sniffer would actually rely on that and be thrown out of whack? I would have expected that the SNF file was self-contained (e.g., contained an internal version id or timestamp) so that it was not subject to outside interference. Best Regards, Andy From: Message Sniffer Community [mailto:[EMAIL PROTECTED] On Behalf Of Pete McNeil Sent: Wednesday, October 08, 2008 1:30 PM To: Message Sniffer Community Subject: [sniffer] Re: Updated getRuleBase.cmd Hello Andy, Wednesday, October 8, 2008, 12:52:59 PM, you wrote: Hi, After resolving the issues with UTC vs. local time (apparently the Sniffer service doesn't actually use a version identifier inside the SNF file, but relies on the Windows file date to determine what rulebase version is in place), here the updated getRuleBase.cmd. snip/ 1. Get the latest CURL.EXE for Win 2000 or higher from http://curl.haxx.se/latest.cgi?curl=win32-nossl-sspi http://curl.haxx.se/latest.cgi?curl=win32-nossl-sspi (don't use older builds to avoid timezone issues). Does this resolve the timestamp issues you indicated in your previous message? SNF gets the timestamp from the file system the using gmtime() of the modification timestamp on the file. The same call is made in the SYNC server software when the rulebase timestamp is provided to the SNF node for comparison. gmtime() provides the UTC time (used to be known as GMT) for any timestamp. _M -- Pete McNeil Chief Scientist, Arm Research Labs, LLC.
