Hello Andy,
First, let me say thanks for sharing all of this. We don't often get detailed feedback on these things. Your valuable insights will be used to make later releases better.
With that said I will add a few comments here and there to explain why things are the way they are and help others achieve their goals (which might be different).
Saturday, October 4, 2008, 2:12:47 AM, you wrote:
|
> |
Hi,
Didn’t realize I had been uninstalled for a few months.
I saw that V3 was released, so I gave it a shot. I unzipped the installation files to a new “/SNF” folder. All files were expanded into the same folder (your zip file had not subfolders!). |
The vast majority of SNF installations on Windows systems keep all SNF components in the same folder. So, for the majority of folks SNF is simply decompressed into "it's own folder", configured, and launched. This makes things simple and there is no question where to find things.
Along the way we have been asked for the ability to put logs in a different location, get rulebsae files from a different location, configuration files, and so forth. We've added those features so that the folks who have reason to move things around can do so.
We decided not to create a presumed directory structure for SNF because the folks who've asked us to provide these features all had their own unique way to divide things and move them. Any structure we created would have been wrong for most folks, so we keep the single folder option as our default since it is what everyone was used to and what most of our customers have been using.
SNF is used on a lot of platforms -- each with their own conventions. Not only that but within each platform administrators and user communities develop their own preferences.
The <paths/> section described next allows folks to manipulate some file locations according to user preferences.
|
> |
Following the instructions I customized the XML files.
I noticed THESE parameters:
<node identity='D:/IMail/declude/SNF/identity.xml'>
<paths> <log path='D:/IMail/declude/SNF/Log/'/> <rulebase path='D:/IMail/declude/SNF/Rulebase/'/> <workspace path='D:/IMail/declude/SNF/Workspace/'/> </paths>
I’m a believer in keeping different data in their distinct subfolders, so I set up the /Log, /Rulebase and /Workspace subfolders by hand and updated the XML file.
The I took a wild guess that SOME files would have to be moved into those subfolders – but there are NO instructions WHAT files go WHERE for things to actually work! |
The current documentation is located here:
http://www.armresearch.com/support/articles/software/snfServer/config/node/paths/index.jsp
The general design is such that log files will be written into the log path, the rulebase file will be read from the rulebase path, and the remaining files should reside in the workspace path.
I will add a task to clarify this in our documentation and provide more detail.
|
> |
I found it annoying that further down in the same XML File was yet another “path” that was NOT included in the “paths” node in the top of the XML file:
<update-script on-off='on' call='D:/IMail/declude/SNF/getRulebase.cmd' guard-time='180'/> |
The configuration file is organized by function. The top of the configuration file and in particular the <paths/> section is concerned with describing the architecture of the SNFServer installation.
The <update-script/> feature is a component of the networking section because it is triggered by SNF network operations, so we put it's configuration information in that section.
This feature is still evolving -- in it's original design it was presumed that the update script would reside in the single SNF directory, or perhaps in the workspace directory -- so only the name of the script would be required in this location. We actually have had quite a few successful installations this way.
However, along the way we've determined that the update script might be located anywhere on the system and that we could not always assume the current workspace for SNFServer indicated the location (or even a relative location) for the udpate script.
To prevent errors we've taken to coding the full path to the script in this section of the configuration.
Another part of the thinking on this is that the update-script feature is completely optional. In fact many of the larger systems that we service use entirely separate update mechanisms and turn this feature off. It seemed to make more sense to put the script path closer to the network features that trigger it.
|
> |
Next I had to customize the “getRuleBase.cmd” – because it too does NOT support the use of the rulebase/workspace paths. Here was yet ANOTHER place where I had to manually configure the same path information again, as well as the license key. Needless to say, I’m not a friend of having redundant path information in several locations as this is an unnecessary source of error. |
This is an unfortunate, but necessary requirement. The price of flexibility is complexity.
The previous version of SNF did not offer any ability to move log and other files to alternate locations. This was one of our top feature requests. The new version allows many things to be moved to new locations if needed.
The default installation of SNF is to put all of the files in the same place -- in keeping with the way SNF has always been installed up to now (so that SNF users would be somewhat familiar with it). When SNF is divided into multiple locations there are many models to chose from...
It is likely that the update script might not be the one we've provided and that it might perform a host of other functions that require it to be located explicitly outside of the SNF working or rulebase directories.
The script we provide is only a minimally functioning example that will do the basics for most of our customers (those that would not normally chose to move directories around for example).
You might be surprised to see some of the housekeeping, logging, analysis, reporting, and other functions that are often tied to update mechanisms used by some of our customers. It doesn't make sense for us to assume that every update mechanism must work like our getRulebase.cmd script nor can we make any hard assumptions about directory structures without reducing SNF's flexibility.
Many systems that use SNF do not or cannot follow the tree structure you are using. For example on some linux systems the paths may end up being something like:
Configuration --> /etc/snf
Workspace --> /var/spool/snfilter
Rulebase --> /home/snfilter
Logs --> /var/log/snf
...and so on. You'll note that there are different roots to many of these paths. The SNF/Log, SNF/Workspace, SNF/Rulebase structure is nice (for Windows systems perhaps), but doesn't work for everyone. Since these configurations can vary so widely any assumptions we might make about them would tend to be obstacles. In fact your experience (as described here) proves that since our default installation paradigm did not fit your preferences. The same would be true if we had selected any other paradigm that did not match your preference-- you would have needed to move things around manually as you have ;-)
The <update-script/> feature is designed to work with any update mechanism while providing some isolation between SNFServer and those programs. Our getRulebase scripts are just basic functional examples. When installed automatically they work fine and no further tweaking is required. If they are moved then certainly they must be reconfigured.
When we considered passing configuration information to the update script through this feature we quickly realized that different scripts would have very different requirements and that passing information out of SNFServer to these scripts might represent a security vulnerability in some installations.
We opted for a simpler, more isolated approach that allows the update mechanism (whether a script or an application) to simply get the signal that an update is available. It is then free to take whatever actions it's designers see fit to achieve the update - and the designers are then responsible for providing the configuration information for that mechanism. Here again we decided not make assumptions that might get in the way.
It's worth noting that our windows installer configures all of these paths automatically. In practice most Windows users never need to adjust any of these paths. The only time they would need to be modified is if/when the administrator wants to do something specific that is outside of the "normal" installation.
|
> |
Through testing I determined that some more files had to be moved to certain sub folders for things to work:
UpdateReady.txt -> /Workspace GBUdbIgnoreList.text -> /Workspace Your .SNF -> /Rulebase |
UpdateReady.txt is created automatically by SNFServer when an update has been detected. You don't need to move it. It will be created in the workspace. The only software that reads this file is the update mechanism -- and even that is optional.
In fact, the UpdateReady.txt semaphore file is a parallel mechanism to the <update-script/> feature. Even if the <update-script/> feature is turned off the UpdateReady.txt file will be generated when a newer rulebase file is ready on our servers. As a result it is possible to create an update mechanism that wakes up periodically (a scheduled task) and looks to see if the UpdateReady.txt file is present-- if it is then that mechanism would download the new rulebase and if not it would go back to sleep.
GBUdbIgnoreLst.txt is part of the working configuration. In most cases the configuration is expected to reside in the workspace. This may be rethought as on some systems (particularly *nix systems) configuration files might be put in yet a different path. While the current program allows the main configuration file to be anywhere -- the GBUdbIgnoreList.txt file is expected to be in the workspace. We may provide a separate configuration option indicating where this file can be located.
Of course, the .snf file (the rulebase file) will need to be located in the rulebase directory.
http://www.armresearch.com/support/articles/software/snfServer/config/node/paths/rulebase.jsp
|
> |
Then I had to further adapt the “getRuleBase.cmd” because throughout this procedure, you need to prefix references to the rulebase and the UpdateReady.* files with the appropriate paths for things to actually work. |
Yes. The getRulebase script does need to know where to find things. (See description above -- the update mechanism is kept somewhat independent from the SNF application as it is handled in many different ways on different systems.
|
> |
At this point, I’m still no clear where the mingwm10.dll, exchndl.dll and AuthenticationProtocol.swf need to reside! I didn’t move them, but I’m not sure if that creates a problem down the road. |
The .dll files are part of the program. They need to stay with SNFServer and SNFclient.
The AuthenticationProtocol.swf file is part of the documentation. You could discard it if you wish. It shows how SNF nodes are authenticated when they connect to exchange GBUdb and rulebase activity information.
|
> |
Here are my suggestions:
a) Snf_engine.xml should have one “ApplPath” parameter where I can just define 'D:/IMail/declude/SNF’. Unless I OVERRIDE any of the other paths, it should know the that by default the other “paths” are all assumed to be below the ApplPath and no extra parameters are necessary: identity.xml getRulebase.cmd Log/ Rulebase/ Workspace/
|
We are heading in this direction and in fact the app will largely work this way. We ran into trouble with some installations that moved the SNFServer application or altered it's environment so we decided to make the standard practice to configure full paths for every option.
An upcoming release will resolve these "unusual cases" and allow SNF to find things based on relative paths reliably. In the mean time the best way to avoid trouble is to provide full paths.
|
> |
b) There should be a simple command line utility (e.g., “SNFClient.exe –Paths”) to automatically create Environment Variables for the paths. This way, this command can just be included at the beginning of the getRuleBase script and one doesn’t have to manually hardcode those same paths into yet another location. |
That's an interesting idea though it might be a challenge to come up with a version that would work reliably on all platforms and there would certainly be some security concerns on some platforms.
Keep in mind that the <update-script/> mechanism is designed to work with a wide variety of applications or scripts. In some cases it might not be appropriate to manipulate environment variables with SNFClient.
For example, on *nix systems where security is often more considered, the SNFClient application may be configured so that it is only accessible by the MTA or filter application that will call it.
Permissions for the update mechanism might be very different. On some systems the SNFClient application may not be present at all. At present the SNFClient application (and XCI protocol) does not provide any information about where any of the application's components might be found -- this is actually a design choice for security reasons.
Another option we have considered is providing the <update-script/> mechanism with some scriptable variables that can be emited into the call for example:
call='/path/to/script %rulebase_path% %license% %authentication%'
or perhaps
call='/path/to/script' parm1='RulebasePath' parm2='LicenseId' parm3='Authentication'
That might result in a system() call with the text:
/path/to/script D:/IMail/declude/SNF/Rulebase licensid authenticationxx
We're still considering options.
|
> |
PS: Here is my corrected version of the getRuleBase CMD file that looks for the files in the correct subfolders: |
With all due respect you mean that it is correct for your configuration.
That said the relative path calculations are a nice modification for folks who want to configure SNF on their system with this directory structure.
I recommend moving the LICENSE_ID and AUTHENTICATION to the top of the parameters section. That way only the top 3 lines need to be modified unless the remaining relative paths (RULEBASE_PATH and WORKSPACE_PATH) also need to be changed for a given installation.
Best,
_M
Remainder for reference...
|
> |
@ECHO OFF SETLOCAL
REM ----- Edit This Section --------
SET SNIFFER_PATH=D:\IMail\declude\SNF SET RULEBASE_PATH=%SNIFFER_PATH%\Rulebase SET WORKSPACE_PATH=%SNIFFER_PATH%\Workspace SET AUTHENTICATION=authenticationxx SET LICENSE_ID=licenseid
REM --------------------------------
CD /d %SNIFFER_PATH%
if not exist %WORKSPACE_PATH%\UpdateReady.txt GOTO DONE
REM The next line may cause trouble if your system stops while this REM script is running. It is not needed when this script is run REM from SNF's <update-script/> feature since only one copy will run REM at a time. However, if you are going to run a version of this REM script as a scheduled task you will want to uncomment the next REM line to make sure only one copy runs at a time-- just be sure to REM clean out any stale .lck files after a restart.
REM if exist %WORKSPACE_PATH%\UpdateReady.lck GOTO DONE
:DOWNLOAD
COPY %WORKSPACE_PATH%\UpdateReady.txt %WORKSPACE_PATH%\UpdateReady.lck wget http://www.sortmonster.net/Sniffer/Updates/%LICENSE_ID%.snf -O %RULEBASE_PATH%\%LICENSE_ID%.new.gz --header=Accept-Encoding:gzip --http-user=sniffer --http-passwd=ki11sp8m
if exist %RULEBASE_PATH%\%LICENSE_ID%.new.gz gzip -d -f %RULEBASE_PATH%\%LICENSE_ID%.new.gz
snf2check.exe %RULEBASE_PATH%\%LICENSE_ID%.new %AUTHENTICATION%
if errorlevel 1 goto CLEANUP
if exist %RULEBASE_PATH%\%LICENSE_ID%.old del %RULEBASE_PATH%\%LICENSE_ID%.old
rename %RULEBASE_PATH%\%LICENSE_ID%.snf %LICENSE_ID%.old rename %RULEBASE_PATH%\%LICENSE_ID%.new %LICENSE_ID%.snf
if exist %WORKSPACE_PATH%\UpdateReady.txt del %WORKSPACE_PATH%\UpdateReady.txt if exist %WORKSPACE_PATH%\UpdateReady.lck del %WORKSPACE_PATH%\UpdateReady.lck
:CLEANUP
if exist %RULEBASE_PATH%\%LICENSE_ID%.new del %RULEBASE_PATH%\%LICENSE_ID%.new if exist %WORKSPACE_PATH%\UpdateReady.lck del %WORKSPACE_PATH%\UpdateReady.lck
:DONE
ENDLOCAL |
--
Pete McNeil
Chief Scientist,
Arm Research Labs, LLC.
############################################################# This message is sent to you because you are subscribed to the mailing list <sniffer@sortmonster.com>. To unsubscribe, E-mail to: <[EMAIL PROTECTED]> To switch to the DIGEST mode, E-mail to <[EMAIL PROTECTED]> To switch to the INDEX mode, E-mail to <[EMAIL PROTECTED]> Send administrative queries to <[EMAIL PROTECTED]>
