Rick Matthews wrote:
The following was also posted to [EMAIL PROTECTED]:Good point
True - but I am betting large hardware and decision caching will
compensate.
Don't forget that those decisions will need to be treated as unique
to source, destination, time, day, and possibly total usage.
Henrik suggested:We are trying to add Value to our service as an ISP. That means we want to share filtering machines with different users or organizations. We do not want to maintain the rule or the lists. We want our users, or their representatives to login to the interface and interact with theirs and the globaly shared config only where apropriate.
I would recommend periodically updating the SquidGuard databases
from MySQL for easier configuration and maintenance, but keep
SquidGuard as it is.
You responded:
I have a hard time seeing that scaling well.
I don't understand your scalability concerns, and maybe that's
because I'm not sure I understand your goal. You say that you
"...are doing this for enterprise scalability reasons", but you also
say "many admins in many orgs" and "many different customers with
different needs". Could you clarify your goals here? What problem(s)
are you addressing?
The webmin module also helps.squidGuard is extremely fast and efficient in its current form. The majority of the problems and complaints that are posted on the squidGuard mailing list can be categorized into three areas:
Installation
------------
The squidGuard documentation is at least one version behind the code.
Following the installation instructions to the letter will result in
a broken squidGuard.
We actualy want to replace the entire configuration aproach with a web front and a SQL back.Configuration ------------- The documentation plays a part in configuration problems, too. It is a good base, but it stops short of providing real-life applications of squidGuard's capabilities and provides little insight to successful configuration and operation.
You could add significant value here by building in operational best practices and by insulating the user from the arcane squidGuard.conf file. This would build in diagnostic processes, and provide a logical approach to problem research. Build in research tools.
I agree. This will have to be done by somebody eventually, maybe even us.
Blacklists
----------
The blacklists are probably the single most critical factor in
deciding squidGuard's success, and yet the users are left floundering.
Add value by providing good blacklists. Provide automated blacklist
updates. Build a tool that will read hundreds of thousands of porn
urls (from the blacklist) and create an expressionlist that can
identify new porn urls with a very high degree of accuracy (without
false positives). Provide an easy process for local blocks and
allows, while still protecting the user from himself.
Once we have to get that deep into it, I think a hack - which the above clearly is - should theoretically be the wrong aproach. What we are looking for will no doubt be anything but simplistic.
I have seen some efforts to this end, but we are talking about
parsing out a SquidGuard config from an sql db - ( not a simplistic
task I would think )
Are you looking for a "simplistic" solution?
New sources, new destinations, new Rules, new times, new redirects, new rewrites. These will all require the master config being reparsed and rebuilt and the squidguards rehupped.
- aside from Berkeley DB updates. Each time this config changes due
to new sources/acl/destination all squidguards need to be hupped.
Now that can be a large hit. As we are trying to take the
configuration to a highly distributed ( many admins in many orgs)
level, this would probaly make the whole thing fall down.
How will the ABC Corporation be affected when the XYZ Corporation
changes their config file?
If you are simply talking about blacklist subscriptions, you'dA diff would work for list maintenance. And yes the hup is fast. But doing a hup for config changes means that it is not wise to download the config changes very frequently.
be much better off using squidGuard/Berkeley as the engine,
and distributing squidGuard-style diff files. (Restarting a
squidGuard that is using pre-built Berkeley db files takes seconds
[my p200 does it in 5 seconds])
That would be a good thing. Anyways, based on the feedback I am getting, it looks like we may have to go back to the drawing board.There are quite a few things that you could do to improve squidGuard and make it a better commercial solution. Rick Matthews
-----Original Message----- From: Joe Maimon [mailto:[EMAIL PROTECTED] Sent: Monday, March 03, 2003 8:09 AM To: Henrik Nordstrom Cc: [EMAIL PROTECTED] Subject: Re: [squid-users] Extending SquidGuard to work out of a SQL database
Henrik Nordstrom wrote:
m�n 2003-03-03 klockan 13.41 skrev Joe Maimon:True - but I am betting large hardware and decsion caching will compensate.
My company is looking to extend the Squid redirector, SquidGuard to work realtime out of a SQL database. We are looking to target the GNU/Linux environment and the MySql database server.
Hmm.. I would be a little worried about the latency of using MySQL in the mix.. it is very hard to beat BerkelyDB when it comes to read latency of static data..
I would recommend periodically updating the SquidGuard databases from MySQL for easier configuration and maintenance, but keep SquidGuard as it is.
I have a hard time seeing that scaling well. I have seen some efforts to this end, but we are talking about parsing out a SquidGuard config from an sql db - ( not a simplistic task I would think ) - aside from Berkely DB updates. Each time this config changes due to new sources/acl/destination all squidguards need to be hupped. Now that can be a large hit. As we are trying to take the configuration to a highly distributed ( many admins in many orgs) level, this would probaly make the whole thing fall down. And users hate latency on config changes.
I actualy practice this concept for my named.conf files. It wasn't simple for that and lost some flexibility there as well.
That being said, your advice merits some consideration.
Regards Henrik
