On 06/11/2013 04:13 AM, Jose-Marcio Martins wrote:
Thanks for the info,
On 06/11/2013 05:26 AM, Amos Jeffries wrote:
On 11/06/2013 9:03 a.m., Jose-Marcio Martins wrote:
When Squid reloads it pauses *everything* it is doing while the reload is
happening.
* 100% of resources get dedicated to the reload ensuring fastest possible
recovery then everything
resumes exactly as before.
OK ! But, IMHO, one doesn't need to reload squid if a helper just needs to
reload its database. It's a design error on huge servers.
There is a big misunderstanding:
in the old days when the only URL filter was squidguard, Squid had the be
reloaded in order for squidguard to reloads its database.
And when Squid reloads, *everything* pauses.
_But things have changed since then_:
- ICAP-based URL filters can reload a URL database without Squid reloading
- ufdbGuard, which is a URL redirector just like squidGuard, can also reload a
URL database without Squid reloading.
The above implies that ICAP-based filters and ufdbGuard are a good alternatives
for squidguard or filtering by ACLs.
When a squid helper reloads it pauses *just* transations which are depending on
it, other
transactions remain processing.
* Some small % of resources get dedicated to the reload.
* each helper instance of that type must do its own reload, multiplying the
work performed during
reload by M times.
OK. So the goal is to MINIMIZE the time taken to reload/reopen databases,
WITHOUT reloading all squid.
This kind of problem arrives on many other kind of online filtering software.
E.g. mail filters (milters, ...).
If the looooong time is the one needed to convert, e.g., a text file into a .db
file, you can do things like the following :
1. move bl.db bl.db.old
2. makedatabase bl.db
3. tell all helpers to reopen database (e.g. using some signal).
On (1.) renaming the file doesn't change file descriptors, so from 1. to 3. the
helper will still use old database. On 3. all helpers will just close the old
database and open the new one. The needed
time to do this is minimal (surely much less than a milisecond). Just lock
database access during reopening database.
If you use in memory databases, you can think the same way, except that you're
using memory pointers instead of file descriptors/database handlers.
Sure, if you have a very big number of helpers, this may be a problem for
memory databases. But in this case, maybe you shall think about why you need so
much helpers. Maybe there are some
optimisation to be done on the programming side, or use some fast disk based
database, or shared memory.
Another situation can arrive when you have different kind of helpers : one kind
doing url filtering and another one doing content filtering (e.g., virus, ...).
So, if when each one need to reload all
squid... it's crazy...
It seems that ICAP allows all this in a cleaner way.
What about multithreading. Which solution can be used on multithreaded helpers ?
ufdbGuard loads the URL database in memory and is multithreaded.
It can do 50,000 URL verifications per second, which is far more than the
number of requests per second that Squid can handle.
ufdbGuard works very well with a large number of helpers (e.g. 64), since the
helpers are lightweight and there is one one physical copy of the URL database
in memory.
Best regards,
Marcus
Thanks for the pointer on hotconf, I'll take a look.
Regards,
José-Marcio
When ICAP reloads it has the option of signalling Squid no more transactions
and completing the
existing ones first, or spawning a new service instance with the new config and
then swapping over
seamlessly.
* the resources of some other server are usually being applied to the problem -
leaving Squid to run
happily
* Squid can failover to other instances of that ICAP service for handling new
transactions.
No matter how you slice it, Squid will eventually need reconfiguring for
something and we come back
to Squid needing to accept new configuration without pausing at all.
There is the "HotConf" project (http://wiki.squid-cache.org/Features/HotConf)
which 3.x releases are
being prepared for through the code cleanup we are doing in the background on
each successive
release. There is also CacheMgr.JS project Kinkie and I have underway to polish
up the manager API,
which will eventually result in some configuration options being configurable
via the web API.
Amos
