Re: [squid-dev] Benchmarking Performance with reuseport
tor 2016-08-04 klockan 23:12 +1200 skrev Amos Jeffries: > > > I imagine that Nginx are seeing latency reduction due to no longer > needing a central worker that receives the connection then spawns a > whole new process to handle it. The behaviour sort of makes sense for > a > web server (which Nginx is at heart still, a copy of Apache) spawning > CGI processes to handle each request. But kind of daft in these > HTTP/1.1 > multiplexed performance-centric days. No, it's only about accepting new connections on existing workers. Many high load sites still run with non-persistent connections to keep worker count down, and these benefit a lot from this change. Sites using persistent connections only benefit marginally. But the larger the worker count the higher the benefit as the load from new connections gets distrubuted by the kernel instead of a stamping herd of workers. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] [PATCH] Make Squid death due to overloaded helpers optional
tis 2016-08-09 klockan 11:47 -0600 skrev Alex Rousskov: > > Yep, that matches both my understanding and motivation to return ERR > in > the explicitly configured on-persistent-overload=err case. I'd say make it configurable. on-persistent-overload=abort/err/ok/unknown to represent all three possible states. But that is based on the assumption that unknown is "outcome is not known" and that Squid then skips the access rule and looks at next rule. Have not looked in detail in how current Squid operated with these conditions. Starting with only abort/err is good. err should then map to the same as a negative helper response which is the most clear buldingblock to work with then bulding access lists. It does not make sense that err in this context should map to different logics depending on the directive. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] HTTP meetup in Stockholm
tis 2016-07-12 klockan 18:34 +1200 skrev Amos Jeffries: > I'm much more in favour of binary formats. The HTTP/2 HPACK design > lends > itself very easily to binary header values (ie sending integers as > interger encoded value). Following PHK's lead on those. json is very ambiguous with no defined schema or type restrictions. It's up to the receiver to guess type information from format while parsing, which in itself is a mess from security point of view. The beauty of json is that it is trivially extensible with new data, and have all basic data constructs you need for arbitrary data. (name tagged, strings, integers, floats, booleans, arrays, dictionaries and maybe something more). But for the same reason it's also unsuitable for HTTP header information which should be consise, terse and un-ambiguous with little room for syntax errors. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] HTTP meetup in Stockholm
sön 2016-07-10 klockan 11:33 +0100 skrev Kinkie: > Hi all, > at the end of the month I will attend the HTTP meetup in Stockholm. > Besides having a chance to see Henrik, I'd like to collect your > feedback and opinions on the topic that are likely to be touched. Oh, This have completely bypassed my radar, most likely beacause it has been practically turned off for a long while. Any reference to more details? I the only near time upcoming HTTP activity I can find is in Berlin the upcoming weekend as part of IETF96. Note that I live a bit outside of Stockholm these days.. about 2 hours by car or train to Surahammar which is close to Västerås. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] Digest related question.
mån 2015-03-16 klockan 04:03 +0200 skrev Eliezer Croitoru: My main concern until now is that if squid would have the cached object in a digest url form such as: http://digest.squid.internal/MD5/xyz123; Squid would in many cases try to verify against the origin server that the cached object has the same ETAG and MODIFICATION time. The Digest alone is only on the body, and says nothing about header authority. You need to get trusted object headers from somewhere else, i.e. the requested origin. Once you have the authoritative headers you can splice in the digest verified response body. This is in some sense similar to the header merging needed in ETag based variant handling on a single URL, but even more so as you must not take headers from one random URL and apply them to another requested URL without permission unless the requested URL permits this. Violating this opens a range of security concerns where headers may be injected giving a different result than intended by the origin. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] [RFC] Squid 4.0 ideas
tis 2015-03-10 klockan 23:58 +1300 skrev Amos Jeffries: There is approximately 8-18 months year between series releases now. A very arbitrary choice whenever it seems reasonable to release a batch of features. My undertanding of the proposal stated was that we keep that current practice, but dropping the 2-part 3.x numeric down to a single part. Yes, except that we should aim in decreasing release cycle a bit. 18 months is not acceptable. imho 6-12 months is a more reasonable cycle. Only longer if there have been no meaningful new features matured in last 6-10 months. Its already clear from past discussions that our views of feature are very different. If we bumped the version with each new thing ... patch releases should be just patch releases. No config changes unless absolutely required due to security issue. version number is bumped after each release with new features. Whatever features gets finished tested in time gets included in next release. One new feature is sufficient for a new release if we want. There should be no need to wait for other features to finish testing. Some features likely will need several releases before becoming user visible, where infrastructure plumbing goes into the first one or two releases before feature is ready. ... last month we had non-trivial feature changes (100+ Lines of code) on the 26th, 22nd, and some lesser (25+ LOC) but still widely user visible new thing on the 11th and 1st, and large-ish but not user visible feature changes across the 1st-5th and 19th. That would be 4-8 major releases of Squid within the month of Feb 2015. There is obviously no need to increase version in trunk/master for every change, just every release we make with new features compared to previous release. I'm not searching for excuses. There is a meta-pattern of previous major numbers changing roughly when the language is changed. Roughly because it is +/- a few years and the next new language this time around can build anything 3.3+ AFAIK, while the old language can't build new tarballs - and there is precedence in the last changover there too with C++ tools building C code fine but _not_ vice-versa. The compiler library requirements is continuously increased slowly, and is fully acceptable as long as we stay within our supported range of operating systems. No need for a major version to indicate this. The Squid-2 - Squid-3 bump was a major rewrite over many years. In many ways a failed project. Lets not repeat that. The development model that works for us is incremental development model in smallish steps. I argue that we are not in a position to take on another major rewrite. I'm seeking opinions. On whether we continue with that pattern in conjunction with the coming trunk activity, or discard it in favour of something else - even if that something else is slipping down the slope towards an eventual 3.999 series (haven't heard anyone advocating for that though). I am advocating for 1234.5. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [squid-dev] [RFC] Squid 4.0 ideas
sön 2015-03-08 klockan 07:49 +0100 skrev Kinkie: Beta releases can be managed just like they are now: MAJ.0.X would be beta MAJ.Y (Y=1) would be stable Correct. Today we have release numbers that is made of major.minor (2.5, 3.0, 3,1.., 3.5), followed by patchlevel where beta releases are 0.betanumber, the proposed versioning change is to replace the release number with a single number, i.e, 4, 5, 6, ..., The rest of our version numbering do not need any changes. So first Squid-4 beta release becomes 4.0.1, second 4.0.2, first production release 4.1, first patch release 4.2, etc. Regards Henrik ___ squid-dev mailing list squid-dev@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-dev
Re: [RFC] bandwidth savigns via header eliding
fre 2014-07-25 klockan 19:05 +0300 skrev Eliezer Croitoru: The response to alex question why would anybody want to drop cteonnt-length: header: Some places do not allow cookies or POST for external services and it's sometimes can looks weird but I still understand why would it be considered a security hole by some minds. Dropping mangled connection-length header is not about security. It's no more than a garbage header carrying no meaning other than an distant echo of it's original form. It is transformed in this manner to avoid being read as connection-length while doing a minimal lightweight rewrite of the TCP/IP payload. Bandwidth saving from dropping this header will be close to unmeasurable. Security impact likewise. But sure, if you have a whitelist policy of only allowing what is explicitly allowed then it would be dropped by the catch-all DROP policy. But no hardwiring in our code is needed for that. The discussion about mangled Connection header may be more interesting, but only if there is bugs in the software that mangled the Connection header leaving what was intended as hop-by-hop headers unmangled. But I don't see much of a point of worrying about that until there is indication that there is problems caused by such headers. From the referenced discussion it's quite clear this rewrite practice is limited to one cache appliance vendor. It is not likely to be implemented by others. Regards Henrik
Re: [RFC] bandwidth savigns via header eliding
lör 2014-07-19 klockan 11:35 -0600 skrev Alex Rousskov: The above email is talking about a nnCoection: close header which appears to be a result of a bug in some 15-year old software. Identifying that rare header would be overall harmful -- Squid would spend more resources on detecting that header presence than it will save by removing that header when it is found. Indeed a valid question. HTTP divides headers in two groups. end-to-end and hop-by-hop. Assuming we intend to operate as a semantically transparent HTTP proxy (meaing one that do not change the meaning of requests/responses) then we MUST NOT touch end-to-end headers. The list of hop-by-hop headers is very short. Anything not in that list is by definition end-to-end unless mentioned in Connetion. There are some gray areas like defense against request smuggling, but even there extreme care should be taken to avoid harming valid HTTP traffic. It is certainly not the area of delete all bad headers in the parser solutions. request/response smuggling is different. request/response smuggling uses incorrectly/ambigiously formatted HTTP to try to make different agents parse the data stream differently. As for bandwidth savings then dropping considered useless headers will do very little on bandwidth savings and risk cause a lot of grief. Much higher bandwidth savings can be acheived by header compression using a dictionary, but that requires supporting agents on both sides of the link. Regards Henrik
Re: [RFC] unified port directive
tor 2014-06-12 klockan 21:01 +1200 skrev Amos Jeffries: When FTP transfer protocol is added is the time to determine if the fpt_ prefix is actually necesary or the *existing* protocol= parameter can be used alone. FWIW the only strong reason http_ and https_ are currently used is to determine which of the split config option lists to add the PortCfg to. And this proposed step removes that need. protocol= means a different thing in current definition; what to use as protocol part in reconstructed URLs. Regards Henrik
Re: [RFC] flag day commits
sön 2014-06-08 klockan 23:05 +1200 skrev Amos Jeffries: When a change can it should avoid being a flag-day. But if breaking into a set of smaller incremental changes is difficult or causes too much annoyance from repeated painful merges or wastes audit time on other patches, then it seems better to end the ongoing pain with a flag-day commit. Breaking a flag-day commit into smaller changes do not avoid it being a flag-day commit. This type of changes has to be viewed in it's entire scope, not individual commits. The exception is if the change can be done in two steps where the first step prepares for the change and allows new code to use new agreed style, leaving the change of existing code for flag-day commit. But only if there is some users of the new way. This gives the flow 1. Preparation change 2. First user(s) 3. Cleanup of old code to use the new way. 4. Removal of any glue supporting old code. There 1 2 can be committed during normal feature window, while 3 should wait until the next flag-day window, and 4 might even be requested to wait one cycle longer. If possible it's preferred if 3 can be scripted. I would also recommend that 4 is from start guarded by a define in step 1, allowing easy compile time check that code is really using the new style. * From the release perspective the week or two prior to branching trunk seems to be the best time for these to occur. During that period new feature and logic commits to trunk are usually heldup in audit we are trying to ensure a stable first beta. We also have minimal pressure/need for back-porting patches across the change to already stable branches by then. There should be sufficient cool-down period between these and first beta. Technically the flag day do not need to be strictly connected to release cycle. Might be anytime while trunk is open. * audit requirements are small. Just checking the patch is restricted to pre-agreed painful act(s) and has no other code changes. The change should have gone thru full review prior to being accepted as a flag-day change. I also propose that this type of patch audit submission be accompanied by clear how-to steps assisting porters of older code to get over the bump. Yes. NP: if we can't provide a simple how-to then re-consider the patch suitability for being a flag-day change. Yes. Regards Henrik
https_port CONNECT
It looks like Mozilla is finally adding support for TLS encrypted proxy connections. So if we still have the problem with CONNECT not properly handling TLS then it might we a good idea to finally fix that. Regards Henrik
Re: How long is a domain or url can be?
ons 2014-04-30 klockan 06:22 +1200 skrev Amos Jeffries: HTTP defines no limit. - squid defines MAX_URL of 8KB, along with a header line limit of 64KB total, and a helper line limit of 32KB total. Unless it has been fixed the UFS based stores also have an implicit limit on cached entries somewhat less than 4KB (whole meta header need to fit in first 4KB). Entries failing this gets cached but can never get hit. DNS defines X.Y.Z segments as being no longer than 255 bytes *each*. For Internet host names the limits are 63 octets per label, and 255 octects in total including dot delimiters. Regards Henrik
Re: [RFC] use libnettle for crypto
tis 2014-03-11 klockan 09:53 +0200 skrev Tsantilas Christos: No objection, just to note that these algorithms are implemented in openssl library too and squid already has dependencies to openSSL. And in Squid-2 used OpenSSL MD5 if linked to OpenSSL, or some system MD5 implementation (i.e. Solaris). From squid-2 squid_md5.h: /* * If Squid is compiled with OpenSSL then we use the MD5 routines * from there via some wrapper macros, and the rest of this file is ignored.. */ #if USE_OPENSSL HAVE_OPENSSL_MD5_H #define USE_SQUID_MD5 0 #include openssl/md5.h /* Hack to adopt Squid to the OpenSSL syntax */ #define SQUID_MD5_DIGEST_LENGTH MD5_DIGEST_LENGTH #define SQUID_MD5Init MD5_Init #define SQUID_MD5Update MD5_Update #define SQUID_MD5Final MD5_Final #define SQUID_MD5_CTX MD5_CTX #elif USE_OPENSSL !HAVE_OPENSSL_MD5_H #error Cannot find OpenSSL MD5 headers #elif (HAVE_SYS_MD5_H || HAVE_MD5_H) HAVE_MD5INIT /* * Solaris 10 provides MD5 as part of the system. */ #if HAVE_MD5_H #include md5.h #else #include sys/md5.h #endif /* * They also define SQUID_MD5_CTX with different field names * fortunately we do not access it directly in the squid code. */ #define SQUID_MD5Init MD5Init #define SQUID_MD5Update MD5Update #define SQUID_MD5Final MD5Final #define SQUID_MD5_CTX MD5_CTX #ifdef MD5_DIGEST_LENGTH #define SQUID_MD5_DIGEST_LENGTH MD5_DIGEST_LENGTH #else #define SQUID_MD5_DIGEST_LENGTH 16 #endif #else /* No system MD5 code found */ The API provided by nettle is very similar to openSSL related API. Which is very similar to the reference implementation shipped with Squid. Regards Henrik
Re: Fwd: [CTTE #727708] Default init system for Debian
tor 2014-02-13 klockan 09:43 -0700 skrev Alex Rousskov: I am not sure which approach is the best, but I am sure we need to clearly define and document the new semantics to avoid repeating -N problems which were caused primarily by folks (including myself) interpreting what -N means differently and changing the code according to their own misguided interpretation. For example, we can define the following three modes: daemon -- Master process forks and uses setsid (what else?). No other effects on Squid. For example, SMP works by forking kid processes as needed. Meant for stand-alone execution. This is the default (enabled when no related command line options are given). yes. no daemon -- Daemonization steps listed in the above paragraph are disabled. Thus, the master process does not fork and does not use setsid (what else?). No other effects on Squid. For example, SMP works by forking kid processes as needed. Meant for systemd and such. In my mind this is what -N is meant to be. no forking -- Squid runs a single Squid process, which works both as a master process and the only worker process. No kid processes, no SMP. Meant for debugging and triage. Exactly. no forking is more of a debug mode. Maybe time to add a --debug=... option to set various debug flags? --debug=nofork And also merge the -X flag into this.
Re: Fwd: [CTTE #727708] Default init system for Debian
ons 2014-02-12 klockan 10:13 +1300 skrev Amos Jeffries: Sigh. Amos Why sigh? systemd have worked well for me for years now.. except Squid.. For Squid the main impact is that we really should have a dont daemonize yourself run mode which don't otherwise limit functionality. The -N option we have today is a little overloaded. systemd just as most other actual system service monitors/managers really want system service processes that don't try to detach themselves from their parent. Regards Henrik
Re: Fwd: [CTTE #727708] Default init system for Debian
ons 2014-02-12 klockan 10:47 +1300 skrev Amos Jeffries:
Exactly. More deep changes needed.
Well, systemd (and most other service monitors) do manage services that
background themselves as well, just not optimally. So it's more changes
desired, not strictly needed.
But seriously, it's a very small change to Squid. All systemd and the
like needs is that Squid stops all the backgrounding, setsid etc done in
watch_child(). The rest is business as usual, including watching childs.
I.e. all or most of the following:
openlog(APP_SHORTNAME, LOG_PID | LOG_NDELAY | LOG_CONS, LOG_LOCAL4);
if ((pid = fork()) 0)
syslog(LOG_ALERT, fork failed: %s, xstrerror());
else if (pid 0)
exit(0);
if (setsid() 0)
syslog(LOG_ALERT, setsid failed: %s, xstrerror());
closelog();
#ifdef TIOCNOTTY
if ((i = open(/dev/tty, O_RDWR | O_TEXT)) = 0) {
ioctl(i, TIOCNOTTY, NULL);
close(i);
}
#endif
/*
* RBCOLLINS - if cygwin stackdumps when squid is run without
* -N, check the cygwin1.dll version, it needs to be AT LEAST
* 1.1.3. execvp had a bit overflow error in a loop..
*/
/* Connect stdio to /dev/null in daemon mode */
nullfd = open(_PATH_DEVNULL, O_RDWR | O_TEXT);
if (nullfd 0)
fatalf(_PATH_DEVNULL %s\n, xstrerror());
dup2(nullfd, 0);
if (Debug::log_stderr 0) {
dup2(nullfd, 1);
dup2(nullfd, 2);
}
Note: Looking at this code I notice that the nullfd is leaked, consuming
one more filedescriptor than needed. There should be a
close(nullfd);
somewhere around here..
for now probably best by using yet another command line option. The
opt_no_daemon flag (-N) is far too overloaded and used at a bit too many
places today and trying to sort that out is likely to give some more
gray hairs.
Regards
Henrik
Re: Fwd: HTTP/1.1 revision status [was: HTTP/1.1bis draft -26]
mån 2014-02-10 klockan 21:48 +1300 skrev Amos Jeffries: FYI for anyone interested in the HTTP/1.1 compliance project or related work. The HTTPbis drafts are now etched in stone. You can find links to the HTTPbis work on 1.1 and 2.0 protocols here: http://trac.tools.ietf.org/wg/httpbis/trac/wiki Is the HTTP/2.0 work also finished? Or is that still work in progress? Have any analysis been made on the impact of HTTP/1.1bis and maybe even HTTP/2.0 on Squid code? Regards Henrik
Re: [RFC] Squid process model and service name impact
ons 2014-01-29 klockan 13:29 +1300 skrev Amos Jeffries:
hard-coded is being used a bit much here. I don't think any of us are
arguing for that.
Lets look at the BNF:
FOO.pid = chroot pid_filename
chroot = {squid.conf chroot} | /
pid_filename = {squid.conf pid_filename} | DEFAULT_PID_FILE
DEFAULT_PID_FILE = {./configure --with-pidfile} | ( PREFIX /squid.pid
)
PREFIX = {./configure --prefix} | /
My veiw is just a default squid.conf setting of
pid_filename @DEFAULT_PID_DIRECTORY@/${service_name}.pid
Where DEFAULT_PID_DIRECTORY is defined when running configure, and
${service_name} is the macro expanding servicename.
The rest of the configuration points you talk about is just different
ways of telling configure what the DEFAULT_PID_DIRECTORY path is.
- explicit
- derived from prefix
- whatever other means the user selects
and seriously, none of that matters for this discussion imho.
There should be NO special cases for pid_filename in the code. Just use
the configured path with suitable macro expansions to find the pid path.
We have no less than 4 configuration points for this one file, some of
which replace others , and some join unless other has been configured. I
am proposing we can do with a simpler setup:
FOO.pid = chroot DEFAULT_PID_PATH / service_name .pid
chroot = {squid.conf chroot} | /
service_name = {squid -n FOO} | squid
DEFAULT_PID_PATH = {./configure --with-pidfile} | PREFIX
PREFIX = {./configure --prefix} | /
Most of the configuration is in path. The file name part is static but
configurable via command line with less of the per-worker problems
brought in by squid.conf flexibility (chroot still suffers those).
Problem above is that your DEFAULT_PID_PATH is static, set at configure
time. i.e. hardcoded at build time.
Benefits:
* multiple instances with same ./configure can run without clashing on
.pid
* multiple instances with default squid.conf can run without clashing
on .pid
* base path is configurable by user from squid.conf (via chroot)
* default path is configurable by distributor (via --prefix or
--with-pidfile)
* script/code running Squid is where -n gets set, so if it needs access
to foo.pid or test.pid separately it already knows whether it ran -n
foo or -n test.
Same with having a suitable default pid_filename setting using
service_name macro, but giving users full flexibility to change to
whatever scheme they prefer even after running configure.
If people do stupid things in squid.conf sure they will shoot themselves
in the foot. Do we need to care? Having a sane and documented default
setting is sufficient imho.
So why do we need to configure pid_filename exactly?
Does a user who is running under their local directory care whether its
called squid.pid or squid3.pid or test.pid with all the rest of the
process overlaps being collisions? I don't think so.
A user wanting to run their own instance without being able to recompile
Squid from source does care.
Same reasoning applies to cache_dir paths etc. Why do we need these
configurable in squid.conf? To me it's as obvious that the pid filename
path needs to be fully configurable in squid.conf just as any cache_dir
path needs to be.
The chroot directive is about another thing entirely, and do not even
apply in the use case discussed. Use of chroot requires full control of
the system where Squid runs as Squid MUST be started as root for chroot
to work.
Perhapse we need to look into providing a lightweight jail/location
directive to solve that use-case rather than forcing admin to know and
configure each component path individually.
I don't see it as a problem. All default paths are visible in
squid.conf.documented, and there is clearly visible error messages if
one forgets one.
* When the admin is running a single instance of Squid with default
locations for their build. pid_filename is not needed, and not used.
There is always a built-in default.
* When the admin is running in a different setup to the build defaults.
pid_filename is a must, and configured. The problem is that must
configure, alongside all other path directives. Squid _works_ but with a
lot of settings that really should not be mandatory / separate just to
change location of the entire instance.
Which boils down to providing a settable prefix macro/variable in
squid.conf and have the relevant directives using that in their default
value.
I.e. something like
set name value
to define new squid.conf macros/variables that can be used in following
directives.
Issue with this is how to handle default values with such expansions.
The default squid.conf settings are parsed before squid.conf if I
remember correctly, which means their macro expansions can't rely on
variables set in squid.conf.
service name is from command line and works.
adding a command line option for prefix? Works.
But do we really want easy relocation by a
Re: [RFC] Squid process model and the need for -n
tis 2014-01-28 klockan 11:25 -0700 skrev Alex Rousskov: Why do we need a command line option to specify a service name? Can the service name be configured via squid.conf? The squid.conf location is already configurable via command line, of course. If I remember correctly the service name is used for deriving squid.conf path. But have not looked at Squid as a Windows service in ages. Memory is a bit dim. Regards Henrik
Re: [RFC] Squid process model and the need for -n
tis 2014-01-28 klockan 12:15 -0700 skrev Alex Rousskov: but squid.conf location is already configurable via the command line. Why another option, that is now propagated to non-Windows builds, without even defining what service is in non-Windows context (AFAIK). If I am not mistaken the service name is passed via other means when running as a service (service API call), and command line when started from command line. Regards Henrik
Re: [RFC] Squid process model and service name impact
sön 2014-01-26 klockan 13:59 +1300 skrev Amos Jeffries:
So how do we safely allow users to configure where the IPC sockets are
located on a per-worker basis while also allowing workers to communicate
over them to each other?
Please elaborate.
How do using a service name for these differ from having a squid.conf
set the name (possibly using the same service name as a macro
expansion)?
Ditto for shared memory spaces.
Named shared memory spaces need consistent naming. Having a squid.conf
directive with a macro expansion using the service name is suitable
there as well.
I am a little doubtful that we should be letting the PID file remain
configurable, due to the same issue.
Please enlight me on what the issue really is here.
We also seem to have a set of
signalling issues due to that being reconfigured or created late in the
startup process.
Then we have a broken process model somewhere.
It seems to me the only reason that is being configured
at all is to get around this same (bug 3608) problem of instance
identification,
Not at all, not for me at least. It's about being able to relocate the
Squid installation, i.e. for running it in a user home directory without
needing to rebuild Squid from source each time you change location.
but in the context of signalling to one of multiple
instances.
Signalling to a multi-instance Squid installation have not been a
problem before, you just need to remember to give the right squid.conf
when signalling so it knows the right pid filename.
The service name was added to Windows apparently to solve the
same problem there where PID file are not used.
And using it as one possible macro/variable expansion in squid.conf
makes great sense.
Having instance dependent stuff hardcoded in the binary and not being
able to set them in squid.conf is just wrong, even if derived somehow
magically from a service name by some means that someone thought was the
right.
FYI: The mention of other directives and uniqueness was not intended to
mean hard-coding that uniqueness. Just to highlight that it needed a
solution as well. Which turns out to probably be ${service_name} macro
suggested by Alex.
Good.
Regards
Henrik
Re: [RFC] Squid process model and service name impact
fre 2014-01-24 klockan 08:57 -0700 skrev Alex Rousskov:
Please note that since Windows admins want a service name even if they
do not run multiple single-build instances on the same box, setting that
service name should not suddenly make their life more difficult by
altering squid.conf location and such. You can make exceptions for
default service name such as squid, but it may get pretty messy
pretty quickly.
Do they really want the service name to have any other implications?
The only implicatin I might want to tie to the service name is the
default squid.conf name, but we do have command line option for
squid.conf name as well and think it is just more confusing than it adds
to automatically derive some squid.conf name from the service name. But
if Windows services only have the service name and no other parameters
available when run as a service then this may be needed. Was too long
since I dealt with Windows services to remember how they define what and
how to run.
As an alternative, you can restrict the scope to something like this:
All --prefix dependent names not already configurable via squid.conf or
command line that may lead to a conflict between multiple concurrent
instances of a single Squid build.
Do we have any?
I am thinking we should not touch anything that is already configurable
via squid.conf or command line options.
Same here.
pid_filename is similar to http_port, cache_dir, and probably other
options. IMO, those few folks that want to run multiple instances of the
same Squid build should be willing to pay the price of carefully
configuring each instance.
Yes.
BTW, you may want to add a ${service_name} macro support to squid.conf
parser to make multi-service configuration easier.
Right, that is an excellent idea. And if that also applies to include
then the first part mentioned above is solved nicely by having the main
squid.conf include service dependent parts which might in fact be the
entire squid.conf for that service if so desired.
Regards
Henrik
Re: [RFC] Squid process model and service name impact
fre 2014-01-24 klockan 14:27 -0700 skrev Alex Rousskov: using the same Squid build (bug 3608). That bug report contains a suggestion to make the currently ./configure-set IPC paths explicitly configurable in squid.conf, but using service name side-effects also works. I am not sure which approach offers the best overall solution. Everything should be set via squid.conf imho. Regards Henrik
Re: BZR local server?repo?
tor 2014-01-23 klockan 09:53 +0200 skrev Eliezer Croitoru: Since I do have a local server I want to have an up-to-date bzr replica. I can just use checkout or whatever but I want it to be be updated etc. I am no bzr expert so any help about the subject is more then welcome. What I do is that I set up a shared bzr repository that collects the branches I want to monitor/backup, then have a cron job to update branches in there from their source repositories. bzr init-repo --no-trees /path/to/shared/repo cd /path/to/shared/repo bzr branch remote_url local_branch_name bzr branch ... [repeat per barach to mirror] then a cron job that runs bzr pull --overwrite in each branch to keep them updated The reason for --overwrite is to handle if/when history of the master repo is tweaked... This is optional, and without --overwrite you will need to manually recover the mirroring in such events, which is also a good thing as it alerts when something bad is done to the mirrored repository history. The reason for --no-trees is to be able to also push working branches to the same repository for backup purposes. And don't really need checked out copies of the sources of each branch on the server If server side checkouts is needed then it's easily created separately bzr checkout --lightweight /path/to/shared/repo/branch --lightweight is entierly optional, and depends on what you want to use the checkout for. Regards Henrik
Re: [PATCH] client-side redesign pt1 - Comm::TcpReceiver
tis 2014-01-21 klockan 22:45 -0700 skrev Alex Rousskov: All the TCP clients and servers you are willing to include (as future TcpReceiver kids) in the current project scope have at least one thing in common -- they all read and write protocol messages over [persistent] TCP connections. Why do you think it is a good idea to focus on just the reading part so much? If you want a common parent for all those agents, why not handle both actions in that common parent, especially if you already find it necessary to add a little bit of send part into the TcpReceiver? I haven't looked in detail at what is being done, but also keep in mind that if you layer TLS ontop of a TCP connection then the TLS layer needs both read write events to the TCP connection. From a code design perspective it should be the same interface for application protocol data for both TCP and TLS. For some food of thought I would throw SCTP into that mix as well, which adds another layer ontop of the socket to track which SCTP channel the application level connection runs on. So we might have aplication / TLS / SCTP channel / SCTP(protocol). Any design that tries to make application level code (i.e. http/ftp protocol handlers etc) needing to be aware of TcpReceiver/Sender is plain wrong imho. If tomorrow we have both TcpReceiver and TcpSender, and all their kids have both classes as parents, with complex inter-parent dependencies that are already showing in TcpReceiver, we are doing something wrong. We should only have channels and listeners at application protocol level code (HTTP, ftp, whatever). To most code it does not matter what transport these use. Details of each transport implementation is more free. Not so much need to argue on details there. But should not be visible in other code. It only exists because tunnel.cc takes over the FD. Really nasty (the dropped data race condition). yes, CONNECT requests will be a mess. But as above, should take over the connection. Any application payload already read need to be either handed back to the connection or given to the CONNECT forwarder. Let's call this method stopReadingXXX() so that it is clear that any caller is doing it wrong and to discourage use. Yes. What is needed is to stop accepting pipelined requests after a CONNECT. Stopping reading is justa brute-force way of accomplishing that. I do not know or do not remember, sorry. The whole delay pools cannot deal with single-byte buffers is a delay pools bug, outside this project scope. I think it's an ancient parser issue which hopefully is long since fixed.
Re: [RFC] FTP gw source layout
tis 2014-01-21 klockan 14:37 -0700 skrev Alex Rousskov: [FWIW, the term FTP Gateway was suggested by Henrik during initial RFC review. Henrik thought that using HTTP semantics internally means we are a gateway. I changed the project name in order to avoid having an argument. Technically, it is not really clear whether we are using HTTP semantics internally or not. We are using HTTP structures, but semantics is a very gray area IMO.] The old code is a gateway, translating between protocol. If you are just piggy-backing on internal http structures to transparently carry FTP semantics without actually being able to translate protocols then it's not a gateway only a relay. Regards henrik
Re: [RFC] FTP gw source layout
mån 2014-01-20 klockan 11:45 -0700 skrev Alex Rousskov: FTP gateway[1] code[2] work well, and we are polishing it for the official submission. The biggest change we need to make is to rearrange where the new code lives in Squid src directory. For the record I have no opinion on either of the proposed layouts, except that sticking multiple server implementations in client_side.c is certainly not the right thing. Pick one and stick to it. But please account for adjusting other protocols to the same layout. Regards Henrik
Re: [RFC] Automating private cache mode in Squid
tis 2013-08-13 klockan 16:01 -0600 skrev Alex Rousskov: the urltag approach used in squid-2 could be used here as general mechanism. It allows http_port or external_acl to add a piece to the object key. This allows splitting cache based on - port received on - username - arbitrary group of IP addresses - anything else that you can come up with in an external acl Can Store ID be used for this? If the Store ID helper has access to receiving port, user name, IP addresses, etc, the helper can manipulate the store key as needed without any Squid modifications, right? Yes. And if it can receive the http_port name then it fully replaces urltag from squid-2 in functionality. Regards Henrik
Re: [RFC] Automating private cache mode in Squid
ons 2013-08-07 klockan 16:51 +1200 skrev Amos Jeffries: My previous thoughts around this have all revolved around the idea of adding the clients IP address to the cache key when private proxy mode is enabled. This has a major issue though in that it does not solve anything for users with multiple devices (and thus multiple IPs) or even devices using IPv4/IPv6 interchangeably (and thus multiple IPs even for localhost traffic). the urltag approach used in squid-2 could be used here as general mechanism. It allows http_port or external_acl to add a piece to the object key. This allows splitting cache based on - port received on - username - arbitrary group of IP addresses - anything else that you can come up with in an external acl Regards Henrik
Re: Windows compilation error
Found this expired in the list moderation queue. Sorry, can't really help much with mingw build environment issues but maybe someone on the list knows? or at least what mingw installation documentation to read? ons 2013-05-15 klockan 14:25 +0100 skrev Srini: Hi, I am trying to compile Squid 3.3.4 on Windows following the instructions found in the following page: http://wiki.squid-cache.org/KnowledgeBase/Windows#Compiling_with_MinGW I was asked to download OpenSSL, libcrypt, TinyCOBOL. The page apparently does mention anything about what to do with the downloaded stuff. I assume that I should copy them to c:\mingw\inclides and lib. However after copying the lib and includes I tried running the ./configure from a MinGW Shell. I get the following error: checking whether to build shared libraries... yes checking whether to build static libraries... yes checking how to run the C++ preprocessor... /lib/cpp configure: error: in `/c/Users/srini/SoftwareInstallers/Windows/SquidCompilation /squid-3.3.4': configure: error: C++ preprocessor /lib/cpp fails sanity check See `config.log' for more details I have also attached the config.log. I have also tried creating a softlink to the C:\MinGW\bin\cpp.exe in /lib from the MinGW Shell Please help me in getting this compiled. Please let me know if you need more information in this regard. Thanks Srini
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 14:20 -0600 skrev Alex Rousskov: The variant of an URL is identified by it's ETag value. Are ETags required by HTTP in a Vary context? Or is it just what you see some origin servers implementing? To identify the variant yes. Without ETag the variant have no identiy. Variants with no identify can not be shared among different request header combinations (unless Key is implemented as well). Caches uses If-None-Match on unknown requests to ask which variant among the known set of cached variants is the right response for this request. The result of an If-None-Match is a 304 listing the ETag of the variant to respond with + updated response headers. What you end up with is a map of * Request headers as indicated by Vary * Which variant (etag value) is the right response for the matching request. * Possibly updated headers for the response (Date, Last-Modied, Set-Cookie, X-Whatever) carried in a 304 response to If-None-Match. Can you express the above in an n:m map form you mentioned earlier? What maps to what? In the ideal world it is an N:1 map where multiple requests maps to the same variant. Basic mechanism: Vary indicated request headers maps to ETag ETag maps to (or is) variant identifier. If there is no ETag then Request headers maps to variant directly. The list of known responses MAY have different set of Vary headers making the same request match multiple entries. In a distributed cache you may also run into multiple differently aged entries with different outcome. In squid-2 there is a special case extension of this to work around mod_deflate and a couple of others mismanaging ETag. When this gets triggered the variant identifier is extended with Accept-Encoding related details to separete gzip from non-gzip variants even when they both have the same ETag. I suspect your n:m map is a side effect of adding ETag optimization to straight Vary support. In the straight implementation, there is no If-None-Match optimization on top, just a hit (if Vary-controlled headers match one of the fully cached responses) or a miss (if they do not). It's sufficient if you think of it as an N:1 map. But if you do a trivial list based implementation with fall-through to next entry on store miss then it becomes N:M by implementation. I also do not see why updated response headers belong to x-vary-object. Would not Squid receive them in the origin server response or load them from the cache? Why store them in x-vary-object? I am not saying that they go into X-Vary object. But they are part of the process and needs to be included in the response. A simple and scalable implementation that would limit the X-Vary churn is to cache the 304 responses separately based on the request headers. This gives a store with 3 kinds of objects key : object Base URL: X-Vary, containing Vary and a list of known ETag values. URL + Vary indicated request headers: 304 response with updated headers + ETag. URL + ETag: Actual object. we might also use a similar design to handle header updates in general until the store can handle them. The 304 object should probably carry a full set of headers, which makes lookup of the full object optional in conditional requests and HEAD. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 23:37 +0300 skrev Eliezer Croitoru: The main problem with ETAG from what I have seen and I need to read more about ETAG, is.. couple mirrors ETAG should be the same for all servers objects in the level of HASH like.. which is not.. an example: ETag uniqueness is only required per URL, not per application level object. Having syncronized ETag values across mirrors is desireable and would simplify manaement of mirrored content in both caches and applications, but is not required by RFC2616. Extending HTTP with ETag values crossing mirrors is relatively easy in terms of specification, but halts on technical grounds on having it implemented. Content-MD5 and specifically Digest headers can address this, but is both lacking in conditional processing capabilities. Note: Content-MD5 have been deprecated as there is some ambigouity on what it really means, plus is limited to MD5 only. So even the content type is not the same... Which means some mirror admin needs to be poked a bit.. they both run apache both run 2.2.22 and they both have right LM headers but they both have different ETAG which is a bit weird but not unacceptable.. To HTTP it is fully acceptable. The two are completely different URLs. The fact that they happen to be mirrors is application layer defined, outside the protocol. As far as HTTP is concerned they are separate rsources. I do hope and know that httpbits tries to make all of the above true.(reading) httpbis do not address mirrors. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
fre 2013-07-26 klockan 14:44 +0300 skrev Eliezer Croitoru: So etag is fine but per server or per url. ETag is per URL. metalink\hashes are more global. Yes. since the list of mirrors is big and there is a list of them it would be very simple to use StoreID helper to identify them faster then a HASH verification. Hasing is required for a distributed system. You also need to worry about people intentionally or by accident trying to inject false material in the cache. It is not uncommon that Fedora mirrors gets out of sync or even having corrupted files. The application layer takes care of that by discarding objects with wrong checksum and trying another mirror. The only problem is that there should be a computer\program friendly list of mirrors for let say fedora.. Yes... Once now squid do IMS then all the mirrors should have the same IMS response for all mirrors(if they use rsync). IMS works in most cases, as mirrors usually preserve modification date. But it is quite weak as cache validation. the public list of fedora contains currently 152 active public mirrors with 240Gbic/sec which a squid forward proxy can pretty easily cache all of the objects in one repo with basic settings. Yes, the Fedora repo is not overly large. Even if you add Fedora, Debian, Ubuntu, CentOS, Arch, Mageia etc it's still manageable in size. Do we have friends there?? Yes. I was thinking before to use a specific mealink file they have on specific resources to just allow the admin update the MIRROR list and then squid will know about all the mirrors. Fedora have very easily parseable official mirror lists. And there is a substantial amount of inofficial mirror sites. Notice that as we publish this feature we go from hacking youtube to allowing admins to prevent de-duplication of objects DB. Good. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
ons 2013-07-24 klockan 18:46 -0600 skrev Alex Rousskov: The full logics (still missing from squid-3) involves an n-m map of requests to response variants. I thought Squid already tries to support that. Or are you talking about responses with varying Vary header values (discussed further below)? Squid-2 does, but Squid-3 only keeps a 1-1 map. This has direct impact on where Vary logics can be performed. Can you elaborate on this point? I think there are ways to support full Vary logics with and without all variants are in one store restrictions. It has an impact on how many store api accesses is needed to resolve a request for a varying URL. * Add mapping of request headers to specific response variant (identified by ETag). AFAIK, this is already supported by adjusting the store key. No, there is no way to query the store which ETags are known for a URL. All it knows is that the URL response varies. * Lookup the response matching this specific request. This is already supported by looking up the computed key in the store index. Sort of. That lookup is today based on URL + response headers which is correct for objects with no ETag. To complicate matters further different responses MAY have different Vary header value. But it is not very likely. True. To support that, we would need to add a list of Vary header values to the special Vary object and iterate it, looking up a cache object for every listed Vary header value. It's a little more complicated than that. Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. It feels like this is kind of separate optimization. It can be restricted to Vary transactions, but does not have to be. I don't think it's separate. It has direct impact on how you can store responses to a varying URL and how to handle cache validation. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 17:23 +1200 skrev Amos Jeffries: We think alike. See my other reply. Although the lookup per-match is not required. HTTP only requires that we identify a whole set of options and pull the most appropriate - with some requirements around defining appropriate. newest match is hightly preferable. Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. It feels like this is kind of separate optimization. It can be restricted to Vary transactions, but does not have to be. I agree. It is not clear which cases of Vary handling this would be appropriate for. We do however need to fix bug #7 once and for all. The map used in squid-2 do not really work well in cache validations. The problem is that the full objects gets revalidated when we get a new 304 response, when actually it's only the response that matches this request that have been revalidated. There is also a problem with too much churn in the x-vary marker object. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 18:53 +1200 skrev Amos Jeffries: Which problem specifically? that churn exists? that it can grow big + churn? races between clients? or that letting it out to disk can cause churn to be slooow? In the design used by Squid-2 there is quite a bit of churn in the x-vary object, and it's seen growing quite big in some extreme cases (Vary: cookie iirc). Races between clients have been seen. Also conflicts between x-vary updates and clients aborting, causing the new x-vary object to also be discarded making Squid forget the map, but that's a bug. Proper handling of cache validations is the main concern. I have been playing with the idea of locking these into memory cache, or using a dedicated memory area just for them to avoid the speed issues. A specialized store for them will also allow us to isolate the secondary-lookup logic in that stores lookup process - it can identify the variant and recurse down to other stores for the final selection using the extra key bits. What to use as permanent store? And you want to store each 304 mapping response separately so a scan can rebuild the map? And what about stores not having an index? IIRC we have as goal to optionally not have an in-memory cache index at all. I believe that they can be generated from a disk scan and if necessary we can add swap.state TLV entries for the missing x-vary meta details to be reloaded quickly. The x-vary meta is not very small. For each request header combination it's - request header contents - timing details for validation - which object variant to map to And there is also a map of known object variants and their ETag values and also Content-Encoding, the latter to work around dynamic gzip brainfart in many major web servers including Apache. That would make them churn particularly badly on startup, but avoid the necessity to store anywhere long-term, and help detect obsolete variants undeleted from disk. The total system churn at startup is already majorly bad with both ufs and rock stores.. caches are growing quite large today with current disk memory prices. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
tor 2013-07-25 klockan 22:06 +1200 skrev Amos Jeffries: I don't see why those are necessary. At the x-vary level all that is necessary is the response details to be searched for in the request headers. ie if x-vary says variant has Content-Encoding:gzip then search for Accept-Encoding:gzip in request headers for a possible match. Vary lists the requests headers the server side variant selection looked at while determining which variant of the requested URL is right for this request. Any number of different request header values can map to the same variant. The variant of an URL is identified by it's ETag value. Caches uses If-None-Match on unknown requests to ask which variant among the known set of cached variants is the right response for this request. The result of an If-None-Match is a 304 listing the ETag of the variant to respond with + updated response headers. What you end up with is a map of * Request headers as indicated by Vary * Which variant (etag value) is the right response for the matching request. * Possibly updated headers for the response (Date, Last-Modied, Set-Cookie, X-Whatever) carried in a 304 response to If-None-Match. Headers from the 304 MAY update the base object identified by ETag if desired (the boundaries is a bit blurred), but you need to handle aging of the map data somehow. But from what was learnt in squid-2 I would today prefer to handle them separately, giving each request variant it's own set of headers and aging, only sharing response body. This reduces X-Vary to just Vary + known etags, where the etag values is only required if the store can not index them by other means. A cache hit would find the Vary, reiterate and look up based on request headers as today, find the response headers, perform cache validation if needed and get the response body from the store if needed. But this requires a somewhat different store, and new pitfalls on what to do if the referenced body suddenly is no longer there. If you have read the Key header specification this should be clearer. I am thinking the x-vary acts like a list of ETag values (for ETag match), Digest (for Digest match), and Key patterns (for Key/Vary matches) with Vary being a special vague case of Key. It stores a set of variants using *only* the exact details causing that variant to exist in the set, none of the request-header garbage like entire Accept: header or ignored field-values. Vary is based on entire headers. We have no indication which parts of the header the server looks at or what rules it applies when selecting the variant. For some headers such as Accept-Encoding we can guess, but not know. Key tries to cut that down a bit, but getting Key correct is non-trivial. Even the Cookie case should be greatly reduced with the use of Key header. Although that will take some time to occur. We can avoid that a bit by hacking an omit for Vary:Cookie sites just like Varnish does, unless they use Key to specify *which* cookie detail to inspect. Imho the Key header is a dream. But sure having support for it as well would be nice. But it is not an escape from supporting Vary. The issue Key tries to address isn't really that big, if ignoring Cookie. And it's very unlikely site owners manages to specify Key correctly, they already have a hard time even doing Vary right. Even the most basic Accept example in the Key draft is flawed. The example will also match Accept: text/html;q=0, plus completely fails in representing quality at all. I.e. consider two clients one preferring HTML one preferring plain text, but both can handle both if absolutely required (i.e. the usual case for Accept*). Regards Henrik
Re: What is The logic of Vary Headers cachiness?
ons 2013-07-24 klockan 10:01 -0600 skrev Alex Rousskov: That is what Squid does today, bugs notwithstanding. If the found store entry is the special Vary entry, then Squid does another lookup, with the appropriate header values added to the store key hash. Yes, but is only part of what is needed for correct Vary support. The full logics (still missing from squid-3) involves an n-m map of requests to response variants. Instead of returning a special Vary object during this first lookup, Squid could return a regular cached object (one of the Vary variants), with some special Vary flag set, Yes, ideally the vary lookup logics should fit in the store, and not upper layers. Was not really an option when the Vary logics was originally designed in Squid-2. From my point of view on the code and after coding StoreID I know that there are two lookups for HEAD and GET and they are not the same object So why not just use 3 object level checking?? GET and HEAD are handled as different objects due to Squid design. It's not meant to from HTTP point of view. I am afraid I do not know what you mean by a 3 object level checking. Neither do I. IIRC Squid does not lookup HEAD and GET for the same request under normal conditions. Lookups with multiple methods happen for HEAD requests and for purging. Both categories are relatively rare. There is some cross-magics between HEAD and GET to try to map Squid store semantics to what HTTP expects, but far from complete. Finally, as we are migrating to per-cache store indexes, more store lookups should be avoided when possible because the number of mandatory lookups has to be multiplied by (the number of cache_dirs plus one for the memory cache index) to check all the indexes. There is a design decision that needs to be taken here.. should it be possible to have different responses for the same Vary:ing URL to be stored in different stores, or should they all need to go into the same store? This has direct impact on where Vary logics can be performed. Full Vary support requires the following store operations * Which responses (ETag values) is known for given URL? This list is needed to construct a If-None-Match validation request as needed to ask upstream which variant is the correct response. * Add mapping of request headers to specific response variant (identified by ETag). Or better yet allow the same response body to be referenced by multiple responses (simplifies validation logics and aging considerably). Response headers is the original response updated with header data from the 304 response in response to If-None-Match. * Lookup the response matching this specific request. To complicate matters further different responses MAY have different Vary header value. But it is not very likely. Regards Henrik
Re: What is The logic of Vary Headers cachiness?
ons 2013-07-17 klockan 23:34 +1200 skrev Amos Jeffries: Also, I think if the variant needs to be invalidated Squid currently coded to drop all variants and/or the main x-vary-* stub object during revalidation. The HTTP/1.1 specs need to be checked to see if that is right or if only the one variant object should be invalidated and the others left for later requests to alter. Invalidation points in HTTP (PUT etc) need to invalidate all variants at that URL. Regards Henrik
Re: [PATCH] Deprecate log_icap and log_access configuration directives
tor 2013-06-13 klockan 23:59 +1200 skrev Amos Jeffries: As a temporary measure, how about adding a directive log_stats allow/deny acl acl acl ... and using it only for the statistcs gathering functions as done by log_access allow? That's fine for me. But should be named something more appropriate than log_stats. It's not about logging. collect_stats maybe? Regards Henrik
Re: [PATCH] Deprecate log_icap and log_access configuration directives
tor 2013-06-13 klockan 08:16 -0600 skrev Alex Rousskov: I suggest stats_collection because it works better with allow/deny keywords that will follow: stats_collection deny internal +1 Regards Henrik
Re: [PATCH] Deprecate log_icap and log_access configuration directives
ons 2013-06-12 klockan 14:25 +1200 skrev Amos Jeffries: Please also note in the descriptive message that this alters the documented behaviour of Requests denied for logging will also not be accounted for in performance counters., and now makes all traffic get performane counters accounting. Are we OK with this? I am okay with the change. It just needs documenting so we don't overlook it in the release documentation. We then may need some other way of excluding certain traffic from accounting. The above was done to exclude internal system monitoring requests from accounting because otherwise these may have a noticeable impact on the collected performance metrics, more so under low traffic conditions. Regards Henrik
Re: I was wondering about SSL\socks support from squid side?
tor 2013-06-13 klockan 02:18 +0300 skrev Eliezer Croitoru: I have seen the wiki: http://wiki.squid-cache.org/Features/Socks Nothing implemented. And have not been given any priority as it's not HTTP related. and was wondering about it very much! I have a situation that I have access to SSL ports only which are being detected by NDPI(Deep Inspection) or similar mechanism. Deep inspection as in that they look that there is actually SSL taking place, or that they decrypt the SSL to inspect what's inside? I want to access my local network resources such as files and other data on a secure channel that is not: 1. SSH port forwarding\tunneling. 2. VPN 3. Layer 2 secure channels. So publish them using https service, with appropriate access controls. I first want to ask for the existing working options that you might know of. The main issue about it that most of the natted(super multi nantted like a government) complex networks doesn't allow the usage of any tunneling protocol that can allow network level routing. Yes... and please don't try to bypass security policies. The situation is a school which has a filtering solution and we want to exploit in order to give them transparent support. Not sure what you mean by exploit or transparent here. This school have a lots of sensitive data which we don't want the third party which is reliable for filtering but not security to be aware of. I do not quite follow.. who want to provide access to whom? Do the school want to provide outside access to internal data while being behind a multi-level-NAT network with deep packet inspection? Or do you want to access your data while being within such messed up network? Regards Henrik
Re: Do we have any connection with RH squid maintainer?
tis 2013-06-04 klockan 11:48 +0300 skrev Eliezer Croitoru: I was wondering about the major bugs that was found in 3.1 that RH and therefor CentOS doesn't have in their RPMS. Double negation there? Have you tried tiling a bug report in redhat bugzilla? It's the best contact method available for reaching RHEL package maintainers. If we had someone that is connected to RH squid maintainer we could push some fixes to the enterprise rings and get more feedback. If I am not mistaken the RHEL Squid maintainer is the same who is main maintainer in Fedora: Michal Luscon mlus...@redhat.com, and if not I am sure Michal known who is maintainin the RHEL package. Jiri Skala have moved on to other tasks I think. I am not suggesting using them as Guinna pigs but puss squid newer versions to the major vendors. RHEL upgrade policy prevents any Guinna pigs. But also prevents most bug fixes.. So who do we care about? or we don't care about anyone? Define care about. Our main concern is the source distribution moving forward. We do not have resources to keep maintaining a frozen stable branch for many years as needed by RHEL and the like. But we should try to keep a clean upgrade path. Regards Henrik
Re: NA - token = fatalf
tis 2013-02-12 klockan 14:41 -0700 skrev Alex Rousskov: Hello, Could somebody with better authentication and helper knowledge clarify whether the token field is indeed required for Nagotiate ERR and NA responses? If not, can we just remove the above quoted fatalf() blob and make the following line conditional on the token presence? Squid-2 negotiate expects NASPACEblobSPACEmessageNEWLINE but do not require any of them to be present. It accepts NASPACENEWLINE as valid response. NTLM is slightly different and do not expect a blob. Simply NASPACEmessageNEWLINE where message may be blank. Regards Henrik
Re: NA - token = fatalf
ons 2013-02-13 klockan 14:41 +1300 skrev Amos Jeffries: Yes you can remove these fatal() if you want, but it needs to result in authentication failure NA is authentication failure. Must not result in anything else. Regards Henrik
Re: eventAdd with zero delay?
ons 2013-01-30 klockan 12:21 + skrev Rainer Weikusat: Thank you. The reason I'm asking is mainly because 0-delay events are, according to the comment in schedule, supposed to be executed in FIFO order. Is this generally true for events, ie is the actual ordering criterion always a pair (delay, sequence number) or just for 0-delay events? Probably true for all events, but should only matter for 0-delay events if any. If this actually matters for some existing sequence of added 0-delay events then I would say that is likely a bug. The more important aspect is tha AsyncCall queue is drained before any events are processed. Regards Henrik
Re: scalable event scheduling
mån 2013-01-28 klockan 22:29 + skrev Rainer Weikusat: Since Mr Rousskov was so kind to publically question my honesty and/or mental sanity when I claimed that replacing the O(n) event scheduler with a more scalable implementation would be easy (and in fact, so easy that even using the STL implementation wouldn't be worth the effort) and since I need this anyway for connection timeouts and possibly other stuff in the squid fork I will be maintaining for at least some time to come and had an hour of free time for some recreational programming yesterday, I thought I could just do that. How large event queue are you expecting to see? The code was designed using a linear list becase a) Simplicity b) Expected queue length is relatively short. Regards Henrik
Re: [RFC] default HTCP
ons 2013-01-23 klockan 18:41 +1300 skrev Amos Jeffries: With HTTP/1.1 functionality Variants are much more common and the ICP failure rate is growing quite high. Yes.. I propose altering the default behaviour of peering to using HTCP queries by default. If HTCP is fixed to handle variants. This would entail adding htcp=off and icp options to cache_peer and altering the build defaults such that HTCP was always available. The no-query option is vague enough that it can be left with its existing behaviour modified to disable the default queries. There should only be at most one query based peering protocol active per peer at a time. no-query means no peering protocol active. Regards Henrik
Re: [RFC] remove negative_ttl directive ?
fre 2013-01-11 klockan 18:53 +1300 skrev Amos Jeffries: Can anyone present any actually useful reason to keep it despite the problems it presents? referse proxies need to be able to cache error responses even if there is no cache-control. But it does not need to be a specific directive. Merging this functionality into refresh_pattern is fine. Regards Henrik
Re: Default behavior for --enable-esi?
tor 2012-12-27 klockan 13:38 +1300 skrev Amos Jeffries: IIRC it is still off because it is not wanted by many installations and adds a large footprint to the binary. The more useful surrogate-capabilities sub-component is auto-enabled on reverse-proxy traffic regardless of ESI. Plus still experimental, and no active maintainer familiar with the code. Regards Henrik
Re: Why tcp_recv_bufsize also acts as tcp_send_bufsize?
tor 2012-12-13 klockan 00:45 -0800 skrev Tianyin Xu: Accoding to the code, when setting tcp_recv_bufsize, Squid also set the same amount as tcp_send_bufsize? And the send log printing message (line# 1794) seems problematic because the failure is caused by setting tcp_send_bufsize (the flag is SO_SNDBUF). If we really want to set the send buffer size, why not add another directive? Memory have faded a bit over the years but commit message says revno: 8451 committer: hno branch nick: HEAD timestamp: Fri 2007-08-24 23:56:45 + message: Make tcp_recv_bufsize apply to send buffers as well, in an attempt to control window scaling the main reason to set tcp_recv_bufsize is to avoid problems related to large windows. For example many firewalls still have problems with window scaling. this change makes the option apply to send buffers as well, and also hints Linux that Squid will never increase the buffer so the window scaling should be clamped to the indicated buffer size. Regards Henrik
Re: [BUG] (13) Permission denied with workers | squid 3.2.3 and 3.3 rev 12500.
tor 2012-12-06 klockan 21:54 +0200 skrev Eliezer Croitoru: On cache log I am getting these: 2012/12/06 21:34:29.780 kid3| commBind: Cannot bind socket FD 10 to [::]: (13) Permission denied Odd.. the debug output indicates this is a request for assigning an ipv6:port. What do strace say? Since there are more users with workers out there, they dont have this issue? I have some issues with workers, but have not seen this issue. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 22:25 -0700 skrev Alex Rousskov: I disagree with this logic. Yes, sawActivity approach supports long chains, but they are not the loop's fault! If some chain is too long, the bug is in that chain's code and should be fixed there. If it cannot be fixed there, then the chain is not too long, but Squid is just being asked to do too much. The event engines we talk about here is SignalEngine EventScheduler StoreEngine Of these only possibly StoreEngine may need repeated work. SignalEngine is happy if it gets polled every now and then at low frequency. EventScheduler do not want to be polled more than once per loop. StoreEngine is about store callbacks. In the past this should not be invoked too often per loop. It is difficult to resolve this disagreement. I argue that the sawActivity loop should stay because breaking call chains by I/O callbacks will lead to bugs, but I cannot show you any specific bug it will cause. You argue that some chains might be so long that they will starve I/O, but you cannot show any specific too long chain (after the heavy event handling bug is fixed). store I/O is known to have done this in the past. Perhaps we can agree to let the sleeping dog lie? The bug you needed to fix was with the heavy events handling. My patch or your next patch fix that bug. Let's not remove the sawActivity loop until its removal becomes the best solution for some other bug. We can discuss the loop's pros and cons in the context of that specific bug then. Ok. I would probably go for both pathes. Solves different aspects of the event loop timings. My patch fixes the delay calculation, both end result and readability, and enabling removal of sawActivity. But it does not fix the comm starvation issue without removal of sawActivity. Today the comm loop delay is often too short in current code. Your patch addresses sawActivity and heavy events. If sawActivity loop is removed in future then your patch have no effect but still makes code more readable. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 23:12 -0700 skrev Alex Rousskov: BTW, once zero-delay events are removed from the code, the event queue will no longer need to be a part of the wasActivity loop and the loop will disappear. Note: Legacy zero-delay addEvent events are meant to be called once per comm loop. Not immediately/chained. This is by design. The fact that zero (or shorter than current workload) delay addEvent events gets called immediately before next comm event check is a bug introduced with the AsyncCall merge when the discussed sawActivity event-loop-in-the-event-loop was added. heavy events in the context of addEvent events are really this is very heavy, and there is only allowed to be one and only one heavy event per comm invocation. If there is multiple heavy jobs competing for time they are meant to get scheduled in a round-robin fashion with comm inbetween each. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
tis 2012-12-04 klockan 08:39 -0700 skrev Alex Rousskov: heavy events in the context of addEvent events are really this is very heavy, and there is only allowed to be one and only one heavy event per comm invocation. If there is multiple heavy jobs competing for time they are meant to get scheduled in a round-robin fashion with comm inbetween each. Agreed. Both patches support that AFAICT. Yes. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
tis 2012-12-04 klockan 08:39 -0700 skrev Alex Rousskov: There are several ways to interpret the designer intent when looking at undocumented code. I cannot say whether all of the currently remaining zero-delay events use your interpretation, but I am certain that I have added zero-delay events in the past that used a comm-independent interpretation (get me out of this deeply nested set of calls but resume work ASAP). And, IMO, that is actually the right definition for the reasons discussed in my response to Amos email. I can only speak for the Squid-2 code base on this, and the intents behind event.c[c] event handling. Agreed. Both patches support that AFAICT. What I ended up with is a combination of both. I kept sawActivity loop for now, pending review of zero-delay events and store callbacks. I still think the sawActivity loop is undesired and not needed, but it's not the bug we are working on right now. But your patch also needs timeTillNextEvent to calculate the loop delay. or the timed events starve due to loop_delay being calculated wrongly (not taking into account for events added just now). Note: There is a slight but critical error in my timeTillNextEvent patch as posted. loop_delay requested should be . Other than this it seems to work. And there was a couple of other problems seen today: *) UFS store rebuilding crashes if there is cache collisions with non-ufs stores competing for the same object. assertion failed: ufs/ufscommon.cc:709: sde have a patch for this. *) some undiagnosed crash assertion failed: comm.cc:1259: isOpen(fd) seen this randomly earlier, but now it hit all the workers at about the same time +/- some seconds. This was randomly seen before event loop rework as well. *) shortly after the above after the workers had restarted and rebuilt their caches all started to spew Worker I/O push queue overflow: ipcIo1.13951w6 and calmed down after some minutes. Continued for ~10 min I think. I/O load on the server was marginal. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
sön 2012-12-02 klockan 22:27 -0700 skrev Alex Rousskov:
My understanding is that the bug is in Squid store rebuilding(?) code
that assumes that Squid will have time to do some I/O before the next
timed event is fired, but actually there is a lot of non-I/O work to do,
In such case the bug is in handling of events added by eventAdd(), not
in the store rebuilding code.
The store rebuilding does
eventAdd(storeRebuild, RebuildStep, rb, 0.01, 1);
which says both
delay = 0.01, avoid saturating things too much
weight = 1, this is a heavy job, slow down event.cc processing when this
is seen. Only one heavy job should be scheduled per event loop.
and the next rebuild step event becomes ready before any I/O can
happen. That event schedules another event that, again, fires before we
can get to I/Os. In this context, I/O is regular select-based I/O
orchestrated by the waitingEngine and not direct disk reading or writing
that Store code might do during rebuild.
events added by eventAdd must not fire until next event loop takes
place.
I doubt this should be fixed by allowing I/O into the process current
non-I/O events engines loop. The problem is not in the lack of I/Os,
but in an unrealistic scheduling expectations by non-I/O code (or lack
of API to schedule those rebuild events on a realistic schedule).
It's not lack of API. Only the event loop not scheduling things in a
sendible manner.
Fixed by this change. I think it's correct but review please.
The change does not really fix the bug if I understand the bug
correctly. It just masks it. Squid would violate the intended wait x
seconds before every rebuld step schedule even more after this change.
No.
More importantly, I am concerned that this change may inject new I/O
callbacks between old async calls if we still have some I/O callbacks
that are not done using async calls API. That is likely to create
hard-to-find bugs because the order of events (in a broad sense) will be
disrupted.
But yes, this may happen and is not good.
1) When entering the non-I/O engines loop, remember now and only
process timed events that are ready at that remembered time. Do not
advance that time value until we exit the loop and let I/O engine run.
This does not change the structure of the engines loop. Some events will
still be late, but there will be no disruptive injections of new I/O
activity that might not use async calls. This is probably a
smaller/easier change, but it also masks the true problem.
event.cc actually does this (and always have), but gets broken by
EventLoop::runOnce() doing this:
do {
// generate calls and events
typedef engine_vector::iterator EVI;
for (EVI i = engines.begin(); i != engines.end(); ++i) {
if (*i != waitingEngine)
checkEngine(*i, false);
}
// dispatch calls accumulated so far
sawActivity = dispatchCalls();
if (sawActivity)
runOnceResult = false;
} while (sawActivity);
if (waitingEngine != NULL)
checkEngine(waitingEngine, true);
This will poll non-comm event loops until they are completely idle
having nothing to do.
The sawActivity loop should go away I think, polling each event engine
once per loop. It's the responsibility of each event engine to run as
many events as it needs per loop.
Looking at the code I think I see why this sawActivity loop got added
around the non-comm event engines. There is a AsyncCalls induced delay
procssing of events fired by checkEngine() making it fail to properly
set the required loop_delay. We need to extend the event engine API with
a call to return the minimum loop delay each engine needs after
dispatchCalls() have been run allowing it to account for new events
added.
An interim solution without changing event engine API to include such
call is to drop loop_delay down to 0 if there is any activity,
disregarding what the event engine said in such case. This will cause a
slight CPU penalty under lower loads as the event loop fires more often
than needed but is a very easy change. I.e. something like this:
(completely untested)
=== modified file 'src/EventLoop.cc'
--- src/EventLoop.cc2012-02-05 06:09:46 +
+++ src/EventLoop.cc2012-12-03 08:42:39 +
@@ -107,19 +107,26 @@
if (!waitingEngine !engines.empty())
waitingEngine = engines.back();
-do {
-// generate calls and events
-typedef engine_vector::iterator EVI;
-for (EVI i = engines.begin(); i != engines.end(); ++i) {
-if (*i != waitingEngine)
+// generate calls and events
+typedef engine_vector::iterator EVI;
+for (EVI i = engines.begin(); i != engines.end(); ++i) {
+if (*i != waitingEngine)
checkEngine(*i, false);
-}
+}
-// dispatch calls accumulated so far
-sawActivity = dispatchCalls();
-if (sawActivity)
-runOnceResult = false;
-}
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 09:07 -0700 skrev Alex Rousskov: Yes. IIRC, it has been broken before my AsyncCall changes to the main loop. The sawActivity code is from the AsyncCall merge in rev 8812. This will poll non-comm event loops until they are completely idle having nothing to do. Yes, and that breaks the weight API. The exhaust all AsyncCalls approach is still needed but we should honor the weight API and stop processing timed events sooner if needed. It breaks not only the weight API but also the limitation to not schedule timed events added after the current loop started. I was going to agree with that wholeheartedly, but then I thought about signals. If we visit each non-waiting engine once, then we will only process signals once per main loop step. Is that OK? Not 100% OK, but not too bad. Causes max 1 seconds delay of processing signals, and the only signals I know of which is not comm related (and also have comm events) are squid -k signals. I think we can still do that provided the AsyncEngine::timeTillNextEvent() API discussed below is adjusted to allow us to skip the waiting engine completely: If the call returns a check me again ASAP AsyncEngine::EVENT_ASAP value, the main loop will skip the waiting engine. Only the signal engine will return such ASAP values. If this is implemented reasonably lightweight in each event engine. I am worried that this will result in 50% or more zero-delay select(2) calls for nearly all Squids because there is usually some AsyncCall activity per I/O loop (but possibly no timed-events). So need to make sure AsyncCall is drained last in RunOnce. Is it sufficient to call dispatchCalls() or do one need to loop over it until no activity remains? And as the code migrates from zero-delay timed events to AsyncCalls, the situation will become worse. That does not sound like a good fix because it is not clear (to me) what kind of side effects this will have on performance. I don't see how this makes a difference. AsyncCalls is fired at least twice per comm loop today, before and after comm. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: True (although the current small maximum select(2) delay is a result of bugs elsewhere in the code and should not really be there). If we can do the right thing here easily, we should (instead of adding more future problems). And I think we can. Signal processing is however always due to these races, unless using a comm pipe to wake up the main loop on receiving a signal. Ah, I see another problem with the each non-waiting engine runs once approach. One engine may create work for other engines or for itself, including async calls and run this now lightweight events. This means we should really continue to use the sawActivity loop so that all lightweight events are processed before I/O wait starts. This design is more complex, but it is actually more correct. Not sure. Same reasoning applies to comm events. There is some good reasons not to do this. A chain of small lightweight events quickly adds up to a heavy event. But might make sense to drain the AsyncCall queue more than once. The new condition for the sawActivity loop should be: while(sawActivity !sawHeavyEvent) What is a heavy event in your view? One call is sufficient -- the async call queue drains itself completely, including any calls scheduled during the draining process itself, but the dispatchCalls() call should remain inside the sawActivity loop (because events create calls create now events create calls create now events ...). would it be possible to make that only fire already queued calls and let the main event loop worry about dealing with chained calls? Does the attached patch makes sense to you? Does it solve the I/O starvation during rebuild problem you found? It probably solves the starvation, but I do not like that sawActivity loop at all. It too easily creates comm starvation if one is careless with event chains. It it is to stay then it need to be limited in number of iterations accepted per main loop. The patch looks good if sawActivity loop-in-the-loop is to stay. If sawActivity loop is not to stay then this patch is not needed, but still makes logic of the code more visible even if the result code doesn't really mean anything particular to the main loop. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 15:53 -0700 skrev Alex Rousskov: We just need to make sure that heavy events (certain timed events and signals) interrupt the sawActivity loop. This will fix the issue you are facing without introducing new problems. Another question on this.. how to fit the concept of heavy with the goal of using AsyncCalls? Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
mån 2012-12-03 klockan 09:07 -0700 skrev Alex Rousskov:
Could you please fix this properly by adding
AsyncEngine::timeTillNextEvent() or similar API and removing loop_delay
manipulation from checkEvents() as discussed above?
The attached patch adds AsyncEngine::timeTillNextEvent() API. It also
kills the discussed sawActivity loop which imho should not be needed.
The patch is only lightly tested.
Regards
Henrik
=== modified file 'src/AsyncEngine.h'
--- src/AsyncEngine.h 2012-02-05 06:09:46 +
+++ src/AsyncEngine.h 2012-12-04 00:06:06 +
@@ -71,16 +71,19 @@
* events.
*
* The return value is the status code of the event checking. If its a
- * non-negative value then it is used as hint for the minimum requested
- * time before checkEvents is called again. I.e. the event engine knows
- * how long it is until the next event will be scheduled - so it will
- * return that time (in milliseconds).
+ * non-negative value then timeTillNextEvent() should be used to check
+ * desired time until next call after other processing have completed.
*
- * The timeout value is a requested timeout for this engine - the engine
- * should not block for more than this period. (If it takes longer than the
- * timeout to do actual checks thats fine though undesirable).
+ * The timeout value is a requested timeout in msec for this engine - the
+ * engine should not block for more than this period. (If it takes longer
+ * than the timeout to do actual checks thats fine though undesirable).
*/
virtual int checkEvents(int timeout) = 0;
+
+/* Returns the desired time in msec until the engine needs to be called
+ * again to continue processing events.
+ */
+virtual int timeTillNextEvent() = 0;
};
#endif /* SQUID_ASYNCENGINE_H */
=== modified file 'src/EventLoop.cc'
--- src/EventLoop.cc 2012-02-05 06:09:46 +
+++ src/EventLoop.cc 2012-12-03 23:58:00 +
@@ -43,17 +43,18 @@
void
EventLoop::checkEngine(AsyncEngine * engine, bool const primary)
{
-int requested_delay;
+int rc;
if (!primary)
-requested_delay = engine-checkEvents(0);
+rc = engine-checkEvents(0);
else
-requested_delay = engine-checkEvents(loop_delay);
+rc = engine-checkEvents(loop_delay);
-if (requested_delay 0)
-switch (requested_delay) {
+if (rc 0)
+switch (rc) {
case AsyncEngine::EVENT_IDLE:
+ runOnceResult = false;
debugs(1, 9, Engine engine is idle.);
break;
@@ -68,9 +69,6 @@
else {
/* not idle or error */
runOnceResult = false;
-
-if (requested_delay loop_delay)
-loop_delay = requested_delay;
}
}
@@ -107,19 +105,24 @@
if (!waitingEngine !engines.empty())
waitingEngine = engines.back();
-do {
-// generate calls and events
-typedef engine_vector::iterator EVI;
-for (EVI i = engines.begin(); i != engines.end(); ++i) {
-if (*i != waitingEngine)
+// generate calls and events
+typedef engine_vector::iterator EVI;
+for (EVI i = engines.begin(); i != engines.end(); ++i) {
+if (*i != waitingEngine)
checkEngine(*i, false);
-}
-
-// dispatch calls accumulated so far
-sawActivity = dispatchCalls();
-if (sawActivity)
-runOnceResult = false;
-} while (sawActivity);
+}
+
+// dispatch calls accumulated so far
+sawActivity = dispatchCalls();
+if (sawActivity)
+runOnceResult = false;
+
+// Figure out how long the waitingEngien is allowed to sleep
+for (EVI i = engines.begin(); i != engines.end(); ++i) {
+ int requested_timeout = (*i)-timeTillNextEvent();
+ if (requested_timeout = 0 loop_delay requested_timeout)
+ loop_delay = requested_timeout;
+}
if (waitingEngine != NULL)
checkEngine(waitingEngine, true);
=== modified file 'src/comm.cc'
--- src/comm.cc 2012-11-26 08:28:09 +
+++ src/comm.cc 2012-12-03 23:59:29 +
@@ -2086,6 +2086,13 @@
};
}
+int
+CommSelectEngine::timeTillNextEvent()
+{
+// We don't know when next event is, so say nothing in particular.
+return EVENT_IDLE;
+}
+
/// Create a unix-domain socket (UDS) that only supports FD_MSGHDR I/O.
int
comm_open_uds(int sock_type,
=== modified file 'src/comm.h'
--- src/comm.h 2012-02-05 06:09:46 +
+++ src/comm.h 2012-12-03 22:51:18 +
@@ -107,7 +107,8 @@
{
public:
-virtual int checkEvents(int timeout);
+int checkEvents(int timeout);
+int timeTillNextEvent();
};
#endif
=== modified file 'src/event.cc'
--- src/event.cc 2012-02-05 06:09:46 +
+++ src/event.cc 2012-12-04 00:03:39 +
@@ -242,10 +242,10 @@
ev_entry *event = NULL;
if (NULL == tasks)
-return checkDelay();
+return EVENT_IDLE;
if (tasks-when current_dtime)
-return checkDelay();
+
Re: [PATCH] Do not send unretriable requests on reused pinned connections
lör 2012-12-01 klockan 18:02 -0700 skrev Alex Rousskov: a) If we are talking about unretriable requests, the client already violated HTTP by sending such a request over the persistent client connection. Some of these clients may fail to handle resets properly. End-user-agents have much more freedom in what is retriable or not. b) To implement 2+1 correctly, Squid will need to close the _client_ connection when the origin server sends Connection: close. We do not do that now IIRC. Yes. If we don't do that today then it's a bug. Regards Henirk
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
fre 2012-11-30 klockan 15:38 -0700 skrev Alex Rousskov:
On 11/30/2012 02:25 PM, Henrik Nordström wrote:
We should look into why it is at all needed. From what I can understand
it should not be needed.
Agreed. Please do that if you can.
Found it. The event loop gets saturated with timed events during the
rebuild and never exectues comm events. There is a little premature
optimization there skipping the primary engine while checking all
others, causing comm to starve to death.
Fixed by this change. I think it's correct but review please.
diff -ru squid-3.2.3/src/EventLoop.cc squid-3.2.3-dpl/src/EventLoop.cc
--- squid-3.2.3/src/EventLoop.cc2012-10-20 15:39:49.0
+0300
+++ squid-3.2.3-dpl/src/EventLoop.cc2012-12-03 01:52:25.381523614
+0200
@@ -111,7 +111,6 @@
// generate calls and events
typedef engine_vector::iterator EVI;
for (EVI i = engines.begin(); i != engines.end(); ++i) {
-if (*i != waitingEngine)
-checkEngine(*i, false);
+checkEngine(*i, false);
}
With this
* original timeouts works fine
* rebuild is now done in background as expected
Unfortunately it uncovered another minor bug resulting in 100% CPU
usage. Fixed by this:
diff -ru --exclude helpers --exclude tools squid-3.2.3/src/comm.cc
squid-3.2.3-dpl/src/comm.cc
--- squid-3.2.3/src/comm.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3-dpl/src/comm.cc 2012-12-03 02:59:34.713503938 +0200
@@ -2071,7 +2071,7 @@
case COMM_OK:
case COMM_TIMEOUT:
-return 0;
+return EVENT_IDLE;
case COMM_IDLE:
Also in testing this I observed that fast ACLs do not do inverse of last
rule on no match. Excplicit allow/deny all rule is needed. Don't think
this is intentional.
Regards
Henrik
Re: [PATCH] Do not send unretriable requests on reused pinned connections
fre 2012-11-30 klockan 23:07 -0700 skrev Alex Rousskov: I am not sure what you are asking about, but I can try to rephrase: This bug is difficult to fix because some pinned connections should be reused and some should not be. Pinned connections that can be re-pinned but have not had any HTTP requests sent on them should be reused, even for unretriable requests. SslBump creates such connections in forward.cc when Squid connects to the origin server to peak at the server certificate. Since no HTTP requests were sent on such connections at the decision time, this is not really a reuse even though it looks like one in all other aspects. It is. You must take care to not reuse a slightly old (1s or so) connection under those conditions. Which it quite likely the wrong thing to do. See above. Does the !flags.canRePin exception address your concern? Yes, if used where needed (TPROXY, NTLM). Regards Henrk
Re: [PATCH] Do not send unretriable requests on reused pinned connections
lör 2012-12-01 klockan 11:42 -0700 skrev Alex Rousskov: come. Why can't TPROXY reopen a server-side connection? Because it tries to use the original source ip:port, but it's in TIME_WAIT from the previous connection and can't be reused. Regards Henrik
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
fre 2012-11-30 klockan 10:47 -0700 skrev Alex Rousskov:
IIRC, Rock store diskers should process queries while rebuilding so db
open should not fail due to rebuild itself. I bet this is actually
related to the problem discussed below.
I think so too.
- kid registration failures at startup (timeout). This seem to be due to
the local cache rebuild in each kid taking more than 6 seconds to
complete, confusing the registration timeout management.
The registration code needs to be redesigned. It is not robust enough,
especially when we have legacy code that blocks Squid for more than 6
seconds at startup.
But the aufs rebuild should not block.. unless it's forced into
foreground rebuild which I think not.
I think the default timeouts SHOULD be quite sufficient I think if it's
only about communication timeouts. Should not get blocked by a normal
background store rebuild.
-const double IpcIoFile::Timeout = 7; // seconds; XXX: ALL,9 may require
more
+const double IpcIoFile::Timeout = 300; // seconds; XXX: ALL,9 may require
more
Do not do that. The timeout is used for both file opening timeout and
regular/ongoing I/O timeout checks. The latter should happen more
frequently than once in 5 minutes, especially on busy Squids. Split this
into OpenTimeout and IoTimeout if you want to increase the former.
Ok.
IpcIoFile::IpcIoFileList IpcIoFile::WaitingForOpen;
IpcIoFile::IpcIoFilesMap IpcIoFile::IpcIoFiles;
std::auto_ptrIpcIoFile::Queue IpcIoFile::queue;
@@ -218,7 +218,7 @@
{
bool ioError = false;
if (!response) {
-debugs(79, 3, HERE error: timeout);
+debugs(79, 1, HERE error: timeout);
Do not do that. It is a common/benign case under overload.
This was needed to show the Open timeout event which is not at all
benign.
if ((nulpos = (char*)memchr(header_start, '\0', header_end -
header_start))) {
-debugs(55, 1, WARNING: HTTP header contains NULL characters {
+debugs(55, 2, WARNING: HTTP header contains NULL characters {
getStringPrefix(header_start, nulpos) }\nNULL\n{
getStringPrefix(nulpos+1, header_end));
goto reset;
}
While I agree that Squid should not warn about malformed headers by
default, others disagree, and this is unrelated to the rest of the changes.
Yes it's unrelated. This is a full unsorted patch of the changes I had.
--- squid-3.2.3/src/ipc/Strand.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/ipc/Strand.cc 2012-11-29 00:12:12.385439783 +0200
@@ -53,7 +53,7 @@
TypedMsgHdr message;
ann.pack(message);
SendMessage(coordinatorAddr, message);
-setTimeout(6, Ipc::Strand::timeoutHandler); // TODO: make 6
configurable?
+setTimeout(600, Ipc::Strand::timeoutHandler); // TODO: make 6
configurable?
}
Please keep the TODO comment in sync.
Yes, except that the change should not really be needed.
This change is discussed at the beginning of this email. In summary, 10
minutes is sometimes worse than 6 seconds (and sometimes is not enough
anyway). The registration algorithm should be adjusted, and the code
that blocks Squid process for 9 minutes should be fixed. Until then, I
am not objecting to this specific change, but perhaps you can add a
comment discussing why this is so large.
We should look into why it is at all needed. From what I can understand
it should not be needed.
Have the foreground/background store rebuild logics been changed
somehow?
Regards
Henrik
Re: [PATCH] Do not send unretriable requests on reused pinned connections
fre 2012-11-30 klockan 15:30 -0700 skrev Alex Rousskov: Squid is sending POST requests on reused pinned connections, and some of those requests fail due to a pconn race, with no possibility for a retry. Yes... and we have to for NTLM, TPROXY and friends or they get in a bit of trouble from connection state mismatch. If sending the request fails we should propagate this to the client by resetting the client connection and let the client retry. When using SslBump, the HTTP request is always forwarded using a server connection pinned to the HTTP client connection. Squid does not reuse a persistent connection from the idle pconn pool for bumped client requests. Ok. Squid uses the dedicated pinned server connection instead. This bypasses pconn race controls even though Squid may be essentially reusing an idle HTTP connection and, hence, may experience the same kind of race conditions. Yes.. However, connections that were just pinned, without sending any requests, are not essentially reused idle pconns so we must be careful to allow unretriable requests on freshly pinned connections. ? The same logic applies to pinned connection outside SslBump. Which it quite likely the wrong thing to do. See above. Regards Henrik
Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
While trying to set up an advanced Squid-3.2.3 with 4 workers, a number
of rock stores for different objects and per-worker aufs caches for
larger files I ran into multiple issues.
- Quite frequent NULL pointer assertions failures in peer selection if
the client aborts before peer selection have completed. Guess this is
seen whenever there is peering enabled and sufficiently impatient
clients around.
- each worker restarting at startup due to rock store DB open failures
(timeout). This seems to settle once the rock store have completed
rebuilding, but not 100% sure that's the cause as logs are a bit
inclusive. Might also be related to the delay due to the kid rebuilding
it's own aufs cache_dir.
- kid registration failures at startup (timeout). This seem to be due to
the local cache rebuild in each kid taking more than 6 seconds to
complete, confusing the registration timeout management.
- cache.log completely flooded with memset() messages at startup with
full debug enabled.
The attached patch tries to address or work around the above issues. But
have not fully understood the timeouts. Suspect the timeours are false
and caused by each kid doing a store rebuild at startup which takes a
bit of time so those changes are likely only a workaround and not
actually addressing the root problem.
Will also set up bug reports for these, but sending here first so it
doesn't get lost by mistake.
Regards
Henrik
--- squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-10-20 15:39:49.0
+0300
+++ squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-11-29 00:47:11.451848433
+0200
@@ -30,7 +30,7 @@
// TODO: make configurable or compute from squid.conf settings if possible
static const int QueueCapacity = 1024;
-const double IpcIoFile::Timeout = 7; // seconds; XXX: ALL,9 may require more
+const double IpcIoFile::Timeout = 300; // seconds; XXX: ALL,9 may require more
IpcIoFile::IpcIoFileList IpcIoFile::WaitingForOpen;
IpcIoFile::IpcIoFilesMap IpcIoFile::IpcIoFiles;
std::auto_ptrIpcIoFile::Queue IpcIoFile::queue;
@@ -218,7 +218,7 @@
{
bool ioError = false;
if (!response) {
-debugs(79, 3, HERE error: timeout);
+debugs(79, 1, HERE error: timeout);
ioError = true; // I/O timeout does not warrant setting error_?
} else {
if (response-xerrno) {
--- squid-3.2.3/src/HttpHeader.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/HttpHeader.cc 2012-11-21 19:51:23.826807754 +0200
@@ -548,7 +548,7 @@
char *nulpos;
if ((nulpos = (char*)memchr(header_start, '\0', header_end -
header_start))) {
-debugs(55, 1, WARNING: HTTP header contains NULL characters {
+debugs(55, 2, WARNING: HTTP header contains NULL characters {
getStringPrefix(header_start, nulpos) }\nNULL\n{
getStringPrefix(nulpos+1, header_end));
goto reset;
}
--- squid-3.2.3/src/ipc/StoreMap.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/ipc/StoreMap.cc 2012-11-20 20:55:41.089500435 +0200
@@ -272,8 +272,8 @@
Ipc::StoreMapSlot::StoreMapSlot(): state(Empty)
{
-xmemset(key, 0, sizeof(key));
-xmemset(basics, 0, sizeof(basics));
+memset(key, 0, sizeof(key));
+memset(basics, 0, sizeof(basics));
}
void
--- squid-3.2.3/src/ipc/Strand.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/ipc/Strand.cc 2012-11-29 00:12:12.385439783 +0200
@@ -53,7 +53,7 @@
TypedMsgHdr message;
ann.pack(message);
SendMessage(coordinatorAddr, message);
-setTimeout(6, Ipc::Strand::timeoutHandler); // TODO: make 6 configurable?
+setTimeout(600, Ipc::Strand::timeoutHandler); // TODO: make 6
configurable?
}
void Ipc::Strand::receive(const TypedMsgHdr message)
--- squid-3.2.3/src/peer_select.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/peer_select.cc 2012-11-21 12:01:06.030883260 +0200
@@ -241,7 +241,7 @@
(req-flags.intercepted ||
req-flags.spoof_client_ip);
const bool useOriginalDst = Config.onoff.client_dst_passthru ||
!req-flags.hostVerified;
const bool choseDirect = fs fs-code == HIER_DIRECT;
-if (isIntercepted useOriginalDst choseDirect) {
+if (isIntercepted useOriginalDst choseDirect
req-clientConnectionManager.valid()) {
// construct a result adding the ORIGINAL_DST to the set instead of
DIRECT
Comm::ConnectionPointer p = new Comm::Connection();
p-remote = req-clientConnectionManager-clientConnection-local;
--- squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-10-20 15:39:49.0 +0300
+++ squid-3.2.3/src/DiskIO/IpcIo/IpcIoFile.cc 2012-11-29 00:47:11.451848433 +0200
@@ -30,7 +30,7 @@
// TODO: make configurable or compute from squid.conf settings if possible
static const int QueueCapacity = 1024;
-const double IpcIoFile::Timeout = 7; // seconds; XXX: ALL,9 may require more
+const double IpcIoFile::Timeout = 300; // seconds; XXX: ALL,9 may require more
Re: Multiple issues in Squid-3.2.3 SMP + rock + aufs + a bit of load
tor 2012-11-29 klockan 15:08 +1300 skrev Amos Jeffries: Does anybody know the rationale for having xmemset at all? I don't see muhc of a meaningful rationale. memory debugging tools already handle it without any wrapper. Regards Henrik
Re: Cant compile squid 3.2.3 with ssl-crtd (FEDORA 17 x64)
tis 2012-11-27 klockan 04:44 +0200 skrev Eliezer Croitoru: I tried to build squid 3.2.3 on fedora 17 x64 and got an error. compiling options -g -O2 -c -o certificate_db.o certificate_db.cc certificate_db.cc: In member function גvoid Ssl::CertificateDb::load()ג: certificate_db.cc:460:76: error: גindex_serial_hash_LHASH_HASHג was not declared in this scope certificate_db.cc:460:106: error: גindex_serial_cmp_LHASH_COMPג was not [...]$ So it's something with the crtd code but I am unable to understand the meaning of it. Fedora 15-17 have borked openssl headers lying about the OpenSSL version, caused by a slightly misguided patch in OpenSSL which they haven't been able to get rid of due to ABI breakage. This causes problems for our crtd which tries do adapt to the detected OpenSSL version but can't on Fedora. Supposedly fixed in Fedora 18 but have not verified yet. Regards Henrik
Re: MD5 and URL validation (continue to other very old thread)
ons 2012-11-21 klockan 21:06 +0200 skrev Eliezer Croitoru: The problem is that it only being checked while a file is being fetched from UFS(what I have checked) while from RAM it wont be checked. There is no risk of object store displacement in RAM. The result is that when store_url_rewrite feature is being used the check points on inconsistency between the request url and the object in cache_dir (naturally). The metadata URL should be the store_url, not requested URL. After a small talk with alex I sat down and made some calculations about MD5 collision risks. The hash used to make the index hash is a string from byte + url. For most caches that I know of there is a very low probability for collision considering the amount of objects and urls. Verifying the MD5 is sufficient. But either MD5 or URL MUST be verified in metadata on objects fetched from disk. Regards Henrik
SNMP mib change
yadi hno: can you confirm that it is okay to simply swap the Counter32 SNMP OID type to Counter64 without changing OID numbers? Generally it's not OK to change existing entries, quite many pollers configure the data type. But it is a balance. Both changing data types and changing the MIB with a new OID is confusing. In case of changing from Counter32 to Counter64 it's better to change the datatype than introducing a new OID. Regards Henrik
Re: bzr repository upgraded to 2a format and restructured
tis 2012-10-16 klockan 05:39 +0200 skrev Eliezer Croitoru:
I did that already and I get error while updating:
##start
$ bzr update
Doing on-the-fly conversion from RepositoryFormat2a() to
RepositoryFormatKnitPack6().
This may take some time. Upgrade the repositories to the same format for
better performance.
bzr: ERROR:
KnitPackRepository('file:///home/eliezer/squid-repo/.bzr/repository/')
is not compatible with
CHKInventoryRepository('http://bzr.squid-cache.org/bzr/squid3/.bzr/repository/')
different rich-root support
##end
Is this a old bound branch automatically pulling changes from the master
repository? Then unbind it using bzr unbind.
There is no need to update any old branches. But it's best if your
working tree is all committed to your local branch.
it's kind of weird since for the new branch seems to work fine with
updates and everything.
Good. Now you just merge over your old branch(es) into the new
repository.
I am assuming you have locally committed your changes to your local
repository? If not then you need to merge changes from the working
directory (--uncommitted flag to bzr merge)
Regards
Henrik
Re: Squid code analysis using Coverity
tor 2012-10-11 klockan 13:30 +1300 skrev Amos Jeffries: Did you mean we should have no cost being part of their program? or that there is an admin control panel (contact person?) you have available to do more than I could? There is no charge to be part of their Open Source program, and they did respond in past when dealing with them. I can try contacting them to reviewe the Squid scan if wanted. Regards Henrik
Re: Squid code analysis using Coverity
tor 2012-10-11 klockan 23:23 +0200 skrev Kinkie: Please once you're done, please share the recipe so I can try to hook it into Jenkins for CI. It's quite simple. Basically a normal build but wrapping the make call in a cov-build tool by them, and then uploading the result written by their tool, using a username password given by coverity. http://scan.coverity.com/steps-for-analysis.html But seems their tool failed badly on 90% of our sources. Investigation pending. Regards Henrik
Re: Squid code analysis using Coverity
Squid is a member of the Coiverity Open Source programme, but we have not been actively using it. tis 2012-10-09 klockan 00:27 +0400 skrev Dmitry Kurochkin: Hello. We are evaluating Coverity tool for static analyzing of Squid code (see bug 3635 for details [1]). Initial results with analysis of recent Squid trunk revisions are ready and now we need to review them. We are looking for volunteers among Squid developers who can review identified defects, classify them using Coverity interfaces, and advise the Squid Project on the overall usefulness of the tool. Kinkie has offered his help (thank you!) and we are looking for two more volunteers. Please add a note to bug 3635 [1] if you want to participate. Regards, Dmitry [1] http://bugs.squid-cache.org/show_bug.cgi?id=3635
Re: Summary of store_url project and some questions before posting some patches.
lör 2012-10-06 klockan 17:24 +0200 skrev Eliezer Croitoru: Code implementation is quite simple but since I was working with the old bzr revision 12317 ant now its about 30 revisions up I dont have a clue on what to do. are you stuck on repository format upgrade, or on incompatible changes in later revisions of squid?
Re: Spaces in ACL values
fre 2012-09-14 klockan 08:23 -0600 skrev Alex Rousskov: Interesting. I did not expect much support for this, but two out of three responses so far suggest this approach, essentially. When the dust settles, perhaps we should post to squid-users as well to get more feedback? clarity have higer priority than backwards compatibility, within reason. but in this case we also have to deal with third party tools, so a toggle may also be needed, but unlike earlier suggestion in this thread it should default to the improved syntax. Regards Henrik
Re: I have a question about the md5 hash.
tor 2012-09-13 klockan 12:05 +1200 skrev Amos Jeffries: Just had a thought. I wonder if this is related to the releases which people suddenly started having cache MISS for a period with no visible reason. That could be the releases where we added/removed methods from the registered set. Only if the new methods were added before METHOD_GET. We might need to make this the actual method string instead of internal code to preserve store keys across squid versions. Yes. Regards Henrik
Re: I have a question about the md5 hash.
tor 2012-09-13 klockan 12:07 +0300 skrev Eliezer Croitoru: but in the logs is see a lot: 2012/09/13 11:48:51.798 kid1| StoreEntry::checkCachable: NO: not cachable Is it a 200 OK response. Suspect it's a 304? Regards Henrik
Re: [RFC] One helper to rewrite them all
ons 2012-09-12 klockan 10:18 +0300 skrev Eliezer Croitoru: For the no cachable situation something should be done to lower performance issues that could come from using store_url, some kind of flag validation before sending it to the helper? That only helps in the case where we know from the request that it's not cacheable. When the non-cache status is due to the response then cache lookup have already taken place and store url rewriting called. Regards Henrik
Re: [RFC] One helper to rewrite them all
tor 2012-09-13 klockan 11:20 +1200 skrev Amos Jeffries: I think long-term we want to have ICAP/eCAP generating ETag+Digest headers for requests and responses and store using those headers as the index key. How do you generate an ETag from a request? ETag is a response header, set by the origin server. Regards Henrik
Re: I have a question about the md5 hash.
ons 2012-09-12 klockan 10:26 +0300 skrev Eliezer Croitoru: if it was a string what will be the string structure? It's not a string. The first hashed octet is the Squid internal method representation in binary integer form. Followed by the requested URL. See storeKeyPublic() function for details http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/store_key_md5.cc#L118 And for method number assignment see HttpRequestMethod http://bazaar.launchpad.net/~squid/squid/3-trunk/view/head:/src/HttpRequestMethod.h#L41 Regards Henrik
bzr repository upgraded to 2a format and restructured
As per prior discussions the bzr repository have now been upgraded to 2a format. New master repository layout is a flat layout with branches trunk 3.2 3.1 3.0 more branches will be migrated over on a need basis. There is also branches/... symlinks in place so old branch URLs continue to work as well for those branches. Unfortunately upgrading to 2a means you need to upgrade your local repository as well if you haven't already or you won't be able to pull changes from the master repository. There is a couple of methods for doing that depending on if you are impatient or lazy The simple but slow method is by running bzr upgrade --2a this will upgrade your repository in-place to 2a format, but due to the fairly long history of Squid it takes a while as it replays the whole history to the new format. A somewhat faster method is to create a new repository branching from the master 2a repository, and then migrate over your branches there. An example of that follows bzr init-repo squid-new bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/trunk squid-new/trunk bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/3.2 squid-new/3.2 [repeat for each version you need] bzr branch path/to/local-branch-1 squid-new/local-branch-1 [repeat for each local branch] exact details differ slightly depending on what kind of local repository branch model you use. Feel free to grab me (or perhaps Robert/lifeless?) on irc if there is any doubt. Note: if you already have some 2a tree but not all then you can branch that into the repository first to speed up the initial download. Regards Henrik
Re: [RFC] One helper to rewrite them all
tis 2012-09-11 klockan 15:52 -0600 skrev Alex Rousskov: Hm... I wonder if we are making a design mistake here by following Squid2 steps: one helper to rewrite request URL, one helper to rewrite store URL, then one helper to rewrite some special HTTP header, etc. Would it be better to extend (in a backward compatible way) the URL rewriter interface so that ONE helper can do all rewriting that is needed (today and tomorrow)? haven't really seen that as a problem. url store url fits nicely in the same helper I think, but requires redefining the interface a bit, currently there is no defined manner for url rewriters to return anything else. not sure about headers, that would require a completely new helper interface, and have significant overhead for url rewriting. What about using eCAP? Is it possible to return extra attributes such as store url? Regards Henrik
Re: [RFC] One helper to rewrite them all
ons 2012-09-12 klockan 11:05 +1200 skrev Amos Jeffries: However, we want also to back-port this feature into 3.2 (3.1?) and maintain upgrade compatibility with 2.7 implementation, at least to start with. Do we really want to backport new features? Regards Henrik
Re: Failure ratio
mån 2012-09-10 klockan 15:11 +0100 skrev Konstantin Korsunov: All working good, but some times 3 times per day I have next error: Failure Ratio at 1.02 Going into hit-only-mode for 5 minutes... That means that an unexpected amount of the proxied requests have failed, and to avoid further network damabe your Squid stops responding with ICP_MISS responses to any cache peers you have. Usually this is due to a networking failure of some kind, but can also have natural causes if some client sends a lot of bad requests, i.e. if infested with a spam bot or similar. There is no difference for normal proxy users. The difference is only for cache peers using this proxy as a parent. Regards Henrik
Re: BZR repository upgrade re-proposal.
sön 2012-09-09 klockan 19:15 +0200 skrev Kinkie: It seems that the schedule was missed. Yes. Been ill since friday. Combination of sleep deprivation and a cold taking it's poll. When is the next chance? Haven't seen any indications of any remaining reasons to not being able to switch. Wednesday maybe? Not feeling too alert tonight and should probably not do anything needing privileges, and monday tuesday is a bit choked already. Regards Henrik
Re: [RFC] CONNECT and use_ssl cache_peer
sön 2012-09-09 klockan 21:34 +1200 skrev Amos Jeffries: Henrik and I seem to disagree on this being a good idea being plugged into BodyPipe. But as I see it BodyPipe is where Upgrade, WebSockets and SSL-Bump should be happening in a node which internally receives the tunnel connection and acts like a ConnStateData reading out of the tunnel. I am open for discussion on how it should fit, but initial reaction is that there may be too much http semantics in the message model there to fit well. When going DIRECT we have to assume the CONNECT is infinite length and handle as per now with closures. You always have to assume that unless there is some other indication. But ... I see a potential gap in the RFC texts that when communicating between two proxies we can do one of several methods (Expect-100 or peer OPTIONS) to negotiate chunking support for any data the CONNECT has following it. Not sure there is a gap, but I see your point. The data of a CONNECT is not the message body, not more than the data following an Upgrade handshake is. There is not much negotiation taking place in CONNECT. Any extensions has to be enabled by configuration. The 1.1+ protocol cache which can be used to negotiate chunked encoding in requests for other methods is useless for CONNECT as we cannot expect HTTP/1.1 servers to accept CONNECT with transfer-encoding. Thus keeping the connection between the peers (only) re-usable after the CONNECT is finished. So long as the transformation (chunked encoding) is stripped away at the recipient proxy this does not introduce problems to the transported end-to-end data stream. Yes... Regards Henrik
Re: [RFC] CONNECT and use_ssl cache_peer
fre 2012-09-07 klockan 21:54 -0600 skrev Alex Rousskov: However, when the proxy receives a CONNECT request, it diverts the request to tunnel.cc. Tunnel.cc is aware of peer selection logic but lacks the code that establishes an SSL connection to SSL peers. Yes, long time bug that no one has gotten arond to fix yet. My current plan is to try #3 to avoid code duplication but if it becomes too complex, go for #1 and duplicate what is necessary. What do you think? Any better ideas? Sounds like a great plan to me. Not too fond of moving CONNECT into ClientStream/StoreEntry intrefaces. It's not having any HTTP body like semantics. Regards Henrik
Re: BZR repository upgrade re-proposal.
ons 2012-09-05 klockan 09:03 +0200 skrev Kinkie: I'm glad that a consensus is forming. The next question would then be who will perform the upgrade, and when - assuming noone speaks up against it. This friday? Regards Henrik
Re: BZR repository upgrade re-proposal.
tis 2012-09-04 klockan 11:00 -0600 skrev Alex Rousskov: It would be nice to have specific instructions on how to update a local branch to support 2a. I see bzr upgrade command but it is not clear whether that is sufficient, especially with all the talk about two repositories (old and 2a). upgrade is sufficient, but quite time consuming even if you only have one branch to upgrade as it needs to redo the complete history from revision 1. An alternative is to reuse the already upgraded repository to avoid upgrading past revisions already upgraded. This is done by using a shared repository (many branches sharing the same store) where you first create branches the already upgraded branches and then your branches. bzr init-repo squid-new cd squid-new bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/trunk bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/3.2 bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/3.1 bzr branch bzr+ssh://bzr.squid-cache.org/bzr/squid3/3.0 bzr branch /path/to/your/repo name [repeat for each branch] you need to reset push/pull locations etc after. you can rename/move things as you please, including a full rename of the main repository path (squid-new). the only requirement is that branches need to stay somewhere below the repository location. Regards Henrik
Re: Native FTP proxying
tis 2012-09-04 klockan 12:22 -0600 skrev Alex Rousskov: A FTP client-side might be accepted, allowing Squid to act as an ftp-ftp gateway using HTTP semantics internally, even accepting ftp-http gatewaying if you want. But not sure it makes sense to add native FTP proxy support. Can you clarify the difference between ftp-ftp gateway and native FTP proxy support? Please assume that we are using HTTP semantics internally in both cases, which I interpret as not altering the forward.cc and Store code much -- the new FTP code will convert FTP commands into HTTP messages for internal processing and the code outside FTP client and server modules will remain pretty much the same. An ftp-ftp gateway using HTTP semantics internally is a FTP server component (client_side..) that accepts FTP commands from FTP clients and maps them to HTTP semantics which is then proxied by Squid. Native FTP proxy support means acting as an FTP proxy relaying FTP commands between client and requested server without going via an internal HTTP sematics mapping. The first is very feasible and maps nicely to Squid. The second is very hard to fit within Squid in a meaningful manner other than to provide basic server level access control, but is required by some.. Regards Henrik
Re: Native FTP proxying
tis 2012-09-04 klockan 16:29 -0600 skrev Alex Rousskov: Why would the second approach be required by some? In other words, what kind of FTP functionality the first approach cannot support? I do not favor the second approach, but I would like to know what we are losing by using the first approach (other than some performance). FTP is a rather messy protocol with many commands including site specific commands which can not be easily mapped to HTTP semantics And server state is also quite complex compared to HTTP. If you do cd /some/path/symlink cd .. then you end up in .. relative to the symlink, not /some/path/, which is a bit of a headache for mapping longer sessions. On servers using drive names/letters state gets even messier as you then usually have one active path per drive. I.e. windows/dos and various old mainframe OS:es. But we do not need to cater for them all unless you plan on intercepting port 21. And even then most of the odd ones can be ignored if there is an acl to bypass interception on selected sites or users. Regards Henrik
Re: BZR repository upgrade re-proposal.
tis 2012-09-04 klockan 10:56 +1200 skrev Amos Jeffries: FWIW: I'm in favour of the upgrade, it is a large part of the build timeout issues on rio Debian sid build slave, it causes timeouts while converting the repo changes and/or downloading large portions of repo changeset data before for each test. It is also the reason I'm only working with patches as merge candidates these days. Yes there is nothing holding back the upgrade to 2a repository format today. There is a 2a repository already prepared in /bzr/squid3-2a since some years back. It's not automatically pulling changes from /bzr/squid3 at the moment but is kept mostly up to date. There is also an alternative layout tree at /bzr/squid3-new-2a/ which flattens the tree to ../ trunk 3.0 3.1 3.2 getting rid of the CVS references and other legacy. Switching to squid3-2a is only a matter of switching the two folders. Switching to the alternative layout requires a bit more discussion as it has effects on branch URLS and our scripts. It is possible to add symlinks to allow both URL styles. Any local branches in old format needs upgrading as well. Due to the number of revisions i'd recommend setting up a 2a shared-repository from the already converted tree and branch your local old format trees into that for conversation. Regards Henrik
Re: [PATCH] Do not reuse persistent connections for PUTs
fre 2012-08-31 klockan 10:58 +0300 skrev Tsantilas Christos: 1) To decide if it can reuse a healthy persistent connection. Inside FwdState::connectStart, we are getting a persistant connection and even if it is healthy, if we want to send eg a PUT request we are closing the persistent connection and we are opening a new one. How do we know it's healthy? The issue here is that we cannot retry some kinds of requests, and HTTP requires us to handle when the server closes the connection while we are sending the request. It is a speed of light problem, we cannot know if the server is just about to close the connection or if it have already closed the connection but the FIN have not yet been seen on our side. From what I can understand there is not any reason to not reuse a healthy persistent connection, for PUT requests. Am I correct? If yes then the problem is only in the case (2). There is, because we often cannot retry the PUT because we have not kep a copy of the already partially sent PUT body, and it's expected to sometimes fail tp send requestso n a persistent connection due to above race condition. THe race condition between HTTP server and client (squid) is fairly big and easily fits many MB of traffic. Regards Henrik
Re: [PATCH] Do not reuse persistent connections for PUTs
fre 2012-08-31 klockan 21:44 +0300 skrev Tsantilas Christos: So looks that a good solution should be (similar to) the solution proposed by Henrik... 100 Continue aviods the race entirely on requests with bodies, leaving only bodyless requests in the we may not retry this on failure but we need to.. problem. Keeping a copy of the sent body allows the request to be retried in cases where 100 Continue were not used or could not be used. Regards Henrik
Re: [PATCH] Do not reuse persistent connections for PUTs
ons 2012-08-29 klockan 12:42 -0600 skrev Alex Rousskov: Hello, I saw bogus ERR_ZERO_SIZE_OBJECT responses while testing Squid v3.1, but the same problem ought to be present in v3.2 as well. A compliant proxy may retry PUTs, but Squid lacks the [rather complicated] code required to protect the PUT request body from being nibbled during the first try, when pconn races are possible. +1 on the patch from me. Regards Henrik
