Hudson build is back to normal : 3.HEAD-amd64-CentOS-5.3 #1101
See http://build.squid-cache.org/job/3.HEAD-amd64-CentOS-5.3/1101/changes
Re: [PATCH] NTLM authentication broken for Mozilla/3.0 User-Agents
On 25/01/11 20:56, Fabian Hugelshofer wrote: Hi, On 12/01/11 00:14, Henrik Nordström wrote: tis 2011-01-11 klockan 11:37 +0100 skrev Fabian Hugelshofer: What do you think about removing the special handling for Mozilla/3 and Netscape/3 agents from HttpMsg.cc? +1 from me. How large is the chance that there is still an affected browser in use? Pretty close to none. And if there are those can be fixed in their local configuration to disable the use of persistent connections. Thank you for your positive feedback. I removed the discussed parts from HttpMsg.cc. See the attached patch. In which versions would this be included? Squid 3.2? Patch applied. Thanks. I think its worth going back to 3.1 as well. Amos -- Please be using Current Stable Squid 2.7.STABLE9 or 3.1.10 Beta testers wanted for 3.2.0.4
[PATCH] external ACL non-caching update - round 2
Admin configure ttl=0 and/or negative_ttl=0 to prevent Squid storing the
ACL lookup results. The problem is that results still get cached and
re-used for the grace= period or one second, whichever is larger.
Also, in the event where two or more requests with identical details
needing to be looked up at the same time there is an optimization which
will merge and share one lookup result for all these requests.
In most situations this result sharing is beneficial, however when a
unique result is wanted it can cause problems.
This patch makes ttl=0 and negative_ttl=0 prevent their respective OK
and ERR results from being stored into the helper result cache. Sharing
is still performed for overlapping duplicate requests.
When cache=0 is configured, no caching or sharing of results is
performed at all.
Amos
=== modified file 'src/external_acl.cc'
--- src/external_acl.cc 2011-01-25 05:31:59 +
+++ src/external_acl.cc 2011-01-25 11:25:02 +
@@ -730,11 +730,16 @@
entry = ch-extacl_entry;
if (entry) {
-if (cbdataReferenceValid(entry) entry-def == acl-def
-strcmp((char *)entry-key, key) == 0) {
+if (cbdataReferenceValid(entry) entry-def == acl-def) {
/* Ours, use it.. */
} else {
/* Not valid, or not ours.. get rid of it */
+debugs(82, 9, aclMatchExternal: entry entry not valid or not ours. Discarded.);
+if (entry) {
+debugs(82, 9, aclMatchExternal: entry def= entry-def , our def= acl-def);
+key = makeExternalAclKey(ch, acl);
+debugs(82, 9, aclMatchExternal: entry key=' (char *)entry-key ', our key=' key ');
+}
cbdataReferenceDone(ch-extacl_entry);
entry = NULL;
}
@@ -743,6 +748,7 @@
external_acl_message = MISSING REQUIRED INFORMATION;
if (!entry) {
+debugs(82, 9, aclMatchExternal: No helper entry available);
if (acl-def-require_auth) {
int ti;
/* Make sure the user is authenticated */
@@ -850,6 +856,10 @@
static void
external_acl_cache_touch(external_acl * def, external_acl_entry * entry)
{
+// this must not be done when nothing is being cached.
+if (def-cache_size = 0 || (def-ttl = 0 entry-result == 1) || (def-negative_ttl = 0 entry-result != 1));
+return;
+
dlinkDelete(entry-lru, def-lru_list);
dlinkAdd(entry, entry-lru, def-lru_list);
}
@@ -1091,6 +1101,9 @@
static int
external_acl_entry_expired(external_acl * def, external_acl_entry * entry)
{
+if (def-cache_size = 0)
+return 1;
+
if (entry-date + (entry-result == 1 ? def-ttl : def-negative_ttl) squid_curtime)
return 1;
else
@@ -1100,6 +1113,9 @@
static int
external_acl_grace_expired(external_acl * def, external_acl_entry * entry)
{
+if (def-cache_size = 0)
+return 1;
+
int ttl;
ttl = entry-result == 1 ? def-ttl : def-negative_ttl;
ttl = (ttl * (100 - def-grace)) / 100;
@@ -1113,12 +1129,24 @@
static external_acl_entry *
external_acl_cache_add(external_acl * def, const char *key, ExternalACLEntryData const data)
{
-ExternalACLEntry *entry = static_castExternalACLEntry *(hash_lookup(def-cache, key));
+ExternalACLEntry *entry;
+
+// do not bother caching this result if TTL is going to expire it immediately
+if (def-cache_size = 0 || (def-ttl = 0 data.result == 1) || (def-negative_ttl = 0 data.result != 1)) {
+debugs(82,6, HERE);
+entry = new ExternalACLEntry;
+entry-key = xstrdup(key);
+entry-update(data);
+entry-def = def;
+return entry;
+}
+
+entry = static_castExternalACLEntry *(hash_lookup(def-cache, key));
debugs(82, 2, external_acl_cache_add: Adding ' key ' = data.result);
if (entry) {
debugs(82, 3, ExternalACLEntry::update: updating existing entry);
-entry-update (data);
+entry-update(data);
external_acl_cache_touch(def, entry);
return entry;
@@ -1126,7 +1154,7 @@
entry = new ExternalACLEntry;
entry-key = xstrdup(key);
-entry-update (data);
+entry-update(data);
def-add(entry);
@@ -1136,7 +1164,7 @@
static void
external_acl_cache_delete(external_acl * def, external_acl_entry * entry)
{
-assert (entry-def == def);
+assert(def-cache_size 0 entry-def == def);
hash_remove_link(def-cache, entry);
dlinkDelete(entry-lru, def-lru_list);
def-cache_entries -= 1;
@@ -1200,7 +1228,7 @@
char *status;
char *token;
char *value;
-char *t;
+char *t = NULL;
ExternalACLEntryData entryData;
entryData.result = 0;
external_acl_entry *entry = NULL;
@@ -1313,12 +1341,15 @@
entry = NULL;
/* Check for a pending lookup to hook into */
-for (node = def-queue.head; node; node = node-next) {
-externalAclState *oldstatetmp =
NTLM passthrough broken in 3.1.3 and higher
In revision 9957 (Remove HTTP/1.1 sent to clients), NTLM passthrough was broken for Outlook clients, which require 1.1 to work properly. When receiving 1.0, Outlook closes the connection (which breaks NTLM since it requires a continued session). Version 3.1.2 works fine for these clients. Note this is likely the cause of bug 3141. In 2.7, the http_port option 'http11' allowed forcing version 1.1 to clients. That was removed when 3.1 defaulted to 1.1, but when this change was reverted, the http11 option was not added back. Squid 3.2 latest works fine, since 1.1 is now the default again. Can we either a) make 1.1 the default in 3.1 again b) add back the 'http11' option Thanks, Phil
Re: NTLM passthrough broken in 3.1.3 and higher
On Tue, 25 Jan 2011 13:26:02 -0800, Phil Oester wrote: In revision 9957 (Remove HTTP/1.1 sent to clients), NTLM passthrough was broken for Outlook clients, which require 1.1 to work properly. When receiving 1.0, Outlook closes the connection (which breaks NTLM since it requires a continued session). Version 3.1.2 works fine for these clients. Note this is likely the cause of bug 3141. In 2.7, the http_port option 'http11' allowed forcing version 1.1 to clients. That was removed when 3.1 defaulted to 1.1, but when this change was reverted, the http11 option was not added back. Squid 3.2 latest works fine, since 1.1 is now the default again. Can we either a) make 1.1 the default in 3.1 again b) add back the 'http11' option The http11 option has never been part of squid-3. It was an experiment in 2.7 to see if sending 1.1 was possible yet. It looked successful so we made it live in 3.1. But had to cut it out again quickly as you noticed. It turns out the common browser software all wants to actually use HTTP/1.1 performance features which 3.1 does not support. Feel free to undo the removal patch for your Squid-3.1 if you find those problems not to be relevant on your network. Or to use 3.2. Amos
Re: /bzr/squid3/trunk/ r11181: Portability fix: icc doesn't like string literal in assert checks
On Tue, 25 Jan 2011 23:19:46 +0100, Francesco Chemolli wrote: revno: 11181 committer: Francesco Chemolli kin...@squid-cache.org branch nick: trunk timestamp: Tue 2011-01-25 23:19:46 +0100 message: Portability fix: icc doesn't like string literal in assert checks modified: lib/MemPool.cc lib/MemPoolChunked.cc lib/MemPoolMalloc.cc This was bug 2868 http://bugs.squid-cache.org/show_bug.cgi?id=2868 If its not too late can you redo this commit please with --fixes squid:2868 ? Otherwise just close the bug as fixed. Amos
Re: /bzr/squid3/trunk/ r11181: Portability fix: icc doesn't like string literal in assert checks
On Tue, Jan 25, 2011 at 11:26 PM, Amos Jeffries squ...@treenet.co.nz wrote: On Tue, 25 Jan 2011 23:19:46 +0100, Francesco Chemolli wrote: revno: 11181 committer: Francesco Chemolli kin...@squid-cache.org branch nick: trunk timestamp: Tue 2011-01-25 23:19:46 +0100 message: Portability fix: icc doesn't like string literal in assert checks modified: lib/MemPool.cc lib/MemPoolChunked.cc lib/MemPoolMalloc.cc This was bug 2868 http://bugs.squid-cache.org/show_bug.cgi?id=2868 If its not too late can you redo this commit please with --fixes squid:2868 ? Otherwise just close the bug as fixed. Sure, I'll try. -- /kinkie
Re: /bzr/squid3/trunk/ r11181: Portability fix: icc doesn't like string literal in assert checks
On Tue, Jan 25, 2011 at 11:39 PM, Kinkie gkin...@gmail.com wrote: On Tue, Jan 25, 2011 at 11:26 PM, Amos Jeffries squ...@treenet.co.nz wrote: On Tue, 25 Jan 2011 23:19:46 +0100, Francesco Chemolli wrote: revno: 11181 committer: Francesco Chemolli kin...@squid-cache.org branch nick: trunk timestamp: Tue 2011-01-25 23:19:46 +0100 message: Portability fix: icc doesn't like string literal in assert checks modified: lib/MemPool.cc lib/MemPoolChunked.cc lib/MemPoolMalloc.cc This was bug 2868 http://bugs.squid-cache.org/show_bug.cgi?id=2868 If its not too late can you redo this commit please with --fixes squid:2868 ? Otherwise just close the bug as fixed. Sure, I'll try. Ok done. What is the process for queueing it up for backporting? Thanks. -- /kinkie
Re: /bzr/squid3/trunk/ r11181: Portability fix: icc doesn't like string literal in assert checks
On Tue, 25 Jan 2011 23:50:40 +0100, Kinkie wrote: On Tue, Jan 25, 2011 at 11:39 PM, Kinkie wrote: On Tue, Jan 25, 2011 at 11:26 PM, Amos Jeffries wrote: On Tue, 25 Jan 2011 23:19:46 +0100, Francesco Chemolli wrote: revno: 11181 committer: Francesco Chemolli branch nick: trunk timestamp: Tue 2011-01-25 23:19:46 +0100 message: Portability fix: icc doesn't like string literal in assert checks modified: lib/MemPool.cc lib/MemPoolChunked.cc lib/MemPoolMalloc.cc This was bug 2868 http://bugs.squid-cache.org/show_bug.cgi?id=2868 If its not too late can you redo this commit please with --fixes squid:2868 ? Otherwise just close the bug as fixed. Sure, I'll try. Ok done. What is the process for queueing it up for backporting? Just a heads-up to me (like this). Otherwise I tend to go by the reported earliest sighting version in the bug report or if it looks important and is easy to port back. Amos
Re: /bzr/squid3/trunk/ r11181: Portability fix: icc doesn't like string literal in assert checks
Ok done. What is the process for queueing it up for backporting? Just a heads-up to me (like this). Otherwise I tend to go by the reported earliest sighting version in the bug report or if it looks important and is easy to port back. Could it be a nice candidate for a custom multi-value bugzilla field? setting it and assigning the bug to you would mark it as backport candidate; you decide what to do and when and close it after you're done. -- /kinkie
Re: /bzr/squid3/trunk/ r11178: Added specific flags for Intel(R)'s icc compiler
*sigh*. An unrelated fix slipped in.
Do you want me to recommit or is it trivial enough that it can be left
like this?
K.
On Tue, Jan 25, 2011 at 10:29 PM, Francesco Chemolli
kin...@squid-cache.org wrote:
revno: 11178 [merge]
committer: Francesco Chemolli kin...@squid-cache.org
branch nick: trunk
timestamp: Tue 2011-01-25 22:29:23 +0100
message:
Added specific flags for Intel(R)'s icc compiler
modified:
acinclude/compiler-flags.m4
lib/smblib/smblib-util.c
=== modified file 'acinclude/compiler-flags.m4'
--- a/acinclude/compiler-flags.m4 2010-08-24 10:35:03 +
+++ b/acinclude/compiler-flags.m4 2011-01-25 21:29:23 +
@@ -86,6 +86,15 @@
#endif
]])],[squid_cv_compiler=sunstudio],[])
fi
+ dnl Intel CC
+ if test -z $squid_cv_compiler ; then
+ AC_COMPILE_IFELSE([
+ AC_LANG_PROGRAM([[
+#if !defined(__ICC)
+#error not Intel(R) C++ Compiler
+#endif
+ ]])],[squid_cv_compiler=icc],[])
+ fi
dnl end of block to be repeated
if test -z $squid_cv_compiler ; then
squid_cv_compiler=none
@@ -122,6 +131,13 @@
squid_cv_cc_option_optimize=-fast
squid_cv_cc_arg_pipe=
;;
+ icc)
+ squid_cv_cxx_option_werror=-Werror
+ squid_cv_cc_option_werror=$squid_cv_cxx_option_werror
+ squid_cv_cc_option_wall=-Wall
+ squid_cv_cc_option_optimize=-O2
+ squid_cv_cc_arg_pipe=
+ ;;
*)
squid_cv_cxx_option_werror=
squid_cv_cc_option_werror=
=== modified file 'lib/smblib/smblib-util.c'
--- a/lib/smblib/smblib-util.c 2010-10-17 03:24:24 +
+++ b/lib/smblib/smblib-util.c 2011-01-11 22:25:34 +
@@ -48,11 +48,13 @@
-1
};
+#if UNDEFINED
char *SMB_DOSTimToStr(int DOS_time);
char *SMB_AtrToStr(int attribs, BOOL verbose);
int SMB_Get_Tree_MBS(SMB_Tree_Handle tree);
int SMB_Get_Max_Buf_Siz(SMB_Handle_Type Con_Handle);
int SMB_Get_Protocol_IDX(SMB_Handle_Type Con_Handle);
+#endif /* UNDEFINED */
int SMB_Get_Protocol(SMB_Handle_Type Con_Handle);
int SMB_Figure_Protocol(const char *dialects[], int prot_index);
int SMB_TreeDisconnect(SMB_Tree_Handle Tree_Handle, BOOL discard);
@@ -80,6 +82,7 @@
/* Convert a DOS Date_Time to a local host type date time for printing */
+#if UNDEFINED
char *SMB_DOSTimToStr(int DOS_time)
{
@@ -172,6 +175,7 @@
}
}
+#endif /* UNDEFINED */
/* Pick up the protocol from the connection structure
*/
--
/kinkie
Re: [PATCH] bug 3081: comm layer cleanups for TCP acceptor
On 01/22/2011 06:46 AM, Amos Jeffries wrote: However we cannot drop the CommCbFunPtrCallT and AsyncCallT copy constructors. Notice how they explicitly call AsyncCall(a,b,c) instead of the AsyncCall copy contructor. Without this we loose the call debugs() and the ID creation. Adding the assignment operator (undefined) and destructor (no-op) to match our big-three requirements. New version attached. This clears up all the points we have been talking about. As well as several duplicate calls to switchTimeoutToData() in FTP. Thank you for polishing this! I do not see any more problems. This is minor, but you do not need to use the inline keyword for methods that are immediately inlined, such as CommCbFunPtrCallT copy constructor. Cheers, Alex.
Re: /bzr/squid3/trunk/ r11178: Added specific flags for Intel(R)'s icc compiler
On Wed, 26 Jan 2011 00:37:56 +0100, Kinkie wrote: *sigh*. An unrelated fix slipped in. Do you want me to recommit or is it trivial enough that it can be left like this? The smblib removals? its a bit far back now, but as you are the only comitter since it could be done. I can split it on the back port. I guess there is good reason to do the maintenance tonight while I still remember it :). On the smblib and rfcnb side... did I already point out that being 3rd party library and I was trying to keep it as close to the original as reasonable? Still keeping ears open for any replacement newer than 1998 :). Some of these removal changes look like things that might show up in only one of the helpers linking against those lib. Please be sure basic_msnt_auth and ntlm_smb_lm_auth are both being built in your test environment. Amos
Re: [PATCH] bug 3081: comm layer cleanups for TCP acceptor
On Tue, 25 Jan 2011 17:42:03 -0700, Alex Rousskov rouss...@measurement-factory.com wrote: On 01/22/2011 06:46 AM, Amos Jeffries wrote: However we cannot drop the CommCbFunPtrCallT and AsyncCallT copy constructors. Notice how they explicitly call AsyncCall(a,b,c) instead of the AsyncCall copy contructor. Without this we loose the call debugs() and the ID creation. Adding the assignment operator (undefined) and destructor (no-op) to match our big-three requirements. New version attached. This clears up all the points we have been talking about. As well as several duplicate calls to switchTimeoutToData() in FTP. Thank you for polishing this! I do not see any more problems. This is minor, but you do not need to use the inline keyword for methods that are immediately inlined, such as CommCbFunPtrCallT copy constructor. Cheers, Alex. Wonderful. Thank you so much for the time you have been putting into the audits. Will fix that inline and commit in a few hours. Amos
Re: Sharing DNS cache among Squid workers
On 01/13/2011 03:54 PM, Amos Jeffries wrote: On 14/01/11 11:20, Robert Collins wrote: On Fri, Jan 14, 2011 at 11:13 AM, Alex Rousskov rouss...@measurement-factory.com wrote: On 01/13/2011 02:18 PM, Robert Collins wrote: Have you considered just having a caching-only local DNS server colocated on the same machine? I am sure that would be an appropriate solution in some environments. On the other hand, sometimes the box has no capacity for another server. Sometimes the traffic from 8-16 Squids can be too much for a single DNS server to handle. And sometimes administration/policy issues would prevent using external caching DNS servers on the Squid box. This surprises me - surely the CPU load for a dedicated caching DNS server is equivalent to the CPU load for squid maintaining a DNS cache itself; and DNS servers are also multithreaded? Anyhow, I've no particular objection to it being in the code base, but it does seem like something we'd get better results by not doing (or having a defined IPC mechanism to a single (possibly multi-core) cache process which isn't a 'squid'. [Even if it is compiled in the squid tree]. -Rob Thats pretty much my opinion too. A shared resolver on the same box where possible is our best-practice anyway. Where DNS speed is important users have their DNS as close as possible to the Squid. It maybe worthwhile instead researching the lightest available DNS resolver and using that as a recommendation to assist people. When the workers are doing shared memory blocks merging these caches would be worthwhile to de-duplicate the entries. But until then its just adding complexity. If we implement DNS cache sharing among workers, then the shared caches will share entries (using shared memory), of course. Or did you mean something else by doing shared memory blocks? Thank you, Alex.
Re: Sharing DNS cache among Squid workers
On 01/13/2011 03:20 PM, Robert Collins wrote: On Fri, Jan 14, 2011 at 11:13 AM, Alex Rousskov rouss...@measurement-factory.com wrote: On 01/13/2011 02:18 PM, Robert Collins wrote: Have you considered just having a caching-only local DNS server colocated on the same machine? I am sure that would be an appropriate solution in some environments. On the other hand, sometimes the box has no capacity for another server. Sometimes the traffic from 8-16 Squids can be too much for a single DNS server to handle. And sometimes administration/policy issues would prevent using external caching DNS servers on the Squid box. This surprises me - surely the CPU load for a dedicated caching DNS server is equivalent to the CPU load for squid maintaining a DNS cache itself; and DNS servers are also multithreaded? A general-purpose caching DNS server usually does more than Squid does, so I would not be surprised if it uses more CPU cycles. And even if a super-optimized DNS server is fast enough, administration/policy issues may still prevent it from being installed on Squid boxes. And FWIW, BIND multi-threading is rather poor. I am sure there are faster servers out there, but the most popular one may not work as fast as the DNS server component of SMP Squid. Thank you, Alex.
Re: Sharing DNS cache among Squid workers
On Tue, 25 Jan 2011 17:58:36 -0700, Alex Rousskov rouss...@measurement-factory.com wrote: On 01/13/2011 03:54 PM, Amos Jeffries wrote: On 14/01/11 11:20, Robert Collins wrote: On Fri, Jan 14, 2011 at 11:13 AM, Alex Rousskov rouss...@measurement-factory.com wrote: On 01/13/2011 02:18 PM, Robert Collins wrote: Have you considered just having a caching-only local DNS server colocated on the same machine? I am sure that would be an appropriate solution in some environments. On the other hand, sometimes the box has no capacity for another server. Sometimes the traffic from 8-16 Squids can be too much for a single DNS server to handle. And sometimes administration/policy issues would prevent using external caching DNS servers on the Squid box. This surprises me - surely the CPU load for a dedicated caching DNS server is equivalent to the CPU load for squid maintaining a DNS cache itself; and DNS servers are also multithreaded? Anyhow, I've no particular objection to it being in the code base, but it does seem like something we'd get better results by not doing (or having a defined IPC mechanism to a single (possibly multi-core) cache process which isn't a 'squid'. [Even if it is compiled in the squid tree]. -Rob Thats pretty much my opinion too. A shared resolver on the same box where possible is our best-practice anyway. Where DNS speed is important users have their DNS as close as possible to the Squid. It maybe worthwhile instead researching the lightest available DNS resolver and using that as a recommendation to assist people. When the workers are doing shared memory blocks merging these caches would be worthwhile to de-duplicate the entries. But until then its just adding complexity. If we implement DNS cache sharing among workers, then the shared caches will share entries (using shared memory), of course. Or did you mean something else by doing shared memory blocks? Yes. mutually accessible memory / shared memory where one writes and all can read from the same memory space was what I meant. I thought you were proposing to do it on top of the existing IPC channels. *copying* data around. If you do implement this shared memory backend, it is probably worth doing it as a generic cache which can be leveraged via inheritance from both ipcache/fqdncache and the helper caches etc, etc. They all have a read-often write-rarely structure with small key/value data records. Amos
Re: /bzr/squid3/trunk/ r11178: Added specific flags for Intel(R)'s icc compiler
On Wed, Jan 26, 2011 at 1:50 AM, Amos Jeffries squ...@treenet.co.nz wrote: On Wed, 26 Jan 2011 00:37:56 +0100, Kinkie wrote: *sigh*. An unrelated fix slipped in. Do you want me to recommit or is it trivial enough that it can be left like this? The smblib removals? its a bit far back now, but as you are the only comitter since it could be done. I can split it on the back port. I guess there is good reason to do the maintenance tonight while I still remember it :). Thanks. On the smblib and rfcnb side... did I already point out that being 3rd party library and I was trying to keep it as close to the original as reasonable? You didn't, but I suspected that it was due to that reason. Still keeping ears open for any replacement newer than 1998 :). Some of these removal changes look like things that might show up in only one of the helpers linking against those lib. Please be sure basic_msnt_auth and ntlm_smb_lm_auth are both being built in your test environment. I grepped the whole source tree for references before removing. There were none. -- /kinkie
Re: Unparseable HTTP header field DNS weirdness
Do you have an upstream proxy configured?
Cheers,
On 21/01/2011, at 3:29 AM, Alex Ray wrote:
This might be nothing, but I notice the following errors in my build
of squid 3.HEAD:
kid1| ctx: enter level 0:
'http://trailers.apple.com/trailers/global/styles/ipad_black.css
kid1| WARNING: unparseable HTTP header field {: , 1.1 cup-www-cache01: 80}
kid1| WARNING: unparseable HTTP header field {: , 1.1 cup-www-cache02: 80}
kid1| ctx: exit level 0
--
Mark Nottingham m...@yahoo-inc.com
