Re: assert(e->mem_status == NOT_IN_MEMORY) versus TCP_MEM_HIT.

[Henrik Nordstrom]

fre 2009-09-25 klockan 23:40 +1200 skrev Amos Jeffries:
> This is the current blocker on trunk...
> 
> Attempting to swap-in an object which is already in memory and on disk.

Right, there is a timing race uncovered by the swapin changes.

1. store client created and type set to disk client.

[some time may pass]

2. First doCopy call triggers the start of the swapin.


I see two ways of fixing this:

a) Move the swapin start to 1.

b) Make store clients automatically switch from disk to mem type if the
object is found to be already swapped in in doCopy

or alternatively

c) Remove the assert. It's harmless.

Regards
Henrik

Re: Segfault in HTCP CLR request on 64-bit

[Adrian Chadd]

Could you please create a bugzilla report for this, complete with a
patch against Squid-2.HEAD and 2.7? I'll then commit it.

2009/9/26 Jason Noble :
> I recently ran into an issue where Squid 2.7 would segfault trying to issue
> HTCP CLR requests.  I found the segfault only occurred on 64-bit machines.
>  While debugging, I found that the value of stuff.S.req_hdrs was not
> initialized but later, strlen was being called on it.  This seems to -- by
> chance -- not fail on 32 bit builds, but always segfaults on 64-bit.  The
> attached patch fixed the problem for me and it seems good programming
> practice to properly initialize pointers to prevent issues such as this.  As
> the htcpStuff struct is used in other places, I have concerns that other
> issues may be lurking as well, although I have yet to run into them.
>
> Regards,
> Jason
>

Squid 3.1 kerb auth helper

[Amos Jeffries]

Hi Markus,
  We've hit another issue with the new 3.1 helper. This time on Debian...

Luigi Gangitano wrote:
>
> At last, I've a small issue with negotiate_auth/squid_kerb_auth, which
> set RPATH in binaries. configure script adds '-Wl,-R' to the linker
> under squid. This is an issue for debian (see
> http://wiki.debian.org/RpathIssue). Do you think this can be fixed? I
> can make a debian patch if needed.

Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.13

Re: Build failed in Hudson: 3.HEAD-i386-FreeBSD-6.4 #72

[Henrik Nordstrom]

fre 2009-09-25 klockan 14:49 +0200 skrev n...@squid-cache.org:

> sed: 1: " s...@default_http_port@% ...": unbalanced brackets ([])
> *** Error code 1

Hmm.. what is this about?

The actual failing line is

sed " s...@default_http_port@%3128%g; s...@default_icp_port@%3130%g; 
s...@default_cache_effective_user@%nobody%g; 
s...@default_mime_table@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc/mime.conf%g;
 
s...@default_dnsserver@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/libexec/`echo
 dnsserver | sed 's,x,x,;s/$//'`%g; 
s...@default_unlinkd@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/libexec/`echo
 unlinkd | sed 's,x,x,;s/$//'`%g; 
s...@default_pinger@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/libexec/`echo
 pinger | sed 's,x,x,;s/$//'`%g; 
s...@default_diskd@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/libexec/`echo
 diskd | sed 's,x,x,;s/$//'`%g; 
s...@default_cache_log@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/logs/cache.log%g;
 
s...@default_access_log@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/logs/access.log%g;
 
s...@default_store_log@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/logs/store.log%g;
 
s...@default_pid_file@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/squid.pid%g;
 
s...@default_netdb_file@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/logs/netdb.state%g;
 
s...@default_swap_dir@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/var/cache%g;
 
s...@default_icon_dir@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/share/icons%g;
 
s...@default_error_dir@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/share/errors%g;
 
s...@default_config_dir@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst/etc%g;
 
s...@default_prefix@%/home/hudson/workspace/3.HEAD-i386-FreeBSD-6.4/btlayer-00-default/squid-3.HEAD-BZR/_inst%g;
 s...@default_hosts@%/etc/hosts%g; s...@[v]ersion@%3.HEAD-BZR%g;"


I can't see anything wrong with that, certainly no unbalanced brackets..

Note: the [V]ERSION thing which is the only brackets in that line is a hack to 
get around VERSION being some magic keyword iirc, and has been there since 2003.

Regards
Henrik

Segfault in HTCP CLR request on 64-bit

[Jason Noble]

I recently ran into an issue where Squid 2.7 would segfault trying to 
issue HTCP CLR requests.  I found the segfault only occurred on 64-bit 
machines.  While debugging, I found that the value of stuff.S.req_hdrs 
was not initialized but later, strlen was being called on it.  This 
seems to -- by chance -- not fail on 32 bit builds, but always segfaults 
on 64-bit.  The attached patch fixed the problem for me and it seems 
good programming practice to properly initialize pointers to prevent 
issues such as this.  As the htcpStuff struct is used in other places, I 
have concerns that other issues may be lurking as well, although I have 
yet to run into them.


Regards,
Jason
diff --git a/src/htcp.c b/src/htcp.c
index b262d0f..5cfe715 100644
--- a/src/htcp.c
+++ b/src/htcp.c
@@ -1265,6 +1265,7 @@ htcpClear(StoreEntry * e, const char *uri, request_t * req
 stuff.f1 = 0;
 stuff.response = 0;
 stuff.msg_id = ++msg_id_counter;
+stuff.S.req_hdrs = 0;
 switch (reason) {
 case HTCP_CLR_INVALIDATION:
stuff.reason = 1;

Build failed in Hudson: 3.HEAD-i386-FreeBSD-6.4 #72

[noc]

See 

Changes:

[Amos Jeffries ] Author: Adrian Chadd 

A tproxy cache cluster (eg behind WCCPv2) can't peer.

The issue stems from the forwarding logic creating source address spoofed
sockets to destinations that are inside the cluster. Since the WCCPv2
router won't redirect packets with an origin of the proxy MAC (at least for
L2 peering), source spoofed packets go out and are routed normally. The
packets back from the destination peer have a remote end of the spoofed IP,
and are instead sent to teh original client rather than the proxy.

The forwarding logic needs to be taught to optionally enable tproxy source
spoofing on connections based on a peer flag.

Just for completeness - tproxy'ed connections to a upstream or peer proxy
which is -outside- of the WCCPv2 tproxy cluster work fine.

[Amos Jeffries ] Author: Markus Moeller 

Add KRB5INCS to INCLUDES

Needs to be global because we use the kerb5 com_err.h hack globally
through the libcompat headers.

NP: There should be a compat/*.am for these I guess...

--
[...truncated 2033 lines...]
checking whether ln -s works... yes
checking how to recognize dependent libraries... pass_all
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... none
checking how to run the C++ preprocessor... g++ -E
checking for g77... no
checking for xlf... no
checking for f77... f77
checking whether we are using the GNU Fortran 77 compiler... yes
checking whether f77 accepts -g... yes
checking the maximum length of command line arguments... 196608
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC
checking if gcc PIC flag -fPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
configure: creating libtool
appending configuration tag "CXX" to libtool
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC
checking if g++ PIC flag -fPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
(cached) (cached) checking how to hardcode library paths into programs... 
immediate
appending configuration tag "F77" to libtool
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for f77 option to produce PIC... -fPIC
checking if f77 PIC flag -fPIC works... yes
checking if f77 static flag -static works... yes
checking if f77 supports -c -o file.o... yes
checking whether the f77 linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
(cached) (cached) checking how to hardcode library paths into programs... 
immediate
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking which extension is used for loadable modules... .so
checking which variable specifies run-time library path... LD_LIBRARY_PATH
checking for the default library search path... /lib /usr/lib
checking for objdir... .libs
checking whether libtool supports -dlopen/-dlpreopen... yes
checking for shl_load... no
checking for shl_load in -ldld... no
checking for dlopen in -ldl... no
checking for dlerror... yes
checking for _ prefix in compiled symbols... no
checking whether deplibs are loaded by dlopen... yes
checking argz.h usability... no
checking argz.h prese

Build failed in Hudson: 3.HEAD-i386-Debian-sid #62

[noc]

See 

--
Started by upstream project "3.HEAD-amd64-CentOS-5.3" build number 116
Building remotely on rio.treenet
bzr: ERROR: Invalid http response for 
http://www.squid-cache.org/bzr/squid3/trunk/.bzr/branch-format: Bad status line 
received
Using saved parent location: http://www.squid-cache.org/bzr/squid3/trunk/
ERROR: Failed to pull

assert(e->mem_status == NOT_IN_MEMORY) versus TCP_MEM_HIT.

[Amos Jeffries]

This is the current blocker on trunk...

Attempting to swap-in an object which is already in memory and on disk.


Program received signal SIGABRT, Aborted.

#3  0x080884c0 in xassert (msg=0x81e0480 "e->mem_status == 
NOT_IN_MEMORY", file=0x81e0551 "store_swapin.cc", line=47) at debug.cc:557

#4  0x08120d9c in storeSwapInStart (sc=0x86b6c38) at store_swapin.cc:47
#5  0x08118e58 in store_client::startSwapin (this=0x86b6c38) at 
store_client.cc:393
#6  0x081191a5 in store_client::doCopy (this=0x86b6c38, 
anEntry=0xb6f3fb90) at store_client.cc:375
#7  0x08119267 in storeClientCopy2 (e=0xb6f3fb90, sc=0x86b6c38) at 
store_client.cc:331
#8  0x0807a867 in clientReplyContext::doGetMoreData (this=0xb6d39018) at 
client_side_reply.cc:1679
#9  0x0807b04b in clientReplyContext::identifyStoreObject 
(this=0xb6d39018) at client_side_reply.cc:1457
#10 0x0807d997 in ClientHttpRequest::httpStart (this=0x86e6c98) at 
client_side_request.cc:1121
#11 0x0807e11d in ClientHttpRequest::doCallouts (this=0x86e6c98) at 
client_side_request.cc:1323

#12 0x0807f5ff in ClientRequestContext::clientRedirectDone (this=0x86f4050,
result=0xb79a800a "http://web.mac.com/tomm..._rollover_0.png";) at 
client_side_request.cc:1047

#13 0x080fe820 in redirectHandleReply (data=0x86f81d8,
reply=0xb79a800a "http://web.mac.com/tomm..._rollover_0.png";) at 
redirect.cc:81

#14 0x080bf55f in helperHandleRead (fd=9,
buf=0xb79a8008 "0 http://web.mac.com/tomm..._rollover_0.png";, 
len=148, flag=COMM_OK, xerrno=0, data=0x84283a8) at helper.cc:913

#15 0x08146d75 in CommIoCbPtrFun::dial (this=0x8761d6c) at CommCalls.cc:183
#16 0x081396e3 in AsyncCall::make (this=0x8761d50) at AsyncCall.cc:34


(gdb) f 4
#4  0x08120d9c in storeSwapInStart (sc=0x86b6c38) at store_swapin.cc:47
47  in store_swapin.cc

(gdb) p *sc
$9 = {cmp_offset = 0, entry = 0xb6f3fb90, swapin_sio = {p_ = 0x0}, flags 
= {disk_io_pending = 0, store_copying = 1, copy_event_pending = 0},
  delayId = {pool_ = 0, compositeId = {p_ = 0x0}, markedAsNoDelay = 
false}, node = {data = 0x86b6c38, prev = 0x0, next = 0x0}, copyInto = {
flags = {error = 0}, length = 4096, offset = 0, data = 0xb6d5b024 
""}, static CBDATA_store_client = 26, type = 2, object_ok = true,
  _callback = {callback_handler = 0x807ba30 
, callback_data = 
0xb6d39018}}



(gdb) p *e
$2 = {<_hash_link> = {key = 0xb6eedc48, next = 0xb6f86074}, 
_vptr.StoreEntry = 0x81de6c8, mem_obj = 0x8618b28, repl = {data = 
0xb701001c},
  timestamp = 1253865128, lastref = 1253865618, expires = 1253865188, 
lastmod = 1237898828, swap_file_sz = 10030, refcount = 474,
  flags = 1122, swap_filen = 97209, swap_dirn = 0, lock_count = 1, 
mem_status = IN_MEMORY, ping_status = PING_NONE, store_status = STORE_OK,

  swap_status = SWAPOUT_DONE, static pool = 0x8616058}

(gdb) p *(e->mem_obj)
$10 = {method = {static RequestMethodStr = 0x8235460, theMethod = 
METHOD_GET, theImage = {static npos = , size_ = 0, len_ = 0,

  buf_ = 0x0}},
  url = 0x876cc18 "http://web.mac.com/tomm..._rollover_0.png";, data_hdr 
= {inmem_hi = 3886, nodes = {head = 0x84abf38, elements = 1}}, inmem_lo 
= 0, clients = {head = 0x86b6c58,
tail = 0x86b6c58}, nclients = 1, swapout = {queue_offset = 0, 
memnode = 0x0, sio = {p_ = 0x0}}, request = 0x0, start_ping = {tv_sec = 0,
tv_usec = 0}, ping_reply_callback = 0, ircb_data = 0x0, abort = 
{callback = 0, data = 0x0},
  log_url = 0x876c080 
"http://www.restorativejustice.com/..._rollover_0.png";,
  repl = {data = 0xb76fd4c0}, id = 0, object_sz = 9820, swap_hdr_sz = 
210, vary_headers = 0x0, _reply = 0x861d208, deferredReads = {

deferredReads = {head = 0x0}}}

(gdb) p *((*(e->mem_obj))->data_hdr->nodes->head)->data
$17 = {nodeBuffer = {flags = {error = 0}, length = 3886, offset = 0,
data = 0xb6a080dc "HTTP/1.0 200 OK\r\nAge: 0\r\nDate: Wed, 05 Aug 
2009 19:38:40 GMT\r\nContent-Length: 9376\r\nContent-Type: 
image/png\r\nCache-Control: max-age=60, must-revalidate\r\nConnection: 
keep-alive\r\nProxy-Connection: keep-"...},
  data = "HTTP/1.0 200 OK\r\nAge: 0\r\nDate: Wed, 05 Aug 2009 19:38:40 
GMT\r\nContent-Length: 9376\r\nContent-Type: image/png\r\nCache-Control: 
max-age=60, must-revalidate\r\nConnection: 
keep-alive\r\nProxy-Connection: keep-alive\r\nServer: 
AppleIDiskServer-1E4123\r\nx-responding-server: hpng034\r\nX-dmUser: 
tomm...\r\nETag: 
\"u-1g3s18hn-9d2p-1d9dyjlk2s-11hl44zdvg0\"\r\nLast-Modified: Tue, 24 Mar 
2009 12:47:08 GMT\r\nVia: 1.1 hpcache003 (NetCache NetApp/6.0.5P2D2)\r\n\r\n



Amos
--
Please be using
  Current Stable Squid 2.7.STABLE7 or 3.0.STABLE19
  Current Beta Squid 3.1.0.13

Build failed in Hudson: 3.HEAD-i386-FreeBSD-6.4 #71

[noc]

See 

Changes:

[Amos Jeffries ] Author: Markus Moeller 

Add KRB5INCS to INCLUDES

Needs to be global because we use the kerb5 com_err.h hack globally
through the libcompat headers.

NP: There should be a compat/*.am for these I guess...

[Henrik Nordstrom ] Bug #2773: Segfault in RFC2069 
Digest authantication

Squid segfaulted if digest authentication is enabled an a client responded
with RFC2069 style response.

--
[...truncated 2025 lines...]
checking whether ln -s works... yes
checking how to recognize dependent libraries... pass_all
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking dlfcn.h usability... yes
checking dlfcn.h presence... yes
checking for dlfcn.h... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... none
checking how to run the C++ preprocessor... g++ -E
checking for g77... no
checking for xlf... no
checking for f77... f77
checking whether we are using the GNU Fortran 77 compiler... yes
checking whether f77 accepts -g... yes
checking the maximum length of command line arguments... 196608
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for objdir... .libs
checking for ar... ar
checking for ranlib... ranlib
checking for strip... strip
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC
checking if gcc PIC flag -fPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking whether the gcc linker (/usr/bin/ld) supports shared libraries... yes
checking whether -lc should be explicitly linked in... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
configure: creating libtool
appending configuration tag "CXX" to libtool
checking for ld used by g++... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fPIC
checking if g++ PIC flag -fPIC works... yes
checking if g++ static flag -static works... yes
checking if g++ supports -c -o file.o... yes
checking whether the g++ linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
(cached) (cached) checking how to hardcode library paths into programs... 
immediate
appending configuration tag "F77" to libtool
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for f77 option to produce PIC... -fPIC
checking if f77 PIC flag -fPIC works... yes
checking if f77 static flag -static works... yes
checking if f77 supports -c -o file.o... yes
checking whether the f77 linker (/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... freebsd6.4 ld.so
(cached) (cached) checking how to hardcode library paths into programs... 
immediate
checking for dirent.h that defines DIR... yes
checking for library containing opendir... none required
checking which extension is used for loadable modules... .so
checking which variable specifies run-time library path... LD_LIBRARY_PATH
checking for the default library search path... /lib /usr/lib
checking for objdir... .libs
checking whether libtool supports -dlopen/-dlpreopen... yes
checking for shl_load... no
checking for shl_load in -ldld... no
checking for dlopen in -ldl... no
checking for dlerror... yes
checking for _ prefix in compiled symbols... no
checking whether deplibs are loaded by dlopen... yes
checking argz.h usability... no
checking argz.h presence... no
checking for argz.h... no
checking for error_t... no
checking for argz_append... no
checking for argz_create_sep... no
checking for argz_insert... no
checking for argz_next... no
checking for argz_stringify... no
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking ctype.h usability... yes
checking ctype.h presence... yes
checking for ctype.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking malloc.h usability... no
checking malloc.h presence... no
checking for malloc.h... no
c

Build failed in Hudson: 3.HEAD-i386-Debian-sid #61

[noc]

See 

--
Started by upstream project "3.HEAD-amd64-CentOS-5.3" build number 115
Building remotely on rio.treenet
bzr: ERROR: Invalid http response for 
http://www.squid-cache.org/bzr/squid3/trunk/.bzr/branch-format: Bad status line 
received
Using saved parent location: http://www.squid-cache.org/bzr/squid3/trunk/
ERROR: Failed to pull