Package: squid
Version: 2.7.STABLE3-4.1
Severity: normal
My main squid reverse proxy suddenly stopped working after some days.
The last time it happened, I managed to dig a bit around and also got a
core dump and analyzed it as far as this works without debugging
symbols. This happened on my own rebuild with SSL enabled, but the
affected code region does not even consider SSL support.
Config excerpt:
| http_port 80 accel vhost defaultsite=example.com
| https_port 443 accel vhost defaultsite=example.com cert=/etc/squid/ssl/all
options=NO_SSLv2
| icp_port 3130
|
| logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %Hs %<st %Ss %Sh/%<A
"%{Referer}>h" "%{User-Agent}>h"
| cache_access_log /srv/squid/prod/log/access.log
| cache_access_log /srv/squid/prod/log/combined.log combined
| cache_log /srv/squid/prod/log/cache.log
| cache_store_log /srv/squid/prod/log/store.log
|
| acl accelerated_domains dstdomain example.com
| acl accelerated_protocols proto http https
|
| external_acl_type zope_auth ttl=0 %PATH %{Cookie:;__ac} /etc/squid/auth/auth
/etc/squid/zope_auth.conf
| acl zope_auth external zope_auth
|
| http_access allow accelerated_domains accelerated_protocols zope_auth
| http_access deny all
Available threads:
| (gdb) info threads
| 17 process 17096 0x00002b7100488bc8 in strcspn () from /lib/libc.so.6
| 16 process 17138 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 15 process 17137 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 14 process 17136 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 13 process 17135 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 12 process 17134 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 11 process 17133 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 10 process 17132 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 9 process 17131 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 8 process 17130 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 7 process 17129 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 6 process 17128 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 5 process 17127 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 4 process 17126 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 3 process 17125 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| 2 process 17124 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
| * 1 process 17123 0x00002b70ffd60d29 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib/libpthread.so.0
So 16 threads suddenly waited for something shared and only the 17th did
something usefull.
Annotated backtrace of thread 17 (I had to reconstruct the function names from
a similar binary):
| (gdb) bt
| #0 0x00002b7100488bc8 in strcspn () from /lib/libc.so.6
| #1 0x0000000000456021 in ?? ()
0000000000455f80 g F .text 0000000000000191 strListGetItem
| #2 0x000000000045395e in ?? ()
00000000004538b0 g F .text 000000000000014a
httpHeaderGetListMember
| #3 0x000000000043923a in ?? ()
0000000000438e60 l F .text 0000000000000648 makeExternalAclKey
| #4 0x0000000000439f6b in ?? ()
0000000000439e70 g F .text 000000000000048c aclMatchExternal
| #5 0x000000000040a24c in ?? ()
0000000000409f30 g F .text 0000000000000eef aclMatchAclList
| #6 0x000000000040ae61 in ?? ()
000000000040ae20 l F .text 000000000000044d aclCheck
| #7 0x000000000042652b in ?? ()
| #8 0x0000000000431105 in ?? ()
| #9 0x00000000004601a0 in ?? ()
| #10 0x00002b710042c1a6 in __libc_start_main () from /lib/libc.so.6
Register dump to show the parameters for strcspn:
| (gdb) info registers
| rax 0x0 0
| rbx 0x199d93d 26859837
| rcx 0x3 3
| rdx 0x199d93d 26859837
| rsi 0x700690 7341712
| rdi 0x199d93d 26859837
| rbp 0x0 0x0
| rsp 0x7fffab99df88 0x7fffab99df88
| r8 0x199d93d 26859837
| r9 0x1 1
| r10 0x353a39353a333220 3835440933431882272
| r11 0x2b71005153a0 47764336628640
| r12 0x7fffab99e130 140736072376624
| r13 0x7fffab99e128 140736072376616
| r14 0x3b 59
| r15 0x7fffab99e13c 140736072376636
| rip 0x2b7100488bc8 0x2b7100488bc8 <strcspn+24>
Parameters are in rsi and rdi:
| (gdb) print {char[5]}0x700690
| $14 = "\";,\000"
| (gdb) print {char[50]}0x199d93d
| $16 = ", 31-Dec-97 23:59:59 GMT; Max-Age=0", '\0' <repeats 14 times>
So it calls
| strcspn("\";,", ", 31-Dec-97 23:59:59 GMT; Max-Age=0")
But suddenly the value starts with ",", which is defined as a field
delimiter in the Cookie-header, but incorrectly used in this case.
More data:
| (gdb) print {char[100]}0x199d910
| $18 = "statusmessages=\"deleted\"; Path=/; Expires=Wed, 31-Dec-97 23:59:59
GMT; Max-Age=0", '\0' <repeats 19 times>
It loops around this strcspn call all the time. As far as I know always
with the same parameters.
Bastian
--
There is an order of things in this universe.
-- Apollo, "Who Mourns for Adonais?" stardate 3468.1
--
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
