Re: [squid-users] Squid SMP workers crash

[Amos Jeffries]

On 18/10/2016 6:16 p.m., Alex Rousskov wrote:
> On 10/17/2016 10:37 PM, Deniz Eren wrote:
>> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote:
>>> On 10/17/2016 02:38 AM, Deniz Eren wrote:
 2016/10/17 11:22:37 kid1| assertion failed:
 ../../src/ipc/AtomicWord.h:71: "Enabled()"
>>>
>>> Either your Squid does not support SMP (a build environment problem) or
>>> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>>>
>>> What does the following command show?
>>>
>>>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> config.status:D["HAVE_ATOMIC_OPS"]=" 0"
>> include/autoconf.h:#define HAVE_ATOMIC_OPS 0
> 
> Your Squid does not support SMP. The ./configure script failed to find
> the necessary APIs for SMP support. I wish Squid would tell you that in
> a less obscure way than an Enabled() assertion; feel free to file a bug
> report about that, but that is a reporting/UI problem; the assertion
> itself is correct.
> 
> I do not know why your build environment lacks atomics support (or why
> Squid cannot detect that support), but I hope that others on the mailing
> list would be able to help you with that investigation.

It is based on Linux 2.6.18, which has some big multi-processor support
issues. What M-P support existed was largely still based on the
"Big-Lock" design which made it horribly slow and inefficient.

> 
> Finally, in the interest of full disclosure, I have to note that, IIRC,
> atomics are not actually required for some of the primitive SMP
> features, but Squid attempts to create a few shared memory tables even
> when those tables are not needed, and those tables do require atomics
> (and will hit the Enabled() assertion you have reported).
> 
> There have been improvements in this area; eventually no unnecessary
> shared memory tables will be created, but it is probably easier for you
> to get a build with working atomics (usually does not require any
> development) than to get rid of those tables (which probably require
> more development).
> 

It may be as simple as the compiler version - CentOS 5 came with GCC 3.
Squid-3 requires GCC 4.

Either way the config.log produced during build will be needed to figure
out the reasons.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Alex Rousskov]

On 10/17/2016 10:37 PM, Deniz Eren wrote:
> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote:
>> On 10/17/2016 02:38 AM, Deniz Eren wrote:
>>> 2016/10/17 11:22:37 kid1| assertion failed:
>>> ../../src/ipc/AtomicWord.h:71: "Enabled()"
>>
>> Either your Squid does not support SMP (a build environment problem) or
>> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>>
>> What does the following command show?
>>
>>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
> config.status:D["HAVE_ATOMIC_OPS"]=" 0"
> include/autoconf.h:#define HAVE_ATOMIC_OPS 0

Your Squid does not support SMP. The ./configure script failed to find
the necessary APIs for SMP support. I wish Squid would tell you that in
a less obscure way than an Enabled() assertion; feel free to file a bug
report about that, but that is a reporting/UI problem; the assertion
itself is correct.

I do not know why your build environment lacks atomics support (or why
Squid cannot detect that support), but I hope that others on the mailing
list would be able to help you with that investigation.


Finally, in the interest of full disclosure, I have to note that, IIRC,
atomics are not actually required for some of the primitive SMP
features, but Squid attempts to create a few shared memory tables even
when those tables are not needed, and those tables do require atomics
(and will hit the Enabled() assertion you have reported).

There have been improvements in this area; eventually no unnecessary
shared memory tables will be created, but it is probably easier for you
to get a build with working atomics (usually does not require any
development) than to get rid of those tables (which probably require
more development).

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov
 wrote:
> On 10/17/2016 02:38 AM, Deniz Eren wrote:
>> 2016/10/17 11:22:37 kid1| assertion failed:
>> ../../src/ipc/AtomicWord.h:71: "Enabled()"
>
> Either your Squid does not support SMP (a build environment problem) or
> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>
> What does the following command show?
>
>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
config.status:D["HAVE_ATOMIC_OPS"]=" 0"
include/autoconf.h:#define HAVE_ATOMIC_OPS 0

>
> (adjust filename paths as needed).
>
> Alex.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] FYI - primitive hit-rate results w/sslbump vs. not

[L. A. Walsh]

Just as an FYI, I did a test today of squid's efficacy with
the ssl-bumping feature.  This is a preliminary result with
little or no review of the logs -- just going by access log
entries. 


I was interested because I've been running squid @ home for over
10 years to try to squeeze speed out of a home connection using
a largish-cache (at least for 1-2 users) of around 80G used
on a dedicated, 128G partition.

Over the years, I've gotten a vague feeling for what to expect and
have generally gotten around a 15-30% cache hit ratio. 


This dropped as google pushed https.  I noticed the web slowing
as my local cache hit-rate dropped and encryption overhead increased
request latency.

This was somewhat unscientific, but not so much in that it
does reflect a part of my traffic. 
I opened a bunch of (30+) news articles from news.google.com w/

my new ssl-bumping enabled and decided I wanted to get an idea of
cache-hit differences.  So changed proxy to go through a
non-bumping port and used the browser's saved-session
to quit the browser and restart all the tabs -- twice --
once for the https test, and a 2nd time for a repeat test.

Results:

Intial opening of the sites w/ssl bump got   730/3365   hits/requests.

The reload in solid https-CONNECT streams showed   40/1588 hits/requests.

And the 2nd reload of the same sites got1268/2263 hits/requests. 


In percentages:

cold-view w/SSL-bump: 22% hit
no SSL-bump:   3% hit
repeat w/SSL-bump:56% hit


Simply inter/intra-site redundancy resulted in 22% cache-hit
ratio, with a "semi-real" case of bringing up the same content
a second time, gave a 56% hit rate.

I'll have to see if how hard it is to get byte counts out of my
logs to get more detail, but since many of these request are small
there is a large delay caused / request.

FWIW, using a FF-clone (64-bit Palemoon) with no local disk cache
(it does have a memory cache, but that would have been cleared
between runs when I restarted the browser).

Initial results look good for using squid to subvert google's
campaign to keep your webtraffic content hidden, but mostly
from "you".













___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid 3.5 conf

[Johnny Lam]

Dear all,


Sorry for missing the conf in my previous mail.

Below is my conf in squid 2.7

acl dest dst 10.68.8.146/32
http_access deny !dest
auth_param basic program ../libexec/ncsa_auth.exe ../etc/password.txt
acl User_Authorized proxy_auth REQUIRED
http_access allow User_Authorized

I tried to use the same in 3.5 but seems not work, any idea how should I
handle it ?
As I found that the default conf for Squid 3.5 is quite long, Can I just
remove them all and put my 2.7 config in ?

Hope you guys can help. Many Thanks!



-- 
Regards,
Johnny
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Additional ecap/icap questions

[James Lay]

Well this has been a pretty amazing bit of learning that's for sure.  
Here's what I'm wanting to accomplish and it's been proving a challenge: 
 Detect keywords (think DLP maybe) in http/https flows.  I've got ecap 
and icap compiled in and working.  My challenges:


a)with icap, it appears that the filter content adapters only work with 
responses, not requestsI need both.
b)with icap, if I use the "echo" adapter I can see everything on the lo 
interface, but decoding it has proven fruitless for me
c)with ecap, I configured per 
http://wiki.squid-cache.org/ConfigExamples/ContentAdaptation/eCAP, but 
I'm confused on the ecap_service line..examples show 
"ecap://www.vigos.com/ecap_gzip", but what do I put in?  I thought I 
didn't need a service for ecap..do I point this to localhost or 
something?


Anyway thank you.
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Sun, Oct 16, 2016 at 2:57 AM, Eliezer Croitoru  wrote:
> Hey,
>
> I can try to replicate the same configuration removing couple settings just 
> to make it simpler to verify if the issue since it's similar to the next 
> testing lab I have planned.
> Can you give more detail about the OS? CentOS, Ubuntu, Other?
CentOS 5

> If it's a self compiled versions then "squid -v" output.
Squid Cache: Version 3.5.20
Service Name: squid
configure options:  '--build=i686-redhat-linux-gnu'
'--host=i686-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--exec-prefix=/opt/squid'
'--datadir=/opt/squid/share' '--libdir=/opt/squid/lib'
'--libexecdir=/opt/squid/libexec' '--localstatedir=/var'
'--sharedstatedir=/opt/squid/com' '--infodir=/usr/share/info'
'--prefix=/opt/squid' '--exec_prefix=/opt/squid'
'--bindir=/opt/squid/bin' '--sbindir=/opt/squid/sbin'
'--sysconfdir=/opt/squid/etc' '--datadir=/opt/squid/share/squid'
'--includedir=/opt/squid/include' '--libdir=/opt/squid/lib/squid'
'--libexecdir=/opt/squid/lib/squid' '--localstatedir=/opt/squid/var'
'--mandir=/opt/squid/share/man' '--infodir=/opt/squid/share/info'
'--enable-epoll' '--disable-dependency-tracking' '--enable-arp-acl'
'--enable-auth' '--enable-auth-negotiate' '--enable-auth-digest'
'--enable-auth-basic' '--enable-auth-ntlm' '--enable-cache-digests'
'--enable-cachemgr-hostname=localhost' '--enable-delay-pools'
'--enable-external-acl-helpers' '--enable-icap-client'
'--with-large-files' '--enable-linux-netfilter' '--enable-referer-log'
'--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl'
'--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log'
'--enable-wccpv2' '--with-aio' '--with-default-user=squid'
'--with-filedescriptors=32768' '--with-dl' '--enable-ssl-crtd'
'--with-openssl=/opt/openssl101' '--with-pthreads'
'--enable-http-violations' '--enable-follow-x-forwarded-for'
'--disable-ipv6' 'build_alias=i686-redhat-linux-gnu'
'host_alias=i686-redhat-linux-gnu'
'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
-fsigned-char -I /usr/kerberos/include -I/opt/openssl101/include -O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables' 'LDFLAGS=-pie -L/opt/openssl101/lib'
'CXXFLAGS=-fPIE -I/opt/openssl101/include -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables'
'PKG_CONFIG_PATH=/opt/squid/lib/pkgconfig:/opt/squid/share/pkgconfig'
--enable-ltdl-convenience


> I have also seen that you are intercepting both http and https traffic, have 
> you tried looking at the logs?
You are right I'm intercepting both http and https traffic. Yes I have
looked at logs and only suspicious thing is this line:
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

>
> If you don't hear me from me fast enough just bump me with an email.
>
> Eliezer
>
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile+WhatsApp: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On 
> Behalf Of Deniz Eren
> Sent: Thursday, October 13, 2016 10:53 AM
> To: squid-users@lists.squid-cache.org
> Subject: [squid-users] Squid SMP workers crash
>
> Hi,
>
> I'm using squid's SMP functionality to distribute requests to many
> squid instances and distribute workload to multiple processors.
> However while running squid's workers after a while worker processes
> crash with the error below and coordinator does not start them again:
> ...
> FATAL: Ipc::Mem::Segment::open failed to
> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
> Squid Cache (Version 3.5.20): Terminated abnormally.
> ...
>
> Does a solution exists for this problem? (permissions are OK in /dev/shm)
>
>
> When everything is OK coordinator listens to http_ports/https_port and
> distributes connections to workers(at least that's the conclusion I
> got from looking access.logs).
> [root@squidbox ~]# netstat -nlp|grep squid
> tcp0  0 0.0.0.0:80800.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31270.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31280.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31300.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:84430.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> udp0  0 0.0.0.0:57850   0.0.0.0:*
>  7897/(squid-1)
> udp0  0 0.0.0.0:33643   0.0.0.0:*
>  7894/(squid-4)
> udp0  0 0.0.0.0:50485   0.0.0.0:*
>  7896/(squid-2)
> udp0  0 

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Fri, Oct 14, 2016 at 1:50 AM, Alex Rousskov
 wrote:
> On 10/13/2016 01:53 AM, Deniz Eren wrote:
>
>> I'm using squid's SMP functionality to distribute requests to many
>> squid instances and distribute workload to multiple processors.
>> However while running squid's workers after a while worker processes
>> crash with the error below and coordinator does not start them again:
>> ...
>> FATAL: Ipc::Mem::Segment::open failed to
>> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
>> Squid Cache (Version 3.5.20): Terminated abnormally.
>> ...
>
> Are you saying that this fatal shm_open() error happens after all
> workers have started serving/logging traffic?
Yes, they are serving.

> I would expect to see it
> at startup (first few minutes at the most if you have IPC timeout
> problems).
Both happen. Sometimes it crashes after seconds, but most of the time
it takes 5-10 minutes.


> Does the error always point to squid-cf__metadata.shm?
This error is solved but, below error still happens.
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

>
> Are you sure that there are no other fatal errors, segmentation faults,
> or similar deathly problems _before_ this error?
> Are you sure your
> startup script does not accidentally start multiple Squid instances that
> compete with each other?
You were right there was a problem with startup script. I'm now
starting with "squid -f /conf/file/path/conffile.conf". However there
is a new problem shown below.
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

Because of this error workers crash couple of times and after that
coordinator gives up creating workers.

> Check system error logs.
>
> FWIW, Segment::open errors without Segment::create errors are often a
> side-effect of other problems that either prevent Squid from creating
> segments or force Squid to remove created segments (both happen in the
> master process).
>
>
>> permissions are OK in /dev/shm
>
> Do you see any Squid segments there (with reasonable timestamps)?
>
>
>> Also is my way of using SMP functionality correct, since I want to
>> distribute all connections between workers and to listen only specific
>> ports?
>
> Adding "workers N" and avoiding SMP-incompatible features is the right
> way; I do not see any SMP-related problems in your configuration.
>
> Alex.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid 2.7 to Squid 3.5

[Johnny Lam]

Some more information,

 - tell us which system / distribution / version you are running this on
I am running it on Windows machine.

Regards,
Johnny


2016-10-17 9:27 GMT+08:00 Johnny Lam :

> Dear all,
>
>
> Sorry for missing the conf in my previous mail.
>
> Below is my conf in squid 2.7
>
> acl dest dst 10.68.8.146/32
> http_access deny !dest
> auth_param basic program ../libexec/ncsa_auth.exe ../etc/password.txt
> acl User_Authorized proxy_auth REQUIRED
> http_access allow User_Authorized
>
> I tried to use the same in 3.5 but seems not work, any idea how should I
> handle it ?
> As I found that the default conf for Squid 3.5 is quite long, Can I just
> remove them all and put my 2.7 config in ?
>
> Hope you guys can help. Many Thanks!
>
> Regards,
> Johnny
>
>
> 2016-10-17 7:20 GMT+08:00 Yuri Voinov :
>
>>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA256
>>
>> You have in the cryochamber, apparently, there was no internet :) :) It's
>> not been :)
>>
>>
>> 17.10.2016 4:17, Yuri Voinov пишет:
>> >
>> > In _your_ environment :) All world uses DNS caches ;)
>> >
>> >
>> > 17.10.2016 3:07, reinerotto пишет:
>> > > Sorry, I forgot: Another difference is, that response times are lower
>> today.
>> > > (BTW: I also did a SM-4 ...)
>> >
>> >
>> >
>> > > --
>> > > View this message in context:
>> > http://squid-web-proxy-cache.1019090.n4.nabble.com/Squid-2-7
>> -to-Squid-3-5-tp4680115p4680120.html
>> > > Sent from the Squid - Users mailing list archive at Nabble.com.
>> > > ___
>> > > squid-users mailing list
>> > > squid-users@lists.squid-cache.org
>> > > http://lists.squid-cache.org/listinfo/squid-users
>> >
>> >
>>
>> -BEGIN PGP SIGNATURE-
>> Version: GnuPG v2
>>
>> iQEcBAEBCAAGBQJYBAtYAAoJENNXIZxhPexG7ZkH/2M//fMMpMAvIJUCn1Ld+0JC
>> WMOqVMaaQ6JHVH6KC2nLRmpqnX/g0baypqy8/z85BRNQJdnv6Re1PGFCKuO45Ne7
>> ackHSX5Q6BZ4s8iHp9Drn3n3yf0siDjjSois31LCMlFZiIwgANYr1dYoFaJgIghw
>> V2kDRauZFK15v/G1FsASEmgWz1r8bd14RH6utHijtGCG+EY9sdtFiqA+mjJ7cFsR
>> l9jukvXyN2jluJZxxh6yeRqSLg1mKA23vbkf4BdQPMZS6YrT44YC4TJjdb7JQzcY
>> 1GGRwiygvOLM/baaSCXTzL3ZSPiBszdwzPJH8UOOYiHc+CY6g2ScjNHTGCK6blQ=
>> =YH6E
>> -END PGP SIGNATURE-
>>
>>
>> ___
>> squid-users mailing list
>> squid-users@lists.squid-cache.org
>> http://lists.squid-cache.org/listinfo/squid-users
>>
>>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users