Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On 10/18/16, Alex Rousskov  wrote:
> On 10/17/2016 10:37 PM, Deniz Eren wrote:
>> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote:
>>> On 10/17/2016 02:38 AM, Deniz Eren wrote:
 2016/10/17 11:22:37 kid1| assertion failed:
 ../../src/ipc/AtomicWord.h:71: "Enabled()"
>>>
>>> Either your Squid does not support SMP (a build environment problem) or
>>> Squid is trying to use SMP features when SMP is not enabled (a Squid
>>> bug).
>>>
>>> What does the following command show?
>>>
>>>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> config.status:D["HAVE_ATOMIC_OPS"]=" 0"
>> include/autoconf.h:#define HAVE_ATOMIC_OPS 0
>
> Your Squid does not support SMP. The ./configure script failed to find
> the necessary APIs for SMP support. I wish Squid would tell you that in
> a less obscure way than an Enabled() assertion; feel free to file a bug
> report about that, but that is a reporting/UI problem; the assertion
> itself is correct.
Yes, you are right. Inspecting more carefully I saw that "Atomic"
support is missing.

>
> I do not know why your build environment lacks atomics support (or why
> Squid cannot detect that support), but I hope that others on the mailing
> list would be able to help you with that investigation.
Fixing system include paths solved the problem. Thanks for pointing me
what the problem is.

>
>
> Finally, in the interest of full disclosure, I have to note that, IIRC,
> atomics are not actually required for some of the primitive SMP
> features, but Squid attempts to create a few shared memory tables even
> when those tables are not needed, and those tables do require atomics
> (and will hit the Enabled() assertion you have reported).
>
> There have been improvements in this area; eventually no unnecessary
> shared memory tables will be created, but it is probably easier for you
> to get a build with working atomics (usually does not require any
> development) than to get rid of those tables (which probably require
> more development).
>
> Alex.
>
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Amos Jeffries]

On 18/10/2016 6:16 p.m., Alex Rousskov wrote:
> On 10/17/2016 10:37 PM, Deniz Eren wrote:
>> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote:
>>> On 10/17/2016 02:38 AM, Deniz Eren wrote:
 2016/10/17 11:22:37 kid1| assertion failed:
 ../../src/ipc/AtomicWord.h:71: "Enabled()"
>>>
>>> Either your Squid does not support SMP (a build environment problem) or
>>> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>>>
>>> What does the following command show?
>>>
>>>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
>> config.status:D["HAVE_ATOMIC_OPS"]=" 0"
>> include/autoconf.h:#define HAVE_ATOMIC_OPS 0
> 
> Your Squid does not support SMP. The ./configure script failed to find
> the necessary APIs for SMP support. I wish Squid would tell you that in
> a less obscure way than an Enabled() assertion; feel free to file a bug
> report about that, but that is a reporting/UI problem; the assertion
> itself is correct.
> 
> I do not know why your build environment lacks atomics support (or why
> Squid cannot detect that support), but I hope that others on the mailing
> list would be able to help you with that investigation.

It is based on Linux 2.6.18, which has some big multi-processor support
issues. What M-P support existed was largely still based on the
"Big-Lock" design which made it horribly slow and inefficient.

> 
> Finally, in the interest of full disclosure, I have to note that, IIRC,
> atomics are not actually required for some of the primitive SMP
> features, but Squid attempts to create a few shared memory tables even
> when those tables are not needed, and those tables do require atomics
> (and will hit the Enabled() assertion you have reported).
> 
> There have been improvements in this area; eventually no unnecessary
> shared memory tables will be created, but it is probably easier for you
> to get a build with working atomics (usually does not require any
> development) than to get rid of those tables (which probably require
> more development).
> 

It may be as simple as the compiler version - CentOS 5 came with GCC 3.
Squid-3 requires GCC 4.

Either way the config.log produced during build will be needed to figure
out the reasons.

Amos

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Alex Rousskov]

On 10/17/2016 10:37 PM, Deniz Eren wrote:
> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote:
>> On 10/17/2016 02:38 AM, Deniz Eren wrote:
>>> 2016/10/17 11:22:37 kid1| assertion failed:
>>> ../../src/ipc/AtomicWord.h:71: "Enabled()"
>>
>> Either your Squid does not support SMP (a build environment problem) or
>> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>>
>> What does the following command show?
>>
>>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
> config.status:D["HAVE_ATOMIC_OPS"]=" 0"
> include/autoconf.h:#define HAVE_ATOMIC_OPS 0

Your Squid does not support SMP. The ./configure script failed to find
the necessary APIs for SMP support. I wish Squid would tell you that in
a less obscure way than an Enabled() assertion; feel free to file a bug
report about that, but that is a reporting/UI problem; the assertion
itself is correct.

I do not know why your build environment lacks atomics support (or why
Squid cannot detect that support), but I hope that others on the mailing
list would be able to help you with that investigation.


Finally, in the interest of full disclosure, I have to note that, IIRC,
atomics are not actually required for some of the primitive SMP
features, but Squid attempts to create a few shared memory tables even
when those tables are not needed, and those tables do require atomics
(and will hit the Enabled() assertion you have reported).

There have been improvements in this area; eventually no unnecessary
shared memory tables will be created, but it is probably easier for you
to get a build with working atomics (usually does not require any
development) than to get rid of those tables (which probably require
more development).

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov
 wrote:
> On 10/17/2016 02:38 AM, Deniz Eren wrote:
>> 2016/10/17 11:22:37 kid1| assertion failed:
>> ../../src/ipc/AtomicWord.h:71: "Enabled()"
>
> Either your Squid does not support SMP (a build environment problem) or
> Squid is trying to use SMP features when SMP is not enabled (a Squid bug).
>
> What does the following command show?
>
>   fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h
config.status:D["HAVE_ATOMIC_OPS"]=" 0"
include/autoconf.h:#define HAVE_ATOMIC_OPS 0

>
> (adjust filename paths as needed).
>
> Alex.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Sun, Oct 16, 2016 at 2:57 AM, Eliezer Croitoru <elie...@ngtech.co.il> wrote:
> Hey,
>
> I can try to replicate the same configuration removing couple settings just 
> to make it simpler to verify if the issue since it's similar to the next 
> testing lab I have planned.
> Can you give more detail about the OS? CentOS, Ubuntu, Other?
CentOS 5

> If it's a self compiled versions then "squid -v" output.
Squid Cache: Version 3.5.20
Service Name: squid
configure options:  '--build=i686-redhat-linux-gnu'
'--host=i686-redhat-linux-gnu' '--target=i386-redhat-linux-gnu'
'--program-prefix=' '--exec-prefix=/opt/squid'
'--datadir=/opt/squid/share' '--libdir=/opt/squid/lib'
'--libexecdir=/opt/squid/libexec' '--localstatedir=/var'
'--sharedstatedir=/opt/squid/com' '--infodir=/usr/share/info'
'--prefix=/opt/squid' '--exec_prefix=/opt/squid'
'--bindir=/opt/squid/bin' '--sbindir=/opt/squid/sbin'
'--sysconfdir=/opt/squid/etc' '--datadir=/opt/squid/share/squid'
'--includedir=/opt/squid/include' '--libdir=/opt/squid/lib/squid'
'--libexecdir=/opt/squid/lib/squid' '--localstatedir=/opt/squid/var'
'--mandir=/opt/squid/share/man' '--infodir=/opt/squid/share/info'
'--enable-epoll' '--disable-dependency-tracking' '--enable-arp-acl'
'--enable-auth' '--enable-auth-negotiate' '--enable-auth-digest'
'--enable-auth-basic' '--enable-auth-ntlm' '--enable-cache-digests'
'--enable-cachemgr-hostname=localhost' '--enable-delay-pools'
'--enable-external-acl-helpers' '--enable-icap-client'
'--with-large-files' '--enable-linux-netfilter' '--enable-referer-log'
'--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl'
'--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log'
'--enable-wccpv2' '--with-aio' '--with-default-user=squid'
'--with-filedescriptors=32768' '--with-dl' '--enable-ssl-crtd'
'--with-openssl=/opt/openssl101' '--with-pthreads'
'--enable-http-violations' '--enable-follow-x-forwarded-for'
'--disable-ipv6' 'build_alias=i686-redhat-linux-gnu'
'host_alias=i686-redhat-linux-gnu'
'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe
-fsigned-char -I /usr/kerberos/include -I/opt/openssl101/include -O2
-g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables' 'LDFLAGS=-pie -L/opt/openssl101/lib'
'CXXFLAGS=-fPIE -I/opt/openssl101/include -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables'
'PKG_CONFIG_PATH=/opt/squid/lib/pkgconfig:/opt/squid/share/pkgconfig'
--enable-ltdl-convenience


> I have also seen that you are intercepting both http and https traffic, have 
> you tried looking at the logs?
You are right I'm intercepting both http and https traffic. Yes I have
looked at logs and only suspicious thing is this line:
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

>
> If you don't hear me from me fast enough just bump me with an email.
>
> Eliezer
>
> 
> Eliezer Croitoru
> Linux System Administrator
> Mobile+WhatsApp: +972-5-28704261
> Email: elie...@ngtech.co.il
>
>
> -Original Message-
> From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On 
> Behalf Of Deniz Eren
> Sent: Thursday, October 13, 2016 10:53 AM
> To: squid-users@lists.squid-cache.org
> Subject: [squid-users] Squid SMP workers crash
>
> Hi,
>
> I'm using squid's SMP functionality to distribute requests to many
> squid instances and distribute workload to multiple processors.
> However while running squid's workers after a while worker processes
> crash with the error below and coordinator does not start them again:
> ...
> FATAL: Ipc::Mem::Segment::open failed to
> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
> Squid Cache (Version 3.5.20): Terminated abnormally.
> ...
>
> Does a solution exists for this problem? (permissions are OK in /dev/shm)
>
>
> When everything is OK coordinator listens to http_ports/https_port and
> distributes connections to workers(at least that's the conclusion I
> got from looking access.logs).
> [root@squidbox ~]# netstat -nlp|grep squid
> tcp0  0 0.0.0.0:80800.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31270.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31280.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:31300.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> tcp0  0 0.0.0.0:84430.0.0.0:*
>  LISTEN  7887/(squid-coord-1
> udp0  0 0.0.0.0:57850   0.0.0.0:*
>  7897/(squid-1)
> u

Re: [squid-users] Squid SMP workers crash

[Deniz Eren]

On Fri, Oct 14, 2016 at 1:50 AM, Alex Rousskov
 wrote:
> On 10/13/2016 01:53 AM, Deniz Eren wrote:
>
>> I'm using squid's SMP functionality to distribute requests to many
>> squid instances and distribute workload to multiple processors.
>> However while running squid's workers after a while worker processes
>> crash with the error below and coordinator does not start them again:
>> ...
>> FATAL: Ipc::Mem::Segment::open failed to
>> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
>> Squid Cache (Version 3.5.20): Terminated abnormally.
>> ...
>
> Are you saying that this fatal shm_open() error happens after all
> workers have started serving/logging traffic?
Yes, they are serving.

> I would expect to see it
> at startup (first few minutes at the most if you have IPC timeout
> problems).
Both happen. Sometimes it crashes after seconds, but most of the time
it takes 5-10 minutes.


> Does the error always point to squid-cf__metadata.shm?
This error is solved but, below error still happens.
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

>
> Are you sure that there are no other fatal errors, segmentation faults,
> or similar deathly problems _before_ this error?
> Are you sure your
> startup script does not accidentally start multiple Squid instances that
> compete with each other?
You were right there was a problem with startup script. I'm now
starting with "squid -f /conf/file/path/conffile.conf". However there
is a new problem shown below.
2016/10/17 11:22:37 kid1| assertion failed:
../../src/ipc/AtomicWord.h:71: "Enabled()"

Because of this error workers crash couple of times and after that
coordinator gives up creating workers.

> Check system error logs.
>
> FWIW, Segment::open errors without Segment::create errors are often a
> side-effect of other problems that either prevent Squid from creating
> segments or force Squid to remove created segments (both happen in the
> master process).
>
>
>> permissions are OK in /dev/shm
>
> Do you see any Squid segments there (with reasonable timestamps)?
>
>
>> Also is my way of using SMP functionality correct, since I want to
>> distribute all connections between workers and to listen only specific
>> ports?
>
> Adding "workers N" and avoiding SMP-incompatible features is the right
> way; I do not see any SMP-related problems in your configuration.
>
> Alex.
>
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Eliezer Croitoru]

Hey,

I can try to replicate the same configuration removing couple settings just to 
make it simpler to verify if the issue since it's similar to the next testing 
lab I have planned.
Can you give more detail about the OS? CentOS, Ubuntu, Other?
If it's a self compiled versions then "squid -v" output.
I have also seen that you are intercepting both http and https traffic, have 
you tried looking at the logs?

If you don't hear me from me fast enough just bump me with an email.

Eliezer


Eliezer Croitoru
Linux System Administrator
Mobile+WhatsApp: +972-5-28704261
Email: elie...@ngtech.co.il


-Original Message-
From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf 
Of Deniz Eren
Sent: Thursday, October 13, 2016 10:53 AM
To: squid-users@lists.squid-cache.org
Subject: [squid-users] Squid SMP workers crash

Hi,

I'm using squid's SMP functionality to distribute requests to many
squid instances and distribute workload to multiple processors.
However while running squid's workers after a while worker processes
crash with the error below and coordinator does not start them again:
...
FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squid-cf__metadata.shm): (2) No such file or directory
Squid Cache (Version 3.5.20): Terminated abnormally.
...

Does a solution exists for this problem? (permissions are OK in /dev/shm)


When everything is OK coordinator listens to http_ports/https_port and
distributes connections to workers(at least that's the conclusion I
got from looking access.logs).
[root@squidbox ~]# netstat -nlp|grep squid
tcp0  0 0.0.0.0:80800.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31270.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31280.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31300.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:84430.0.0.0:*
 LISTEN  7887/(squid-coord-1
udp0  0 0.0.0.0:57850   0.0.0.0:*
 7897/(squid-1)
udp0  0 0.0.0.0:33643   0.0.0.0:*
 7894/(squid-4)
udp0  0 0.0.0.0:50485   0.0.0.0:*
 7896/(squid-2)
udp0  0 0.0.0.0:46427   0.0.0.0:*
 7887/(squid-coord-1
udp0  0 0.0.0.0:58938   0.0.0.0:*
 7895/(squid-3)


Also is my way of using SMP functionality correct, since I want to
distribute all connections between workers and to listen only specific
ports?

I have attached the squid.conf.

Regards,

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

Re: [squid-users] Squid SMP workers crash

[Alex Rousskov]

On 10/13/2016 01:53 AM, Deniz Eren wrote:

> I'm using squid's SMP functionality to distribute requests to many
> squid instances and distribute workload to multiple processors.
> However while running squid's workers after a while worker processes
> crash with the error below and coordinator does not start them again:
> ...
> FATAL: Ipc::Mem::Segment::open failed to
> shm_open(/squid-cf__metadata.shm): (2) No such file or directory
> Squid Cache (Version 3.5.20): Terminated abnormally.
> ...

Are you saying that this fatal shm_open() error happens after all
workers have started serving/logging traffic? I would expect to see it
at startup (first few minutes at the most if you have IPC timeout
problems). Does the error always point to squid-cf__metadata.shm?

Are you sure that there are no other fatal errors, segmentation faults,
or similar deathly problems _before_ this error? Are you sure your
startup script does not accidentally start multiple Squid instances that
compete with each other? Check system error logs.

FWIW, Segment::open errors without Segment::create errors are often a
side-effect of other problems that either prevent Squid from creating
segments or force Squid to remove created segments (both happen in the
master process).


> permissions are OK in /dev/shm

Do you see any Squid segments there (with reasonable timestamps)?


> Also is my way of using SMP functionality correct, since I want to
> distribute all connections between workers and to listen only specific
> ports?

Adding "workers N" and avoiding SMP-incompatible features is the right
way; I do not see any SMP-related problems in your configuration.

Alex.

___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users

[squid-users] Squid SMP workers crash

[Deniz Eren]

Hi,

I'm using squid's SMP functionality to distribute requests to many
squid instances and distribute workload to multiple processors.
However while running squid's workers after a while worker processes
crash with the error below and coordinator does not start them again:
...
FATAL: Ipc::Mem::Segment::open failed to
shm_open(/squid-cf__metadata.shm): (2) No such file or directory
Squid Cache (Version 3.5.20): Terminated abnormally.
...

Does a solution exists for this problem? (permissions are OK in /dev/shm)


When everything is OK coordinator listens to http_ports/https_port and
distributes connections to workers(at least that's the conclusion I
got from looking access.logs).
[root@squidbox ~]# netstat -nlp|grep squid
tcp0  0 0.0.0.0:80800.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31270.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31280.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:31300.0.0.0:*
 LISTEN  7887/(squid-coord-1
tcp0  0 0.0.0.0:84430.0.0.0:*
 LISTEN  7887/(squid-coord-1
udp0  0 0.0.0.0:57850   0.0.0.0:*
 7897/(squid-1)
udp0  0 0.0.0.0:33643   0.0.0.0:*
 7894/(squid-4)
udp0  0 0.0.0.0:50485   0.0.0.0:*
 7896/(squid-2)
udp0  0 0.0.0.0:46427   0.0.0.0:*
 7887/(squid-coord-1
udp0  0 0.0.0.0:58938   0.0.0.0:*
 7895/(squid-3)


Also is my way of using SMP functionality correct, since I want to
distribute all connections between workers and to listen only specific
ports?

I have attached the squid.conf.

Regards,


squid.conf
Description: Binary data
___
squid-users mailing list
squid-users@lists.squid-cache.org
http://lists.squid-cache.org/listinfo/squid-users