Re: [squid-users] Squid SMP workers crash
On 10/18/16, Alex Rousskov wrote: > On 10/17/2016 10:37 PM, Deniz Eren wrote: >> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote: >>> On 10/17/2016 02:38 AM, Deniz Eren wrote: 2016/10/17 11:22:37 kid1| assertion failed: ../../src/ipc/AtomicWord.h:71: "Enabled()" >>> >>> Either your Squid does not support SMP (a build environment problem) or >>> Squid is trying to use SMP features when SMP is not enabled (a Squid >>> bug). >>> >>> What does the following command show? >>> >>> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h >> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h >> config.status:D["HAVE_ATOMIC_OPS"]=" 0" >> include/autoconf.h:#define HAVE_ATOMIC_OPS 0 > > Your Squid does not support SMP. The ./configure script failed to find > the necessary APIs for SMP support. I wish Squid would tell you that in > a less obscure way than an Enabled() assertion; feel free to file a bug > report about that, but that is a reporting/UI problem; the assertion > itself is correct. Yes, you are right. Inspecting more carefully I saw that "Atomic" support is missing. > > I do not know why your build environment lacks atomics support (or why > Squid cannot detect that support), but I hope that others on the mailing > list would be able to help you with that investigation. Fixing system include paths solved the problem. Thanks for pointing me what the problem is. > > > Finally, in the interest of full disclosure, I have to note that, IIRC, > atomics are not actually required for some of the primitive SMP > features, but Squid attempts to create a few shared memory tables even > when those tables are not needed, and those tables do require atomics > (and will hit the Enabled() assertion you have reported). > > There have been improvements in this area; eventually no unnecessary > shared memory tables will be created, but it is probably easier for you > to get a build with working atomics (usually does not require any > development) than to get rid of those tables (which probably require > more development). > > Alex. > > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On 18/10/2016 6:16 p.m., Alex Rousskov wrote: > On 10/17/2016 10:37 PM, Deniz Eren wrote: >> On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote: >>> On 10/17/2016 02:38 AM, Deniz Eren wrote: 2016/10/17 11:22:37 kid1| assertion failed: ../../src/ipc/AtomicWord.h:71: "Enabled()" >>> >>> Either your Squid does not support SMP (a build environment problem) or >>> Squid is trying to use SMP features when SMP is not enabled (a Squid bug). >>> >>> What does the following command show? >>> >>> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h >> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h >> config.status:D["HAVE_ATOMIC_OPS"]=" 0" >> include/autoconf.h:#define HAVE_ATOMIC_OPS 0 > > Your Squid does not support SMP. The ./configure script failed to find > the necessary APIs for SMP support. I wish Squid would tell you that in > a less obscure way than an Enabled() assertion; feel free to file a bug > report about that, but that is a reporting/UI problem; the assertion > itself is correct. > > I do not know why your build environment lacks atomics support (or why > Squid cannot detect that support), but I hope that others on the mailing > list would be able to help you with that investigation. It is based on Linux 2.6.18, which has some big multi-processor support issues. What M-P support existed was largely still based on the "Big-Lock" design which made it horribly slow and inefficient. > > Finally, in the interest of full disclosure, I have to note that, IIRC, > atomics are not actually required for some of the primitive SMP > features, but Squid attempts to create a few shared memory tables even > when those tables are not needed, and those tables do require atomics > (and will hit the Enabled() assertion you have reported). > > There have been improvements in this area; eventually no unnecessary > shared memory tables will be created, but it is probably easier for you > to get a build with working atomics (usually does not require any > development) than to get rid of those tables (which probably require > more development). > It may be as simple as the compiler version - CentOS 5 came with GCC 3. Squid-3 requires GCC 4. Either way the config.log produced during build will be needed to figure out the reasons. Amos ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On 10/17/2016 10:37 PM, Deniz Eren wrote: > On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote: >> On 10/17/2016 02:38 AM, Deniz Eren wrote: >>> 2016/10/17 11:22:37 kid1| assertion failed: >>> ../../src/ipc/AtomicWord.h:71: "Enabled()" >> >> Either your Squid does not support SMP (a build environment problem) or >> Squid is trying to use SMP features when SMP is not enabled (a Squid bug). >> >> What does the following command show? >> >> fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h > fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h > config.status:D["HAVE_ATOMIC_OPS"]=" 0" > include/autoconf.h:#define HAVE_ATOMIC_OPS 0 Your Squid does not support SMP. The ./configure script failed to find the necessary APIs for SMP support. I wish Squid would tell you that in a less obscure way than an Enabled() assertion; feel free to file a bug report about that, but that is a reporting/UI problem; the assertion itself is correct. I do not know why your build environment lacks atomics support (or why Squid cannot detect that support), but I hope that others on the mailing list would be able to help you with that investigation. Finally, in the interest of full disclosure, I have to note that, IIRC, atomics are not actually required for some of the primitive SMP features, but Squid attempts to create a few shared memory tables even when those tables are not needed, and those tables do require atomics (and will hit the Enabled() assertion you have reported). There have been improvements in this area; eventually no unnecessary shared memory tables will be created, but it is probably easier for you to get a build with working atomics (usually does not require any development) than to get rid of those tables (which probably require more development). Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On Mon, Oct 17, 2016 at 7:43 PM, Alex Rousskov wrote: > On 10/17/2016 02:38 AM, Deniz Eren wrote: >> 2016/10/17 11:22:37 kid1| assertion failed: >> ../../src/ipc/AtomicWord.h:71: "Enabled()" > > Either your Squid does not support SMP (a build environment problem) or > Squid is trying to use SMP features when SMP is not enabled (a Squid bug). > > What does the following command show? > > fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h config.status:D["HAVE_ATOMIC_OPS"]=" 0" include/autoconf.h:#define HAVE_ATOMIC_OPS 0 > > (adjust filename paths as needed). > > Alex. > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On 10/17/2016 02:38 AM, Deniz Eren wrote: > 2016/10/17 11:22:37 kid1| assertion failed: > ../../src/ipc/AtomicWord.h:71: "Enabled()" Either your Squid does not support SMP (a build environment problem) or Squid is trying to use SMP features when SMP is not enabled (a Squid bug). What does the following command show? fgrep -RI HAVE_ATOMIC_OPS config.status include/autoconf.h (adjust filename paths as needed). Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On Sun, Oct 16, 2016 at 2:57 AM, Eliezer Croitoru wrote: > Hey, > > I can try to replicate the same configuration removing couple settings just > to make it simpler to verify if the issue since it's similar to the next > testing lab I have planned. > Can you give more detail about the OS? CentOS, Ubuntu, Other? CentOS 5 > If it's a self compiled versions then "squid -v" output. Squid Cache: Version 3.5.20 Service Name: squid configure options: '--build=i686-redhat-linux-gnu' '--host=i686-redhat-linux-gnu' '--target=i386-redhat-linux-gnu' '--program-prefix=' '--exec-prefix=/opt/squid' '--datadir=/opt/squid/share' '--libdir=/opt/squid/lib' '--libexecdir=/opt/squid/libexec' '--localstatedir=/var' '--sharedstatedir=/opt/squid/com' '--infodir=/usr/share/info' '--prefix=/opt/squid' '--exec_prefix=/opt/squid' '--bindir=/opt/squid/bin' '--sbindir=/opt/squid/sbin' '--sysconfdir=/opt/squid/etc' '--datadir=/opt/squid/share/squid' '--includedir=/opt/squid/include' '--libdir=/opt/squid/lib/squid' '--libexecdir=/opt/squid/lib/squid' '--localstatedir=/opt/squid/var' '--mandir=/opt/squid/share/man' '--infodir=/opt/squid/share/info' '--enable-epoll' '--disable-dependency-tracking' '--enable-arp-acl' '--enable-auth' '--enable-auth-negotiate' '--enable-auth-digest' '--enable-auth-basic' '--enable-auth-ntlm' '--enable-cache-digests' '--enable-cachemgr-hostname=localhost' '--enable-delay-pools' '--enable-external-acl-helpers' '--enable-icap-client' '--with-large-files' '--enable-linux-netfilter' '--enable-referer-log' '--enable-removal-policies=heap,lru' '--enable-snmp' '--enable-ssl' '--enable-storeio=aufs,diskd,ufs' '--enable-useragent-log' '--enable-wccpv2' '--with-aio' '--with-default-user=squid' '--with-filedescriptors=32768' '--with-dl' '--enable-ssl-crtd' '--with-openssl=/opt/openssl101' '--with-pthreads' '--enable-http-violations' '--enable-follow-x-forwarded-for' '--disable-ipv6' 'build_alias=i686-redhat-linux-gnu' 'host_alias=i686-redhat-linux-gnu' 'target_alias=i386-redhat-linux-gnu' 'CFLAGS=-fPIE -Os -g -pipe -fsigned-char -I /usr/kerberos/include -I/opt/openssl101/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' 'LDFLAGS=-pie -L/opt/openssl101/lib' 'CXXFLAGS=-fPIE -I/opt/openssl101/include -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic -fasynchronous-unwind-tables' 'PKG_CONFIG_PATH=/opt/squid/lib/pkgconfig:/opt/squid/share/pkgconfig' --enable-ltdl-convenience > I have also seen that you are intercepting both http and https traffic, have > you tried looking at the logs? You are right I'm intercepting both http and https traffic. Yes I have looked at logs and only suspicious thing is this line: 2016/10/17 11:22:37 kid1| assertion failed: ../../src/ipc/AtomicWord.h:71: "Enabled()" > > If you don't hear me from me fast enough just bump me with an email. > > Eliezer > > > Eliezer Croitoru > Linux System Administrator > Mobile+WhatsApp: +972-5-28704261 > Email: elie...@ngtech.co.il > > > -Original Message- > From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On > Behalf Of Deniz Eren > Sent: Thursday, October 13, 2016 10:53 AM > To: squid-users@lists.squid-cache.org > Subject: [squid-users] Squid SMP workers crash > > Hi, > > I'm using squid's SMP functionality to distribute requests to many > squid instances and distribute workload to multiple processors. > However while running squid's workers after a while worker processes > crash with the error below and coordinator does not start them again: > ... > FATAL: Ipc::Mem::Segment::open failed to > shm_open(/squid-cf__metadata.shm): (2) No such file or directory > Squid Cache (Version 3.5.20): Terminated abnormally. > ... > > Does a solution exists for this problem? (permissions are OK in /dev/shm) > > > When everything is OK coordinator listens to http_ports/https_port and > distributes connections to workers(at least that's the conclusion I > got from looking access.logs). > [root@squidbox ~]# netstat -nlp|grep squid > tcp0 0 0.0.0.0:80800.0.0.0:* > LISTEN 7887/(squid-coord-1 > tcp0 0 0.0.0.0:31270.0.0.0:* > LISTEN 7887/(squid-coord-1 > tcp0 0 0.0.0.0:31280.0.0.0:* > LISTEN 7887/(squid-coord-1 > tcp0 0 0.0.0.0:31300.0.0.0:* > LISTEN 7887/(squid-coord-1 > tcp0 0 0.0.0.0:84430.0.0.0:* > LISTEN 7887/(squid-coord-1 > udp0 0 0.0.0.0:57850 0.0.0.0:* > 7897/(squid-1) > udp0 0 0.0.0.0:33643 0.0.0.0:* > 7894/(squid-4) > udp0 0 0.0.0.0:50485 0.0.0.0:* > 7896/(squid-2) > udp0 0 0.0.0.0:46427
Re: [squid-users] Squid SMP workers crash
On Fri, Oct 14, 2016 at 1:50 AM, Alex Rousskov wrote: > On 10/13/2016 01:53 AM, Deniz Eren wrote: > >> I'm using squid's SMP functionality to distribute requests to many >> squid instances and distribute workload to multiple processors. >> However while running squid's workers after a while worker processes >> crash with the error below and coordinator does not start them again: >> ... >> FATAL: Ipc::Mem::Segment::open failed to >> shm_open(/squid-cf__metadata.shm): (2) No such file or directory >> Squid Cache (Version 3.5.20): Terminated abnormally. >> ... > > Are you saying that this fatal shm_open() error happens after all > workers have started serving/logging traffic? Yes, they are serving. > I would expect to see it > at startup (first few minutes at the most if you have IPC timeout > problems). Both happen. Sometimes it crashes after seconds, but most of the time it takes 5-10 minutes. > Does the error always point to squid-cf__metadata.shm? This error is solved but, below error still happens. 2016/10/17 11:22:37 kid1| assertion failed: ../../src/ipc/AtomicWord.h:71: "Enabled()" > > Are you sure that there are no other fatal errors, segmentation faults, > or similar deathly problems _before_ this error? > Are you sure your > startup script does not accidentally start multiple Squid instances that > compete with each other? You were right there was a problem with startup script. I'm now starting with "squid -f /conf/file/path/conffile.conf". However there is a new problem shown below. 2016/10/17 11:22:37 kid1| assertion failed: ../../src/ipc/AtomicWord.h:71: "Enabled()" Because of this error workers crash couple of times and after that coordinator gives up creating workers. > Check system error logs. > > FWIW, Segment::open errors without Segment::create errors are often a > side-effect of other problems that either prevent Squid from creating > segments or force Squid to remove created segments (both happen in the > master process). > > >> permissions are OK in /dev/shm > > Do you see any Squid segments there (with reasonable timestamps)? > > >> Also is my way of using SMP functionality correct, since I want to >> distribute all connections between workers and to listen only specific >> ports? > > Adding "workers N" and avoiding SMP-incompatible features is the right > way; I do not see any SMP-related problems in your configuration. > > Alex. > ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
Hey, I can try to replicate the same configuration removing couple settings just to make it simpler to verify if the issue since it's similar to the next testing lab I have planned. Can you give more detail about the OS? CentOS, Ubuntu, Other? If it's a self compiled versions then "squid -v" output. I have also seen that you are intercepting both http and https traffic, have you tried looking at the logs? If you don't hear me from me fast enough just bump me with an email. Eliezer Eliezer Croitoru Linux System Administrator Mobile+WhatsApp: +972-5-28704261 Email: elie...@ngtech.co.il -Original Message- From: squid-users [mailto:squid-users-boun...@lists.squid-cache.org] On Behalf Of Deniz Eren Sent: Thursday, October 13, 2016 10:53 AM To: squid-users@lists.squid-cache.org Subject: [squid-users] Squid SMP workers crash Hi, I'm using squid's SMP functionality to distribute requests to many squid instances and distribute workload to multiple processors. However while running squid's workers after a while worker processes crash with the error below and coordinator does not start them again: ... FATAL: Ipc::Mem::Segment::open failed to shm_open(/squid-cf__metadata.shm): (2) No such file or directory Squid Cache (Version 3.5.20): Terminated abnormally. ... Does a solution exists for this problem? (permissions are OK in /dev/shm) When everything is OK coordinator listens to http_ports/https_port and distributes connections to workers(at least that's the conclusion I got from looking access.logs). [root@squidbox ~]# netstat -nlp|grep squid tcp0 0 0.0.0.0:80800.0.0.0:* LISTEN 7887/(squid-coord-1 tcp0 0 0.0.0.0:31270.0.0.0:* LISTEN 7887/(squid-coord-1 tcp0 0 0.0.0.0:31280.0.0.0:* LISTEN 7887/(squid-coord-1 tcp0 0 0.0.0.0:31300.0.0.0:* LISTEN 7887/(squid-coord-1 tcp0 0 0.0.0.0:84430.0.0.0:* LISTEN 7887/(squid-coord-1 udp0 0 0.0.0.0:57850 0.0.0.0:* 7897/(squid-1) udp0 0 0.0.0.0:33643 0.0.0.0:* 7894/(squid-4) udp0 0 0.0.0.0:50485 0.0.0.0:* 7896/(squid-2) udp0 0 0.0.0.0:46427 0.0.0.0:* 7887/(squid-coord-1 udp0 0 0.0.0.0:58938 0.0.0.0:* 7895/(squid-3) Also is my way of using SMP functionality correct, since I want to distribute all connections between workers and to listen only specific ports? I have attached the squid.conf. Regards, ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users
Re: [squid-users] Squid SMP workers crash
On 10/13/2016 01:53 AM, Deniz Eren wrote: > I'm using squid's SMP functionality to distribute requests to many > squid instances and distribute workload to multiple processors. > However while running squid's workers after a while worker processes > crash with the error below and coordinator does not start them again: > ... > FATAL: Ipc::Mem::Segment::open failed to > shm_open(/squid-cf__metadata.shm): (2) No such file or directory > Squid Cache (Version 3.5.20): Terminated abnormally. > ... Are you saying that this fatal shm_open() error happens after all workers have started serving/logging traffic? I would expect to see it at startup (first few minutes at the most if you have IPC timeout problems). Does the error always point to squid-cf__metadata.shm? Are you sure that there are no other fatal errors, segmentation faults, or similar deathly problems _before_ this error? Are you sure your startup script does not accidentally start multiple Squid instances that compete with each other? Check system error logs. FWIW, Segment::open errors without Segment::create errors are often a side-effect of other problems that either prevent Squid from creating segments or force Squid to remove created segments (both happen in the master process). > permissions are OK in /dev/shm Do you see any Squid segments there (with reasonable timestamps)? > Also is my way of using SMP functionality correct, since I want to > distribute all connections between workers and to listen only specific > ports? Adding "workers N" and avoiding SMP-incompatible features is the right way; I do not see any SMP-related problems in your configuration. Alex. ___ squid-users mailing list squid-users@lists.squid-cache.org http://lists.squid-cache.org/listinfo/squid-users