Bug#894827: apache2: Apache workers are hanging after reload

[Rob Hooper]

Hello,

No this is not a new bug. We have been investigating this since January.

Rob (Dan's colleague).


On April 05, 2018 08:18 Stefan Fritsch wrote:

Hi Dan,


On April 04, 2018 17:18 Debian Bug Tracking System wrote:

Thank you for filing a new Bug report with Debian.

You can follow progress on this Bug here: 894827: 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=894827.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 Debian Apache Maintainers 

If you wish to submit further information on this problem, please
send it to 894...@bugs.debian.org.

Please do not send mail to ow...@bugs.debian.org unless you wish
to report a problem with the Bug-tracking system.


On April 04, 2018 16:56 Dogsbody Technology Support wrote:

Package: apache2
Version: 2.4.10-10+deb8u12
Severity: normal

Dear Maintainer,

I am finding some Apache processes stuck in a gracefully finishing state after 
an Apache reload. The only way to clear these processes down is to restart 
Apache or killing the individual processes.
The way these processes are being left the stuck thread 100% CPU and eventually 
crashing the server. 
I have seen these hung processes stay running for over 12 hours before I 
manually kill them. 

This seems to only be when the reload catches a large request to my website. 
Unfortunately I have been unsuccessful to replicate this issue anywhere outside 
of my production environment.

We are using Apache to process PHP behind a NGINX proxy.

-- Package-specific info:

-- System Information:
Debian Release: 8.10
  APT prefers oldstable
  APT policy: (500, 'oldstable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2 depends on:
ii  apache2-bin2.4.10-10+deb8u12
ii  apache2-data   2.4.10-10+deb8u12
ii  apache2-utils  2.4.10-10+deb8u12
ii  dpkg   1.17.27
ii  lsb-base   4.1+Debian13+nmu1
ii  mime-support   3.58
ii  perl   5.20.2-3+deb8u9
ii  procps 2:3.3.9-9

Versions of packages apache2 recommends:
ii  ssl-cert  1.0.35

Versions of packages apache2 suggests:
pn  apache2-doc  
pn  apache2-suexec-pristine | apache2-suexec-custom  
ii  w3m [www-browser]0.5.3-19+deb8u2

Versions of packages apache2-bin depends on:
ii  libapr1  1.5.1-3
ii  libaprutil1  1.5.4-1
ii  libaprutil1-dbd-sqlite3  1.5.4-1
ii  libaprutil1-ldap 1.5.4-1
ii  libc62.19-18+deb8u10
ii  libldap-2.4-22.4.40+dfsg-1+deb8u3
ii  liblua5.1-0  5.1.5-7.1
ii  libpcre3 2:8.35-3.3+deb8u4
ii  libssl1.0.0  1.0.1t-1+deb8u8
ii  libxml2  2.9.1+dfsg1-5+deb8u6
ii  perl 5.20.2-3+deb8u9
ii  zlib1g   1:1.2.8.dfsg-2+b1

Versions of packages apache2-bin suggests:
pn  apache2-doc  
pn  apache2-suexec-pristine | apache2-suexec-custom  
ii  w3m [www-browser]0.5.3-19+deb8u2

Versions of packages apache2 is related to:
ii  apache2  2.4.10-10+deb8u12
ii  apache2-bin  2.4.10-10+deb8u12

-- Configuration Files:
/etc/apache2/mods-available/mpm_prefork.conf changed:

StartServers 5
MinSpareServers   5
MaxSpareServers  10
MaxRequestWorkers 110
MaxConnectionsPerChild   3
GracefulShutdownTimeout 60


/etc/apache2/mods-available/mpm_worker.conf changed:

StartServers 2
MinSpareThreads  25
MaxSpareThreads  75
ThreadLimit  64
ThreadsPerChild  25
MaxRequestWorkers 300
MaxConnectionsPerChild   0


/etc/apache2/mods-available/status.conf changed:

# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the "192.0.2.0/24" to allow access from other 
hosts.

SetHandler server-status
Require local
#Require ip 192.0.2.0/24

# Keep track of extended status information for each request
ExtendedStatus On
# Determine if mod_status displays the first 63 characters of a 

Bug#894827: apache2: Apache workers are hanging after reload

[Stefan Fritsch]

Hi Dan,

On Wed, 4 Apr 2018, Dan Benton wrote:

> Package: apache2
> Version: 2.4.10-10+deb8u12
> Severity: normal

Is this a new issue with version 2.4.10-10+deb8u12 (from the security 
update a few days ago) or have you also observed it with the previous 
version 2.4.10-10+deb8u11?

Cheers,
Stefan

> Dear Maintainer,
> 
> I am finding some Apache processes stuck in a gracefully finishing state 
> after an Apache reload. The only way to clear these processes down is to 
> restart Apache or killing the individual processes. The way these 
> processes are being left the stuck thread 100% CPU and eventually 
> crashing the server. I have seen these hung processes stay running for 
> over 12 hours before I manually kill them.
> 
> This seems to only be when the reload catches a large request to my 
> website. Unfortunately I have been unsuccessful to replicate this issue 
> anywhere outside of my production environment.

Bug#894760: apache2-bin: trying to overwrite '/usr/lib/apache2/modules/mod_md.so', which is also in package libapache2-mod-md 1.1.0-1

[Andreas Beckmann]

Followup-For: Bug #894760

Hi,

similar problems exist with libapache2-mod-proxy-uwsgi:

  Preparing to unpack .../13-apache2-bin_2.4.33-1_amd64.deb ...
  Unpacking apache2-bin (2.4.33-1) over (2.4.29-2) ...
  dpkg: error processing archive 
/tmp/apt-dpkg-install-B7VtPD/13-apache2-bin_2.4.33-1_amd64.deb (--unpack):
   trying to overwrite '/usr/lib/apache2/modules/mod_proxy_uwsgi.so', which is 
also in package libapache2-mod-proxy-uwsgi 2.0.15-10.4
  dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)

  Selecting previously unselected package apache2.
  Preparing to unpack .../21-apache2_2.4.33-1_amd64.deb ...
  Unpacking apache2 (2.4.33-1) ...
  dpkg: error processing archive 
/tmp/apt-dpkg-install-B7VtPD/21-apache2_2.4.33-1_amd64.deb (--unpack):
   trying to overwrite '/etc/apache2/mods-available/proxy_uwsgi.load', which is 
also in package libapache2-mod-proxy-uwsgi 2.0.15-10.4
  dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)


Andreas

re: Bug#894827: apache2: Apache workers are hanging after reload

[Kai Storbeck]

> Hi Dan,
>
> On Wed, 4 Apr 2018, Dan Benton wrote:
>
> > Package: apache2
> > Version: 2.4.10-10+deb8u12
> > Severity: normal
>
> Is this a new issue with version 2.4.10-10+deb8u12 (from the security
> update a few days ago) or have you also observed it with the previous
> version 2.4.10-10+deb8u11?
>
> Cheers,
> Stefan
Hello List,

We are experiencing roughly the same issue since upgrading to deb8u12. I
was searching this ML archive and bumped in this report, happily?.


We noticed that reverting to deb8u11 restores normal behaviour: "apache
reloads" (graceful) work without interruption again.

We have pinpointed this problem in the slotmem_shm module that is
automatically loaded when we use the proxy_balancer. (all done via our
wonderful puppet forge apache module).

The exact log message after graceful restarts in deb8u12:

[Thu Apr 05 16:19:38.963500 2018] [slotmem_shm:error] [pid 18315:tid
140146439718784] (28)No space left on device: AH02611: create:
apr_shm_create(/var/run/apache2/slotmem-shm-pe0653d6a_apidocs_cluster.shm)
failed
[Thu Apr 05 16:19:38.963553 2018] [:emerg] [pid 18315:tid
140146439718784] AH00020: Configuration Failed, exiting

Is this info helpful? Or not at all? Can I assist with providing more
information?

Thanks,

Kai Storbeck

XS4ALL (Happy apache2 users provided by Debian)

PS: should I be ccing the bug report?



signature.asc
Description: OpenPGP digital signature

Re: Bug#894827: apache2: Apache workers are hanging after reload

[Kai Storbeck]

Hello,

Sorry. I assumed everything was fine, but alas. deb8u11 shows the same
problem. We'll be investigating.

I'm terrible sorry for the noise.

Kai

On 05-04-18 17:16, Kai Storbeck wrote:
>> Hi Dan,
>>
>> On Wed, 4 Apr 2018, Dan Benton wrote:
>>
>>> Package: apache2
>>> Version: 2.4.10-10+deb8u12
>>> Severity: normal
>>
>> Is this a new issue with version 2.4.10-10+deb8u12 (from the security
>> update a few days ago) or have you also observed it with the previous
>> version 2.4.10-10+deb8u11?
>>
>> Cheers,
>> Stefan
> Hello List,
> 
> We are experiencing roughly the same issue since upgrading to deb8u12. I
> was searching this ML archive and bumped in this report, happily?.
> 
> 
> We noticed that reverting to deb8u11 restores normal behaviour: "apache
> reloads" (graceful) work without interruption again.
> 
> We have pinpointed this problem in the slotmem_shm module that is
> automatically loaded when we use the proxy_balancer. (all done via our
> wonderful puppet forge apache module).
> 
> The exact log message after graceful restarts in deb8u12:
> 
> [Thu Apr 05 16:19:38.963500 2018] [slotmem_shm:error] [pid 18315:tid
> 140146439718784] (28)No space left on device: AH02611: create:
> apr_shm_create(/var/run/apache2/slotmem-shm-pe0653d6a_apidocs_cluster.shm)
> failed
> [Thu Apr 05 16:19:38.963553 2018] [:emerg] [pid 18315:tid
> 140146439718784] AH00020: Configuration Failed, exiting
> 
> Is this info helpful? Or not at all? Can I assist with providing more
> information?
> 
> Thanks,
> 
> Kai Storbeck
> 
> XS4ALL (Happy apache2 users provided by Debian)
> 
> PS: should I be ccing the bug report?
> 



signature.asc
Description: OpenPGP digital signature