[Bug 1028470] Re: apache2+ssl hangs on high load
I can confirm, that upgrading the kernel to the most current version solves the problem: $ sudo apt-get dist-upgrade Thank you! -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
** Description changed: Apache2 stops accepting connections when using mod_ssl and having more than 1000 processes running. This is only happening on ubuntu 12.04 and only with mod_ssl enabled. Steps to reproduce: - take a clean install of ubuntu 12.04 server 64bit (i use english installer and all standard settings) - execute following commands as root: $ apt-get update $ apt-get upgrade $ apt-get install apache2-mpm-prefork - change /etc/apache2/apache2.conf to start at least 1001 processes: ServerLimit 1500 StartServers 1500 MinSpareServers 1400 MaxSpareServers 1500 MaxClients 1500 MaxRequestsPerChild 1200 - enable mod_ssl and restart apache: $ a2enmod ssl $ service apache2 restart [no further configuration changes requred, i did not configure any ssl hosts, only enabled the module] - verify, that apache is running at least 1001 processes $ ps ax | grep apache | wc -l 1502 - verify you can connect to localhost: $ curl http://localhost/ It works! This is the default web page for this server. The web server software is running but no content has been added, yet. - start high load: $ ab -n 5000 -c 1000 http://localhost/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 500 requests apr_poll: The timeout specified has expired (70007) Total of 998 requests completed - ready, now apache is not working properly: $ curl -v http://localhost/ * About to connect() to localhost port 80 (#0) * Trying 127.0.0.1... connected > GET / HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: localhost > Accept: */* > . silence There are no errors to find in the logs. After restarting apache it will work for some time, but continue crashing regurally, if you have some traffic coming to the server. In my tests i sometimes had crashes even with very few users connecting to the servers. For greater reproducibility however you will need this high connections number for ab. This is reproducible, happening every time and i also tested this on 3 different machines. - This is specific to 12.04, as i have the same setup working properly on 11.10. - I'm aware of that 1000 Processes will consume a plenty of ram. The machine that is supposed + This is specific to 12.04, as i have the same setup working properly on 11.10 and 12.10. + I'm aware of that 1000 Processes will consume a lot of ram. The machine that is supposed to run this config has 32GB, so this should not be the problem here. Notice: - apache crashed only with mod_ssl enabled - apache crashed only with >1000 processes: 1000 processes runs fine, 1001 will produce a crash Additional information: 1) The release of Ubuntu you are using $ lsb_release -rd Description: Ubuntu 12.04 LTS Release: 12.04 2) The version of the package you are using $ apt-cache policy apache2-mpm-prefork apache2-mpm-prefork: Installed: 2.2.22-1ubuntu1 Candidate: 2.2.22-1ubuntu1 Version table: *** 2.2.22-1ubuntu1 0 500 http://de.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages 100 /var/lib/dpkg/status 3) What you expected to happen i expect apache to handle the 5000 requests as usual and continue accepting connections afterwards 4) What happened instead apache handles only 1000 requests and stops accepting new connections at all, which is a disaster for any website running on the host -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
I have made some further experiments: the bug is still occuring with the latest updates. I have traced the problem down to the main apache functions. It is not a mod_ssl issue! What is actually happening, is that due to "a2enmod ssl", the server have to listen on two ports: 80 and 443. This activates AcceptMutex to synchronize running processes. And here some problem occures under Ubuntu 12.04 So an alternative way to reproduce is: $ a2dismod ssl $ echo "Listen 81" > /etc/apache2/ports.conf $ service apache2 restart $ ab -c 1000 -n 5000 http://localhost/ Changing the "AcceptMutex" config option causes different results. For example with: AcceptMutex posixsem i get all 5000 requests done without error. Then trying to run ab again causes apache to stop accepting requests. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
Any news about this bug? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
Unfortunately i do not get any PID with this method. The other end of the connection is simply "-", not associated with any apache2 process: root@ubuntu:/home/jeff# netstat -tnp |grep curl tcp0161 127.0.0.1:33399 127.0.0.1:80ESTABLISHED 347/curl root@ubuntu:/home/jeff# netstat -tnp | grep 33399 tcp0 0 127.0.0.1:80127.0.0.1:33399 SYN_RECV - tcp0161 127.0.0.1:33399 127.0.0.1:80ESTABLISHED 347/curl I have tried capturing the http traffic to get some insight: tcpdump -p -s0 -w dump.cap -i lo port 80 This is the dump of the last curl request. Wireshark shows me multiple tcp retransmits, but no reply from the server. So it may be something on the tcp level, that is going wrong. Could someone with a deeper understanding of tcp take a look on the dump? If someone is interested, i can provide a downloadable virtual appliance running ubuntu 12.04 with the reported bug (.OVF) from my virtualbox for debugging purposes. ** Attachment added: "dump.cap" https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+attachment/3238977/+files/dump.cap -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
I also agree with you, that this feature has medium importance. I think our install of apache is not very common. We are running mod_php and as a consequence relying on the prefork modell, because php is not thread safe. Also please notice that the presented config was changed to maximize reproducibility. On our production system we have about 700 processes running the most of the time, but on peak traffic this number rises to 1000-1200 producing the hanging that i have described. So while systems with such load may be uncommon, the reported problem is still existing in the real world. Thanks for paying attention to it! -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
I could verify getting the same log messages on my system. This however seems to be not directly related to the problem. I see the same messages, when testing it on Ubuntu 11.10. But the apache remains stable and repsonsive on this older version of ubuntu. I have found some interesting behavior, when using the -k switch with ab to ensure there are really only the specified amount of processes used and no process is busy waiting for the connection to close: $ ab -k -n 5000 -c 999 http://localhost/ all requests will complete without error $ curl http://localhost/ It works! This is the default web page for this server. The web server software is running but no content has been added, yet. $ curl http://localhost/ ... no answer so here you can literally see how the connection #1001 (the "magic number", that appeared before) is breaking apache. i think maybe its some kind of buffer running full? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] Re: apache2+ssl hangs on high load
As i have written, this is a clean ubuntu install, i did not change any config files upon mentioned. /etc/security/limits.conf ist standard config file with no limits inside (only comments). I can't figure out how to determine the pid of apache child process that curl connects to. If you have an idea, please let me know. Here is some output, that i get with gdb for a hanging apache: main apache process (determined with pstree -p): #0 0x7f9b07745803 in select () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x7f9b07c630fd in apr_sleep () from /usr/lib/libapr-1.so.0 No symbol table info available. #2 0x7f9b0853bc69 in ap_wait_or_timeout () No symbol table info available. #3 0x7f9b08548e79 in ap_mpm_run () No symbol table info available. #4 0x7f9b0851e4a4 in main () No symbol table info available. some child process (gdb) bt full #0 0x7f9b0774e5b7 in semop () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x7f9b07c4f68e in ?? () from /usr/lib/libapr-1.so.0 No symbol table info available. #2 0x7f9b07c504a6 in apr_proc_mutex_lock () from /usr/lib/libapr-1.so.0 No symbol table info available. #3 0x7f9b085480dd in ?? () No symbol table info available. #4 0x7f9b0854893a in ?? () No symbol table info available. #5 0x7f9b085489f7 in ?? () No symbol table info available. #6 0x7f9b08549374 in ap_mpm_run () No symbol table info available. #7 0x7f9b0851e4a4 in main () No symbol table info available. This is basically the same as for the working instance before the crash. The output changes, if i disable mod_ssl. Any ideas? -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to apache2 in Ubuntu. https://bugs.launchpad.net/bugs/1028470 Title: apache2+ssl hangs on high load To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1028470/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1028470] [NEW] apache2+ssl hangs on high load
Public bug reported: Apache2 stops accepting connections when using mod_ssl and having more than 1000 processes running. This is only happening on ubuntu 12.04 and only with mod_ssl enabled. Steps to reproduce: - take a clean install of ubuntu 12.04 server 64bit (i use english installer and all standard settings) - execute following commands as root: $ apt-get update $ apt-get upgrade $ apt-get install apache2-mpm-prefork - change /etc/apache2/apache2.conf to start at least 1001 processes: ServerLimit 1500 StartServers 1500 MinSpareServers 1400 MaxSpareServers 1500 MaxClients 1500 MaxRequestsPerChild 1200 - enable mod_ssl and restart apache: $ a2enmod ssl $ service apache2 restart [no further configuration changes requred, i did not configure any ssl hosts, only enabled the module] - verify, that apache is running at least 1001 processes $ ps ax | grep apache | wc -l 1502 - verify you can connect to localhost: $ curl http://localhost/ It works! This is the default web page for this server. The web server software is running but no content has been added, yet. - start high load: $ ab -n 5000 -c 1000 http://localhost/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 500 requests apr_poll: The timeout specified has expired (70007) Total of 998 requests completed - ready, now apache is not working properly: $ curl -v http://localhost/ * About to connect() to localhost port 80 (#0) * Trying 127.0.0.1... connected > GET / HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 > zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: localhost > Accept: */* > . silence There are no errors to find in the logs. After restarting apache it will work for some time, but continue crashing regurally, if you have some traffic coming to the server. In my tests i sometimes had crashes even with very few users connecting to the servers. For greater reproducibility however you will need this high connections number for ab. This is reproducible, happening every time and i also tested this on 3 different machines. This is specific to 12.04, as i have the same setup working properly on 11.10. I'm aware of that 1000 Processes will consume a plenty of ram. The machine that is supposed to run this config has 32GB, so this should not be the problem here. Notice: - apache crashed only with mod_ssl enabled - apache crashed only with >1000 processes: 1000 processes runs fine, 1001 will produce a crash Additional information: 1) The release of Ubuntu you are using $ lsb_release -rd Description:Ubuntu 12.04 LTS Release:12.04 2) The version of the package you are using $ apt-cache policy apache2-mpm-prefork apache2-mpm-prefork: Installed: 2.2.22-1ubuntu1 Candidate: 2.2.22-1ubuntu1 Version table: *** 2.2.22-1ubuntu1 0 500 http://de.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages 100 /var/lib/dpkg/status 3) What you expected to happen i expect apache to handle the 5000 requests as usual and continue accepting connections afterwards 4) What happened instead apache handles only 1000 requests and stops accepting new connections at all, which is a disaster for any website running on the host ** Affects: apache2 (Ubuntu) Importance: Undecided Status: New ** Tags: apache2 connection crash oneiric processes serverlimit ssl ** Description changed: Apache2 stops accepting connections when using mod_ssl and having more than 1000 processes running. This is only happening on ubuntu 12.04 and only with mod_ssl enabled. Steps to reproduce: - take a clean install of ubuntu 12.04 server 64bit (i use english installer and all standard settings) - execute following commands as root: - apt-get update - apt-get upgrade - apt-get install apache2-mpm-prefork + $ apt-get update + $ apt-get upgrade + $ apt-get install apache2-mpm-prefork + - change /etc/apache2/apache2.conf to start at least 1001 processes: - ServerLimit 1500 - StartServers 1500 - MinSpareServers 1400 - MaxSpareServers 1500 - MaxClients 1500 - MaxRequestsPerChild 1200 + ServerLimit 1500 + StartServers 1500 + MinSpareServers 1400 + MaxSpareServers 1500 + MaxClients 1500 + MaxRequestsPerChild 1200 - enable mod_ssl and restart apache: - a2enmod ssl - service apache2 restart + $ a2enmod ssl + $ service apache2 restart [no further configuration changes requred, i did not configure any ssl hosts, only enabled the module] - verify, that apache is running at least 1001 processes $ ps ax | grep apache | wc -l 1502 - verify you can connect to localho
