Steps to reproduce/verify on Xenial:
===
References:
-
https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
-
https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process
-
https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
-
https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/
Create a container with apache2/mod-wsgi in daemon mode,
w/ keep-alive timeout long enough, and wsgi hello-world:
---
$ lxc launch ubuntu:xenial lp1863232x
$ lxc exec lp1863232x -- su - ubuntu
$ sudo apt update
$ sudo apt install -y apache2 libapache2-mod-wsgi
$ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf
$ cat <<EOF | sudo tee /etc/apache2/conf-available/wsgi.conf
WSGIScriptAlias /hello-world /var/www/html/hello-world.py
WSGIDaemonProcess 127.0.0.1 processes=2 threads=16 display-name=%{GROUP}
WSGIProcessGroup 127.0.0.1
EOF
$ sudo a2enconf wsgi
$ sudo systemctl reload apache2
$ cat <<EOF | sudo tee /var/www/html/hello-world.py
def application(environ, start_response):
status = '200 OK'
output = b'Hello World!\n'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
return [output]
EOF
$ curl 127.0.0.1/hello-world
Hello World!
Check WSGI socket filename changes when
reloading apache2 (aka graceful restart):
---
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.0.1.sock
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.1.1.sock
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.2.1.sock
Create script and shell one-liner
to send two HTTP requests in the
same connection (keep-alive used)
---
$ cat <<EOF >http-request
GET /hello-world HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive
EOF
One connection/One request:
$ (cat http-request; sleep 1) | telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Mon, 17 Feb 2020 23:33:19 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 13
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain
Hello World!
Connection closed by foreign host.
One connection/Two requests (added timestamps to check timeout values)
$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80
2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:34:42 == Trying 127.0.0.1...
23:34:42 == Connected to 127.0.0.1.
23:34:42 == Escape character is '^]'.
23:34:42 == HTTP/1.1 200 OK
23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT
23:34:42 == Server: Apache/2.4.18 (Ubuntu)
23:34:42 == Content-Length: 13
23:34:42 == Keep-Alive: timeout=5, max=100
23:34:42 == Connection: Keep-Alive
23:34:42 == Content-Type: text/plain
23:34:42 ==
23:34:42 == Hello World!
23:34:43 == HTTP/1.1 200 OK
23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT
23:34:43 == Server: Apache/2.4.18 (Ubuntu)
23:34:43 == Content-Length: 13
23:34:43 == Keep-Alive: timeout=5, max=99
23:34:43 == Connection: Keep-Alive
23:34:43 == Content-Type: text/plain
23:34:43 ==
23:34:43 == Hello World!
23:34:58 == Connection closed by foreign host.
Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module:
mpm_event/mpm_worker/mpm_prefork)
---
$ lsb_release -cs
xenial
$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1build1
For reference on socket filename:
$ sudo systemctl restart apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.0.1.sock
$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat
http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo
"$(date +'%T') == $line"; done
23:48:29 == Trying 127.0.0.1...
23:48:29 == Connected to 127.0.0.1.
23:48:29 == Escape character is '^]'.
23:48:29 == HTTP/1.1 200 OK
23:48:29 == Date: Mon, 17 Feb 2020 23:48:29 GMT
23:48:29 == Server: Apache/2.4.18 (Ubuntu)
23:48:29 == Content-Length: 13
23:48:29 == Keep-Alive: timeout=15, max=100
23:48:29 == Connection: Keep-Alive
23:48:29 == Content-Type: text/plain
23:48:29 ==
23:48:29 == Hello World!
23:48:35 == HTTP/1.1 503 Service Unavailable
23:48:35 == Date: Mon, 17 Feb 2020 23:48:35 GMT
23:48:35 == Server: Apache/2.4.18 (Ubuntu)
23:48:35 == Content-Length: 374
23:48:35 == Connection: close
23:48:35 == Content-Type: text/html; charset=iso-8859-1
23:48:35 ==
23:48:35 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:48:35 == <html><head>
23:48:35 == <title>503 Service Unavailable</title>
23:48:35 == </head><body>
23:48:35 == <h1>Service Unavailable</h1>
23:48:35 == <p>The server is temporarily unable to service your
23:48:35 == request due to maintenance downtime or capacity
23:48:35 == problems. Please try again later.</p>
23:48:35 == <hr>
23:48:35 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port
80</address>
23:48:35 == </body></html>
23:48:35 == Connection closed by foreign host.
Notice the error log mention socket file name not found:
$ sudo tail /var/log/apache2/error.log
...
[Mon Feb 17 23:47:43.366910 2020] [mpm_event:notice] [pid 5892:tid
140660633114496] AH00493: SIGUSR1 received. Doing graceful restart
[Mon Feb 17 23:47:43.422167 2020] [mpm_event:notice] [pid 5892:tid
140660633114496] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12
configured -- resuming normal operations
[Mon Feb 17 23:47:43.422207 2020] [core:notice] [pid 5892:tid 140660633114496]
AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:47:46.384026 2020] [wsgi:error] [pid 5898:tid 140660447377152]
(2)No such file or directory: [client 127.0.0.1:33536] mod_wsgi (pid=5898):
Unable to connect to WSGI daemon process '127.0.0.1' on
'/var/run/apache2/wsgi.5892.0.1.sock'.
The socket file name has changed:
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.1.1.sock
With the fix/patch for Xenial:
---
$ sudo add-apt-repository ppa:mfo/lp1863232v2
$ sudo apt update
$ sudo apt install -y libapache2-mod-wsgi
$ sudo systemctl restart apache2
$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1ubuntu1
Same behavior continues by default:
---
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.0.1.sock
$ ps 4149
PID TTY STAT TIME COMMAND
4149 ? Ss 0:00 /usr/sbin/apache2 -k start
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.1.1.sock
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.2.1.sock
Manually changing the setting:
---
$ echo 'WSGISocketRotation Off' | sudo tee -a
/etc/apache2/conf-enabled/wsgi.conf
$ sudo systemctl restart apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock
Notice name pattern change as in patch. (u33 = UID 33 = www-data:)
$ id -un 33
www-data
And socket file name remains constant:
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock
$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock
Check the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 500 now)
---
The second HTTP request (after reload) no longer hits Error 503,
but now Error 500, and the log file no longer mentions file not
found, but a different message.
The Error 500 is observed in/consistent with Eoan as well, thus
not a missing patch or something -- this is an improvement.
$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat
http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo
"$(date +'%T') == $line"; done
23:51:50 == Trying 127.0.0.1...
23:51:50 == Connected to 127.0.0.1.
23:51:50 == Escape character is '^]'.
23:51:50 == HTTP/1.1 200 OK
23:51:50 == Date: Mon, 17 Feb 2020 23:51:50 GMT
23:51:50 == Server: Apache/2.4.18 (Ubuntu)
23:51:50 == Content-Length: 13
23:51:50 == Keep-Alive: timeout=15, max=100
23:51:50 == Connection: Keep-Alive
23:51:50 == Content-Type: text/plain
23:51:50 ==
23:51:50 == Hello World!
23:51:56 == HTTP/1.1 500 Internal Server Error
23:51:56 == Date: Mon, 17 Feb 2020 23:51:56 GMT
23:51:56 == Server: Apache/2.4.18 (Ubuntu)
23:51:56 == Content-Length: 607
23:51:56 == Connection: close
23:51:56 == Content-Type: text/html; charset=iso-8859-1
23:51:56 ==
23:51:56 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:51:56 == <html><head>
23:51:56 == <title>500 Internal Server Error</title>
23:51:56 == </head><body>
23:51:56 == <h1>Internal Server Error</h1>
23:51:56 == <p>The server encountered an internal error or
23:51:56 == misconfiguration and was unable to complete
23:51:56 == your request.</p>
23:51:56 == <p>Please contact the server administrator at
23:51:56 == webmaster@localhost to inform them of the time this error occurred,
23:51:56 == and the actions you performed just before this error.</p>
23:51:56 == <p>More information about this error may be available
23:51:56 == in the server error log.</p>
23:51:56 == <hr>
23:51:56 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port
80</address>
23:51:56 == </body></html>
23:51:56 == Connection closed by foreign host.
$ sudo tail /var/log/apache2/error.log
...
[Mon Feb 17 23:51:51.260775 2020] [mpm_event:notice] [pid 6554:tid
140543840806784] AH00493: SIGUSR1 received. Doing graceful restart
[Mon Feb 17 23:51:51.326786 2020] [mpm_event:notice] [pid 6554:tid
140543840806784] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12
configured -- resuming normal operations
[Mon Feb 17 23:51:51.326873 2020] [core:notice] [pid 6554:tid 140543840806784]
AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:51:56.277943 2020] [wsgi:alert] [pid 7039:tid 140543732401920]
mod_wsgi (pid=7039): Request origin could not be validated.
[Mon Feb 17 23:51:56.278181 2020] [wsgi:error] [pid 6907:tid 140543690430208]
[client 127.0.0.1:33556] Truncated or oversized response headers received from
daemon process '127.0.0.1': /var/www/html/hello-world.py
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1863232
Title:
daemon rotates socket on restart
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mod-wsgi/+bug/1863232/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
