Steps to reproduce/verify on Xenial:
===

References:
- 
https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html
- 
https://modwsgi.readthedocs.io/en/develop/user-guides/quick-configuration-guide.html#delegation-to-daemon-process
- 
https://github.com/GrahamDumpleton/mod_wsgi/commit/13169f2a0610d7451fae92a414e8e20b91e348c9#diff-3e8b16b2885169dcec2dac843521e12cR29
- 
https://www.howtoforge.com/tutorial/how-to-run-python-scripts-with-apache-and-mod_wsgi-on-ubuntu-18-04/

Create a container with apache2/mod-wsgi in daemon mode,
w/ keep-alive timeout long enough, and wsgi hello-world:
---

$ lxc launch ubuntu:xenial lp1863232x
$ lxc exec lp1863232x -- su - ubuntu

$ sudo apt update
$ sudo apt install -y apache2 libapache2-mod-wsgi

$ sudo sed -i '/^KeepAliveTimeout/ s/ .*/ 15/' /etc/apache2/apache2.conf

$ cat <<EOF | sudo tee /etc/apache2/conf-available/wsgi.conf
WSGIScriptAlias /hello-world /var/www/html/hello-world.py
WSGIDaemonProcess 127.0.0.1 processes=2 threads=16 display-name=%{GROUP}
WSGIProcessGroup 127.0.0.1
EOF

$ sudo a2enconf wsgi
$ sudo systemctl reload apache2

$ cat <<EOF | sudo tee /var/www/html/hello-world.py
def application(environ, start_response):
    status = '200 OK'
    output = b'Hello World!\n'

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]

    start_response(status, response_headers)
    return [output]
EOF

$ curl 127.0.0.1/hello-world
Hello World!


Check WSGI socket filename changes when
reloading apache2 (aka graceful restart):
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.0.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.1.1.sock

$ sudo systemctl reload apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.2476.2.1.sock


Create script and shell one-liner
to send two HTTP requests in the
same connection (keep-alive used)
---

$ cat <<EOF >http-request
GET /hello-world HTTP/1.1
Host: 127.0.0.1
Connection: keep-alive

EOF

One connection/One request:

$ (cat http-request; sleep 1) | telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
HTTP/1.1 200 OK
Date: Mon, 17 Feb 2020 23:33:19 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 13
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/plain

Hello World!
Connection closed by foreign host.


One connection/Two requests (added timestamps to check timeout values)

$ (cat http-request; sleep 1; cat http-request; sleep 9) | telnet 127.0.0.1 80 
2>&1 | while read line; do echo "$(date +'%T') == $line"; done
23:34:42 == Trying 127.0.0.1...
23:34:42 == Connected to 127.0.0.1.
23:34:42 == Escape character is '^]'.
23:34:42 == HTTP/1.1 200 OK
23:34:42 == Date: Mon, 17 Feb 2020 23:34:42 GMT
23:34:42 == Server: Apache/2.4.18 (Ubuntu)
23:34:42 == Content-Length: 13
23:34:42 == Keep-Alive: timeout=5, max=100
23:34:42 == Connection: Keep-Alive
23:34:42 == Content-Type: text/plain
23:34:42 == 
23:34:42 == Hello World!
23:34:43 == HTTP/1.1 200 OK
23:34:43 == Date: Mon, 17 Feb 2020 23:34:43 GMT
23:34:43 == Server: Apache/2.4.18 (Ubuntu)
23:34:43 == Content-Length: 13
23:34:43 == Keep-Alive: timeout=5, max=99
23:34:43 == Connection: Keep-Alive
23:34:43 == Content-Type: text/plain
23:34:43 == 
23:34:43 == Hello World!
23:34:58 == Connection closed by foreign host.


Reproduce the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 503,
depending on apache2 MPM module:
mpm_event/mpm_worker/mpm_prefork)
---

$ lsb_release -cs
xenial

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1build1

For reference on socket filename:

$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.0.1.sock

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat 
http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo 
"$(date +'%T') == $line"; done
23:48:29 == Trying 127.0.0.1...
23:48:29 == Connected to 127.0.0.1.
23:48:29 == Escape character is '^]'.
23:48:29 == HTTP/1.1 200 OK
23:48:29 == Date: Mon, 17 Feb 2020 23:48:29 GMT
23:48:29 == Server: Apache/2.4.18 (Ubuntu)
23:48:29 == Content-Length: 13
23:48:29 == Keep-Alive: timeout=15, max=100
23:48:29 == Connection: Keep-Alive
23:48:29 == Content-Type: text/plain
23:48:29 == 
23:48:29 == Hello World!
23:48:35 == HTTP/1.1 503 Service Unavailable
23:48:35 == Date: Mon, 17 Feb 2020 23:48:35 GMT
23:48:35 == Server: Apache/2.4.18 (Ubuntu)
23:48:35 == Content-Length: 374
23:48:35 == Connection: close
23:48:35 == Content-Type: text/html; charset=iso-8859-1
23:48:35 == 
23:48:35 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:48:35 == <html><head>
23:48:35 == <title>503 Service Unavailable</title>
23:48:35 == </head><body>
23:48:35 == <h1>Service Unavailable</h1>
23:48:35 == <p>The server is temporarily unable to service your
23:48:35 == request due to maintenance downtime or capacity
23:48:35 == problems. Please try again later.</p>
23:48:35 == <hr>
23:48:35 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 
80</address>
23:48:35 == </body></html>
23:48:35 == Connection closed by foreign host.

Notice the error log mention socket file name not found:

$ sudo tail /var/log/apache2/error.log 
...
[Mon Feb 17 23:47:43.366910 2020] [mpm_event:notice] [pid 5892:tid 
140660633114496] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 17 23:47:43.422167 2020] [mpm_event:notice] [pid 5892:tid 
140660633114496] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 
configured -- resuming normal operations
[Mon Feb 17 23:47:43.422207 2020] [core:notice] [pid 5892:tid 140660633114496] 
AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:47:46.384026 2020] [wsgi:error] [pid 5898:tid 140660447377152] 
(2)No such file or directory: [client 127.0.0.1:33536] mod_wsgi (pid=5898): 
Unable to connect to WSGI daemon process '127.0.0.1' on 
'/var/run/apache2/wsgi.5892.0.1.sock'.

The socket file name has changed:

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6244.1.1.sock


With the fix/patch for Xenial:
---

$ sudo add-apt-repository ppa:mfo/lp1863232v2
$ sudo apt update

$ sudo apt install -y libapache2-mod-wsgi
$ sudo systemctl restart apache2

$ dpkg -s libapache2-mod-wsgi | grep ^Version
Version: 4.3.0-1.1ubuntu1


Same behavior continues by default:
---

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.0.1.sock

$ ps 4149
  PID TTY      STAT   TIME COMMAND
 4149 ?        Ss     0:00 /usr/sbin/apache2 -k start

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.1.1.sock

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.4149.2.1.sock


Manually changing the setting:
---

$ echo 'WSGISocketRotation Off' | sudo tee -a 
/etc/apache2/conf-enabled/wsgi.conf 
$ sudo systemctl restart apache2

$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock

Notice name pattern change as in patch. (u33 = UID 33 = www-data:)

$ id -un 33
www-data

And socket file name remains constant:

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock

$ sudo systemctl reload apache2
$ ls -1 /var/run/apache2/wsgi.*.sock
/var/run/apache2/wsgi.6554.u33.1.sock


Check the problem by placing
'sudo systemctl reload apache2'
between the two HTTP requests
(second request hits Error 500 now)
---

The second HTTP request (after reload) no longer hits Error 503,
but now Error 500, and the log file no longer mentions file not
found, but a different message.

The Error 500 is observed in/consistent with Eoan as well, thus
not a missing patch or something -- this is an improvement.

$ (cat http-request; sleep 1; sudo systemctl reload apache2; sleep 5; cat 
http-request; sleep 9) | telnet 127.0.0.1 80 2>&1 | while read line; do echo 
"$(date +'%T') == $line"; done
23:51:50 == Trying 127.0.0.1...
23:51:50 == Connected to 127.0.0.1.
23:51:50 == Escape character is '^]'.
23:51:50 == HTTP/1.1 200 OK
23:51:50 == Date: Mon, 17 Feb 2020 23:51:50 GMT
23:51:50 == Server: Apache/2.4.18 (Ubuntu)
23:51:50 == Content-Length: 13
23:51:50 == Keep-Alive: timeout=15, max=100
23:51:50 == Connection: Keep-Alive
23:51:50 == Content-Type: text/plain
23:51:50 == 
23:51:50 == Hello World!
23:51:56 == HTTP/1.1 500 Internal Server Error
23:51:56 == Date: Mon, 17 Feb 2020 23:51:56 GMT
23:51:56 == Server: Apache/2.4.18 (Ubuntu)
23:51:56 == Content-Length: 607
23:51:56 == Connection: close
23:51:56 == Content-Type: text/html; charset=iso-8859-1
23:51:56 == 
23:51:56 == <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
23:51:56 == <html><head>
23:51:56 == <title>500 Internal Server Error</title>
23:51:56 == </head><body>
23:51:56 == <h1>Internal Server Error</h1>
23:51:56 == <p>The server encountered an internal error or
23:51:56 == misconfiguration and was unable to complete
23:51:56 == your request.</p>
23:51:56 == <p>Please contact the server administrator at
23:51:56 == webmaster@localhost to inform them of the time this error occurred,
23:51:56 == and the actions you performed just before this error.</p>
23:51:56 == <p>More information about this error may be available
23:51:56 == in the server error log.</p>
23:51:56 == <hr>
23:51:56 == <address>Apache/2.4.18 (Ubuntu) Server at 127.0.0.1 Port 
80</address>
23:51:56 == </body></html>
23:51:56 == Connection closed by foreign host.


$ sudo tail /var/log/apache2/error.log 
...
[Mon Feb 17 23:51:51.260775 2020] [mpm_event:notice] [pid 6554:tid 
140543840806784] AH00493: SIGUSR1 received.  Doing graceful restart
[Mon Feb 17 23:51:51.326786 2020] [mpm_event:notice] [pid 6554:tid 
140543840806784] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/2.7.12 
configured -- resuming normal operations
[Mon Feb 17 23:51:51.326873 2020] [core:notice] [pid 6554:tid 140543840806784] 
AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 17 23:51:56.277943 2020] [wsgi:alert] [pid 7039:tid 140543732401920] 
mod_wsgi (pid=7039): Request origin could not be validated.
[Mon Feb 17 23:51:56.278181 2020] [wsgi:error] [pid 6907:tid 140543690430208] 
[client 127.0.0.1:33556] Truncated or oversized response headers received from 
daemon process '127.0.0.1': /var/www/html/hello-world.py

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1863232

Title:
  daemon rotates socket on restart

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mod-wsgi/+bug/1863232/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to