Since Python is embedded inside of Apache by mod_wsgi, ie., the Python library is linked into the process and the CLI Python is not run/used, all virtual environments you use with it must be created from the same version/directory installation of Python as mod_wsgi was compiled for. So it is sort of understandable that things would break or behave strangely if your two virtual environments were created off different Python installations.
> On 13 May 2024, at 6:15 PM, A McBain <[email protected]> wrote: > > On Sun, May 12, 2024 at 11:37 PM A McBain <[email protected] > <mailto:[email protected]>> wrote: >> >> On Sun, May 12, 2024 at 11:19 PM Graham Dumpleton >> <[email protected]> wrote: >>> >>> The mod_wsgi.so specified by LoadFile is what dictates version (and in what >>> location) of Python is being used. >>> >>> Right now you are using one under your home directory. I would not be >>> surprised if it is compiled for the version of Python you built yourself >>> from source code and that is causing the problem. >>> >>> Ensuring you have no virtual environments activated, run the following: >>> >>> pip3 uninstall mod_wsgi >>> >>> Run this multiple times until it says nothing to uninstall. >>> >>> Then try rebuilding/reinstall mod_wsgi again but when you run pip install >>> use the --no-cache-dir option so it doesn't use old cached build. >>> >>> Then update LoadFile line to use whatever location it now gets installed to >>> if you use a different location (eg., a dedicated virtual environment, or >>> system Python directory. >>> >>> Ensure you stop and start Apache when changed. >>> >>> Graham >> >> OK I uninstalled the old one (there was only the .local copy). I ran >> the install (with the flag) and it's now residing in the more expected >> /usr/local/... python3.11 site-packages directory. >> >> Unfortunately no dice. >> >> I also did a full shutdown and boot. Nope. 😕 > > OK so for transparency it is working now! But heck if I know exactly why. > > I know I probably caused this mess and I'm not sure how to fix it > exactly, so I'm just going to live in my mess for the time being, > since this is just a home dev server, and proper projects go to an > offsite server I purposefully haven't messed up. > > Also, thank you very much for helping me debug this, I really appreciate it. > > For anyone else I'll go through what I found in case it helps them > with their own weird environments. > > Python versions on my server: > 2.7.18 - the system one > 3.11.1 - the one I compiled, only available at /usr/local/bin/python3.11 > 3.12.2 - the package/distro Python, and what is used when I do > "python3 --version", available at /usr/bin/python3 and > /usr/bin/python3.11 > > Inside the virtual environment folder of each app, which I'll just > call Old App and New App, there's a pyvenv.cfg file. Old App appears > to have been created with virtualenv, and New App was created with > venv, but testing shows that doesn't matter. > > Old App had different file contents in pyvenv.cfg, presumably because > it was created prior to installing the package Python (3.11.2): > > home = /usr/local/bin > implementation = CPython > version_info = 3.11.1.final.0 > virtualenv = 20.17.1 > include-system-site-packages = false > base-prefix = /usr/local > base-exec-prefix = /usr/local > base-executable = /usr/local/bin/python3.11 > > New App had this in pyvenv.cfg (after I recreated it using virtualenv): > > home = /usr/bin > implementation = CPython > version_info = 3.11.2.final.0 > virtualenv = 20.17.1+ds > include-system-site-packages = false > base-prefix = /usr > base-exec-prefix = /usr > base-executable = /usr/bin/python3 > > Once I changed /usr to /usr/local and python3 to python3.11 in the New > App, the import error disappeared and left me only with logical errors > due to mistakes in wsgi.prod.py and settings-prod.py leftover from > debugging. Fixing those let the app start up just fine. > > Something weird is going on with the Python 3.11.2, or the interaction > between it and the Python 3.11.1, but as per above since I have the > app working I don't have the energy right this minute to untangle > that. > > Thanks again, > Alastair > >>> On 13 May 2024, at 4:10 PM, A McBain <[email protected]> wrote: >>> >>> On Sun, May 12, 2024 at 11:08 PM A McBain <[email protected]> wrote: >>> >>> >>> >>> >>> On Sun, May 12, 2024 at 10:57 PM Graham Dumpleton >>> <[email protected]> wrote: >>> >>> >>> Ensure you do a complete stop and start of Apache and not just reload or >>> restart in case Apache had cached a reference to a broken Python library >>> variant from your self compiled build. >>> >>> If going to build your own Python from source code ensure you read. A bit >>> dated, but still relevant. Ignore that talks about Docker, still applies >>> for any Python build. >>> >>> Installing a custom Python version into a Docker image. >>> blog.dscpl.com.au >>> >>> Also, where is LoadFile line in Apache picking up mod_wsgi.so from? >>> >>> Graham >>> >>> >>> I don't intend to build from source again if I can help it. That was a >>> PITA. 😄 I only did it way back since I believe at the time Python 3.11 >>> wasn't available as a package for Devuan/Debian yet...? I honestly don't >>> remember. Thank you for the link though if I ever do need. 🙂 >>> >>> I had been just doing "service apache2 restart" yeah, but I just tried a >>> proper stop, then a start, and the state is the same. 😑 >>> >>> So best I can tell this is where I'm at: >>> >>> CLI-run Python, the older site, and new site, have identical paths except >>> for the first entry (CLI = '', apps = their app dir) >>> Both apps don't import cmath and a bunch of other modules but the CLI >>> Python does >>> All prefixes point to the correct virtual env root >>> The old app still runs even after a full stop and start of Apache >>> >>> I apologize for all this. I don't have any idea what's going on and I've >>> tried everything I can think of, what you've asked / suggested, and it's >>> still breaking my brain. >>> >>> - Alastair >>> >>> >>> Oh, the load command. Right, so that's at >>> "/etc/apache2/mods-enabled/wsgi.load" >>> >>> The contents are this: >>> >>> #LoadModule wsgi_module /usr/lib/apache2/modules/mod_wsgi.so >>> LoadModule wsgi_module >>> /home/asmcbain/.local/lib/python3.11/site-packages/mod_wsgi/server/mod_wsgi-py311.cpython-311-x86_64-linux-gnu.so >>> >>> On 13 May 2024, at 3:35 PM, A McBain <[email protected]> wrote: >>> >>> On Sun, May 12, 2024 at 10:22 PM A McBain <[email protected]> wrote: >>> >>> >>> On Sun, May 12, 2024 at 10:05 PM A McBain <[email protected]> wrote: >>> >>> >>> On Sun, May 12, 2024 at 9:47 PM Graham Dumpleton >>> <[email protected]> wrote: >>> >>> >>> >>> >>> On 13 May 2024, at 2:33 PM, A McBain <[email protected]> wrote: >>> >>> The old one is running on the same physical server, just with its own >>> Apache config and own subdomain. As to having multiple things running, >>> that's why I set WSGIApplicationGroup to %{SERVER} but just in case I >>> changed it explicitly to "enfilade.asmcbain.net" (no effect, but shouldn't >>> hurt anything to keep the change). >>> >>> >>> The WSGIApplicationGroup directive sets which Python interpreter context is >>> used. It is distinct from the daemon process group. >>> >>> You want each WSGI application instance to run in a different mod_wsgi >>> daemon process group if using the main Python interpreter context >>> (application group %{GLOBAL}). >>> >>> So what you want is for each VirtualHost to use a different named mod_wsgi >>> dameon process group. >>> >>> <VirtualHost ...> >>> ServerName site-1.example.com >>> WSGIDaemonProcess site-1 \ >>> user=asmcbain group=asmcbain \ >>> display-name='%{GROUP}' \ >>> lang='en_US.UTF-8' \ >>> locale='en_US.UTF-8' \ >>> threads=5 \ >>> queue-timeout=45 \ >>> socket-timeout=60 \ >>> connect-timeout=15 \ >>> request-timeout=60 \ >>> inactivity-timeout=0 \ >>> startup-timeout=15 \ >>> deadlock-timeout=60 \ >>> graceful-timeout=15 \ >>> eviction-timeout=0 \ >>> restart-interval=0 \ >>> shutdown-timeout=5 \ >>> maximum-requests=0 \ >>> python-home=/home/asmcbain/enfilade/.env \ >>> python-path=/home/asmcbain/enfilade >>> WSGIProcessGroup site-1 >>> WSGIApplicationGroup %{GLOBAL} >>> WSGIScriptAlias / /home/asmcbain/enfilade/atcid/wsgi.py >>> </VirtualHost> >>> >>> <VirtualHost ...> >>> ServerName site-2.example.com >>> WSGIDaemonProcess site-2 \ >>> user=asmcbain group=asmcbain \ >>> display-name='%{GROUP}' \ >>> lang='en_US.UTF-8' \ >>> locale='en_US.UTF-8' \ >>> threads=5 \ >>> queue-timeout=45 \ >>> socket-timeout=60 \ >>> connect-timeout=15 \ >>> request-timeout=60 \ >>> inactivity-timeout=0 \ >>> startup-timeout=15 \ >>> deadlock-timeout=60 \ >>> graceful-timeout=15 \ >>> eviction-timeout=0 \ >>> restart-interval=0 \ >>> shutdown-timeout=5 \ >>> maximum-requests=0 \ >>> python-home=/home/asmcbain/enfilade/.env \ >>> python-path=/home/asmcbain/enfilade >>> WSGIProcessGroup site-2 >>> WSGIApplicationGroup %{GLOBAL} >>> WSGIScriptAlias / /home/asmcbain/enfilade/atcid/wsgi.py >>> </VirtualHost> >>> >>> So each site should have different name to WSGIDaemonProcess, with matching >>> WSGIProcessGroup, and WSGIApplicationGroup should be %{GLOBAL} for both, >>> not {SERVER}. >>> >>> The {SERVER} value has a different meaning and best avoiding it. >>> >>> So lets make sure each is running in separate daemon process groups. >>> >>> BTW, also make sure that somewhere outside of all VirtualHost definitions >>> you add: >>> >>> WSGIRestrictEmbedded On >>> >>> >>> OK, I updated both the old site and the new one to make >>> WSGIApplicationGroup %{GLOBAL}. They always had different >>> WSGIProcessGroup values. >>> >>> I also added WSGIRestrictEmbedded to the base Apache config. >>> >>> Apache restarts fine, and the old site loads up but the new one still >>> prints the import math error even after the changes. 😕 >>> >>> I've attached the relevant bits to both. >>> >>> - Alastair >>> >>> >>> Originally I'd compiled this Python 3.11 myself, but I've just double >>> checked and verified Python 3.11 and the -dev package are from >>> official packages and not my own manual build+install. >>> >>> I also checked that inside or outside of the virtual env (results are >>> the same) I can run the Python interpreter shell and do: >>> >>> import math >>> math.ceil(1.1) >>> >>> I don't get any errors. >>> >>> - Alastair >>> >>> >>> Update to that. The modules loaded are different between the daemon >>> one and the regular interpreter. >>> >>> Doing "print(sys.builtin_module_names)" results in the following for >>> the daemon version: >>> >>> [Sun May 12 22:31:33.744235 2024] [wsgi:error] [pid 7430:tid >>> 140134038087360] ('_abc', '_ast', '_codecs', '_collections', >>> '_functools', '_imp', '_io', '_locale', '_operator', '_signal', >>> '_sre', '_stat', '_string', '_symtable', '_thread', '_tokenize', >>> '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', >>> 'errno', 'faulthandler', >>> 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype') >>> >>> And this substantially longer list for the one I just run from the CLI >>> myself: >>> >>> ('_abc', '_ast', '_bisect', '_blake2', '_codecs', '_collections', >>> '_csv', '_datetime', '_elementtree', '_functools', '_heapq', '_imp', >>> '_io', '_locale', '_md5', '_opcode', '_operator', '_pickle', >>> '_posixsubprocess', '_random', '_sha1', '_sha256', '_sha3', '_sha512', >>> '_signal', '_socket', '_sre', '_stat', '_statistics', '_string', >>> '_struct', '_symtable', '_thread', '_tokenize', '_tracemalloc', >>> '_warnings', '_weakref', 'array', 'atexit', 'binascii', 'builtins', >>> 'cmath', 'errno', 'faulthandler', 'fcntl', 'gc', 'grp', 'itertools', >>> 'marshal', 'math', 'posix', 'pwd', 'pyexpat', 'select', 'spwd', 'sys', >>> 'syslog', 'time', 'unicodedata', 'xxsubtype', 'zlib') >>> >>> Now I just have to figure out why. >>> >>> >>> >>> This will cause an error if mod_wsgi daemon process groups aren't being >>> configured properly. >>> >>> Graham >>> >>> >>> This is the result of the wsgi.prod.py printouts: >>> >>> [Sun May 12 21:29:11.788518 2024] [wsgi:error] [pid 5671:tid >>> 140047682610880] sys.prefix '/home/asmcbain/enfilade/.env' >>> [Sun May 12 21:29:11.788639 2024] [wsgi:error] [pid 5671:tid >>> 140047682610880] sys.path ['/home/asmcbain/enfilade', >>> '/usr/lib/python311.zip', '/usr/lib/python3.11', >>> '/usr/lib/python3.11/lib-dynload', >>> '/home/asmcbain/enfilade/.env/lib/python3.11/site-packages'] >>> >>> Alastair >>> >>> On Sun, May 12, 2024 at 9:24 PM Graham Dumpleton >>> <[email protected]> wrote: >>> >>> >>> >>> On 13 May 2024, at 2:02 PM, A McBain <[email protected]> wrote: >>> >>> >>> Hi, I looked at previous messages and others on StackOverflow but none seem >>> to solve my issue. >>> >>> I have an app I wrote working perfectly fine under Python 3.11 with >>> mod_wsgi and Apache 2. >>> >>> I did a bunch of development on the app (upgraded django, new features), >>> and set up a new checkout of that on my server, with its own virtual >>> environment (using venv). It uses effectively the same config (different >>> subdomain) in Apache2 as the original older copy, but the new one fails >>> with an import error while the old one is still chugging along. >>> >>> >>> When you say "old one is still chugging along" can you confirm you are >>> saying that the existing one is still running on the same server at the >>> same time, or it is a completely new server you have set up. >>> >>> I double checked the instructions at >>> https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/modwsgi/ >>> >>> I've also re-verified the mod_wsgi I installed (via pip) matches the Python >>> version I'm using (both are Python 3.11). >>> >>> I also tried: >>> >>> Removing the python-path argument >>> >>> The mod_wsgi docs suggest I don't need that if I specify python-home? >>> >>> Removing python-path if it refers to site-packages directory of virtual >>> environment specified to python-home is okay as path would be redundant. If >>> python-path is used to tell it where your Django project is, that still >>> needs to stay. >>> >>> Setting WSGIApplicationGroup to %{GLOBAL} >>> >>> That is recommended, but if you are hosting multiple WSGI applications on >>> the same server, they would need to be delegated to different mod_wsgi >>> daemon process groups. >>> >>> Unfortunately the error didn't change at all after trying those. >>> >>> I've attached the relevant apache2 config section which includes all the >>> mod_wsgi-setup (the rest is just redirects, ssl stuff, aliases, etc.) >>> >>> I also attached the error from the log. It looks like it's trying to use >>> the system Python instead of the one that exists in my .env (virtual >>> environment) directory. >>> >>> >>> The virtual environment doesn't have a copy of Python stdlib files so they >>> are still imported from the system Python the virtual environment was >>> created from. The virtual environment effectively only has its own >>> site-packages directory. >>> >>> I'm banging my head as to why this worked before but not now so any help is >>> much appreciated. Thank you! 🙂 >>> >>> Other details: >>> >>> wsgi.prod.py is the default wsgi.py, just modified to load settings.prod.py >>> The server is Devuan Daedalus >>> >>> >>> At the start of your WSGI script file add: >>> >>> import sys >>> print('sys.prefix', repr(sys.prefix)) >>> print('sys.path', repr(sys.path)) >>> >>> and check Apache logs for what paths it is using. >>> >>> Compare that to the working site. >>> >>> Post what you learn from that. >>> >>> Graham >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/7184C263-1522-4583-A98B-EC9806B0FA66%40gmail.com. >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/CAGV_ScocfNc%3DSv7W7Ha3MbrM_0wq43v8U0AM3JOuNtsfP9vK%3Dw%40mail.gmail.com. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/41CA1CF3-8A06-46BF-A7AE-2CA74FCC2196%40gmail.com. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/CAGV_ScoxJu%3DxKwijvbPjE78BnZZD4s2ppRBT0-jCErCgdr8v%2Bg%40mail.gmail.com. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/55163B95-DCAF-4129-BD81-4D477F4FC9F6%40gmail.com. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/CAGV_Scr3A_PxtgAtamReunLZ9zjG03OFG%2BZZFpQWV5aSzmo16Q%40mail.gmail.com. >>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "modwsgi" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/modwsgi/IJp7zr6SjtY/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/F97967BE-2139-4E76-A9DC-4C7A9B3A397E%40gmail.com. > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/CAGV_ScqPGkJ2r_UKL0rQHf6RHXLvfoSWrpmVk-tfc0gqCtvb%2BA%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/60D4F577-15FB-4C20-AC85-0B2C73BE0753%40gmail.com.
