Hi Graham! Thank you for your assistance! I had one more question, we have some applications that used to run before because they thought they were running in the main thread but now they don't. One of the things that they did was register signal handlers. We also use Nginx with uwsgi and have the main thread serve the requests by using process=X and threads=1 in the uwsgi config. That serves all the requests in main threads and things work fine there.
Is there any alternative that would replicate the same behavior in the Apache+mod_wsgi setup? Thanks On Tuesday, May 18, 2021 at 9:07:53 AM UTC+5:30 Graham Dumpleton wrote: > That should just require an instance of threading.Lock object and having > the code acquire the lock before doing the changes (and reading if > necessary to ensure changes can't modify data while being used) of the > common data. > > On 17 May 2021, at 6:31 pm, Swapnil Ojha <[email protected]> wrote: > > Hi Graham! > > The check was done to ensure that there isn't a race condition when > multiple threads modify the global context. We were able to figure out that > the check is done in a naive way and are working on improving the check. > Once that is done, we will be able to run the application in the side > thread too. > > Thanks again for your help! > > On Wednesday, May 12, 2021 at 4:06:46 AM UTC+5:30 Graham Dumpleton wrote: > >> >> On 12 May 2021, at 1:35 am, Swapnil Ojha <[email protected]> wrote: >> >> Thank you for taking a look and the detailed explanation. >> >> > So the question now is what are you doing that has to be done in the >> main thread and can't be in a secondary thread? If can understand that can >> explain what you need to do to change your code to handle that requests >> always run in secondary >> > threads. >> >> This is part of the non-public code. It modifies the current context and >> only allows to do so if being done from the main thread. >> >> >> But why does it need that check? >> >> The only limitations I know of in Python in regards to stuff that can >> only be done in the main Python thread relates to setting up signal >> handlers, but under mod_wsgi it overrides things and effectively disables >> you from doing stuff with signal handlers so you don't break Apache signal >> handling. >> >> So there must be a technical reason that check exists and would need to >> know what it is to help further. Is it to try and avoid proper thread mutex >> locking or something? >> >> On Tuesday, May 11, 2021 at 4:15:58 PM UTC+5:30 Graham Dumpleton wrote: >> >>> In daemon mode requests are never handled in the true main Python >>> thread. Only the pre-loading occurs in the main Python thread. >>> >>> What you were likely seeing was the result in what could be argued is a >>> bug in the Python threading module in that it calculates incorrectly what >>> is the main Python thread. >>> >>> The problem with Python threading module is that it assumes that >>> whatever thread first imports the threading module is the main thread, yet >>> the Python interpreter on initialisation doesn't actually import the >>> threading module, so that can't actually be guaranteed in an embedded >>> system, where the first call into the interpreter to execute Python code >>> and which imports the threading module may be an external thread and not >>> the same main thread which created the Python interpreter in the first >>> place. >>> >>> In your original case what was occurring was that the first, and only >>> request thread, got designated due to this incorrect assumption in the >>> threading module as the main Python thread when it actually wasn't. >>> >>> When you enabled preloading, where the script is preloaded using the >>> true main Python thread, it correctly gets designated as the main thread by >>> the threading module. The request thread then is properly seen as not being >>> the main Python thread. >>> >>> The latest develop branch of mod_wsgi actually has a change which >>> overrides this wrong behaviour of the Python threading module by forcibly >>> importing the threading module in the main thread just after the Python >>> interpreter is initialised. Thus it ensures that request threads never >>> wrongly get reported as the Python main thread when pre-loading isn't being >>> used. >>> >>> In other words, the behaviour you see when you enable pre-loading is >>> actually the correct behaviour and means the wrong behaviour of the >>> threading module doesn't happen. >>> >>> So the question now is what are you doing that has to be done in the >>> main thread and can't be in a secondary thread? If can understand that can >>> explain what you need to do to change your code to handle that requests >>> always run in secondary threads. >>> >>> Also be aware that when you specific process-group and application-group >>> to WSGIScriptAlias, you don't actually need to set WSGIProcessGroup and >>> WSGIApplicationGroup as the options on WSGIScriptAlias take precedence and >>> override them anyway. >>> >>> Graham >>> >>> On 11 May 2021, at 8:27 pm, Swapnil Ojha <[email protected]> wrote: >>> >>> I am using apache(2.4.6) with mod_wsgi(4.6.5) in daemon mode. I am >>> currently working on a project which needs to preload the WSGI application. >>> >>> This is the configuration i was using before: >>> <VirtualHost *:8080> >>> DocumentRoot /path/to/repro/wwwroot >>> WSGIDaemonProcess repro processes=1 threads=1 display-name=repro >>> maximum-requests=0 >>> WSGIProcessGroup repro >>> WSGIScriptAlias /repro /path/to/repro.py process-group=repro >>> WSGIApplicationGroup %{GLOBAL} >>> </VirtualHost> >>> >>> This is the repro.py file: >>> import os >>> import web >>> import sys >>> import threading >>> urls = ( '/', 'index' ) >>> class index: >>> def GET(self): >>> print(threading.main_thread()) >>> print(threading.current_thread()) >>> print(threading.enumerate()) >>> return "Hello, world!" >>> >>> application = web.application(urls, globals()).wsgifunc() >>> >>> According to the following two resources: >>> >>> - enable to initialize other python module before first request >>> (Django + mod_wsgi + apache) >>> >>> <https://stackoverflow.com/questions/43941192/enable-to-initialize-other-python-module-before-first-request-django-mod-wsgi> >>> - >>> >>> https://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIScriptAlias.html >>> >>> I changed the configuration to this: >>> <VirtualHost *:8080> >>> DocumentRoot /path/to/repro/wwwroot >>> WSGIDaemonProcess repro processes=1 threads=1 display-name=repro >>> maximum-requests=0 >>> WSGIScriptAlias /repro /path/to/repro.py process-group=repro >>> application-group=%{GLOBAL} >>> WSGIProcessGroup repro >>> WSGIApplicationGroup %{GLOBAL} >>> </VirtualHost> >>> >>> i.e specified both *application-group* and *process-group* directive. >>> This solved the pre-loading problem for us. >>> >>> But, the difference here is that in the first case all the requests were >>> being handled by the main thread but after the configuration change, the >>> requests are now being handled in the non-main thread. We have a bunch of >>> code that requires things to be run on the main thread and were running >>> previously but after the change, it is causing them to fail. >>> >>> I tried reordering WSGIApplicationGroup and WSGIProcess group directive >>> but that didn't work. The only thing that worked was the removal of >>> application-group from WSGIScriptAlias but that then would not preload the >>> wsgi app. Could you please suggest to me what other things I could try? >>> >>> Thank you for your time! >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/modwsgi/eb747f16-10b2-48c3-9ce3-2da6a3ef383fn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/modwsgi/eb747f16-10b2-48c3-9ce3-2da6a3ef383fn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/9cbf2cb7-9592-4fc4-9b8e-4a81f934fc06n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/9cbf2cb7-9592-4fc4-9b8e-4a81f934fc06n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/373b764c-0b53-4039-848d-020d08f32d56n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/373b764c-0b53-4039-848d-020d08f32d56n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/5be34c29-ad70-40ef-a4ff-ac434b40792en%40googlegroups.com.
