2009/2/10 Joshua J. Kugler <[email protected]>: > > On Sunday 08 February 2009, Graham Dumpleton said something like: >> In PHP, it preloads all its extension modules in the Apache parent >> process. That is, they have been loaded even before Apache forks off >> the child processes which handle requests. As a result, creation of >> new child processes capable of handling PHP requests is very quick >> with neglible overhead. > > And, as has been mentioned, all those modules share memory, so memory > usage is much lower.
Correct and what various people who rubbish Python embedded within Apache seem not to understand the subtlety of. If one looks at a typical PHP .so bundle with all extensions it can take up 7MB or more of memory. Being shared though and not appearing as resident memory local to process, they ignore the fact that PHP does actually require a lot of memory in itself. > So, in relation to this question, would there be any way to tell > mod_wsgi to pre-load a set of modules, and then allow Apache to fork? > That would bring memory usage WAY down as Python only loads a module > once, and it seems each instance could share. I have been thinking a little about doing something like that. That is, a directive which would allow one to give a list of Python modules which would be preloaded in Apache parent before the fork. There are various issues to be considered though. First is that Apache parent typically runs as root. It may therefore only be advisable to be listing any C extension modules associated with the Python installation itself and not third party modules, even if in site-packages. The second issue is that the sharing only helps with the .so part of a Python module. That is, you wouldn't gain much from the Python code parts and any objects they create in memory. Even if you could initially benefit from copy on write semantics of memory pages, because objects embed the reference count, as soon as the objects/code is used, reference counts likely need to be manipulated and so page has to be copied into child process memory space. The third is that in Apache parent process only the main Python interpreter instance is created. The actual application that may use a module may use a sub interpreter which is only created in child process context. I would rather not be creating additional sub interpreters in Apache parent process as it adds unwanted complexity to code as they have to be destroyed properly when an Apache restart is performed. But then, since the .so is only loaded once for whole process, no matter how many sub interpreters later created, may be okay. Does mean that only preloading into main interpreter. The fourth issue is that arbitrary third party extension modules may not be implemented correctly so as to work properly when Python is destroyed within context of a process and then reinitialised, as happens with an Apache restart. The result could be crashes when extension module is reinitialised, or wrong behaviour. > An alternative, but related: could mod_wsgi execute the WSGI script file > before allowing Apache to fork? Our script file loads just about all > our modules when it does a bootstrap of our environment, so would be an > ideal candidate for pre-loading all our modules. Lot of the same issues as above, the worst possibly being that would run as root, meaning that files created as a side effect of WSGI script wouldn't have correct permissions to then be usable by application when run in child process. Anyway, something that needs a bit more thought. Graham --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en -~----------~----~----~----~------~----~------~--~---
