> > > BTW, if you run 'ldd' on the mod_python.so file from the Apache > > > modules directory, does it use Python as a shared library or is there > > > no reference to libpython2.?.so at all, meaning it is embedded with in > > > mod_python.so? What is the actual size of your mod_python.so file? > > > mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of > > the shared libraries. > > Which shows that the Python installation on the box is really crappy. > One of the problems with using RedHat derived distributions. > > That the mod_python.so is so large means that there is no shared > library built and provided with the Python installation, or at least > not with the version that mod_python was compiled against. > Alternatively, there is a shared library, but it is only in /usr/lib > and not along side the libpython2.?.a file in /usr/lib/python2.?/ > config directory. > > In other words, when mod_python was built, all it found in the /usr/ > lib/python.2?/config directory was the static library archive. As a > consequence, 'libtool' when it creates the mod_python.so file has no > choice but to extract all the object files from the static Python > library and embed them into mod_python.so itself. Thus, instead of > mod_python.so being less than 0.5MB through being able to link > dynamically to the Python library it is much larger. > > Worse is that the size of mod_python.so indicates that debug may have > been compiled into the Python library. For an optimised build of the > Python library it is normally only about 1MB, not over 3MB as in this > case. > > Even more worse than that, is what now happens when that mod_python.so > is loaded by Apache. Now the way I understand it, normally a > dynamically loaded module should really be shared and thus memory cost > only applies once across all processes that load it, however, if the > object files in that static Python library are not truly position > independent, when mod_python.so gets loaded, the dynamic linker will > have to perform address relocations to fixup references to function > and data from the Python library objects. As a consequence, rather > than being shared memory, the mod_python.so becomes private memory to > the parent Apache process. When Apache forks off the child processes, > they in turn will each end up with their own private copy of up to 4MB > of memory that mod_python.so consumes when it could all have been > shared. So, even on a fresh restart, the Apache processes are going to > be much larger than they need to be because of the way that Python was > built in the first place. > > Now whether this is going to happen like this I am not 100% sure as it > can depend on the operating system and perhaps the age of the > operating system version as well and how they handle loading of > dynamically loadable modules. I am led to believe that this can occur > on Solaris, to what degree it applies on different Linux versions I am > not as sure. > > Either way, that debug seems to be built into your Python library is > not good. IfWebfactionused a better build of Python with > optimisation and a shared library they could quite easily cut down on > memory use allowing your applications to have more space within their > set limit. > > What is even more annoying is that various people run around > continually saying how crap mod_python is because of how much memory > it uses, when in practice a lot of it has to do with poorly built > Python versions as I show above. :-( > > > > Base Django setup can take up 5-7MB without even your own stuff so it > > > can jump up quite a bit. Is that 35MB in total across all processes or > > > per process? > > > About 4MB for the parent process, and 13MB for the two child > > processes. > > > > What version of mod_python are you using? > > > 3.2.8. > > Which has various memory leaks. Even 3.2.10 has memory leaks. I highly > recommend upgrading to mod_python 3.3.1 if you can do it as it > eliminates a lot of memory leaks. If you also do a rebuild of Python > with optimisation and a shared library you should hopefully see less > memory being used then at startup.
Just a quick update to mention that we've taken these comments into account and we've optimized our Django setup at WebFaction: - Django apps now comes with mod_python-3.3.1 by default - mod_python is now compiled with a shared libpython.so (the size of mod_python.so went from 3MB down to 300KB). - you can now choose between Python-2.4 or Python-2.5 when creating a Django app. Remi -- WebFaction - Hosting for an agile web http://www.webfaction.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~----------~----~----~----~------~----~------~--~---
