If I exclude your mathsite modules, on MacOS X at least, importing all the other modules under mod_wsgi only results in a process size under 12MB.
What sort of data are your mathsite modules loading as part of the code, or preloading in some way from somewhere else. For them to take 30MB is quite a lot. Right now because you are preloading mathsite into the task-queue-manager.py file at global scope, that process itself is quite larger to begin with. At the least what you should do is organise the algorithm function being called under the task manager as a wrapper which only lazily imports the mathsite module when it is called. That is, mathsite would not be preloaded in task-queue-manager.py, but would only be imported after a worker process is forked from it. That cuts back the size of the manager process, but you will need to work out why your mathsite code may be using so much memory even before it does anything. Graham On 28/02/2015, at 10:02 PM, Paul Royik <[email protected]> wrote: > Here you go: > ['Queue', 'UserDict', '__builtin__', '__future__', '__main__', '_abcoll', > '_codecs', '_collections', '_functools', '_hashlib', '_heapq', '_io', > '_locale', '_mod_wsgi_f07fbb7150823bae22c26382ad841b54', '_multiprocessing', > '_random', '_socket', '_sre', '_ssl', '_struct', '_sysconfigdata', > '_warnings', '_weakref', '_weakrefset', 'abc', 'apache', 'array', 'atexit', > 'binascii', 'cPickle', 'cStringIO', 'codecs', 'collections', 'copy_reg', > 'encodings', 'encodings.__builtin__', 'encodings.aliases', 'encodings.ascii', > 'encodings.codecs', 'encodings.encodings', 'encodings.utf_8', 'errno', > 'exceptions', 'fcntl', 'functools', 'gc', 'genericpath', 'hashlib', 'heapq', > 'imp', 'io', 'itertools', 'keyword', 'linecache', 'marshal', 'math', > 'mathsite', 'mathsite.timeout', 'mathsite.timeout.exceptions', > 'mathsite.timeout.mathsite', 'mathsite.timeout.multiprocessing', > 'mathsite.timeout.os', 'mathsite.timeout.signal', 'mathsite.timeout.timeout', > 'mod_wsgi', 'mod_wsgi.pkgutil', 'multiprocessing', 'multiprocessing.Queue', > 'multiprocessing._multiprocessing', 'multiprocessing.array', > 'multiprocessing.atexit', 'multiprocessing.cPickle', > 'multiprocessing.connection', 'multiprocessing.errno', > 'multiprocessing.forking', 'multiprocessing.functools', > 'multiprocessing.itertools', 'multiprocessing.managers', > 'multiprocessing.multiprocessing', 'multiprocessing.os', > 'multiprocessing.pickle', 'multiprocessing.process', > 'multiprocessing.signal', 'multiprocessing.socket', > 'multiprocessing.subprocess', 'multiprocessing.sys', > 'multiprocessing.tempfile', 'multiprocessing.threading', > 'multiprocessing.time', 'multiprocessing.traceback', 'multiprocessing.util', > 'multiprocessing.weakref', 'operator', 'os', 'os.path', 'pickle', 'pkgutil', > 'posix', 'posixpath', 'random', 're', 'select', 'signal', 'site', > 'sitecustomize', 'socket', 'sre_compile', 'sre_constants', 'sre_parse', > 'stat', 'struct', 'subprocess', 'sys', 'sysconfig', 'tempfile', 'thread', > 'threading', 'time', 'traceback', 'types', 'warnings', 'weakref', 'zipimport'] > > On Saturday, February 28, 2015 at 11:36:37 AM UTC+2, Graham Dumpleton wrote: > What code modules does this algorithm require. > > If that is all the (wsgi:service:tasks) processes, then it shows that just > loading all the code and before the process even really does anything is > using 41MB of memory or there about. > > That is quite a lot just for code. You may well still be dragging it too much > stuff you not need. > > That said, there are a lot of Python packages that are really badly > structured and just to use a single function from a module they have a side > effect of dragging in huge amounts of other stuff you do not need in there > parts of the package. > > In the task-queue-manager.py file, just before you you run > 's.serve_forever()', add: > > import sys > print(sorted(sys.modules.keys())) > > Then from Apache error log, get the list of modules you are importing and > post them. > > Graham > > On 28/02/2015, at 12:29 AM, Paul Royik <[email protected]> wrote: > > I isolated algorithm and it worked. > > However, I don't see any improvement in memory usage. > Actually, things become even worse. > > 27242 80580 (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27243 45108 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27244 3076 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27345 2860 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27393 3024 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30377 41568 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30902 41620 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30914 41604 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > Total 359.777 MB > > Perhaps, because I use external library? I can't isolate it. > > Also, I noticed, that when algorithm finishes tasks process is not killed. > So, after some time I have a huge list of tasks processes. This doesn't > happen with previous version of timeout. > > On Friday, February 27, 2015 at 3:14:48 PM UTC+2, Paul Royik wrote: > I isolated algorithm and i > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
