What code modules does this algorithm require. If that is all the (wsgi:service:tasks) processes, then it shows that just loading all the code and before the process even really does anything is using 41MB of memory or there about.
That is quite a lot just for code. You may well still be dragging it too much stuff you not need. That said, there are a lot of Python packages that are really badly structured and just to use a single function from a module they have a side effect of dragging in huge amounts of other stuff you do not need in there parts of the package. In the task-queue-manager.py file, just before you you run 's.serve_forever()', add: import sys print(sorted(sys.modules.keys())) Then from Apache error log, get the list of modules you are importing and post them. Graham On 28/02/2015, at 12:29 AM, Paul Royik <[email protected]> wrote: > I isolated algorithm and it worked. > > However, I don't see any improvement in memory usage. > Actually, things become even worse. > > 27242 80580 (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27243 45108 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27244 3076 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27345 2860 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27393 3024 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30377 41568 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30902 41620 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30914 41604 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > Total 359.777 MB > > Perhaps, because I use external library? I can't isolate it. > > Also, I noticed, that when algorithm finishes tasks process is not killed. > So, after some time I have a huge list of tasks processes. This doesn't > happen with previous version of timeout. > > On Friday, February 27, 2015 at 3:14:48 PM UTC+2, Paul Royik wrote: > I isolated algorithm and it worked. > > However, I don't see any improvement in memory usage. > Actually, things become even worse. > > 27242 80580 (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27243 45108 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27244 3076 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27345 2860 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 27393 3024 httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30377 41568 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30902 41620 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > 30914 41604 (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > Total 359.777 MB > > Perhaps, because I use external library? I can't isolate it. > > On Friday, February 27, 2015 at 2:11:44 PM UTC+2, Paul Royik wrote: > OK. I will try > > On Friday, February 27, 2015 at 2:08:34 PM UTC+2, Graha > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
