On 12/03/2015, at 4:06 AM, Paul Royik <[email protected]> wrote:
> I still experience problems with memory overflow on webfaction > But it seems that problem is not in service:tasks (it takes only 9 MB). > I don't know why there are so many other threads. > > simamura - 1MB - 23:37:52 - 1571 - > /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f > /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start > simamura - 43MB - 23:37:52 - 1587 - > /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f > /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start > simamura - 43MB - 23:37:52 - 1588 - > /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f > /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start > simamura - 2MB - 23:37:52 - 1589 - > /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f > /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start Your other Apache instance. > simamura - 2MB - 23:42:53 - 23836 - httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start Apache parent process. > simamura - 3MB - 11:08:36 - 4462 - httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > simamura - 3MB - 13:37:49 - 18634 - httpd (mod_wsgi-express) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start Apache child worker processes which proxy to mod_wsgi daemon processes. > simamura - 114MB - 0:16:12 - 10076 - (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > simamura - 120MB - 0:15:15 - 11838 - (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > simamura - 131MB - 0:11:15 - 20119 - (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > simamura - 98MB - 0:10:48 - 20908 - (wsgi:localhost:20241:1092) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start The mod_wsgi daemon processes. > simamura - 9MB - 23:42:53 - 23854 - (wsgi:service:tasks) -f > /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG > -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE > -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start The service process which acts as parent for the task processes. What are you currently using for the --processes and --threads option to mod_wsgi-express when setting up configuration? The above suggests you are using '--processes 4' and looks completely normal. So I don't know what you mean by are so many threads. What the above doesn't show is how big your transient task worker processes are getting when running your algorithm. That is important as right now there is no limit on how many of those can run concurrently except that of how many concurrent processes your mod_wsgi daemon processes can handle. You need to understand realistically how many concurrent instances of your algorithm might handle and if you need to implement a queuing mechanism to restrict the number then the queue manager will need to change. Graham > On Thursday, March 5, 2015 at 9:46:34 AM UTC+2, Paul Royik wrote: > If you have time, please respond. > > On Monday, March 2, 2015 at 2:04:46 PM UTC+2, Paul Royik wrote: > I made a research and found no suspicious imports. > I give you contents of mathsite.timeout module for further research: > > exceptions.py > class TimeoutException(Exception): > pass > > manager.py > from mathsite.timeout.timeout import MyManager > import os > def get_manager(): > sockpath = os.path.join(os.path.dirname(__file__), > 'task-queue-manager.sock') > if True: > return > try: > m = MyManager(address=sockpath, authkey='abracadabra') > m.connect() > return m.Maths() > except: > return None > > task-queue-manager.py > import os > > from mathsite.timeout.timeout import MyManager > > sockpath = os.path.join(os.path.dirname(__file__), 'task-queue-manager.sock') > > try: > os.unlink(sockpath) > except OSError: > pass > > m = MyManager(address=sockpath, authkey='abracadabra') > s = m.get_server() > s.serve_forever() > > timeout.py > import multiprocessing > import os > import signal > from multiprocessing.managers import BaseManager > from mathsite.timeout.exceptions import TimeoutException > > > class RunableProcessing(multiprocessing.Process): > def __init__(self, func, *args, **kwargs): > self.queue = multiprocessing.Manager().Queue(maxsize=1) > args = (func,) + args > multiprocessing.Process.__init__(self, target=self.run_func, > args=args, kwargs=kwargs) > > def run_func(self, func, *args, **kwargs): > try: > > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
