Does WebFaction provide the 'pstree' program? If it does, learn how to use it to show parent/child relation ships.
Else work out how to get 'ps' to show both process ID and parent process ID. The extra processes could be the result of something int your web application forking child processes to do stuff still, even though you have tried to farm stuff out to the separate task manager. It is too hard to tell based on just the information you have given. Graham On 12/03/2015, at 11:18 PM, Paul Royik <[email protected]> wrote: > Here is my express-config > mod_wsgi-express setup-server > --working-directory ${SITE_NAME} > --server-root express --port ${SITE_PORT} > --log-directory ${LOG_DIR} --access-log --access-log-format "${LOG_FORMAT}" > --access-log-name access_${APPL_NAME}.log --error-log-name > error_${APPL_NAME}.log > --python-path ${SITE_ROOT}/lib/python${PYTHON_VERSION} > --threads 10 > --service-script tasks $SITE_NAME/$SITE_NAME/timeout/task-queue-manager.py > $SITE_NAME/$SITE_NAME/wsgi.py > > Don't see where I can tune number of processes. > Also, I will find out about how many instances can external library handle > and let you know. > > On Thursday, March 12, 2015 at 3:38:17 AM UTC+2, Graham Dumpleton wrote: > > On 12/03/2015, at 4:06 AM, Paul Royik <[email protected]> wrote: > >> I still experience problems with memory overflow on webfaction >> But it seems that problem is not in service:tasks (it takes only 9 MB). >> I don't know why there are so many other threads. >> >> simamura - 1MB - 23:37:52 - 1571 - >> /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f >> /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start >> simamura - 43MB - 23:37:52 - 1587 - >> /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f >> /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start >> simamura - 43MB - 23:37:52 - 1588 - >> /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f >> /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start >> simamura - 2MB - 23:37:52 - 1589 - >> /home/simamura/webapps/django_gradis/apache2/bin/httpd.worker -f >> /home/simamura/webapps/django_gradis/apache2/conf/httpd.conf -k start > > Your other Apache instance. > >> simamura - 2MB - 23:42:53 - 23836 - httpd (mod_wsgi-express) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > > Apache parent process. > >> simamura - 3MB - 11:08:36 - 4462 - httpd (mod_wsgi-express) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start >> simamura - 3MB - 13:37:49 - 18634 - httpd (mod_wsgi-express) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > > Apache child worker processes which proxy to mod_wsgi daemon processes. > >> simamura - 114MB - 0:16:12 - 10076 - (wsgi:localhost:20241:1092) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start >> simamura - 120MB - 0:15:15 - 11838 - (wsgi:localhost:20241:1092) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start >> simamura - 131MB - 0:11:15 - 20119 - (wsgi:localhost:20241:1092) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start >> simamura - 98MB - 0:10:48 - 20908 - (wsgi:localhost:20241:1092) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > > The mod_wsgi daemon processes. > >> simamura - 9MB - 23:42:53 - 23854 - (wsgi:service:tasks) -f >> /home/simamura/webapps/django_math/express/httpd.conf -DWSGI_ACCESS_LOG >> -DWSGI_MPM_ENABLE_EVENT_MODULE -DWSGI_MPM_EXISTS_EVENT_MODULE >> -DWSGI_MPM_EXISTS_WORKER_MODULE -DWSGI_MPM_EXISTS_PREFORK_MODULE -k start > > The service process which acts as parent for the task processes. > > What are you currently using for the --processes and --threads option to > mod_wsgi-express when setting up configuration? > > The above suggests you are using '--processes 4' and looks completely normal. > So I don't know what you mean by are so many threads. > > What the above doesn't show is how big your transient task worker processes > are getting when running your algorithm. > > That is important as right now there is no limit on how many of those can run > concurrently except that of how many concurrent processes your mod_wsgi > daemon processes can handle. > > You need to understand realistically how many concurrent instances of your > algorithm might handle and if you need to implement a queuing mechanism to > restrict the number then the queue manager will need to change. > > Graham > >> On Thursday, March 5, 2015 at 9:46:34 AM UTC+2, Paul Royik wrote: >> If you have time, please respond. >> >> On Monday, March 2, 2015 at 2:04:46 PM UTC+2, Paul Royik wrote: >> I made a research and found no suspicious imports. >> I give you contents of mathsite.timeout module for further research: >> >> exceptions.py >> class TimeoutException(Exception): >> pass >> >> manager.py >> from mathsite.timeout.timeout import MyManager >> import os >> def get_manager(): >> sockpath = os.path.join(os.path.dirname(__file__), >> 'task-queue-manager.sock') >> if True: >> return >> try: >> m = MyManager(address=sockpath, authkey='abracadabra') >> m.connect() >> return m.Maths() >> except: >> return None >> >> task-queue-manager.py >> import os >> >> from mathsite.timeout.timeout import MyManager >> >> sockpath = os.path.join(os.path.dirname(__file__), 'task-queue-manager.sock') >> >> try: >> os.unlink(sockpath) >> except OSError: >> pass >> >> m = MyManager(address=sockpath, authkey='abracadabra') >> s = m.get_server() >> s.serve_forever() >> >> timeout.py >> import multiprocessing >> import os >> import signal >> from multiprocessing.managers import BaseManager >> from mathsite.timeout.exceptions import TimeoutException >> >> >> class RunableProcessing(multiprocessing.Process): >> def __init__(self, func, *args, **kwargs): >> self.queue = multiprocessing.Manager().Queue(maxsize=1) >> args = (func,) + args >> multiprocessing.Process.__init__(self, target=self.run_func, >> args=args, kwargs=kwargs) >> >> def run_func(self, func, *args, **kwargs): >> try: >> >> ... >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
