Hi all, We have a setup of uwsgi running in emperor mode, so that we can have multiple applications (mostly Python, few Ruby) running on a single machine. For most of the application setup, we use reload-on-rss = 256 and evil-reload-on-rss = 512 in case the application misbehaves.
Recently we found out the machine is having extremely high ram usage, and the high ram consuming process ended up being one of the uwsgi worker process which belongs to a Python program and it’s consuming about 16G of memory. Not sure why uwsgi master process didn’t kill the worker based on the evil-reload-on-rss setting, we ran gdb on the worker process. After some quick digging, we realized that worker process is actually stuck in Python interrupter, which mostly likely means it’s because of some of Python code goes bad, and it’s running into a infinite loop and keeps allocating memory. In order to test it, we wrote a simple Python wsgi program which does exactly that. And we get the same behavior, the application keeps allocating memory until it’s killed by Linux's OOM killer. Also we found that even if we enable threads (i.e. set “threads = 8”), we still get that behavior most of the time, only few other times, it’s killed by uwsgi’s evil-reload-on-rss. After some digging into the uwsgi, we found out that the master process kills the worker process based on the ram usage reported by the worker process (in uwsgi_master_check_workers_deadline() of core/master_checks.c). And the worker process only updates its ram usage after a request ends (in uwsgi_close_request() of core/utils.c), but if the request never ends, it means the ram usage is never updated, which means it will never be killed by the master process. So after enabling threads, in theory, other threads of the worker process should run uwsgi_close_request() and update the ram, but it’s not the case, I am not sure why it’s happening, and it still needs more digging. I am wondering shouldn’t it be the case that the master process checks worker’s ram usage when doing evil reload? Is there any reason why it’s not doing that? Thanks, Kaiwen Xu _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
