Looks like this *might* be related http://bugs.python.org/issue1404925 http://bugs.python.org/issue14308
Cal On Mon, Nov 12, 2012 at 7:59 PM, Cal Leeming [Simplicity Media Ltd] < [email protected]> wrote: > Further update on this, I disabled new relic and still getting the issue; > > > So I enabled thread dumping and caught this; > Mon Nov 12 19:50:45 2012 - spawned uWSGI master process (pid: 28456) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 1 (pid: 28457, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 2 (pid: 28461, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 3 (pid: 28462, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 4 (pid: 28469, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 5 (pid: 28470, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 6 (pid: 28477, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 7 (pid: 28481, cores: 4) > Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 8 (pid: 28485, cores: 4) > Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] > during GET /login (177.36.243.7) > Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] > during GET /login (190.29.22.247) > Mon Nov 12 19:50:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] > during GET /favicon.ico (134.93.74.121) > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > Exception AttributeError: AttributeError("'_DummyThread' object has no > attribute '_Thread__block'",) in <module 'threading' from > '/usr/lib/python2.7/threading.pyc'> ignored > > > <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored > File: > "/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py", > line 51, in encrypt > env={envvar: self._password}) > File: > "/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py", > line 26, in check_output > process = Popen(stdout=PIPE, stdin=stdin, *((cmd,) + popenargs), **kwargs) > File: "/usr/lib/python2.7/subprocess.py", line 679, in __init__ > errread, errwrite) > File: "/usr/lib/python2.7/subprocess.py", line 1234, in _execute_child > data = _eintr_retry_call(os.read, errpipe_read, 1048576) > File: "/usr/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call > return func(*args) > ### END THREAD DUMP > ### BEGIN THREAD DUMP > > So I removed all calls to SimpleAES/subprocess.Popen, and the problem went > away :( > > Sooooo, it would seem that using uwsgi with subprocess.Popen is a no go... > > Any thoughts? > > Cal > > > On Mon, Nov 12, 2012 at 7:42 PM, Cal Leeming [Simplicity Media Ltd] < > [email protected]> wrote: > >> Got a bit of a strange problem when using virtualenv / New Relic / uWSGI >> / threading.. >> >> If I disable threading, the problem seems to go away - but when I use >> threading, all of hell breaks loose. >> >> Debug is below - any ideas? >> >> Cal >> >> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \ >> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \ >> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \ >> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \ >> --module django_wsgi \ >> --listen 1024 \ >> --socket-timeout 2 \ >> --idle 3600 \ >> --reaper \ >> --disable-logging \ >> --chmod-socket 666 \ >> --harakiri 3600 \ >> --harakiri-verbose \ >> --max-requests 30000 \ >> --logdate true \ >> --limit-as 1024 \ >> --post-buffering 8192 \ >> --single-interpreter \ >> --enable-threads \ >> --threads 4 \ >> --workers 8 \ >> --master \ >> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/ >> >> RESULTS IN: >> >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector-7.newrelic.com >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21356, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252185598, in 0.15 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector-7.newrelic.com >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21378, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252185602, in 0.12 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector-7.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector-7.newrelic.com >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21348, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252185605, in 0.13 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21362, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252185606, in 0.20 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector.newrelic.com >> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >> HTTP connection (1): collector-7.newrelic.com >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21344, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252185607, in 0.14 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> >> >> >> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \ >> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \ >> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \ >> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \ >> --module django_wsgi \ >> --listen 1024 \ >> --socket-timeout 2 \ >> --idle 3600 \ >> --reaper \ >> --disable-logging \ >> --chmod-socket 666 \ >> --harakiri 3600 \ >> --harakiri-verbose \ >> --max-requests 30000 \ >> --logdate true \ >> --limit-as 1024 \ >> --post-buffering 8192 \ >> --single-interpreter \ >> --enable-threads \ >> --threads 1 \ >> --workers 8 \ >> --master \ >> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/ >> >> RESULTS IN; >> INFO:newrelic.core.data_collector:Successfully registered New Relic >> Python agent where app_name='DDMA Production', pid=21524, redirect_host=' >> collector-7.newrelic.com' and agent_run_id=252186112, in 0.14 seconds. >> INFO:newrelic.core.data_collector:Reporting to: >> https://rpm.newrelic.com/accounts/69850/applications/886440 >> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (210.83.86.162) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x3b9b0a0> ignored >> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.96.64.234) >> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.248.67.146) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x3c0a820> ignored >> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (200.195.136.150) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x23eff00> ignored >> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (197.157.218.50) >> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (200.105.237.94) >> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.121.135.178) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x7f11740d1460> ignored >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (112.125.120.145) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x2401af0> ignored >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (195.200.199.98) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (195.3.254.159) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.111.17.161) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x7f11740f3e10> ignored >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.128.57.22) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x3d2bdc0> ignored >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /videos/7221 (65.30.62.10) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (222.92.117.219) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x3a91820> ignored >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (202.46.151.59) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (110.139.118.95) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (125.88.125.201) >> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (2.179.143.167) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x2401af0> ignored >> Mon Nov 12 19:35:55 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during POST /login (217.27.145.94) >> Mon Nov 12 19:35:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /videos/7221 (65.30.62.10) >> Mon Nov 12 19:36:01 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.0.57.98) >> Exception AssertionError: AssertionError() in <generator object __iter__ >> at 0x2859a00> ignored >> Mon Nov 12 19:36:04 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /videos/7221 (65.30.62.10) >> >> (venv)ddcmsprod@dande-xen2-web3 /home/ddcmsprod/webapps/ddma.prod $ >> python >> Python 2.7.3 (default, Oct 10 2012, 04:15:59) >> [GCC 4.6.3] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> >>> >> >> Sun Nov 11 17:51:26 2012 - *** Starting uWSGI 1.3 (64bit) on [Sun Nov 11 >> 17:51:26 2012] *** >> Sun Nov 11 17:51:26 2012 - compiled with version: 4.6.3 on 06 November >> 2012 11:23:15 >> Sun Nov 11 17:51:26 2012 - os: Linux-3.6.6-gentoo #1 SMP Wed Nov 7 >> 15:03:29 EST 2012 >> Sun Nov 11 17:51:26 2012 - nodename: dande-xen2-web3.isprime.com >> Sun Nov 11 17:51:26 2012 - machine: x86_64 >> Sun Nov 11 17:51:26 2012 - clock source: unix >> Sun Nov 11 17:51:26 2012 - detected number of CPU cores: 2 >> Sun Nov 11 17:51:26 2012 - current working directory: >> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp >> Sun Nov 11 17:51:26 2012 - writing pidfile to >> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid >> Sun Nov 11 17:51:26 2012 - detected binary path: >> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi >> Sun Nov 11 17:51:26 2012 - your processes number limit is 13443 >> Sun Nov 11 17:51:26 2012 - limiting address space of processes... >> Sun Nov 11 17:51:26 2012 - your process address space limit is 1073741824 >> bytes (1024 MB) >> Sun Nov 11 17:51:26 2012 - your memory page size is 4096 bytes >> Sun Nov 11 17:51:26 2012 - detected max file descriptor number: 1024 >> Sun Nov 11 17:51:26 2012 - lock engine: pthread robust mutexes >> Sun Nov 11 17:51:26 2012 - chmod() socket to 666 for lazy and brave users >> Sun Nov 11 17:51:26 2012 - uwsgi socket 0 bound to UNIX address >> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock fd 3 >> Sun Nov 11 17:51:26 2012 - Python version: 2.7.3 (default, Oct 10 2012, >> 04:15:59) [GCC 4.6.3] >> Sun Nov 11 17:51:26 2012 - Set PythonHome to >> /home/ddcmsprod/webapps/ddma.prod/venv/ >> Sun Nov 11 17:51:26 2012 - Python main interpreter initialized at >> 0x2406190 >> Sun Nov 11 17:51:26 2012 - python threads support enabled >> Sun Nov 11 17:51:26 2012 - your server socket listen backlog is limited >> to 1024 connections >> Sun Nov 11 17:51:26 2012 - mapped 673920 bytes (658 KB) for 16 cores >> Sun Nov 11 17:51:26 2012 - *** Operational MODE: preforking+threaded *** >> Sun Nov 11 17:51:26 2012 - added >> /home/ddcmsprod/webapps/ddma.prod/webapp/ to pythonpath. >> IOError creating stack dump trigger >> '/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp/sock/uwsgi.dump' >> Sun Nov 11 22:51:28 2012 - WSGI app 0 (mountpoint='') ready in 2 seconds >> on interpreter 0x2406190 pid: 13976 (default app) >> Sun Nov 11 22:51:28 2012 - spawned uWSGI master process (pid: 13976) >> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 1 (pid: 13979, cores: 4) >> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 2 (pid: 13983, cores: 4) >> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 3 (pid: 13987, cores: 4) >> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 4 (pid: 13991, cores: 4) >> Sun Nov 11 22:51:40 2012 - SIGINT/SIGQUIT received...killing workers... >> >> >> >
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
