I just replaced this library with one that doesn't use subprocess/popen, and the problem went away.
Looks like uwsgi definitely isn't compatible with subprocess.Popen when using threads. Cal On Mon, Nov 12, 2012 at 8:00 PM, Cal Leeming [Simplicity Media Ltd] < [email protected]> wrote: > Looks like this *might* be related > > http://bugs.python.org/issue1404925 > http://bugs.python.org/issue14308 > > Cal > > > On Mon, Nov 12, 2012 at 7:59 PM, Cal Leeming [Simplicity Media Ltd] < > [email protected]> wrote: > >> Further update on this, I disabled new relic and still getting the issue; >> >> >> So I enabled thread dumping and caught this; >> Mon Nov 12 19:50:45 2012 - spawned uWSGI master process (pid: 28456) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 1 (pid: 28457, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 2 (pid: 28461, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 3 (pid: 28462, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 4 (pid: 28469, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 5 (pid: 28470, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 6 (pid: 28477, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 7 (pid: 28481, cores: 4) >> Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 8 (pid: 28485, cores: 4) >> Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (177.36.243.7) >> Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /login (190.29.22.247) >> Mon Nov 12 19:50:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124] >> during GET /favicon.ico (134.93.74.121) >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> Exception AttributeError: AttributeError("'_DummyThread' object has no >> attribute '_Thread__block'",) in <module 'threading' from >> '/usr/lib/python2.7/threading.pyc'> ignored >> >> >> <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored >> File: >> "/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py", >> line 51, in encrypt >> env={envvar: self._password}) >> File: >> "/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py", >> line 26, in check_output >> process = Popen(stdout=PIPE, stdin=stdin, *((cmd,) + popenargs), >> **kwargs) >> File: "/usr/lib/python2.7/subprocess.py", line 679, in __init__ >> errread, errwrite) >> File: "/usr/lib/python2.7/subprocess.py", line 1234, in _execute_child >> data = _eintr_retry_call(os.read, errpipe_read, 1048576) >> File: "/usr/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call >> return func(*args) >> ### END THREAD DUMP >> ### BEGIN THREAD DUMP >> >> So I removed all calls to SimpleAES/subprocess.Popen, and the problem >> went away :( >> >> Sooooo, it would seem that using uwsgi with subprocess.Popen is a no go... >> >> Any thoughts? >> >> Cal >> >> >> On Mon, Nov 12, 2012 at 7:42 PM, Cal Leeming [Simplicity Media Ltd] < >> [email protected]> wrote: >> >>> Got a bit of a strange problem when using virtualenv / New Relic / uWSGI >>> / threading.. >>> >>> If I disable threading, the problem seems to go away - but when I use >>> threading, all of hell breaks loose. >>> >>> Debug is below - any ideas? >>> >>> Cal >>> >>> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \ >>> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \ >>> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \ >>> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \ >>> --module django_wsgi \ >>> --listen 1024 \ >>> --socket-timeout 2 \ >>> --idle 3600 \ >>> --reaper \ >>> --disable-logging \ >>> --chmod-socket 666 \ >>> --harakiri 3600 \ >>> --harakiri-verbose \ >>> --max-requests 30000 \ >>> --logdate true \ >>> --limit-as 1024 \ >>> --post-buffering 8192 \ >>> --single-interpreter \ >>> --enable-threads \ >>> --threads 4 \ >>> --workers 8 \ >>> --master \ >>> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/ >>> >>> RESULTS IN: >>> >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector-7.newrelic.com >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21356, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252185598, in 0.15 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector-7.newrelic.com >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21378, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252185602, in 0.12 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector-7.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector-7.newrelic.com >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21348, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252185605, in 0.13 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21362, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252185606, in 0.20 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector.newrelic.com >>> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new >>> HTTP connection (1): collector-7.newrelic.com >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21344, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252185607, in 0.14 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> Exception AttributeError: AttributeError("'_DummyThread' object has no >>> attribute '_Thread__block'",) in <module 'threading' from >>> '/usr/lib/python2.7/threading.pyc'> ignored >>> >>> >>> >>> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \ >>> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \ >>> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \ >>> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \ >>> --module django_wsgi \ >>> --listen 1024 \ >>> --socket-timeout 2 \ >>> --idle 3600 \ >>> --reaper \ >>> --disable-logging \ >>> --chmod-socket 666 \ >>> --harakiri 3600 \ >>> --harakiri-verbose \ >>> --max-requests 30000 \ >>> --logdate true \ >>> --limit-as 1024 \ >>> --post-buffering 8192 \ >>> --single-interpreter \ >>> --enable-threads \ >>> --threads 1 \ >>> --workers 8 \ >>> --master \ >>> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/ >>> >>> RESULTS IN; >>> INFO:newrelic.core.data_collector:Successfully registered New Relic >>> Python agent where app_name='DDMA Production', pid=21524, redirect_host=' >>> collector-7.newrelic.com' and agent_run_id=252186112, in 0.14 seconds. >>> INFO:newrelic.core.data_collector:Reporting to: >>> https://rpm.newrelic.com/accounts/69850/applications/886440 >>> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (210.83.86.162) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x3b9b0a0> ignored >>> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.96.64.234) >>> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.248.67.146) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x3c0a820> ignored >>> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (200.195.136.150) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x23eff00> ignored >>> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (197.157.218.50) >>> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (200.105.237.94) >>> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.121.135.178) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x7f11740d1460> ignored >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (112.125.120.145) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x2401af0> ignored >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (195.200.199.98) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (195.3.254.159) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.111.17.161) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x7f11740f3e10> ignored >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.128.57.22) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x3d2bdc0> ignored >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /videos/7221 (65.30.62.10) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (222.92.117.219) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x3a91820> ignored >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (202.46.151.59) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (110.139.118.95) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (125.88.125.201) >>> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (2.179.143.167) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x2401af0> ignored >>> Mon Nov 12 19:35:55 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during POST /login (217.27.145.94) >>> Mon Nov 12 19:35:56 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /videos/7221 (65.30.62.10) >>> Mon Nov 12 19:36:01 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /login (190.0.57.98) >>> Exception AssertionError: AssertionError() in <generator object __iter__ >>> at 0x2859a00> ignored >>> Mon Nov 12 19:36:04 2012 - writev(): Broken pipe [proto/uwsgi.c line >>> 124] during GET /videos/7221 (65.30.62.10) >>> >>> (venv)ddcmsprod@dande-xen2-web3 /home/ddcmsprod/webapps/ddma.prod $ >>> python >>> Python 2.7.3 (default, Oct 10 2012, 04:15:59) >>> [GCC 4.6.3] on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> >>> >>> >>> Sun Nov 11 17:51:26 2012 - *** Starting uWSGI 1.3 (64bit) on [Sun Nov 11 >>> 17:51:26 2012] *** >>> Sun Nov 11 17:51:26 2012 - compiled with version: 4.6.3 on 06 November >>> 2012 11:23:15 >>> Sun Nov 11 17:51:26 2012 - os: Linux-3.6.6-gentoo #1 SMP Wed Nov 7 >>> 15:03:29 EST 2012 >>> Sun Nov 11 17:51:26 2012 - nodename: dande-xen2-web3.isprime.com >>> Sun Nov 11 17:51:26 2012 - machine: x86_64 >>> Sun Nov 11 17:51:26 2012 - clock source: unix >>> Sun Nov 11 17:51:26 2012 - detected number of CPU cores: 2 >>> Sun Nov 11 17:51:26 2012 - current working directory: >>> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp >>> Sun Nov 11 17:51:26 2012 - writing pidfile to >>> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid >>> Sun Nov 11 17:51:26 2012 - detected binary path: >>> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi >>> Sun Nov 11 17:51:26 2012 - your processes number limit is 13443 >>> Sun Nov 11 17:51:26 2012 - limiting address space of processes... >>> Sun Nov 11 17:51:26 2012 - your process address space limit is >>> 1073741824 bytes (1024 MB) >>> Sun Nov 11 17:51:26 2012 - your memory page size is 4096 bytes >>> Sun Nov 11 17:51:26 2012 - detected max file descriptor number: 1024 >>> Sun Nov 11 17:51:26 2012 - lock engine: pthread robust mutexes >>> Sun Nov 11 17:51:26 2012 - chmod() socket to 666 for lazy and brave users >>> Sun Nov 11 17:51:26 2012 - uwsgi socket 0 bound to UNIX address >>> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock fd 3 >>> Sun Nov 11 17:51:26 2012 - Python version: 2.7.3 (default, Oct 10 2012, >>> 04:15:59) [GCC 4.6.3] >>> Sun Nov 11 17:51:26 2012 - Set PythonHome to >>> /home/ddcmsprod/webapps/ddma.prod/venv/ >>> Sun Nov 11 17:51:26 2012 - Python main interpreter initialized at >>> 0x2406190 >>> Sun Nov 11 17:51:26 2012 - python threads support enabled >>> Sun Nov 11 17:51:26 2012 - your server socket listen backlog is limited >>> to 1024 connections >>> Sun Nov 11 17:51:26 2012 - mapped 673920 bytes (658 KB) for 16 cores >>> Sun Nov 11 17:51:26 2012 - *** Operational MODE: preforking+threaded *** >>> Sun Nov 11 17:51:26 2012 - added >>> /home/ddcmsprod/webapps/ddma.prod/webapp/ to pythonpath. >>> IOError creating stack dump trigger >>> '/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp/sock/uwsgi.dump' >>> Sun Nov 11 22:51:28 2012 - WSGI app 0 (mountpoint='') ready in 2 seconds >>> on interpreter 0x2406190 pid: 13976 (default app) >>> Sun Nov 11 22:51:28 2012 - spawned uWSGI master process (pid: 13976) >>> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 1 (pid: 13979, cores: 4) >>> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 2 (pid: 13983, cores: 4) >>> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 3 (pid: 13987, cores: 4) >>> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 4 (pid: 13991, cores: 4) >>> Sun Nov 11 22:51:40 2012 - SIGINT/SIGQUIT received...killing workers... >>> >>> >>> >> >
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
