Further update on this, I disabled new relic and still getting the issue;
So I enabled thread dumping and caught this;
Mon Nov 12 19:50:45 2012 - spawned uWSGI master process (pid: 28456)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 1 (pid: 28457, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 2 (pid: 28461, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 3 (pid: 28462, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 4 (pid: 28469, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 5 (pid: 28470, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 6 (pid: 28477, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 7 (pid: 28481, cores: 4)
Mon Nov 12 19:50:45 2012 - spawned uWSGI worker 8 (pid: 28485, cores: 4)
Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (177.36.243.7)
Mon Nov 12 19:50:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.29.22.247)
Mon Nov 12 19:50:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /favicon.ico (134.93.74.121)
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
<module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
File:
"/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py",
line 51, in encrypt
env={envvar: self._password})
File:
"/home/ddcmsprod/webapps/ddma.prod/venv/lib/python2.7/site-packages/SimpleAES/__init__.py",
line 26, in check_output
process = Popen(stdout=PIPE, stdin=stdin, *((cmd,) + popenargs), **kwargs)
File: "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File: "/usr/lib/python2.7/subprocess.py", line 1234, in _execute_child
data = _eintr_retry_call(os.read, errpipe_read, 1048576)
File: "/usr/lib/python2.7/subprocess.py", line 478, in _eintr_retry_call
return func(*args)
### END THREAD DUMP
### BEGIN THREAD DUMP
So I removed all calls to SimpleAES/subprocess.Popen, and the problem went
away :(
Sooooo, it would seem that using uwsgi with subprocess.Popen is a no go...
Any thoughts?
Cal
On Mon, Nov 12, 2012 at 7:42 PM, Cal Leeming [Simplicity Media Ltd] <
[email protected]> wrote:
> Got a bit of a strange problem when using virtualenv / New Relic / uWSGI /
> threading..
>
> If I disable threading, the problem seems to go away - but when I use
> threading, all of hell breaks loose.
>
> Debug is below - any ideas?
>
> Cal
>
> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \
> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \
> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \
> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \
> --module django_wsgi \
> --listen 1024 \
> --socket-timeout 2 \
> --idle 3600 \
> --reaper \
> --disable-logging \
> --chmod-socket 666 \
> --harakiri 3600 \
> --harakiri-verbose \
> --max-requests 30000 \
> --logdate true \
> --limit-as 1024 \
> --post-buffering 8192 \
> --single-interpreter \
> --enable-threads \
> --threads 4 \
> --workers 8 \
> --master \
> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/
>
> RESULTS IN:
>
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector-7.newrelic.com
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21356, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252185598, in 0.15 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector-7.newrelic.com
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21378, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252185602, in 0.12 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector-7.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector-7.newrelic.com
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21348, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252185605, in 0.13 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21362, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252185606, in 0.20 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector.newrelic.com
> INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
> HTTP connection (1): collector-7.newrelic.com
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21344, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252185607, in 0.14 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
> Exception AttributeError: AttributeError("'_DummyThread' object has no
> attribute '_Thread__block'",) in <module 'threading' from
> '/usr/lib/python2.7/threading.pyc'> ignored
>
>
>
> /home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \
> --pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \
> --socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \
> --pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \
> --module django_wsgi \
> --listen 1024 \
> --socket-timeout 2 \
> --idle 3600 \
> --reaper \
> --disable-logging \
> --chmod-socket 666 \
> --harakiri 3600 \
> --harakiri-verbose \
> --max-requests 30000 \
> --logdate true \
> --limit-as 1024 \
> --post-buffering 8192 \
> --single-interpreter \
> --enable-threads \
> --threads 1 \
> --workers 8 \
> --master \
> --virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/
>
> RESULTS IN;
> INFO:newrelic.core.data_collector:Successfully registered New Relic Python
> agent where app_name='DDMA Production', pid=21524, redirect_host='
> collector-7.newrelic.com' and agent_run_id=252186112, in 0.14 seconds.
> INFO:newrelic.core.data_collector:Reporting to:
> https://rpm.newrelic.com/accounts/69850/applications/886440
> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (210.83.86.162)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x3b9b0a0> ignored
> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.96.64.234)
> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.248.67.146)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x3c0a820> ignored
> Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (200.195.136.150)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x23eff00> ignored
> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (197.157.218.50)
> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (200.105.237.94)
> Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.121.135.178)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x7f11740d1460> ignored
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (112.125.120.145)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x2401af0> ignored
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (195.200.199.98)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (195.3.254.159)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.111.17.161)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x7f11740f3e10> ignored
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.128.57.22)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x3d2bdc0> ignored
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /videos/7221 (65.30.62.10)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (222.92.117.219)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x3a91820> ignored
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (202.46.151.59)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (110.139.118.95)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (125.88.125.201)
> Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (2.179.143.167)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x2401af0> ignored
> Mon Nov 12 19:35:55 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during POST /login (217.27.145.94)
> Mon Nov 12 19:35:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /videos/7221 (65.30.62.10)
> Mon Nov 12 19:36:01 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /login (190.0.57.98)
> Exception AssertionError: AssertionError() in <generator object __iter__
> at 0x2859a00> ignored
> Mon Nov 12 19:36:04 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
> during GET /videos/7221 (65.30.62.10)
>
> (venv)ddcmsprod@dande-xen2-web3 /home/ddcmsprod/webapps/ddma.prod $ python
> Python 2.7.3 (default, Oct 10 2012, 04:15:59)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>>
>
> Sun Nov 11 17:51:26 2012 - *** Starting uWSGI 1.3 (64bit) on [Sun Nov 11
> 17:51:26 2012] ***
> Sun Nov 11 17:51:26 2012 - compiled with version: 4.6.3 on 06 November
> 2012 11:23:15
> Sun Nov 11 17:51:26 2012 - os: Linux-3.6.6-gentoo #1 SMP Wed Nov 7
> 15:03:29 EST 2012
> Sun Nov 11 17:51:26 2012 - nodename: dande-xen2-web3.isprime.com
> Sun Nov 11 17:51:26 2012 - machine: x86_64
> Sun Nov 11 17:51:26 2012 - clock source: unix
> Sun Nov 11 17:51:26 2012 - detected number of CPU cores: 2
> Sun Nov 11 17:51:26 2012 - current working directory:
> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp
> Sun Nov 11 17:51:26 2012 - writing pidfile to
> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid
> Sun Nov 11 17:51:26 2012 - detected binary path:
> /nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi
> Sun Nov 11 17:51:26 2012 - your processes number limit is 13443
> Sun Nov 11 17:51:26 2012 - limiting address space of processes...
> Sun Nov 11 17:51:26 2012 - your process address space limit is 1073741824
> bytes (1024 MB)
> Sun Nov 11 17:51:26 2012 - your memory page size is 4096 bytes
> Sun Nov 11 17:51:26 2012 - detected max file descriptor number: 1024
> Sun Nov 11 17:51:26 2012 - lock engine: pthread robust mutexes
> Sun Nov 11 17:51:26 2012 - chmod() socket to 666 for lazy and brave users
> Sun Nov 11 17:51:26 2012 - uwsgi socket 0 bound to UNIX address
> /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock fd 3
> Sun Nov 11 17:51:26 2012 - Python version: 2.7.3 (default, Oct 10 2012,
> 04:15:59) [GCC 4.6.3]
> Sun Nov 11 17:51:26 2012 - Set PythonHome to
> /home/ddcmsprod/webapps/ddma.prod/venv/
> Sun Nov 11 17:51:26 2012 - Python main interpreter initialized at 0x2406190
> Sun Nov 11 17:51:26 2012 - python threads support enabled
> Sun Nov 11 17:51:26 2012 - your server socket listen backlog is limited to
> 1024 connections
> Sun Nov 11 17:51:26 2012 - mapped 673920 bytes (658 KB) for 16 cores
> Sun Nov 11 17:51:26 2012 - *** Operational MODE: preforking+threaded ***
> Sun Nov 11 17:51:26 2012 - added /home/ddcmsprod/webapps/ddma.prod/webapp/
> to pythonpath.
> IOError creating stack dump trigger
> '/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp/sock/uwsgi.dump'
> Sun Nov 11 22:51:28 2012 - WSGI app 0 (mountpoint='') ready in 2 seconds
> on interpreter 0x2406190 pid: 13976 (default app)
> Sun Nov 11 22:51:28 2012 - spawned uWSGI master process (pid: 13976)
> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 1 (pid: 13979, cores: 4)
> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 2 (pid: 13983, cores: 4)
> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 3 (pid: 13987, cores: 4)
> Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 4 (pid: 13991, cores: 4)
> Sun Nov 11 22:51:40 2012 - SIGINT/SIGQUIT received...killing workers...
>
>
>
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
