Got a bit of a strange problem when using virtualenv / New Relic / uWSGI /
threading..
If I disable threading, the problem seems to go away - but when I use
threading, all of hell breaks loose.
Debug is below - any ideas?
Cal
/home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \
--pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \
--socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \
--pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \
--module django_wsgi \
--listen 1024 \
--socket-timeout 2 \
--idle 3600 \
--reaper \
--disable-logging \
--chmod-socket 666 \
--harakiri 3600 \
--harakiri-verbose \
--max-requests 30000 \
--logdate true \
--limit-as 1024 \
--post-buffering 8192 \
--single-interpreter \
--enable-threads \
--threads 4 \
--workers 8 \
--master \
--virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/
RESULTS IN:
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector-7.newrelic.com
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21356, redirect_host='
collector-7.newrelic.com' and agent_run_id=252185598, in 0.15 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector-7.newrelic.com
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21378, redirect_host='
collector-7.newrelic.com' and agent_run_id=252185602, in 0.12 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector-7.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector-7.newrelic.com
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21348, redirect_host='
collector-7.newrelic.com' and agent_run_id=252185605, in 0.13 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21362, redirect_host='
collector-7.newrelic.com' and agent_run_id=252185606, in 0.20 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector.newrelic.com
INFO:newrelic.lib.requests.packages.urllib3.connectionpool:Starting new
HTTP connection (1): collector-7.newrelic.com
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21344, redirect_host='
collector-7.newrelic.com' and agent_run_id=252185607, in 0.14 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
Exception AttributeError: AttributeError("'_DummyThread' object has no
attribute '_Thread__block'",) in <module 'threading' from
'/usr/lib/python2.7/threading.pyc'> ignored
/home/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi \
--pythonpath /home/ddcmsprod/webapps/ddma.prod/webapp \
--socket /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock \
--pidfile /var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid \
--module django_wsgi \
--listen 1024 \
--socket-timeout 2 \
--idle 3600 \
--reaper \
--disable-logging \
--chmod-socket 666 \
--harakiri 3600 \
--harakiri-verbose \
--max-requests 30000 \
--logdate true \
--limit-as 1024 \
--post-buffering 8192 \
--single-interpreter \
--enable-threads \
--threads 1 \
--workers 8 \
--master \
--virtualenv=/home/ddcmsprod/webapps/ddma.prod/venv/
RESULTS IN;
INFO:newrelic.core.data_collector:Successfully registered New Relic Python
agent where app_name='DDMA Production', pid=21524, redirect_host='
collector-7.newrelic.com' and agent_run_id=252186112, in 0.14 seconds.
INFO:newrelic.core.data_collector:Reporting to:
https://rpm.newrelic.com/accounts/69850/applications/886440
Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (210.83.86.162)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x3b9b0a0> ignored
Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.96.64.234)
Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.248.67.146)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x3c0a820> ignored
Mon Nov 12 19:35:51 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (200.195.136.150)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x23eff00> ignored
Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (197.157.218.50)
Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (200.105.237.94)
Mon Nov 12 19:35:52 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.121.135.178)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x7f11740d1460> ignored
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (112.125.120.145)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x2401af0> ignored
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (195.200.199.98)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (195.3.254.159)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.111.17.161)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x7f11740f3e10> ignored
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.128.57.22)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x3d2bdc0> ignored
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /videos/7221 (65.30.62.10)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (222.92.117.219)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x3a91820> ignored
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (202.46.151.59)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (110.139.118.95)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (125.88.125.201)
Mon Nov 12 19:35:53 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (2.179.143.167)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x2401af0> ignored
Mon Nov 12 19:35:55 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during POST /login (217.27.145.94)
Mon Nov 12 19:35:56 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /videos/7221 (65.30.62.10)
Mon Nov 12 19:36:01 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /login (190.0.57.98)
Exception AssertionError: AssertionError() in <generator object __iter__ at
0x2859a00> ignored
Mon Nov 12 19:36:04 2012 - writev(): Broken pipe [proto/uwsgi.c line 124]
during GET /videos/7221 (65.30.62.10)
(venv)ddcmsprod@dande-xen2-web3 /home/ddcmsprod/webapps/ddma.prod $ python
Python 2.7.3 (default, Oct 10 2012, 04:15:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
Sun Nov 11 17:51:26 2012 - *** Starting uWSGI 1.3 (64bit) on [Sun Nov 11
17:51:26 2012] ***
Sun Nov 11 17:51:26 2012 - compiled with version: 4.6.3 on 06 November 2012
11:23:15
Sun Nov 11 17:51:26 2012 - os: Linux-3.6.6-gentoo #1 SMP Wed Nov 7 15:03:29
EST 2012
Sun Nov 11 17:51:26 2012 - nodename: dande-xen2-web3.isprime.com
Sun Nov 11 17:51:26 2012 - machine: x86_64
Sun Nov 11 17:51:26 2012 - clock source: unix
Sun Nov 11 17:51:26 2012 - detected number of CPU cores: 2
Sun Nov 11 17:51:26 2012 - current working directory:
/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp
Sun Nov 11 17:51:26 2012 - writing pidfile to
/var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.pid
Sun Nov 11 17:51:26 2012 - detected binary path:
/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/venv/bin/uwsgi
Sun Nov 11 17:51:26 2012 - your processes number limit is 13443
Sun Nov 11 17:51:26 2012 - limiting address space of processes...
Sun Nov 11 17:51:26 2012 - your process address space limit is 1073741824
bytes (1024 MB)
Sun Nov 11 17:51:26 2012 - your memory page size is 4096 bytes
Sun Nov 11 17:51:26 2012 - detected max file descriptor number: 1024
Sun Nov 11 17:51:26 2012 - lock engine: pthread robust mutexes
Sun Nov 11 17:51:26 2012 - chmod() socket to 666 for lazy and brave users
Sun Nov 11 17:51:26 2012 - uwsgi socket 0 bound to UNIX address
/var/run/uwsgi/ddcmsprod/ddma.prod.uwsgi.sock fd 3
Sun Nov 11 17:51:26 2012 - Python version: 2.7.3 (default, Oct 10 2012,
04:15:59) [GCC 4.6.3]
Sun Nov 11 17:51:26 2012 - Set PythonHome to
/home/ddcmsprod/webapps/ddma.prod/venv/
Sun Nov 11 17:51:26 2012 - Python main interpreter initialized at 0x2406190
Sun Nov 11 17:51:26 2012 - python threads support enabled
Sun Nov 11 17:51:26 2012 - your server socket listen backlog is limited to
1024 connections
Sun Nov 11 17:51:26 2012 - mapped 673920 bytes (658 KB) for 16 cores
Sun Nov 11 17:51:26 2012 - *** Operational MODE: preforking+threaded ***
Sun Nov 11 17:51:26 2012 - added /home/ddcmsprod/webapps/ddma.prod/webapp/
to pythonpath.
IOError creating stack dump trigger
'/nfs/homedirs/cluster1/ddcmsprod/webapps/ddma.prod/deployment/current/ddcms/webapp/sock/uwsgi.dump'
Sun Nov 11 22:51:28 2012 - WSGI app 0 (mountpoint='') ready in 2 seconds on
interpreter 0x2406190 pid: 13976 (default app)
Sun Nov 11 22:51:28 2012 - spawned uWSGI master process (pid: 13976)
Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 1 (pid: 13979, cores: 4)
Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 2 (pid: 13983, cores: 4)
Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 3 (pid: 13987, cores: 4)
Sun Nov 11 22:51:28 2012 - spawned uWSGI worker 4 (pid: 13991, cores: 4)
Sun Nov 11 22:51:40 2012 - SIGINT/SIGQUIT received...killing workers...
_______________________________________________
uWSGI mailing list
[email protected]
http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
