One possibility: If cx_oracle is using connection pooling and it is not round robin (or you just have large inactivity periods) AND There is a firewall between cx_oracle and oracle that times out unused connections.
Then the firewall could be locking traffic for connections that cx_oracle/oracle still consider to be open. Forcing round robin connection pooling plus keep alive traffic may then help. (We think that we've seen this..) On Mon, Sep 28, 2015 at 9:36 AM, 'Lukasz Szajkowski' via modwsgi < [email protected]> wrote: > Hi , > > I have a problem with mod_wsgi (3.4-14) / Apache 2.4.12 / cx_oracle 5.2 / > Oracle 12 / Django 1.8.2 hanging under bigger load. > The configuration was used for the last several month in dev / testing. > > Now when deployed to production environment Apache hangs after couple of > minutes under bigger load (it works fine with small or moderate load). > > I have a cluster of 3 Django/Apache/mod_wsgi servers and when this > problems happens each server stops responding (one by one) within 5 - 15 > minutes. > > > Here is my configuration > > I am using Python 3.3 and Apache 2.4.12 and mod_wsgi-3.4-14 from Red Hat > (6.7) Software Collection > > Apache Virtual Host > > <VirtualHost *:8080> > ... > > WSGIDaemonProcess app.prod processes=2 threads=25 > display-name=%{GROUP} user=MY_USER python-path=MY_PATH > WSGIProcessGroup app.prod > WSGIScriptAlias / /opt/hosts/app/app/wsgi.py > </VirtualHost> > > > Apache version > > Server version: Apache/2.4.12 (Red Hat) > Server built: Aug 11 2015 08:12:59 > Server's Module Magic Number: 20120211:41 > Server loaded: APR 1.5.1, APR-UTIL 1.5.4 > Compiled using: APR 1.5.1, APR-UTIL 1.5.4 > Architecture: 64-bit > Server MPM: worker > threaded: yes (fixed thread count) > forked: yes (variable process count) > Server compiled with.... > -D APR_HAS_SENDFILE > -D APR_HAS_MMAP > -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) > -D APR_USE_SYSVSEM_SERIALIZE > -D APR_USE_PTHREAD_SERIALIZE > -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT > -D APR_HAS_OTHER_CHILD > -D AP_HAVE_RELIABLE_PIPED_LOGS > -D DYNAMIC_MODULE_LIMIT=256 > -D HTTPD_ROOT="/opt/rh/httpd24/root/etc/httpd" > -D SUEXEC_BIN="/opt/rh/httpd24/root/usr/sbin/suexec" > -D DEFAULT_PIDLOG="/opt/rh/httpd24/root/var/run/httpd/httpd.pid" > -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" > -D DEFAULT_ERRORLOG="logs/error_log" > -D AP_TYPES_CONFIG_FILE="conf/mime.types" > -D SERVER_CONFIG_FILE="conf/httpd.conf" > > python33-mod_wsgi-3.4-14.el6.x86_64 > > Django db configuration > > 'default': { > 'ENGINE': 'django.db.backends.oracle', > 'NAME': 'DB_NAME', > 'USER': 'webuser', > 'PASSWORD': 'webpassword', > 'HOST': '', > 'PORT': '', > 'CONN_MAX_AGE': None, > 'OPTIONS': { > 'threaded': True, > }, > }, > > > > [root@taseapp-em1-02 ~]# pstack 17044 > Thread 4 (Thread 0x7fb6ffb4b700 (LWP 17053)): > #0 0x00007fb70cf90423 in select () from /lib64/libc.so.6 > #1 0x00007fb70d488535 in apr_sleep () from > /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 > #2 0x00007fb7021d4340 in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 > Thread 3 (Thread 0x7fb6ff14a700 (LWP 17054)): > #0 0x00007fb70cf90423 in select () from /lib64/libc.so.6 > #1 0x00007fb70d488535 in apr_sleep () from > /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 > #2 0x00007fb7021d3ffb in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 > Thread 2 (Thread 0x7fb6fe749700 (LWP 17055)): > #0 0x00007fb70d2519b8 in fcntl () from /lib64/libpthread.so.0 > #1 0x00007fb70d47b3a2 in ?? () from > /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 > #2 0x00007fb7021de4d8 in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 > #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 > Thread 1 (Thread 0x7fb70e6e57e0 (LWP 17044)): > #0 0x00007fb70cf8e183 in poll () from /lib64/libc.so.6 > #1 0x00007fb70d484480 in apr_poll () from > /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 > #2 0x00007fb7021d8cf7 in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #3 0x00007fb7021d9b51 in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #4 0x00007fb7021df017 in ?? () from > /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so > #5 0x00007fb70e73d4f9 in ap_run_post_config () > #6 0x00007fb70e71c78f in main () > > > If I reduce the number of mod_wsgi then the problem goes away but the > performance is not acceptable > > > WSGIDaemonProcess app.prod processes=4 threads=1 display-name=%{GROUP} > user=MY_USER python-path=MY_PATH > > > Could anybody suggest what could be the cause or how could I debug it > please? > > Thanks, > Lukasz > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
