Thanks, Interesting hypothesis but: - we use dedicated connection with persistent connection in Django - the problem is observed in performance testing under constant heavy load
On Monday, 28 September 2015 17:08:28 UTC+1, Bill, KE1G wrote: > > One possibility: > > If cx_oracle is using connection pooling and it is not round robin (or > you just have large inactivity periods) > AND > There is a firewall between cx_oracle and oracle that times out unused > connections. > > Then the firewall could be locking traffic for connections that > cx_oracle/oracle still consider to be open. > > Forcing round robin connection pooling plus keep alive traffic may then > help. (We think that we've seen this..) > > On Mon, Sep 28, 2015 at 9:36 AM, 'Lukasz Szajkowski' via modwsgi < > [email protected] <javascript:>> wrote: > >> Hi , >> >> I have a problem with mod_wsgi (3.4-14) / Apache 2.4.12 / cx_oracle 5.2 / >> Oracle 12 / Django 1.8.2 hanging under bigger load. >> The configuration was used for the last several month in dev / testing. >> >> Now when deployed to production environment Apache hangs after couple of >> minutes under bigger load (it works fine with small or moderate load). >> >> I have a cluster of 3 Django/Apache/mod_wsgi servers and when this >> problems happens each server stops responding (one by one) within 5 - 15 >> minutes. >> >> >> Here is my configuration >> >> I am using Python 3.3 and Apache 2.4.12 and mod_wsgi-3.4-14 from Red Hat >> (6.7) Software Collection >> >> Apache Virtual Host >> >> <VirtualHost *:8080> >> ... >> >> WSGIDaemonProcess app.prod processes=2 threads=25 >> display-name=%{GROUP} user=MY_USER python-path=MY_PATH >> WSGIProcessGroup app.prod >> WSGIScriptAlias / /opt/hosts/app/app/wsgi.py >> </VirtualHost> >> >> >> Apache version >> >> Server version: Apache/2.4.12 (Red Hat) >> Server built: Aug 11 2015 08:12:59 >> Server's Module Magic Number: 20120211:41 >> Server loaded: APR 1.5.1, APR-UTIL 1.5.4 >> Compiled using: APR 1.5.1, APR-UTIL 1.5.4 >> Architecture: 64-bit >> Server MPM: worker >> threaded: yes (fixed thread count) >> forked: yes (variable process count) >> Server compiled with.... >> -D APR_HAS_SENDFILE >> -D APR_HAS_MMAP >> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) >> -D APR_USE_SYSVSEM_SERIALIZE >> -D APR_USE_PTHREAD_SERIALIZE >> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT >> -D APR_HAS_OTHER_CHILD >> -D AP_HAVE_RELIABLE_PIPED_LOGS >> -D DYNAMIC_MODULE_LIMIT=256 >> -D HTTPD_ROOT="/opt/rh/httpd24/root/etc/httpd" >> -D SUEXEC_BIN="/opt/rh/httpd24/root/usr/sbin/suexec" >> -D DEFAULT_PIDLOG="/opt/rh/httpd24/root/var/run/httpd/httpd.pid" >> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" >> -D DEFAULT_ERRORLOG="logs/error_log" >> -D AP_TYPES_CONFIG_FILE="conf/mime.types" >> -D SERVER_CONFIG_FILE="conf/httpd.conf" >> >> python33-mod_wsgi-3.4-14.el6.x86_64 >> >> Django db configuration >> >> 'default': { >> 'ENGINE': 'django.db.backends.oracle', >> 'NAME': 'DB_NAME', >> 'USER': 'webuser', >> 'PASSWORD': 'webpassword', >> 'HOST': '', >> 'PORT': '', >> 'CONN_MAX_AGE': None, >> 'OPTIONS': { >> 'threaded': True, >> }, >> }, >> >> >> >> [root@taseapp-em1-02 ~]# pstack 17044 >> Thread 4 (Thread 0x7fb6ffb4b700 (LWP 17053)): >> #0 0x00007fb70cf90423 in select () from /lib64/libc.so.6 >> #1 0x00007fb70d488535 in apr_sleep () from >> /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 >> #2 0x00007fb7021d4340 in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 >> #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 >> Thread 3 (Thread 0x7fb6ff14a700 (LWP 17054)): >> #0 0x00007fb70cf90423 in select () from /lib64/libc.so.6 >> #1 0x00007fb70d488535 in apr_sleep () from >> /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 >> #2 0x00007fb7021d3ffb in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 >> #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 >> Thread 2 (Thread 0x7fb6fe749700 (LWP 17055)): >> #0 0x00007fb70d2519b8 in fcntl () from /lib64/libpthread.so.0 >> #1 0x00007fb70d47b3a2 in ?? () from >> /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 >> #2 0x00007fb7021de4d8 in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #3 0x00007fb70d24aa51 in start_thread () from /lib64/libpthread.so.0 >> #4 0x00007fb70cf979ad in clone () from /lib64/libc.so.6 >> Thread 1 (Thread 0x7fb70e6e57e0 (LWP 17044)): >> #0 0x00007fb70cf8e183 in poll () from /lib64/libc.so.6 >> #1 0x00007fb70d484480 in apr_poll () from >> /opt/rh/httpd24/root/usr/lib64/libapr-1-httpd24.so.0 >> #2 0x00007fb7021d8cf7 in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #3 0x00007fb7021d9b51 in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #4 0x00007fb7021df017 in ?? () from >> /opt/rh/httpd24/root/etc/httpd/modules/mod_python33-wsgi.so >> #5 0x00007fb70e73d4f9 in ap_run_post_config () >> #6 0x00007fb70e71c78f in main () >> >> >> If I reduce the number of mod_wsgi then the problem goes away but the >> performance is not acceptable >> >> >> WSGIDaemonProcess app.prod processes=4 threads=1 display-name=%{GROUP} >> user=MY_USER python-path=MY_PATH >> >> >> Could anybody suggest what could be the cause or how could I debug it >> please? >> >> Thanks, >> Lukasz >> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at http://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
