Everything worked as expected once I changed the default configuration
of WSGIDaemonProcess.
Now I can serve up to 3000 clients.
Thanks!
El miércoles, 19 de febrero de 2014 21:17:21 UTC-6, Graham Dumpleton
escribió:
>
> One immediate concern is that your daemon process group will only be able
> to handle 15 concurrent requests (the default), yet you have configured the
> main Apache worker processes, which funnel requests into the daemon process
> group to be able to handle up to 4000 concurrent requests.
>
> That is one rather big bottleneck.
>
> I would very much suggest trying different settings for processes/threads
> for WSGIDaemonProcess directive. Right now it is defaulting to a single
> process with 15 threads.
>
> A second problem is that you are duplicating your application with two
> separate sets of instances for HTTP and HTTPS traffic. This can be wasteful
> on memory, although not entirely sure whether that is intentional or not
> since the WSGIScriptAlias for 443 and 80 actually refer to different WSGI
> script files.
>
> If you have 80/443 pair, you would normally do:
>
> <VirtualHost *:443>
> ServerName xxx
> WSGIDaemonProcess application
> WSGIProcessGroup application
> WSGIApplicationGroup %{GLOBAL}
> WSGIScriptAlias / /some/path/wsgi.py
> </VirtualHost>
>
> <VirtualHost *:80>
> ServerName xxx
> WSGIProcessGroup application
> WSGIApplicationGroup %{GLOBAL}
> WSGIScriptAlias / /some/path/wsgi.py
> </VirtualHost>
>
>
> That is, you would only define WSGIDaemonProcess in the first VirtiualHost
> for that ServerName seen by Apache when reading the configuration file and
> then referring to that using WSGIProcessGroup in the second by name rather
> than having a second WSGIDaemonProcess directive.
>
> You would only do this though if they are the same code base, especially
> if Django. As I said you have separate WSGI script files, so not sure if it
> is the same code base.
>
> If you haven't watched them already, would suggest you watch:
>
>
> - http://lanyrd.com/2012/pycon/spcdg/
> - http://lanyrd.com/2013/pycon/scdyzk/
>
> These mention the funnelling issue with daemon mode and give other tips on
> tuning.
>
> Graham
>
> On 20/02/2014, at 12:47 PM, Mario Adrian Lopez Aleman
> <[email protected]<javascript:>>
> wrote:
>
> Sorry, I have been working in that.
> I have two VirtualHosts.
>
> <VirtualHost *:443>
> SSLEngine On
> SSLCompression Off
> SSLProtocol +TLSv1 +SSLv3
> SSLHonorCipherOrder On
> SSLCipherSuite
> ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:HIGH:!MD5:!aNULL:!EDH
> SSLCertificateFile /var/lib/application/.ssl/ssl-cert-application.pem
> SSLCertificateKeyFile /var/lib/application/.ssl/ssl-cert-application-pk.pem
> SSLCertificateChainFile
> /var/lib/application/.ssl/ssl-cert-application-ca.pem
> ServerName ztubuntuxlarge
> ServerAlias application
> ServerAdmin webmaster@ztubuntuxlarge
>
> WSGIApplicationGroup %{GLOBAL}
> WSGIDaemonProcess application display-name=%{GROUP}
> WSGIProcessGroup application
>
> WSGIScriptAlias / /usr/share/application-server/www/index.wsgi
>
> <Directory /usr/share/application-server/www>
> Order allow,deny
> Allow from all
> </Directory>
>
> Alias /api/ /usr/share/doc/application-docs/html/
> <Directory /usr/share/doc/application-docs/html/>
> AuthUserFile /var/lib/application/.htpasswd-api
> AuthName "Authorization Required"
> AuthType Basic
> require valid-user
> Order allow,deny
> Allow from all
> </Directory>
>
> ErrorLog /var/log/application/webapp.log
> LogLevel info
> CustomLog /var/log/application/access.log combined
>
> </VirtualHost>
>
> <VirtualHost *:80>
> ServerName ztubuntuxlarge
> ServerAlias hkp
> ServerAdmin webmaster@ztubuntuxlarge
>
> WSGIApplicationGroup %{GLOBAL}
> WSGIDaemonProcess hkp display-name=%{GROUP}
> WSGIProcessGroup hkp
>
> WSGIScriptAlias / /usr/share/application-server/www/hkp.wsgi
>
> <Directory /usr/share/application-server/www>
> Order allow,deny
> Allow from all
> </Directory>
>
> LogLevel info
> ErrorLog /var/log/application/hkp.log
> </VirtualHost>
>
>
> Apache config:
> KeepAlive On
> KeepAliveTimeout 1200
> MaxKeepAliveRequests 0
>
> I modified my configuration in MPM
> <IfModule mpm_worker_module>
> ThreadLimit 8000
> ServerLimit 40
> StartServers 40
> MaxClients 4000
> MinSpareThreads 2000
> MaxSpareThreads 2500
> ThreadsPerChild 100
> MaxRequestsPerChild 0
> </IfModule>
>
>
> I'm not using a common testing tool.
> I have a Python script that creates 500 clients in 4 machines (2000
> clients).
>
> The server usually connects with a client, and this client can upload and
> download files.
> In this particular test I execute a registration (Involves a download and
> upload) and after that it downloads a shared file.
>
> I have been using up to 2000 concurrent clients right now, and everything
> seems to work fine (of 2000 just 11 failed).
> The actual response time is of 1 hour with all the 2000 clients.
> (From the registration to the get, it takes 1 hour to finish, so each
> client takes around 50 to 60 minutes)
>
> If I'm not clear in something please let me know.
>
>
> El miércoles, 19 de febrero de 2014 19:13:01 UTC-6, Graham Dumpleton
> escribió:
>>
>> Can you please provide the rest of your mod_wsgi configuration.
>>
>> You do not show what you have set for WSGIDaemonProcess, nor enough about
>> the structure of stuff in your VirtualHost to validate that things are set
>> up correct to delegate requests to that daemon process group.
>>
>> For the Apache configuration, provide what you are using for KeepAlive,
>> KeepAliveTimeout and KeepAliveRequests directives.
>>
>> Also provide information about how long your average and worst case
>> response times typically are for your web application. Plus details on what
>> load testing tool you are using and what options you are using with that in
>> regard to concurrency, use of keep alive etc.
>>
>> Right now your MPM configuration settings seem to be excessively high.
>>
>> Graham
>>
>> On 20/02/2014, at 5:29 AM, Mario Adrian Lopez Aleman <[email protected]>
>> wrote:
>>
>> Hi everyone.
>>
>> I'm using a big server to run my site and I'm having issues when trying
>> to do some performance tests in the server.
>> Here are the specs of the server:
>> Ubuntu 12.04.3 x64 3.2.0-55-virtual
>> 60GB RAM
>> Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (16 cores)
>>
>>
>> *Python 2.7.3*
>>
>>
>> *libapache2-mod-wsgi 3.3-4build1*
>> *apache2 2.2.22-1ubuntu1.4*
>> *apache2-mpm-worker 2.2.22-1ubuntu1.4*
>>
>> *Server version: Apache/2.2.22 (Ubuntu)*
>> Server built: Jul 12 2013 13:37:15
>> Server's Module Magic Number: 20051115:30
>> Server loaded: APR 1.4.6, APR-Util 1.3.12
>> Compiled using: APR 1.4.6, APR-Util 1.3.12
>> Architecture: 64-bit
>> Server MPM: Worker
>> threaded: yes (fixed thread count)
>> forked: yes (variable process count)
>> Server compiled with....
>> -D APACHE_MPM_DIR="server/mpm/worker"
>> -D APR_HAS_SENDFILE
>> -D APR_HAS_MMAP
>> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
>> -D APR_USE_SYSVSEM_SERIALIZE
>> -D APR_USE_PTHREAD_SERIALIZE
>> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
>> -D APR_HAS_OTHER_CHILD
>> -D AP_HAVE_RELIABLE_PIPED_LOGS
>> -D DYNAMIC_MODULE_LIMIT=128
>> -D HTTPD_ROOT="/etc/apache2"
>> -D SUEXEC_BIN="/usr/lib/apache2/suexec"
>> -D DEFAULT_PIDLOG="/var/run/apache2.pid"
>> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
>> -D DEFAULT_ERRORLOG="logs/error_log"
>> -D AP_TYPES_CONFIG_FILE="mime.types"
>> -D SERVER_CONFIG_FILE="apache2.conf
>>
>> <IfModule mpm_worker_module>
>> ServerLimit 900
>> StartServers 900
>> MinSpareThreads 100
>> MaxSpareThreads 500
>> ThreadLimit 6000
>> ThreadsPerChild 32
>> MaxClients 28800
>> MaxRequestsPerChild 0
>> </IfModule>
>>
>> www-data limits
>> time(seconds) unlimited
>> file(blocks) unlimited
>> data(kbytes) unlimited
>> stack(kbytes) 8192
>> coredump(blocks) 0
>> memory(kbytes) unlimited
>> locked memory(kbytes) 64
>> process unlimited
>> nofiles 65535
>> vmemory(kbytes) unlimited
>> locks unlimited
>>
>> The problem I'm seeing in my logs are these messages:
>> *[Wed Feb 19 03:31:04 2014] [error] [client 192.168.6.17] (2)No such file
>> or directory: mod_wsgi (pid=11996): Unable to connect to WSGI daemon
>> process 'ztrustee' on '/var/run/apache2/wsgi.1342.0.1.sock' after multiple
>> attempts.*
>> *[Wed Feb 19 03:31:04 2014] [error] [client 192.168.6.16] Premature end
>> of script headers: index.wsgi*
>> *[Wed Feb 19 18:01:02 2014] [error] [client 192.168.6.8] (4)Interrupted
>> system call: mod_wsgi (pid=29224): Unable to connect to WSGI daemon process
>> 'ztrustee' on '/var/run/apache2/wsgi.1331.0.1.sock' after multiple
>> attempts.*
>>
>> I already added this directive:
>> *WSGIApplicationGroup %{GLOBAL}*
>>
>> And also checked the permission/prefix problem in:
>> */var/run/apache2*
>>
>>
>> These problems appear when doing performance tests (using 1000 ~ 1500
>> clients), however the load in the server is not that much.
>> MAX RAM ~5 G
>> MAX CPU ~200% (MAX is 1600%)
>>
>> And the clients are seeing this message:
>> 500 Internal Server Error
>> Internal Server Error
>> The server encountered an internal error or
>> misconfiguration and was unable to complete
>> your request.
>> Please contact the server administrator,
>> webmaster@ztubuntuxlarge and inform them of the time the error
>> occurred,
>> and anything you might have done that may have
>> caused the error.
>>
>> More information about this error may be available
>> in the server error log.
>> Apache/2.2.22 (Ubuntu) Server at 192.168.6.13 Port 443
>>
>> Hopefully someone can point me in the right direction.
>> Regards!
>>
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To post to this group, send email to [email protected].
>> Visit this group at http://groups.google.com/group/modwsgi.
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected] <javascript:>
> .
> Visit this group at http://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/groups/opt_out.
