Ok, I will try that and post back. Difficulty has been due to the fact that it happens only in one instance and not in the other. That is on the box which is deployed on linux in AWS US, never see any deadlocks even though the number of requests are higher.
However in the box deployed on linux in AWS EU, no matter the number of process and threads, the deadlock always happens. Further, if the number of processes and threads are increased the deadlock happens frequently. As of now I have restricted to 2 process and 15 threads for WSGI. So the system chugs along fine for 3-8 hours and then goes into a tail spin of deadlock/restart for about 3 or 4 times and then it continues again for 3-8 hours. In the AWS US box, it never deadlocks and goes on for more than a couple week. Is there a way query the .sock file? Something similar to cat /proc/locks? Santhosh On Monday, 29 August 2016 18:21:55 UTC-7, Graham Dumpleton wrote: > > > On 30 Aug 2016, at 10:17 AM, kusimari share <[email protected] > <javascript:>> wrote: > > Hi, > > I have two instances of a apache, mod_wsgi and flask based http rest > service. The only difference between the two services are the files being > loaded (python pickle files on flask start) and AWS regions, one is in US > while the other is in EU. The flask service itself is a wrapper over scikit > learn predictions which are the reason for the python pickle files > mentioned earlier. > > Confusingly the instance in US works fine. No issues. However the EU > instance keeps getting into a deadlock situation because of which mod_wsgi > restarts. Apache is still up. I have had no luck using the different > mod_wsgi debug mechanisms, apparently there is no deadlock in the flask, > scikit learn side of the equation. > > Today, found that on deadlock mod_wsgi emitted: > > 21720 [Mon Aug 29 21:31:26.979825 2016] [wsgi:crit] [pid 4317:tid > 140495687612160] (35)Resource deadlock avoided: │EOE > > mod_wsgi (pid=4317): Couldn't acquire accept mutex > '/path-to-apache/var/state.14843.0.1.sock'. > Shutting down daemon process > > Apache conf contains > Mutex file:/path-to-pache/var/state mpm-accept > > My suspicion is that one of the instance is using fcntl while the other > uses flock, and probably the one using fcntl has deadlocks based on apache > documentation. But I have no way to verify it. > > Is there a way to find which of fcntl or flock is being used by Apache? > > httpd -V shows > -D APR_HAS_SENDFILE > -D APR_HAS_MMAP > -D APR_USE_SYSVSEM_SERIALIZE > -D APR_USE_PTHREAD_SERIALIZE > -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT > -D APR_HAS_OTHER_CHILD > -D AP_HAVE_RELIABLE_PIPED_LOGS > -D DYNAMIC_MODULE_LIMIT=256 > > > If you start up the Apache web server with LogLevel set to ‘debug’, you > should see a line like: > > [mpm_prefork:debug] [pid 82122] prefork.c(1027): AH00165: Accept > mutex: none (default: flock) > > I think this is what mod_wsgi will also use. There was an issue at some > point where Apache stopped allowing me to see the choice it made and so I > had to start calculating it myself. > > That in itself shouldn’t cause a problem though as Apache’s own cross > process mutex locks are separate to those mod_wsgi creates, so there cannot > be a clash if types end up being different. > > I would suggest the issue may be an ownership/permissions issue on the > directory rather than a mismatch of lock types. The locks on this are all > managed in mod_wsgi so can’t see how the type of locking used could differ > at different times. > > So what are the ownership/permissions on the directory: > > /path-to-apache/var/ > > Can you try setting the directive: > > WSGISocketPrefix /tmp > > as a test to see if that helps. The sockets and lock files should then be > placed in /tmp instead, which if this is a single user machine should be > fine. > > You can also override what the mutex type used by mod_wsgi for its own > locks is by using: > > WSGIAcceptMutex flock > > Graham > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
