I'll go back over and read email again when I have a chance and see if anything else can suggest, but right now suggest a couple of checks you can make which don't require changing any code.
First up, if you didn't feel comfortable about adding that code I linked before for dumping stack traces, at least do an external check for how many threads are running in the process just to make sure it looks right. For threads=15 you should actually see about 18 threads used by mod_wsgi daemon process from memory. There is the main process thread, 15 request handler threads and a number of background threads used by mod_wsgi for checking liveness and handling process shutdown. Can't remember off hand the exact number of the later, so may actually be 19, but also depends on the configuration. On top of that you will have your background threads. So just check occasionally to make sure the number is what you expect on top of those used by mod_wsgi. Creating background threads like you are from requests is not usually a good idea. How you are creating them doesn't look to be itself thread safe and so technically if multiple requests hit at the same time initially I think you could end up creating more than one background thread. Really needs some locking around thread creation. I will look at what you are doing and comment more on that later. Anyway, you can check the number of threads in a process using 'ps' or looking in /proc for process. https://stackoverflow.com/a/35421908/128141 <https://stackoverflow.com/a/35421908/128141> The other thing you might also check is how many temporary files are under /tmp and /var/tmp. This is just a hunch. For some stuff one can get temp files created as a side effect of doing stuff and my thinking here is maybe they are growing in number over time and not getting cleaned up properly. So am wondering whether this may be inadvertently slowing the machine done if doing things in the tmp directories. Although /tmp is usually cleared on reboot, the /var/tmp directory usually isn't. Graham > On 17 Dec 2020, at 3:31 pm, Gary Conley <g...@goldeneraproductions.org> wrote: > > Graham, > > It is people like you that make the open-source world go round. Thank you for > pointing out my errors and solving them at the same time. > > I implemented the new config and the app is running as expected in terms of > functionality, which is great. > > I still have a bit of an issue with speed, but nothing near as bad as it was > before. I have at times seen performance in the sub 0.5 second per file > range. Same hardware, same app, same code. And it would run for hours at that > speed, but gradually slow down. I am now seeing performance in the 0.8 to 1 > second per file range. I do understand the issues with python multi-threading > and have successfully implemented multi-processing in other (albeit simpler) > standalone applications. But it seems odd that I would have had better > performance under the same conditions and see a gradual slow down. Rebooting > the server made no difference. If it had been a memory leak or something of > that nature one would think rebooting would make a difference. > > If it is a simple matter to implement celery in my app then I may go that > route. Unfortunately I'm not familiar enough with it yet to make an accurate > assessment. It may not be worth the effort. My app is working as it is and > the performance may be adequate. If I could achieve consistent 0.5 seconds > per file performance it would be nice and would utilize my hardware, which is > basically loafing along right now. CPU nevers gets above about 30% and there > are other services running on this machine! So there's plenty of performance > to be gained... And there is a phenomena where it seems to slow down over > time. > > But I'm now into straight python and off topic for this mailing list. > > If you have any ideas I welcome them. But once again thanks for your help. > Really appreciated. > > Best, > > Gary > > > > On Wednesday, December 16, 2020 at 4:45:35 PM UTC-8 Graham Dumpleton wrote: > You have an incorrect path in the configuration meaning you are actually > running embedded mode and not daemon mode. This means you are subject to > Apache dynamic process management and thus you can end up with more than one > process. > > Your config is: > > Listen 82 > > #/etc/apache2/sites-available/phrasea_upload_flask.conf > <VirtualHost *:82> > ServerName phrasea_upload > ServerAlias phrasea_upload.avlib.net <http://phrasea_upload.avlib.net/> > > WSGIDaemonProcess phrasea_uploadapp user=apache group=apache threads=15 > home=/var/www/upload-flask > WSGIScriptAlias / /var/www/upload-flask/phrasea_upload_flask.wsgi > > Alias /static /var/www/upload-flask/static > Alias /templates /var/www/upload-flask/templates > > <Directory "/var/www/phrasea_upload_flask"> > WSGIProcessGroup phrasea_uploadapp > Require all granted > WSGIScriptReloading On > </Directory> > > LogLevel info > ErrorLog /etc/httpd/logs/phrasea_upload_flask.log > </VirtualHost> > > The path which is wrong is: > > /var/www/phrasea_upload_flask > > given in the Directory directive. It doesn't actually match the path for the > WSGI script file, meaning that the WSGIProcessGroup directive was ignored. > > Change this to: > > Listen 82 > > # Ensure that mod_wsgi embedded mode is disabled so don't accidentally run > stuff in embedded mode. > WSGIRestrictEmbedded On > > #/etc/apache2/sites-available/phrasea_upload_flask.conf > <VirtualHost *:82> > ServerName phrasea_upload > ServerAlias phrasea_upload.avlib.net <http://phrasea_upload.avlib.net/> > > WSGIDaemonProcess phrasea_uploadapp user=apache group=apache threads=15 > home=/var/www/upload-flask > # Set the daemon mode process group and application interpreter context > here explicitly. > WSGIScriptAlias / /var/www/upload-flask/phrasea_upload_flask.wsgi > process-group=phrasea_uploadapp application-group=%{GLOBAL} > > Alias /static /var/www/upload-flask/static > Alias /templates /var/www/upload-flask/templates > > <Directory "/var/www/upload-flask"> > Require all granted > </Directory> > > LogLevel info > ErrorLog /etc/httpd/logs/phrasea_upload_flask.log > </VirtualHost> > > Rather than use WSGIProcessGroup, have set process-group on WSGIScriptAlias > instead. Also set application-group to uses the main interpreter context and > not a sub interpreter. This can avoid problems with third party Python > modules that don't work in sub interpreters properly. > > You also didn't need WSGIScriptReloading as that is default for daemon mode. > > That the path didn't match meant that "Require all granted" wasn't being > applied either. That it worked without that being applied means you are > likely on one of the Linux distributions which somehow break Apache access > controls and set access for the whole filesystem or URL namespace at higher > scope somewhere. > > Since you have "info" for LogLevel, with daemon mode now being properly > applied, you should clearly see WSGI script file being loaded in daemon mode > process. Right now you have: > > [Sun Dec 13 03:29:39.291197 2020] [wsgi:info] [pid 16283:tid 139637154567936] > [client 10.12.17.31:41458 <http://10.12.17.31:41458/>] mod_wsgi (pid=16283, > process='', application='phrasea_upload|'): Loading Python script file > '/var/www/upload-flask/phrasea_upload_flask.wsgi'. > > See how "process" is an empty string. This means that it was using embedded > mode. With fixed config that "process" should show "phrasea_uploadapp" and > application should be empty string, with the latter indicating main > interpreter context rather than sub interpreter. > > Graham > > >> On 17 Dec 2020, at 10:53 am, Gary Conley <ga...@goldeneraproductions.org >> <applewebdata://FFAB7441-F624-49F2-9559-02138738A43C>> wrote: >> > >> Hi Graham >> >> Thanks for the rapid reply. >> >> You will have to bear with me a bit as this is my first flask app... I've >> attached what I think you are asking for, plus a log file and my main app.py >> file. Probably a bit rough around the edges in places. The file attached is >> a zip file. >> >> This is the only app running under httpd on this server by the way. And in >> checking the logs I noticed I have 4 PIDs, tending to indicate I have 4 >> processes running. You'll probably see from my app that it really can only >> have one process running as I'm really only trying to do one thing, process >> images for upload to a DAM, and there is only one user doing this, me. >> >> You will see the entries in the log file from yesterday morning showing an >> abort requested. I was the only user and had only one page open to run the >> app. You will see at 9.47 am the app reports there are 0 jobs running, and >> then 5 minutes later reports 1 job running 3 in the queue. That is accessing >> two global variables in app.py, phrasea_upload_runnables and upload_queue. >> What makes no sense at all is that both calls, 5 minutes apart should have >> been addressing the same variables with the same values. The PIDs on both >> those calls are the same. >> >> One other thing I should mention. When we first ran into the performance >> issues caused by the huge directories, we tried running the app under nginx, >> which we never got working. nginx is installed but not running on the >> server. I included my nginx ini file which specifies 4 processes. >> >> Thanks for taking the time to look this over. >> >> Best regards, >> >> Gary >> >> > >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to modwsgi+u...@googlegroups.com >> <applewebdata://FFAB7441-F624-49F2-9559-02138738A43C>. > >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/modwsgi/0ad301ab-ff44-4b83-b261-4d9a2db0e33an%40googlegroups.com >> >> <https://groups.google.com/d/msgid/modwsgi/0ad301ab-ff44-4b83-b261-4d9a2db0e33an%40googlegroups.com?utm_medium=email&utm_source=footer>. >> <phrasea_upload> > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+unsubscr...@googlegroups.com > <mailto:modwsgi+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/4ed52559-f19d-4023-be7a-bffd6cce4af3n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/4ed52559-f19d-4023-be7a-bffd6cce4af3n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/A346F9E2-48AC-43DD-B9AF-6C53A9966FDF%40gmail.com.
