Hi Graham,
I finally figured out what was causing my slows. It had nothing to do with
my flask app or mod_wsgi but was related to the DAM and how it handles file
uploads. That's all in PHP code, which is now fixed and the performance is
back to what I was expecting to begin with. Better actually.
Your help in getting my flask app running properly in daemon mode enabled
me to trace this down. So thank you for that. And I do see that I have an
issue with not being thread safe and will dig in and get that sorted out.
Best,
Gary
On Thursday, December 17, 2020 at 9:59:06 AM UTC-8 Gary Conley wrote:
> Thanks very much Graham,
>
> You are spot on. The logic is not correct. I should actually return a list
> and then iterate through the list in _start_queue. I've modified the code.
> There is a large job running that I need to let finish and then I'll try it
> out. This should provide a slight performance improvement.
>
> I'm working on some debug code that will tell me see how long each step of
> the image processing is slowing down. For each image I upload the original
> image to the DAM using an API call, generate a preview and a thumbnail,
> calculate a sha256 hash, extract metadata with exiftool, upload preview and
> thumbnail to the DAM, also with API calls and record all these transactions
> in a mysql database. And there are several other steps involved to qualify
> the images before upload. So any one of these could be where I'm losing
> performance. Once I have a grip on that I'll have a better idea where to go.
>
> But again, the big mystery is that I have had twice the performance I'm
> getting now, somewhat out of the blue without any obvious change. And so we
> investigate!!
>
> I'll check into your other suggestions to see where that leads me.
>
> Thanks again.
>
> Gary
>
>
>
>
>
> On Wednesday, December 16, 2020 at 9:22:30 PM UTC-8 Graham Dumpleton wrote:
>
>> BTW, this looks like it could be short circuited.
>>
>> for loaddir in loaddirs:
>> if os.path.isdir(loaddir):
>> # filecheck = glob(loaddir + '/**/*.*', recursive=True)
>> filecheck = []
>> for root, _, files in os.walk(loaddir):
>> for f in files:
>> filecheck.append(os.path.join(root, f))
>> if filecheck:
>> # searchpath, startdir, phraseaserver, auth, refspath,
>> jsondir, threadcount, db
>> return PhraseaLoader(upload_dir, to_phrasea_dir,
>> phrasea_server, auth, refspath, json_meta_path, threadcount, db,
>> make_previews, debug_mode)
>> else:
>> print('Empty folder. Nothing to do here. Load directory:
>> {}'.format(loaddir))
>> return None
>>
>> What I mean by that is that since the contents of file check list is
>> never actually used, you don't need to completely populate it. Return as
>> soon as you find an entry.
>>
>> So could you instead just do the following, which is return as soon as
>> you find any directory which has a non empty files list.
>>
>> for loaddir in loaddirs:
>> if os.path.isdir(loaddir):
>> for root, _, files in os.walk(loaddir):
>> if files:
>> return PhraseaLoader(upload_dir, to_phrasea_dir,
>> phrasea_server, auth, refspath, json_meta_path, threadcount, db,
>> make_previews, debug_mode)
>>
>> print('Empty folder. Nothing to do here. Load directory:
>> {}'.format(loaddir))
>> return None
>>
>> The logic also seems not to be quite right either, as it appears to bail
>> out as soon as it finds an empty directory tree, rather than going onto the
>> next one in loaddirs. Should that "return None" really exist? Or is it
>> meant to stop when finds first empty directory tree?
>>
>> Graham
>>
>> On 17 Dec 2020, at 4:08 pm, Graham Dumpleton <[email protected]>
>> wrote:
>>
>> I'll go back over and read email again when I have a chance and see if
>> anything else can suggest, but right now suggest a couple of checks you can
>> make which don't require changing any code.
>>
>> First up, if you didn't feel comfortable about adding that code I linked
>> before for dumping stack traces, at least do an external check for how many
>> threads are running in the process just to make sure it looks right.
>>
>> For threads=15 you should actually see about 18 threads used by mod_wsgi
>> daemon process from memory.
>>
>> There is the main process thread, 15 request handler threads and a number
>> of background threads used by mod_wsgi for checking liveness and handling
>> process shutdown. Can't remember off hand the exact number of the later, so
>> may actually be 19, but also depends on the configuration.
>>
>> On top of that you will have your background threads. So just check
>> occasionally to make sure the number is what you expect on top of those
>> used by mod_wsgi.
>>
>> Creating background threads like you are from requests is not usually a
>> good idea. How you are creating them doesn't look to be itself thread safe
>> and so technically if multiple requests hit at the same time initially I
>> think you could end up creating more than one background thread. Really
>> needs some locking around thread creation. I will look at what you are
>> doing and comment more on that later.
>>
>> Anyway, you can check the number of threads in a process using 'ps' or
>> looking in /proc for process.
>>
>> https://stackoverflow.com/a/35421908/128141
>>
>> The other thing you might also check is how many temporary files are
>> under /tmp and /var/tmp. This is just a hunch. For some stuff one can get
>> temp files created as a side effect of doing stuff and my thinking here is
>> maybe they are growing in number over time and not getting cleaned up
>> properly. So am wondering whether this may be inadvertently slowing the
>> machine done if doing things in the tmp directories. Although /tmp is
>> usually cleared on reboot, the /var/tmp directory usually isn't.
>>
>> Graham
>>
>> On 17 Dec 2020, at 3:31 pm, Gary Conley <[email protected]>
>> wrote:
>>
>> Graham,
>>
>> It is people like you that make the open-source world go round. Thank you
>> for pointing out my errors and solving them at the same time.
>>
>> I implemented the new config and the app is running as expected in terms
>> of functionality, which is great.
>>
>> I still have a bit of an issue with speed, but nothing near as bad as it
>> was before. I have at times seen performance in the sub 0.5 second per file
>> range. Same hardware, same app, same code. And it would run for hours at
>> that speed, but gradually slow down. I am now seeing performance in the 0.8
>> to 1 second per file range. I do understand the issues with python
>> multi-threading and have successfully implemented multi-processing in other
>> (albeit simpler) standalone applications. But it seems odd that I would
>> have had better performance under the same conditions and see a gradual
>> slow down. Rebooting the server made no difference. If it had been a memory
>> leak or something of that nature one would think rebooting would make a
>> difference.
>>
>> If it is a simple matter to implement celery in my app then I may go that
>> route. Unfortunately I'm not familiar enough with it yet to make an
>> accurate assessment. It may not be worth the effort. My app is working as
>> it is and the performance may be adequate. If I could achieve consistent
>> 0.5 seconds per file performance it would be nice and would utilize my
>> hardware, which is basically loafing along right now. CPU nevers gets above
>> about 30% and there are other services running on this machine! So there's
>> plenty of performance to be gained... And there is a phenomena where it
>> seems to slow down over time.
>>
>> But I'm now into straight python and off topic for this mailing list.
>>
>> If you have any ideas I welcome them. But once again thanks for your
>> help. Really appreciated.
>>
>> Best,
>>
>> Gary
>>
>>
>>
>> On Wednesday, December 16, 2020 at 4:45:35 PM UTC-8 Graham Dumpleton
>> wrote:
>>
>>> You have an incorrect path in the configuration meaning you are actually
>>> running embedded mode and not daemon mode. This means you are subject to
>>> Apache dynamic process management and thus you can end up with more than
>>> one process.
>>>
>>> Your config is:
>>>
>>> Listen 82
>>>
>>> #/etc/apache2/sites-available/phrasea_upload_flask.conf
>>> <VirtualHost *:82>
>>> ServerName phrasea_upload
>>> ServerAlias phrasea_upload.avlib.net
>>>
>>> WSGIDaemonProcess phrasea_uploadapp user=apache group=apache
>>> threads=15 home=/var/www/upload-flask
>>> WSGIScriptAlias / /var/www/upload-flask/phrasea_upload_flask.wsgi
>>>
>>> Alias /static /var/www/upload-flask/static
>>> Alias /templates /var/www/upload-flask/templates
>>>
>>> <Directory "/var/www/phrasea_upload_flask">
>>> WSGIProcessGroup phrasea_uploadapp
>>> Require all granted
>>> WSGIScriptReloading On
>>> </Directory>
>>>
>>> LogLevel info
>>> ErrorLog /etc/httpd/logs/phrasea_upload_flask.log
>>> </VirtualHost>
>>>
>>>
>>> The path which is wrong is:
>>>
>>> /var/www/phrasea_upload_flask
>>>
>>> given in the Directory directive. It doesn't actually match the path for
>>> the WSGI script file, meaning that the WSGIProcessGroup directive was
>>> ignored.
>>>
>>> Change this to:
>>>
>>> Listen 82
>>>
>>> # Ensure that mod_wsgi embedded mode is disabled so don't accidentally
>>> run stuff in embedded mode.
>>> WSGIRestrictEmbedded On
>>>
>>> #/etc/apache2/sites-available/phrasea_upload_flask.conf
>>> <VirtualHost *:82>
>>> ServerName phrasea_upload
>>> ServerAlias phrasea_upload.avlib.net
>>>
>>> WSGIDaemonProcess phrasea_uploadapp user=apache group=apache
>>> threads=15 home=/var/www/upload-flask
>>> # Set the daemon mode process group and application interpreter
>>> context here explicitly.
>>> WSGIScriptAlias / /var/www/upload-flask/phrasea_upload_flask.wsgi
>>> process-group=phrasea_uploadapp application-group=%{GLOBAL}
>>>
>>> Alias /static /var/www/upload-flask/static
>>> Alias /templates /var/www/upload-flask/templates
>>>
>>> <Directory "/var/www/upload-flask">
>>> Require all granted
>>> </Directory>
>>>
>>> LogLevel info
>>> ErrorLog /etc/httpd/logs/phrasea_upload_flask.log
>>> </VirtualHost>
>>>
>>>
>>> Rather than use WSGIProcessGroup, have set process-group on
>>> WSGIScriptAlias instead. Also set application-group to uses the main
>>> interpreter context and not a sub interpreter. This can avoid problems with
>>> third party Python modules that don't work in sub interpreters properly.
>>>
>>> You also didn't need WSGIScriptReloading as that is default for daemon
>>> mode.
>>>
>>> That the path didn't match meant that "Require all granted" wasn't being
>>> applied either. That it worked without that being applied means you are
>>> likely on one of the Linux distributions which somehow break Apache access
>>> controls and set access for the whole filesystem or URL namespace at higher
>>> scope somewhere.
>>>
>>> Since you have "info" for LogLevel, with daemon mode now being properly
>>> applied, you should clearly see WSGI script file being loaded in daemon
>>> mode process. Right now you have:
>>>
>>> [Sun Dec 13 03:29:39.291197 2020] [wsgi:info] [pid 16283:tid
>>> 139637154567936] [client 10.12.17.31:41458] mod_wsgi (pid=16283,
>>> process='', application='phrasea_upload|'): Loading Python script file
>>> '/var/www/upload-flask/phrasea_upload_flask.wsgi'.
>>>
>>> See how "process" is an empty string. This means that it was using
>>> embedded mode. With fixed config that "process" should show
>>> "phrasea_uploadapp" and application should be empty string, with the latter
>>> indicating main interpreter context rather than sub interpreter.
>>>
>>> Graham
>>>
>>> On 17 Dec 2020, at 10:53 am, Gary Conley <[email protected]>
>>> wrote:
>>>
>>> Hi Graham
>>>
>>> Thanks for the rapid reply.
>>>
>>> You will have to bear with me a bit as this is my first flask app...
>>> I've attached what I think you are asking for, plus a log file and my main
>>> app.py file. Probably a bit rough around the edges in places. The file
>>> attached is a zip file.
>>>
>>> This is the only app running under httpd on this server by the way. And
>>> in checking the logs I noticed I have 4 PIDs, tending to indicate I have 4
>>> processes running. You'll probably see from my app that it really can only
>>> have one process running as I'm really only trying to do one thing, process
>>> images for upload to a DAM, and there is only one user doing this, me.
>>>
>>> You will see the entries in the log file from yesterday morning showing
>>> an abort requested. I was the only user and had only one page open to run
>>> the app. You will see at 9.47 am the app reports there are 0 jobs running,
>>> and then 5 minutes later reports 1 job running 3 in the queue. That is
>>> accessing two global variables in app.py, phrasea_upload_runnables and
>>> upload_queue. What makes no sense at all is that both calls, 5 minutes
>>> apart should have been addressing the same variables with the same values.
>>> The PIDs on both those calls are the same.
>>>
>>> One other thing I should mention. When we first ran into the performance
>>> issues caused by the huge directories, we tried running the app under
>>> nginx, which we never got working. nginx is installed but not running on
>>> the server. I included my nginx ini file which specifies 4 processes.
>>>
>>> Thanks for taking the time to look this over.
>>>
>>> Best regards,
>>>
>>> Gary
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "modwsgi" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>>
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/modwsgi/0ad301ab-ff44-4b83-b261-4d9a2db0e33an%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/modwsgi/0ad301ab-ff44-4b83-b261-4d9a2db0e33an%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>> <phrasea_upload>
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "modwsgi" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/modwsgi/4ed52559-f19d-4023-be7a-bffd6cce4af3n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/modwsgi/4ed52559-f19d-4023-be7a-bffd6cce4af3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
>>
>>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/modwsgi/e6b7fa8a-b746-4582-a365-f4032fa3b27an%40googlegroups.com.
