Is the image processing being done within the web application processes, or in a separate set of processes which operate only based on seeing what is stuck in the upload directory used to queue up images?
Just trying to understand better how the work is broken up. > On 8 Dec 2020, at 9:29 am, Gary Conley <g...@goldeneraproductions.org> wrote: > > I have a flask app running with mod_wsgi version 4.7.1 on Centos7 (httpd). > Python 3.6. > > The server is brand new, 64GB RAM, Dual Xeon 3Ghz, all SSD drives, 8gbs fiber > to a Stornext SAN. All very fast hardware. > > I have a problem where the app tends to slow down dramatically under certain > circumstances and have had no success to date in finding the cause. > > The app itself is fairly simple. It processes images from a directory and > uploads them to a DAM (php based - under NGINX) running on another server. > The processing consists of generating preview and thumbnail images from the > original images (mostly raw file formats - NEF, CR2 etc) using imageMagick, > ufraw and exiftool, extracting xmp data using exiftool and uploading these > elements to the DAM through the DAM's API. > > The processing is multithreaded using python queues and workers, nothing > fancy. > > The app gathers data about the images from a mysql database and persists > transactional data to mysql for workflow management purposes. > > There is a simple html template with controls for selecting which directories > to upload and provide the user with feedback on progress, errors and so on, > using ajax calls. The user can also abort the upload process. > > Under normal circumstances the app will process 2 images per second. There > have been instances however where the app slows way down, taking 7-10 seconds > to upload 1 image, a factor of 15 to 20 times slower. > > I have run metrics on the various steps of the upload procedure and it > appears that every aspect of the app slows down. Generating a preview image, > which normally takes less than a second takes 40 seconds, extracting metadata > with exiftool, typically less than half a second takes 7-10 seconds. Database > response seems to remain constant. Upload to the DAM also takes much longer. > > When the slow down occurs requests from the browser to the app time out. > Aborting the upload procedure becomes impossible and the only way to stop it > is to stop Apache (httpd) directly on the server. > > We have checked CPU usage (less than 10%), memory usage (less than 8 GB on a > 64GB machine) and also checked IO, confirming that we were able read and > write at up to 9.5gbs while the app was running at a snail's pace. > > The only thing we have been able to isolate as having any effect on the > upload speed is the size of the upload queue. When a user selects a directory > to be uploaded the files in that directory (and all sub-directories) are > sorted by size and put into a python queue from which they are then uploaded > with between 10 and 20 threads, which is user configurable. We have tested > with queues up to 10,000 files with no issue at all. We had a slow down with > a queue that was over 35,000 images. > > The content of the images makes little to no difference in speed. Processing > huge image files (such as 5GB PSB files) does slow down the upload process, > but only to 1.5 seconds per file. The last instance of slow down occurred on > 5MB jpgs. The speed on 60MB NEF and 200KB jpgs is virtually the same under > normal circumstances. > > We suspect a memory issue, but don't see any increase in memory usage using > htop, top or glances. > > We restarted httpd in the hopes it would clear up any memory leak, with no > improvement. We even rebooted the machine with similar hopes, again with no > improvement. > > We tried change the number of threads in our wsgi config (from 5 to 15) and > also changed the number of processes to 2. We even tried setting threads to > 1, which had disastrous effects. None of this made any improvement. We put > our settings back to 1 process and 5 threads. > > Any clues on what could be causing this slow down or ideas on how to isolate > what is causing it would be much appreciated. We've spent days trying to > track it down with the only solution being to break up our jobs into smaller > chunks, which is very non-optimum for our workflow as it has to be done > manually due to the nature of the content. > > We can send files if needed, but are not sure what to send. > > Thank you. > > Gary > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+unsubscr...@googlegroups.com > <mailto:modwsgi+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/modwsgi/389a39bd-2ad1-4c65-a784-0762029d0825n%40googlegroups.com > > <https://groups.google.com/d/msgid/modwsgi/389a39bd-2ad1-4c65-a784-0762029d0825n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/DE991B69-45FB-4A1F-B4E2-168AFF726F6D%40gmail.com.
