I am currently processing the data when i receive it and then returning it within the same http request. The extended time out allows it to work for one single request so thank you! I agree that this does not look scalable... I have so many options. Am looking into video streaming and web sockets as well other than celery and rabbit mq.
On Monday, 30 October 2017 09:56:38 UTC+8, Graham Dumpleton wrote: > > Are you trying to process the data in realtime? > > A problem you might have is that if you are processing data in real time > and that cannot keep up with the rate at which data is being sent, then > buffering will occur between the Apache child worker processes doing the > proxying, and the mod_wsgi daemon processes. If things run slow enough, > then eventually you will trip Apache's timeout on blocked on write to a > connection. This timeout is dictated by the Apache Timeout directive. In > mod_wsgi-express that is set to 60 seconds by default. > > You could be exacerbating the issue if your processing of the video stream > is done in the daemon process and you are handling concurrent requests, > because that sounds like a CPU intensive activity. The global interpreter > lock in Python means that multithreading and CPU intensive tasks do not mix > well. > > First up I would suggest using 10 processes, but set threads to 1, to > ensure issue not caused by high CPU and multiple requests hitting the same > process. > > Next you could try overriding the socket timeout using > the --socket-timeout option to be larger than 60 seconds, even if only > initially to see if that helps the problem go away. In older versions of > Apache this was set as 300 seconds, but that is ridiculously high for the > majority of web applications. > > After that, should perhaps look at how you are processing the video > stream. Whether should save to a file and then queue a job in a background > task queuing system such as Celery to process it, with the original web > request returning as soon as the upload finishes. Then rely on slow polling > or other mechanisms to find out when video processing finished if need to > show result. > > Graham > > On 30 Oct 2017, at 12:44 pm, Isabelle <islan...@gmail.com <javascript:>> > wrote: > > I am attempting to upload a 1GB video file to my server and I was using > default. But now I have tried 10 processes with default threads and it > still gives a 504. Could there still be any component that needs a time > limit to be opened up? I expect to wait 10 minutes for this extracting to > be completed. > > [Mon Oct 30 01:20:47.352358 2017] [wsgi:error] [pid 25153:tid > 139819841537792] [remote 192.168.20.104:58706] Extracting frames from > video file. > [Mon Oct 30 01:21:36.750641 2017] [wsgi:error] [pid 25158:tid > 139820955944704] [client 192.168.20.104:58706] Timeout when reading > response headers from daemon process 'localhost:5001': > /tmp/mod_wsgi-localhost:5001:0/htdocs/videohandler > > > On Friday, 27 October 2017 18:40:44 UTC+8, Graham Dumpleton wrote: >> >> A 504 is going to be because you exceeded capacity of what daemon >> processes could handle. >> >> What are you setting processes/threads to, or are you using default, with >> is 1 process with 5 threads? >> >> On 27 Oct 2017, at 8:55 pm, Isabelle <islan...@gmail.com> wrote: >> >> Thanks that works, then i run into a 504 time out. Is there a max >> execution time setting too? I tried --max-execution-time but it doesnt >> have that option... ( is there a complete listing somewhere for options or >> am I missing it? ) >> >> On Friday, 27 October 2017 12:06:29 UTC+8, Graham Dumpleton wrote: >>> >>> >>> On 27 Oct 2017, at 2:58 pm, Isabelle <islan...@gmail.com> wrote: >>> >>> Hi it turns out I had to give my new user access to some folders to do >>> read/writes. >>> >>> I am having a problem with it throwing a 413 error. I checked my apache >>> config and there was no LimitRequestBody >>> <http://httpd.apache.org/docs/2.0/mod/core.html#limitrequestbody> field >>> in it. I purposely set it to unlimited but i still get the 413. So just >>> wondering does mod_wsgi-express control the size of input files sent >>> over POST and where to set it? >>> >>> >>> Yes, mod_wsgi-express does set a default for it, because having no >>> default is a bad idea as it can actually be used as a denial of service >>> attack vector. >>> >>> The option to change it is: >>> >>> --limit-request-body NUMBER >>> The maximum number of bytes which are allowed >>> in a >>> request body. Defaults to 10485760 (10MB). >>> >>> If you set it to 0, it will be back to unlimited. >>> >>> The better thing to do is to work out what URLs actually require a >>> higher limit and change it for just those. >>> >>> This can be done by creating a limits.conf file which contains: >>> >>> <Location /some/url >>> # 100MB >>> LimitRequestBody 104857600 >>> </Location> >>> >>> Then use the option '--include-file limits.conf' to have the partial >>> configuration file included at the end of the generated Apache >>> configuration file. >>> >>> Another option is to set overall limit higher, but then make it lower >>> for just GET requests, which usually wouldn't have request content. >>> >>> # By default unlimited. >>> >>> LimitRequestBody 0 >>> >>> <Limit GET> >>> # Set limit for GET to 1MB. >>> LimitRequestBody 1048576 >>> </Limit> >>> >>> Graham >>> >>> On Thursday, 19 October 2017 18:41:52 UTC+8, Graham Dumpleton wrote: >>>> >>>> >>>> On 19 Oct 2017, at 8:16 pm, Isabelle <islan...@gmail.com> wrote: >>>> >>>> Thank you. I was able to run this hello world >>>> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> >>>> (saved as wsgi.py) >>>> >>>> def application(environ, start_response): >>>> >>>> status = '200 OK' >>>> output = 'Hello World!' >>>> >>>> response_headers = [('Content-type', 'text/plain'), >>>> ('Content-Length', str(len(output)))] >>>> >>>> start_response(status, response_headers) >>>> >>>> return [output] >>>> >>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>> >>>> with these added to my docker file: >>>> >>>> RUN apt-get update && \ >>>> apt-get install -y --no-install-recommends apache2 apache2-dev locales >>>> && \ >>>> apt-get clean && \ >>>> rm -r /var/lib/apt/lists/* >>>> >>>> RUN pip install --no-cache-dir mod_wsgi >>>> >>>> >>>> Copy your requirements.txt to the image and use: >>>> >>>> RUN pip install --no-cache-dir -r requirements.txt >>>> >>>> RUN adduser --disabled-password --gecos "apache" --uid 1001 --gid 0 >>>> --home /app apache && \ >>>> chmod g+w /etc/passwd >>>> >>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>> >>>> mod_wsgi-express start-server --port "8080" --user apache >>>> --log-to-terminal wsgi.py >>>> >>>> and then i tried my own >>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>> (saved as wsgi2.py) >>>> >>>> import sys >>>> >>>> #Expand Python classes path with your app's path >>>> sys.path.insert(0, "/app") >>>> >>>> from manage import app as application >>>> >>>> >>>>>>>>>>>>>>>> >>>> >>>> mod_wsgi-express start-server --port "8080" --user apache >>>> --log-to-terminal wsgi2.py >>>> >>>> >>>> And got something running but with some dependencies not found, which >>>> is strange. But I was indeed running things from root without a python >>>> virtual environment previously. So will probably have to dig around there >>>> to figure how it works now that the dockerfile set up a new user account. >>>> >>>> >>>> >>>> On Thursday, 19 October 2017 09:36:57 UTC+8, Isabelle wrote: >>>>> >>>>> Hi, I have been attempting to do the hello world on a docker-ce but >>>>> failling and would like to make a hello world for mod wsgi work. I am >>>>> currently stuck and not sure how to proceed because I don't see error >>>>> messages from the mod wsgi >>>>> >>>>> >>>>> Client: >>>>> Version: 17.09.0-ce >>>>> API version: 1.32 >>>>> Go version: go1.8.3 >>>>> Git commit: afdb6d4 >>>>> Built: Tue Sep 26 22:42:18 2017 >>>>> OS/Arch: linux/amd64 >>>>> >>>>> Server: >>>>> Version: 17.09.0-ce >>>>> API version: 1.32 (minimum version 1.12) >>>>> Go version: go1.8.3 >>>>> Git commit: afdb6d4 >>>>> Built: Tue Sep 26 22:40:56 2017 >>>>> OS/Arch: linux/amd64 >>>>> Experimental: false >>>>> >>>>> with ubuntu >>>>> >>>>> more /etc/lsb-release >>>>> DISTRIB_ID=Ubuntu >>>>> DISTRIB_RELEASE=16.04 >>>>> DISTRIB_CODENAME=xenial >>>>> DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS" >>>>> >>>>> and apache and python2 >>>>> >>>>> apache2ctl -M >>>>> AH00558: apache2: Could not reliably determine the server's fully >>>>> qualified domain name, using 172.17.0.2. Set the 'ServerName' directive >>>>> globally to suppress this message >>>>> Loaded Modules: >>>>> core_module (static) >>>>> so_module (static) >>>>> watchdog_module (static) >>>>> http_module (static) >>>>> log_config_module (static) >>>>> logio_module (static) >>>>> version_module (static) >>>>> unixd_module (static) >>>>> access_compat_module (shared) >>>>> alias_module (shared) >>>>> auth_basic_module (shared) >>>>> authn_core_module (shared) >>>>> authn_file_module (shared) >>>>> authz_core_module (shared) >>>>> authz_host_module (shared) >>>>> authz_user_module (shared) >>>>> autoindex_module (shared) >>>>> deflate_module (shared) >>>>> dir_module (shared) >>>>> env_module (shared) >>>>> filter_module (shared) >>>>> mime_module (shared) >>>>> mpm_event_module (shared) >>>>> negotiation_module (shared) >>>>> setenvif_module (shared) >>>>> status_module (shared) >>>>> wsgi_module (shared) >>>>> >>>>> dpkg -s libapache2-mod-wsgi >>>>> Package: libapache2-mod-wsgi >>>>> Status: install ok installed >>>>> Priority: optional >>>>> Section: httpd >>>>> Installed-Size: 242 >>>>> Maintainer: Ubuntu Developers >>>>> Architecture: amd64 >>>>> Source: mod-wsgi >>>>> Version: 4.3.0-1.1build1 >>>>> Provides: httpd-wsgi >>>>> Depends: libc6 (>= 2.14), libpython2.7 (>= 2.7), apache2-api-20120211, >>>>> apache2-bin (>= 2.4.16), python (>= 2.7), python (<< 2.8) >>>>> Conffiles: >>>>> /etc/apache2/mods-available/wsgi.conf c4ca5be35d0820b5d5cc2892097b476b >>>>> /etc/apache2/mods-available/wsgi.load 06d2b4d2c95b28720f324bd650b7cbd6 >>>>> Description: Python WSGI adapter module for Apache >>>>> The mod_wsgi adapter is an Apache module that provides a WSGI (Web >>>>> Server >>>>> Gateway Interface, a standard interface between web server software and >>>>> web applications written in Python) compliant interface for hosting >>>>> Python >>>>> based web applications within Apache. The adapter provides >>>>> significantly >>>>> better performance than using existing WSGI adapters for mod_python or >>>>> CGI. >>>>> . >>>>> This package provides module for Python 2.X. >>>>> Original-Maintainer: Debian Python Modules Team >>>>> Homepage: http://www.modwsgi.org/ >>>>> >>>>> root@e3807b2573b3:/usr/lib/apache2/modules# ldd mod_wsgi.so >>>>> linux-vdso.so.1 => (0x00007fff4fbd3000) >>>>> libpython2.7.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0 >>>>> (0x00007f0041a33000) >>>>> libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 >>>>> (0x00007f0041816000) >>>>> libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f004144b000) >>>>> libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f0041231000) >>>>> libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f004102d000) >>>>> libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f0040e29000) >>>>> libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0040b20000) >>>>> /lib64/ld-linux-x86-64.so.2 (0x000056530c32f000) >>>>> >>>>> I have created the following files >>>>> >>>>> /usr/local/www/wsgi-scripts/myapp.wsgi >>>>> >>>>> def application(environ, start_response): >>>>> status = '200 OK' >>>>> output = 'Hello World!\n' >>>>> response_headers = [('Content-type', 'text/plain'), >>>>> ('Content-Length', str(len(output)))] >>>>> start_response(status, response_headers) >>>>> return [output] >>>>> >>>>> /etc/apache2/sites-available/myapp.conf >>>>> >>>>> >>>>> ServerName example.com >>>>> >>>>> DocumentRoot /usr/local/www/documents >>>>> >>>>> Order allow,deny >>>>> Allow from all >>>>> >>>>> WSGIScriptAlias /myapp /usr/local/www/wsgi-scripts/myapp.wsgi >>>>> >>>>> Order allow,deny >>>>> Allow from all >>>>> >>>>> I have run the command, "a2ensite myapp" >>>>> >>>>> And then I restart apache2 "/etc/init.d/apache2 restart" >>>>> >>>>> And I attempt to view 172.17.0.2/myapp from my browser and it gives >>>>> me a 404 >>>>> >>>>> 172.17.0.2 shows me the apache homepage >>>>> >>>>> My apache log file is giving print outs from my previous failed >>>>> attempt to try out flask modwsgi and apache ... I am not sure what I am >>>>> looking at. >>>>> >>>>> [Thu Oct 19 01:19:46.036918 2017] [wsgi:warn] [pid 4016:tid >>>>> 140327866734464] mod_wsgi: Compiled for Python/2.7.11. >>>>> <http://2.7.0.11/> >>>>> [Thu Oct 19 01:19:46.036971 2017] [wsgi:warn] [pid 4016:tid >>>>> 140327866734464] mod_wsgi: Runtime using Python/2.7.12. >>>>> <http://2.7.0.12/> >>>>> [Thu Oct 19 01:19:46.037195 2017] [wsgi:debug] [pid 4016:tid >>>>> 140327866734464] src/server/mod_wsgi.c(7362): mod_wsgi (pid=4016): Socket >>>>> for 'webtool' is '/var/run/apache2 >>>>> /wsgi.4016.0.1.sock'. >>>>> [Thu Oct 19 01:19:46.037229 2017] [wsgi:debug] [pid 4016:tid >>>>> 140327866734464] src/server/mod_wsgi.c(7420): mod_wsgi (pid=4016): Listen >>>>> backlog for socket '/var/run/apach >>>>> e2/wsgi.4016.0.1.sock' is '100'. >>>>> [Thu Oct 19 01:19:46.037465 2017] [wsgi:info] [pid 4019:tid >>>>> 140327866734464] mod_wsgi (pid=4019): Starting process 'webtool' with >>>>> uid=33, gid=33 and threads=5. >>>>> [Thu Oct 19 01:19:46.037594 2017] [wsgi:info] [pid 4019:tid >>>>> 140327866734464] mod_wsgi (pid=4019): Initializing Python. >>>>> [Thu Oct 19 01:19:46.037890 2017] [mpm_event:notice] [pid 4016:tid >>>>> 140327866734464] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 >>>>> Python/2.7.12 configured -- resuming n >>>>> ormal operations >>>>> [Thu Oct 19 01:19:46.037904 2017] [mpm_event:info] [pid 4016:tid >>>>> 140327866734464] AH00490: Server built: 2017-09-18T15:09:02 >>>>> [Thu Oct 19 01:19:46.037915 2017] [core:notice] [pid 4016:tid >>>>> 140327866734464] AH00094: Command line: '/usr/sbin/apache2' >>>>> [Thu Oct 19 01:19:46.037919 2017] [core:debug] [pid 4016:tid >>>>> 140327866734464] log.c(1546): AH02639: Using SO_REUSEPORT: yes (1) >>>>> [Thu Oct 19 01:19:46.037954 2017] [wsgi:info] [pid 4020:tid >>>>> 140327866734464] mod_wsgi (pid=4020): Initializing Python. >>>>> [Thu Oct 19 01:19:46.038284 2017] [wsgi:info] [pid 4021:tid >>>>> 140327866734464] mod_wsgi (pid=4021): Initializing Python. >>>>> [Thu Oct 19 01:19:46.044567 2017] [wsgi:info] [pid 4019:tid >>>>> 140327866734464] mod_wsgi (pid=4019): Attach interpreter ''. >>>>> [Thu Oct 19 01:19:46.044844 2017] [wsgi:debug] [pid 4019:tid >>>>> 140327762720512] src/server/mod_wsgi.c(7971): mod_wsgi (pid=4019): >>>>> Started >>>>> thread 0 in daemon process 'webto >>>>> ol'. >>>>> [Thu Oct 19 01:19:46.044904 2017] [wsgi:debug] [pid 4019:tid >>>>> 140327745910528] src/server/mod_wsgi.c(7971): mod_wsgi (pid=4019): >>>>> Started >>>>> thread 2 in daemon process 'webto >>>>> ol'. >>>>> [Thu Oct 19 01:19:46.044896 2017] [wsgi:debug] [pid 4019:tid >>>>> 140327754319616] src/server/mod_wsgi.c(7971): mod_wsgi (pid=4019): >>>>> Started >>>>> thread 1 in daemon process 'webto >>>>> ol'. >>>>> [Thu Oct 19 01:19:46.044935 2017] [wsgi:debug] [pid 4019:tid >>>>> 140327737509632] src/server/mod_wsgi.c(7971): mod_wsgi (pid=4019): >>>>> Started >>>>> thread 3 in daemon process 'webto >>>>> ol'. >>>>> [Thu Oct 19 01:19:46.044961 2017] [wsgi:debug] [pid 4019:tid >>>>> 140327729108736] src/server/mod_wsgi.c(7971): mod_wsgi (pid=4019): >>>>> Started >>>>> thread 4 in daemon process 'webto >>>>> ol'. >>>>> [Thu Oct 19 01:19:46.054970 2017] [wsgi:info] [pid 4020:tid >>>>> 140327866734464] mod_wsgi (pid=4020): Attach interpreter ''. >>>>> [Thu Oct 19 01:19:46.054970 2017] [wsgi:info] [pid 4021:tid >>>>> 140327866734464] mod_wsgi (pid=4021): Attach interpreter ''. >>>>> [Thu Oct 19 01:19:46.055565 2017] [mpm_event:debug] [pid 4021:tid >>>>> 140327779505920] event.c(2094): AH02471: start_threads: Using epoll >>>>> [Thu Oct 19 01:19:46.055571 2017] [mpm_event:debug] [pid 4020:tid >>>>> 140327779505920] event.c(2094): AH02471: start_threads: Using epoll >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "modwsgi" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to modwsgi+u...@googlegroups.com. >>>> To post to this group, send email to mod...@googlegroups.com. >>>> Visit this group at https://groups.google.com/group/modwsgi. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "modwsgi" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to modwsgi+u...@googlegroups.com. >>> To post to this group, send email to mod...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/modwsgi. >>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "modwsgi" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to modwsgi+u...@googlegroups.com. >> To post to this group, send email to mod...@googlegroups.com. >> Visit this group at https://groups.google.com/group/modwsgi. >> For more options, visit https://groups.google.com/d/optout. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to modwsgi+u...@googlegroups.com <javascript:>. > To post to this group, send email to mod...@googlegroups.com <javascript:> > . > Visit this group at https://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+unsubscr...@googlegroups.com. To post to this group, send email to modwsgi@googlegroups.com. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
