I have some scenarios where I need to do some processing (envelope decryption) on a file from s3 prior to download and then let the user download it and this is how I do it as well.
1. Download giant file from S3 to temporary file. 2. Process file to another temporary file. 3. Return a FileResponse wrapping the temporary file. 4. via WSGI iterator protocol the server will invoke the close() method on the iterator when the request is cleaned up and this will bubble up to delete the temporary file being wrapped. My solution is not ideal, there is a lag while the file is downloaded from storage into a temporary file and processed, before I return the iterator. However in practice it doesn't blow out memory (yay) and since the throughput between S3 and EC2 is great it does >1GB files with only a slight lag (a couple seconds iirc). I would say that if you don't need to do any processing then Theron's S3 suggestion is definitely better assuming you can expose those endpoint details to clients. - Michael > On Jul 14, 2021, at 10:57, Theron Luhn <[email protected]> wrote: > > Pyramid has FileResponse > https://docs.pylonsproject.org/projects/pyramid/en/latest/api/response.html#pyramid.response.FileResponse > > <https://docs.pylonsproject.org/projects/pyramid/en/latest/api/response.html#pyramid.response.FileResponse>, > which does use wsgi.file_wrapper you linked to (if available). > > Generally in this situation I offload the file to S3 or similar, and have > Pyramid generate a signed URL to redirect to. Operationally much simpler. > > — Theron > > > >> On Jul 14, 2021, at 3:17 AM, Mikko Ohtamaa <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> I need to serve large authenticated files (several gigabytes). I will do an >> API key check before the user can download a file. >> >> What is the most efficient way to serve these out from Pyramid? Assuming I >> do not want to block processes or threads - is it possible? >> >> - Zope 2 used to have a sendfile ( >> https://www.python.org/dev/peps/pep-0333/#id36 >> <https://www.python.org/dev/peps/pep-0333/#id36>) - is there anything >> equivalent for waitress >> >> - Any ideas about caching the file in the frontend servers (Caddy, >> Cloudflare) and then just creating a short-lived HTTP redirect (<1h) to the >> actual target >> >> - Other ideas >> >> Cheers, >> Mikko >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUuTG7xZ9tFf5raSmotjOWPJx31dcXyDujk4222_GDq74w%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUuTG7xZ9tFf5raSmotjOWPJx31dcXyDujk4222_GDq74w%40mail.gmail.com?utm_medium=email&utm_source=footer>. > > > -- > You received this message because you are subscribed to the Google Groups > "pylons-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pylons-discuss/8B634DFB-996C-47AB-9623-EC0DFF5FD50D%40luhn.com > > <https://groups.google.com/d/msgid/pylons-discuss/8B634DFB-996C-47AB-9623-EC0DFF5FD50D%40luhn.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/1A776BE4-DB33-4E9E-A652-9A2B7F76E04B%40gmail.com.
