I haven't done anything similar, but this upload in chunks then reassemble
idea bears some similarity to peer to peer file sharing software. I would
look there for ideas in case I got stuck with some problem.
J
----------------------------------------
From: "Hansel Dunlop" <han...@interpretthis.org>
Sent: 11 April 2017 11:22
To: "UK Python Users" <python-uk@python.org>
Subject: [python-uk] Handling large file uploads
Hello all
I'm working on an application that has to accept large uploads. Think ~
2GB+ size files getting uploaded over slowish connections. These files are
eventually going to end up in S3.
Uploading smallish files is not a problem. But things get a bit
complicated when you're dealing with large files and load balanced servers.
Servers that can be replaced at any time. Has anyone done something
similar?
My current plan is:
1. Accept chunked uploads. So the app/browser sends individual POST
requests with ~10mb chunks. Once that upload is complete the server
responds with a chunk id and the current offset
2. The server stores each intermediate chunk in a temporary S3 bucket
3. Once the final chunk has been uploaded the server kicks off another
process that stitches the pieces together and puts the whole file into it's
final location. And then deletes the intermediate pieces.
I think I have to do the file in chunks like this but maybe there is some
way to stream the files somewhere?
--
Hansel
_______________________________________________
python-uk mailing list
python-uk@python.org
https://mail.python.org/mailman/listinfo/python-uk
