Thanks everyone, Uploading directly to S3 sounds the avenue to pursue. It removes a whole load of complications!
Cheers Hansel On Tue, Apr 11, 2017 at 12:18 PM, Roger Gammans <rgamm...@gammascience.co.uk > wrote: > Indeed. > > Both the rsync and bittorrent protocols split files into chunks. There are > a lot of similarities bwtween these protocols. > > Also it might be worth looking at are de-duping backup tools like > duplicity. > > On Tue, 2017-04-11 at 11:33 +0100, Javier Llopis wrote: > > I haven't done anything similar, but this upload in chunks then reassemble > idea bears some similarity to peer to peer file sharing software. I would > look there for ideas in case I got stuck with some problem. > > J > > > ------------------------------ > *From*: "Hansel Dunlop" <han...@interpretthis.org> > *Sent*: 11 April 2017 11:22 > *To*: "UK Python Users" <python-uk@python.org> > *Subject*: [python-uk] Handling large file uploads > > Hello all > > I'm working on an application that has to accept large uploads. Think ~ > 2GB+ size files getting uploaded over slowish connections. These files are > eventually going to end up in S3. > > Uploading smallish files is not a problem. But things get a bit > complicated when you're dealing with large files and load balanced servers. > Servers that can be replaced at any time. Has anyone done something > similar? > > My current plan is: > > 1. Accept chunked uploads. So the app/browser sends individual POST > requests with ~10mb chunks. Once that upload is complete the server > responds with a chunk id and the current offset > 2. The server stores each intermediate chunk in a temporary S3 bucket > 3. Once the final chunk has been uploaded the server kicks off another > process that stitches the pieces together and puts the whole file into it's > final location. And then deletes the intermediate pieces. > > I think I have to do the file in chunks like this but maybe there is some > way to stream the files somewhere? > > -- > > Hansel > > _______________________________________________ > python-uk mailing > listpython-uk@python.orghttps://mail.python.org/mailman/listinfo/python-uk > > > > _______________________________________________ > python-uk mailing list > python-uk@python.org > https://mail.python.org/mailman/listinfo/python-uk > > -- Hansel
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
