Indeed. Both the rsync and bittorrent protocols split files into chunks. There are a lot of similarities bwtween these protocols.
Also it might be worth looking at are de-duping backup tools like duplicity. On Tue, 2017-04-11 at 11:33 +0100, Javier Llopis wrote: > I haven't done anything similar, but this upload in chunks then > reassemble idea bears some similarity to peer to peer file sharing > software. I would look there for ideas in case I got stuck with some > problem. > > > > > > J > > > > > > > > > > From: "Hansel Dunlop" <han...@interpretthis.org> > > Sent: 11 April 2017 11:22 > > To: "UK Python Users" <python-uk@python.org> > > Subject: [python-uk] Handling large file uploads > > > > Hello all > > > I'm working on an application that has to accept large uploads. Think > ~ 2GB+ size files getting uploaded over slowish connections. These > files are eventually going to end up in S3. > > > > Uploading smallish files is not a problem. But things get a bit > complicated when you're dealing with large files and load balanced > servers. Servers that can be replaced at any time. Has anyone done > something similar? > > > > My current plan is: > > > > 1. Accept chunked uploads. So the app/browser sends individual POST > requests with ~10mb chunks. Once that upload is complete the server > responds with a chunk id and the current offset > > 2. The server stores each intermediate chunk in a temporary S3 bucket > > 3. Once the final chunk has been uploaded the server kicks off > another process that stitches the pieces together and puts the whole > file into it's final location. And then deletes the intermediate > pieces. > > > > I think I have to do the file in chunks like this but maybe there is > some way to stream the files somewhere? > > > > -- > > > > > Hansel > > > > > _______________________________________________ > python-uk mailing list > python-uk@python.org > https://mail.python.org/mailman/listinfo/python-uk
_______________________________________________ python-uk mailing list python-uk@python.org https://mail.python.org/mailman/listinfo/python-uk
