On 03/07/14 13:52, Scott Wheeler wrote: > I ended up cleaning up and gemifying our little tool for running parallel > extracts: > > https://github.com/directededge/redsnapper > > http://rubygems.org/gems/redsnapper > > Doing this will use 25 parallel clients (from testing 25 seemed to be about > the place where the benefits maxed out) each extracting up to 50 files to > extract an archive: > > sudo redsnapper ARCHIVE > > We see about a 5x speedup doing things that way. It’s slower than the normal > tool for small archives since it has to fetch the file list first.
Thanks for publishing this. One suggestion -- I'd change the "up to 50 files" to "up to 1000". Otherwise with lots of files you end up spawning extra processes which each download the same set of metadata. -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
