On 17/11/2018 21:07, Jeff Ross wrote:
In my experience, it's best to split a large file up into whatever
chunk size you can upload to S3 in 1 minute to avoid that socket time
out error. From my server on a pretty good pipe, I use
--multipart-chunk-size-mb=100.
Thanks Jeff
I've probably been living on the edge for a long time.
Until earlier this year I was using the default settings, It was taking
me about 3 minutes to upload a chunk. I then introduced the
--limit-rate=36k, to half the upload speed, this change I suspect threw
me even closer to the edge as it was taking about 6 minutes to upload a
chunk. I introduced this change because there were a couple of occasions
earlier this year when the upload did effect people working late and
other systems running overnight.
I'm now using --multipart-chunk-size-mb=5 and limit-rate=36K. With my
system and network it now takes about 2 minutes to upload a 5Mb chuck,
but it's stable which is to me is what counts. If I encounter more
problems I can either increase the limit-rate or decrease the chunk size.
I run the upload overnight which generally doesn't impact anyone and
completes easily within 8 hours.
Thanks again for your help and advice.
Alex
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
