AWS Java S3 SDK does the following:
public static long calculateOptimalPartSize(PutObjectRequest
putObjectRequest, TransferManagerConfiguration configuration) {
double contentLength =
TransferManagerUtils.getContentLength(putObjectRequest);
double optimalPartSize = (double)contentLength /
(double)MAXIMUM_UPLOAD_PARTS;
// round up so we don't push the upload over the maximum number of
parts
optimalPartSize = Math.ceil(optimalPartSize);
return (long)Math.max(optimalPartSize,
configuration.getMinimumUploadPartSize());
}
AWS SDK defaults to a maximum of 10000 parts. The minimum default part size is
5MB. So, uploading a 51GB file, for example, would use 8500 6MB parts.
Jclouds could use a similar mechanism. It would probably make sense to expose
the configuration parameters to be able to change the default behavior.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/762#issuecomment-108633453
