On Tue, Mar 24, 2015 at 10:34:55AM +0100, Veit Guna wrote: > It seems that in general it is somehow possible to upload files without > knowing its content length in advance like implemented here: > > https://github.com/libin/s3distcp/blob/master/src/main/java/com/amazon/external/elasticmapreduce/s3distcp/MultipartUploadOutputStream.java > > I'm wondering if such functionality is already implemented in jclouds or, if > not, are there're any plans > to do so? > > Using the jclouds MultiPartUpload still forces me to specify the > content-length.
jclouds uses the multipart size to avoid buffering parts in-memory since each requires an explicit size: https://github.com/jclouds/jclouds/blob/master/apis/s3/src/main/java/org/jclouds/s3/blobstore/strategy/internal/SequentialMultipartUploadStrategy.java The s3distcp implementation calculates these sizes by buffering each part on-disk, incurring three times as much IO as the jclouds equivalent (read and write to the temporary file). Several users have asked for an improved multipart interface although this is tricky to do efficiently. Could you add your thoughts to: https://issues.apache.org/jira/browse/JCLOUDS-769 -- Andrew Gaul http://gaul.org/
