We rewrote multi-part uploads in jclouds 2.0. You should pass an ExecutorService and via PutOptions in your call to BlobStore.putBlob.
On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote: > Hi > > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart > uploads are parallel now ? > RegardsArchana > > On Friday, 3 February 2017, 18:39, Archana C <[email protected]> > wrote: > > > Thanks it helped > RegardsArchana > > On Friday, 3 February 2017, 12:06, Ignasi Barrera <[email protected]> wrote: > > > It looks like the OOM exception is thrown when writing the wire logs. When > using the blob store apis you might see binary data in the logs, as the > "jclouds.wire" logger logs the response/request payloads which might be huge > for some blobs and can cause this kind of exceptions. > Could you try disabling the wire logs? (I recommend doing this for production > environments). > Perhaps for your use case the "jclouds.headers" are enough; that will log all > request/reponse path and headers but skip the bodies. > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187 > https://issues.apache.org/jira/browse/JCLOUDS-932 > > > HTH! > I. > On Feb 3, 2017 06:22, "Archana C" <[email protected]> wrote: > > Hi > > I have written a sample code for multipart upload using jClouds-2.0 > Properties overrides = new Properties(); > BlobStoreContext context = ContextBuilder.newBuilder(" > openstack-swift") > .endpoint("http://x.xxx.xx.xx: 5000/v2.0") > .credentials("xx:xx", "xx") > .overrides(overrides) > .modules(modules) > .buildView(BlobStoreContext. class); > BlobStore blobStore = context.getBlobStore(); > blobStore. createContainerInLocation( null, CONTAINER_NAME); > Path path = Paths.get("test2"); > File f = new File("test2"); > byte []byteArray = Files.readAllBytes(path); > Payload payload = newByteSourcePayload(wrap( byteArray)); > PutOptions opt = new PutOptions(); > opt.multipart(); > Blob blob = blobStore.blobBuilder(OBJECT_ NAME) > .payload(payload). contentLength(f.length()) > .build(); > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt); > test2 is the file I am trying to upload which is of size 36MB and I am > getting the following exception > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - > Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1 > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT > http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP > HTTP/1.1 > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: > fd72b74db90c46cabcca3f317d5a09 d4 > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - > Receiving response 1344471693: HTTP/1.1 201 Created > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 > 04:51:53 GMT > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: > tx83ba6249347c43c99bb41- 0058940c68 > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; > charset=UTF-8 > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 > -------> Container Creation Successful > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> > invoking object:put > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> > invoking object:put > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ > ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ > 1486097513.327000/0/33554432/ 00000001 HTTP/1.1 > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ > ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ > 1486097513.327000/0/33554432/ 00000000 HTTP/1.1 > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - over > limit 3145728/262144: wrote temp file > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - over > limit 33554432/262144: wrote temp fileException in thread "main" > com.google.common.util. concurrent.ExecutionError: > java.lang.OutOfMemoryError: Java heap space > at com.google.common.util. concurrent.Futures. > wrapAndThrowUnchecked(Futures. java:1380) > at com.google.common.util. concurrent.Futures. getUnchecked(Futures.java: > 1373) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putMultipartBlob( RegionScopedSwiftBlobStore. java:650) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putMultipartBlob( RegionScopedSwiftBlobStore. java:628) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putBlob( RegionScopedSwiftBlobStore. java:274) > at jclouds20.App.main(App.java: 83) > Caused by: java.lang.OutOfMemoryError: Java heap space > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342) > at java.lang.StringBuilder. append(StringBuilder.java:208) > at org.jclouds.logging.internal. Wire.wire(Wire.java:68) > at org.jclouds.logging.internal. Wire.copy(Wire.java:99) > at org.jclouds.logging.internal. Wire.output(Wire.java:176) > at org.jclouds.logging.internal. Wire.output(Wire.java:143) > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296) > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( > BaseHttpCommandExecutorService .java:97) > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( > InvokeHttpMethod.java:90) > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > InvokeHttpMethod.java:73) > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > InvokeHttpMethod.java:44) > at org.jclouds.reflect. FunctionalReflection$ > FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117) > at com.google.common.reflect. AbstractInvocationHandler. invoke( > AbstractInvocationHandler. java:87) > at com.sun.proxy.$Proxy68.put( Unknown Source) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > uploadMultipartPart( RegionScopedSwiftBlobStore. java:529) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ > BlobUploader.call( RegionScopedSwiftBlobStore. java:666) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ > BlobUploader.call( RegionScopedSwiftBlobStore. java:653) > at java.util.concurrent. FutureTask.run(FutureTask. java:274) > at java.util.concurrent. ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1157) > at java.util.concurrent. ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:627) > at java.lang.Thread.run(Thread. java:798) > at java.lang.Thread. getStackTraceImpl(Native Method) > at java.lang.Thread. getStackTrace(Thread.java: 1133) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putMultipartBlob( RegionScopedSwiftBlobStore. java:647) > ... 3 more > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ > CombinedFuture setExceptionAndMaybeLog > SEVERE: input future failed. > java.lang.OutOfMemoryError: Java heap space > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342) > at java.lang.StringBuilder. append(StringBuilder.java:208) > at org.jclouds.logging.internal. Wire.wire(Wire.java:68) > at org.jclouds.logging.internal. Wire.copy(Wire.java:99) > at org.jclouds.logging.internal. Wire.output(Wire.java:176) > at org.jclouds.logging.internal. Wire.output(Wire.java:143) > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296) > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( > BaseHttpCommandExecutorService .java:97) > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( > InvokeHttpMethod.java:90) > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > InvokeHttpMethod.java:73) > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > InvokeHttpMethod.java:44) > at org.jclouds.reflect. FunctionalReflection$ > FunctionalInvocationHandler. handleInvocation( FunctionalReflection.java:117) > at com.google.common.reflect. AbstractInvocationHandler. invoke( > AbstractInvocationHandler. java:87) > at com.sun.proxy.$Proxy68.put( Unknown Source) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > uploadMultipartPart( RegionScopedSwiftBlobStore. java:529) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ > BlobUploader.call( RegionScopedSwiftBlobStore. java:666) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore$ > BlobUploader.call( RegionScopedSwiftBlobStore. java:653) > at java.util.concurrent. FutureTask.run(FutureTask. java:274) > at java.util.concurrent. ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1157) > at java.util.concurrent. ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:627) > at java.lang.Thread.run(Thread. java:798) > at java.lang.Thread. getStackTraceImpl(Native Method) > at java.lang.Thread. getStackTrace(Thread.java: 1133) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putMultipartBlob( RegionScopedSwiftBlobStore. java:647) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putMultipartBlob( RegionScopedSwiftBlobStore. java:628) > at org.jclouds.openstack.swift. v1.blobstore. RegionScopedSwiftBlobStore. > putBlob( RegionScopedSwiftBlobStore. java:274) > at jclouds20.App.main(App.java: 83) > > Should putBlob with multipart enabled in PutOptions not be used for uploading > files greater than 32MB from jClouds-2.0 ? > Is it mandatory to use Swift API's initiate, upload and complete multipart > from jClouds-2.0 ? > > RegardsArchana > > > > > > > > -- Andrew Gaul http://gaul.org/
