It will use an internal executorservice, if i remember correctly. I don't remember how it was configured, though, I'll have to look. Probably slow if not even sequential.
-Zack ________________________________ From: Archana C <carchan...@yahoo.co.uk> Sent: Saturday, February 4, 2017 12:37 AM To: user@jclouds.apache.org Subject: Re: jClouds 2.0 MultiPart Upload Does it mean if we are not specifying executorservice, the upload happens in sequential ? Regards Archana On Saturday, 4 February 2017, 10:34, Andrew Gaul <g...@apache.org> wrote: PutOptions *takes* an ExecutorService which allows multiple threads to concurrently upload multiple parts. On Sat, Feb 04, 2017 at 03:34:30AM +0000, Archana C wrote: > Hi > > I think the question was not clear. Parallel upload of multiple file is fine > and that can be achieved by using executorservice. > The question here is, does multipartUpload i.e uploading of each part is > happening in parallel ? > Does sequential upload of part deprecated ? > RegardsArchana > > > > On Saturday, 4 February 2017, 1:30, Andrew Gaul > <g...@apache.org<mailto:g...@apache.org>> wrote: > > > We rewrote multi-part uploads in jclouds 2.0. You should pass an > ExecutorService and via PutOptions in your call to BlobStore.putBlob. > > On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote: > > Hi > > > > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart > > uploads are parallel now ? > > RegardsArchana > > > > On Friday, 3 February 2017, 18:39, Archana C > > <carchan...@yahoo.co.uk<mailto:carchan...@yahoo.co.uk>> wrote: > > > > > > Thanks it helped > > RegardsArchana > > > > On Friday, 3 February 2017, 12:06, Ignasi Barrera > > <n...@apache.org<mailto:n...@apache.org>> wrote: > > > > > > It looks like the OOM exception is thrown when writing the wire logs. When > > using the blob store apis you might see binary data in the logs, as the > > "jclouds.wire" logger logs the response/request payloads which might be > > huge for some blobs and can cause this kind of exceptions. > > Could you try disabling the wire logs? (I recommend doing this for > > production environments). > > Perhaps for your use case the "jclouds.headers" are enough; that will log > > all request/reponse path and headers but skip the bodies. > > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187 > > https://issues.apache.org/jira/browse/JCLOUDS-932 > > > > > > HTH! > > I. > > On Feb 3, 2017 06:22, "Archana C" > > <carchan...@yahoo.co.uk<mailto:carchan...@yahoo.co.uk>> wrote: > > > > Hi > > > > I have written a sample code for multipart upload using jClouds-2.0 > > Properties overrides = new Properties(); > > BlobStoreContext context = ContextBuilder.newBuilder(" > > openstack-swift") > > .endpoint("http://x.xxx.xx.xx: <http://x.xxx.xx.xx/> > > 5000/v2.0") > > .credentials("xx:xx", "xx") > > .overrides(overrides) > > .modules(modules) > > .buildView(BlobStoreContext. class); > > BlobStore blobStore = context.getBlobStore(); > > blobStore. createContainerInLocation( null, CONTAINER_NAME); > > Path path = Paths.get("test2"); > > File f = new File("test2"); > > byte []byteArray = Files.readAllBytes(path); > > Payload payload = newByteSourcePayload(wrap( byteArray)); > > PutOptions opt = new PutOptions(); > > opt.multipart(); > > Blob blob = blobStore.blobBuilder(OBJECT_ NAME) > > .payload(payload). contentLength(f.length()) > > .build(); > > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt); > > test2 is the file I am trying to upload which is of size 36MB and I am > > getting the following exception > > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - > > Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_ > > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1 > > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT > > http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP > > HTTP/1.1 > > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json > > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: > > fd72b74db90c46cabcca3f317d5a09 d4 > > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice - > > Receiving response 1344471693: HTTP/1.1 201 Created > > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created > > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 > > 04:51:53 GMT > > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: > > tx83ba6249347c43c99bb41- 0058940c68 > > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive > > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; > > charset=UTF-8 > > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0 > > -------> Container Creation Successful > > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >> > > invoking object:put > > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >> > > invoking object:put > > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ > > ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_ > > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ > > 1486097513.327000/0/33554432/ 00000001 HTTP/1.1 > > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ > > ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_ > > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/ > > 1486097513.327000/0/33554432/ 00000000 HTTP/1.1 > > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire - > > over limit 3145728/262144: wrote temp file > > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire - > > over limit 33554432/262144: wrote temp fileException in thread "main" > > com.google.common.util. concurrent.ExecutionError: > > java.lang.OutOfMemoryError: Java heap space > > at com.google.common.util. concurrent.Futures. > > wrapAndThrowUnchecked(Futures. java:1380) > > at com.google.common.util. concurrent.Futures. > > getUnchecked(Futures.java: 1373) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. > > java:650) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. > > java:628) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274) > > at jclouds20.App.main(App.java: 83) > > Caused by: java.lang.OutOfMemoryError: Java heap space > > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342) > > at java.lang.StringBuilder. append(StringBuilder.java:208) > > at org.jclouds.logging.internal. Wire.wire(Wire.java:68) > > at org.jclouds.logging.internal. Wire.copy(Wire.java:99) > > at org.jclouds.logging.internal. Wire.output(Wire.java:176) > > at org.jclouds.logging.internal. Wire.output(Wire.java:143) > > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296) > > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( > > BaseHttpCommandExecutorService .java:97) > > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( > > InvokeHttpMethod.java:90) > > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > > InvokeHttpMethod.java:73) > > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > > InvokeHttpMethod.java:44) > > at org.jclouds.reflect. FunctionalReflection$ > > FunctionalInvocationHandler. handleInvocation( > > FunctionalReflection.java:117) > > at com.google.common.reflect. AbstractInvocationHandler. invoke( > > AbstractInvocationHandler. java:87) > > at com.sun.proxy.$Proxy68.put( Unknown Source) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. uploadMultipartPart( > > RegionScopedSwiftBlobStore. java:529) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. > > java:666) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. > > java:653) > > at java.util.concurrent. FutureTask.run(FutureTask. java:274) > > at java.util.concurrent. ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1157) > > at java.util.concurrent. ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:627) > > at java.lang.Thread.run(Thread. java:798) > > at java.lang.Thread. getStackTraceImpl(Native Method) > > at java.lang.Thread. getStackTrace(Thread.java: 1133) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. > > java:647) > > ... 3 more > > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$ > > CombinedFuture setExceptionAndMaybeLog > > SEVERE: input future failed. > > java.lang.OutOfMemoryError: Java heap space > > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342) > > at java.lang.StringBuilder. append(StringBuilder.java:208) > > at org.jclouds.logging.internal. Wire.wire(Wire.java:68) > > at org.jclouds.logging.internal. Wire.copy(Wire.java:99) > > at org.jclouds.logging.internal. Wire.output(Wire.java:176) > > at org.jclouds.logging.internal. Wire.output(Wire.java:143) > > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296) > > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke( > > BaseHttpCommandExecutorService .java:97) > > at org.jclouds.rest.internal. InvokeHttpMethod.invoke( > > InvokeHttpMethod.java:90) > > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > > InvokeHttpMethod.java:73) > > at org.jclouds.rest.internal. InvokeHttpMethod.apply( > > InvokeHttpMethod.java:44) > > at org.jclouds.reflect. FunctionalReflection$ > > FunctionalInvocationHandler. handleInvocation( > > FunctionalReflection.java:117) > > at com.google.common.reflect. AbstractInvocationHandler. invoke( > > AbstractInvocationHandler. java:87) > > at com.sun.proxy.$Proxy68.put( Unknown Source) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. uploadMultipartPart( > > RegionScopedSwiftBlobStore. java:529) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. > > java:666) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore. > > java:653) > > at java.util.concurrent. FutureTask.run(FutureTask. java:274) > > at java.util.concurrent. ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1157) > > at java.util.concurrent. ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:627) > > at java.lang.Thread.run(Thread. java:798) > > at java.lang.Thread. getStackTraceImpl(Native Method) > > at java.lang.Thread. getStackTrace(Thread.java: 1133) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. > > java:647) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore. > > java:628) > > at org.jclouds.openstack.swift. v1.blobstore. > > RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274) > > at jclouds20.App.main(App.java: 83) > > > > Should putBlob with multipart enabled in PutOptions not be used for > > uploading files greater than 32MB from jClouds-2.0 ? > > Is it mandatory to use Swift API's initiate, upload and complete multipart > > from jClouds-2.0 ? > > > > RegardsArchana > > > > > > > > > > > > > > > > > > -- > Andrew Gaul > http://gaul.org/ > > > -- Andrew Gaul http://gaul.org/
