Does it mean if we are not specifying executorservice, the upload happens in
sequential ?
Regards
Archana
On Saturday, 4 February 2017, 10:34, Andrew Gaul <[email protected]> wrote:
PutOptions *takes* an ExecutorService which allows multiple threads to
concurrently upload multiple parts.
On Sat, Feb 04, 2017 at 03:34:30AM +0000, Archana C wrote:
> Hi
>
> I think the question was not clear. Parallel upload of multiple file is fine
> and that can be achieved by using executorservice.
> The question here is, does multipartUpload i.e uploading of each part is
> happening in parallel ?
> Does sequential upload of part deprecated ?
> RegardsArchana
>
>
>
> On Saturday, 4 February 2017, 1:30, Andrew Gaul <[email protected]> wrote:
>
>
> We rewrote multi-part uploads in jclouds 2.0. You should pass an
> ExecutorService and via PutOptions in your call to BlobStore.putBlob.
>
> On Fri, Feb 03, 2017 at 01:11:15PM +0000, Archana C wrote:
> > Hi
> >
> > Is SequentialMultiPartUpload deprecated in jClouds2.0. Is all the multipart
> > uploads are parallel now ?
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 18:39, Archana C <[email protected]>
> >wrote:
> >
> >
> > Thanks it helped
> > RegardsArchana
> >
> > On Friday, 3 February 2017, 12:06, Ignasi Barrera <[email protected]>
> >wrote:
> >
> >
> > It looks like the OOM exception is thrown when writing the wire logs. When
> >using the blob store apis you might see binary data in the logs, as the
> >"jclouds.wire" logger logs the response/request payloads which might be huge
> >for some blobs and can cause this kind of exceptions.
> > Could you try disabling the wire logs? (I recommend doing this for
> > production environments).
> > Perhaps for your use case the "jclouds.headers" are enough; that will log
> > all request/reponse path and headers but skip the bodies.
> > More on this here:https://issues.apache.org/jira/browse/JCLOUDS-1187
> > https://issues.apache.org/jira/browse/JCLOUDS-932
> >
> >
> > HTH!
> > I.
> > On Feb 3, 2017 06:22, "Archana C" <[email protected]> wrote:
> >
> > Hi
> >
> > I have written a sample code for multipart upload using jClouds-2.0
> > Properties overrides = new Properties();
> > BlobStoreContext context = ContextBuilder.newBuilder("
> > openstack-swift")
> > .endpoint("http://x.xxx.xx.xx: 5000/v2.0")
> > .credentials("xx:xx", "xx")
> > .overrides(overrides)
> > .modules(modules)
> > .buildView(BlobStoreContext. class);
> > BlobStore blobStore = context.getBlobStore();
> > blobStore. createContainerInLocation( null, CONTAINER_NAME);
> > Path path = Paths.get("test2");
> > File f = new File("test2");
> > byte []byteArray = Files.readAllBytes(path);
> > Payload payload = newByteSourcePayload(wrap( byteArray));
> > PutOptions opt = new PutOptions();
> > opt.multipart();
> > Blob blob = blobStore.blobBuilder(OBJECT_ NAME)
> > .payload(payload). contentLength(f.length())
> > .build();
> > String etag = blobStore.putBlob(CONTAINER_ NAME, blob, opt);
> > test2 is the file I am trying to upload which is of size 36MB and I am
> > getting the following exception
> > 10:21:52.355 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice -
> > Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_
> > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT
> > http://x.x.x.x:8091/v1/AUTH_ 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP
> > HTTP/1.1
> > 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> > 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token:
> > fd72b74db90c46cabcca3f317d5a09 d4
> > 10:21:53.129 [main] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ ice -
> > Receiving response 1344471693: HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017
> > 04:51:53 GMT
> > 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id:
> > tx83ba6249347c43c99bb41- 0058940c68
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html;
> > charset=UTF-8
> > 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0
> > -------> Container Creation Successful
> > 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal. InvokeHttpMethod - >>
> > invoking object:put
> > 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal. InvokeHttpMethod - >>
> > invoking object:put
> > 10:21:53.374 [user thread 1] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ
> > ice - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_
> > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/
> > 1486097513.327000/0/33554432/ 00000001 HTTP/1.1
> > 10:21:53.376 [user thread 0] DEBUG o.j.h.i. JavaUrlHttpCommandExecutorServ
> > ice - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_
> > 0909ac10e7024847b1a9fe9787c7de 8f/arctestMP/arc/slo/
> > 1486097513.327000/0/33554432/ 00000000 HTTP/1.1
> > 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal. HttpWire -
> > over limit 3145728/262144: wrote temp file
> > 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal. HttpWire -
> > over limit 33554432/262144: wrote temp fileException in thread "main"
> > com.google.common.util. concurrent.ExecutionError:
> > java.lang.OutOfMemoryError: Java heap space
> > at com.google.common.util. concurrent.Futures.
> > wrapAndThrowUnchecked(Futures. java:1380)
> > at com.google.common.util. concurrent.Futures.
> > getUnchecked(Futures.java: 1373)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore.
> > java:650)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore.
> > java:628)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke(
> > BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke(
> > InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply(
> > InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply(
> > InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$
> > FunctionalInvocationHandler. handleInvocation(
> > FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke(
> > AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. uploadMultipartPart(
> > RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore.
> > java:666)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore.
> > java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore.
> > java:647)
> > ... 3 more
> > Feb 03, 2017 10:22:03 AM com.google.common.util. concurrent.Futures$
> > CombinedFuture setExceptionAndMaybeLog
> > SEVERE: input future failed.
> > java.lang.OutOfMemoryError: Java heap space
> > at java.lang.StringBuilder. ensureCapacityImpl( StringBuilder.java:342)
> > at java.lang.StringBuilder. append(StringBuilder.java:208)
> > at org.jclouds.logging.internal. Wire.wire(Wire.java:68)
> > at org.jclouds.logging.internal. Wire.copy(Wire.java:99)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:176)
> > at org.jclouds.logging.internal. Wire.output(Wire.java:143)
> > at org.jclouds.http.HttpUtils. wirePayloadIfEnabled( HttpUtils.java:296)
> > at org.jclouds.http.internal. BaseHttpCommandExecutorService .invoke(
> > BaseHttpCommandExecutorService .java:97)
> > at org.jclouds.rest.internal. InvokeHttpMethod.invoke(
> > InvokeHttpMethod.java:90)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply(
> > InvokeHttpMethod.java:73)
> > at org.jclouds.rest.internal. InvokeHttpMethod.apply(
> > InvokeHttpMethod.java:44)
> > at org.jclouds.reflect. FunctionalReflection$
> > FunctionalInvocationHandler. handleInvocation(
> > FunctionalReflection.java:117)
> > at com.google.common.reflect. AbstractInvocationHandler. invoke(
> > AbstractInvocationHandler. java:87)
> > at com.sun.proxy.$Proxy68.put( Unknown Source)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. uploadMultipartPart(
> > RegionScopedSwiftBlobStore. java:529)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore.
> > java:666)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore$ BlobUploader.call( RegionScopedSwiftBlobStore.
> > java:653)
> > at java.util.concurrent. FutureTask.run(FutureTask. java:274)
> > at java.util.concurrent. ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1157)
> > at java.util.concurrent. ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:627)
> > at java.lang.Thread.run(Thread. java:798)
> > at java.lang.Thread. getStackTraceImpl(Native Method)
> > at java.lang.Thread. getStackTrace(Thread.java: 1133)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore.
> > java:647)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putMultipartBlob( RegionScopedSwiftBlobStore.
> > java:628)
> > at org.jclouds.openstack.swift. v1.blobstore.
> > RegionScopedSwiftBlobStore. putBlob( RegionScopedSwiftBlobStore. java:274)
> > at jclouds20.App.main(App.java: 83)
> >
> > Should putBlob with multipart enabled in PutOptions not be used for
> > uploading files greater than 32MB from jClouds-2.0 ?
> > Is it mandatory to use Swift API's initiate, upload and complete multipart
> > from jClouds-2.0 ?
> >
> > RegardsArchana
> >
> >
> >
> >
> >
> >
> >
> >
>
> --
> Andrew Gaul
> http://gaul.org/
>
>
>
--
Andrew Gaul
http://gaul.org/
