Re: jClouds 2.0 MultiPart Upload

2017-02-02 Thread Ignasi Barrera 
It looks like the OOM exception is thrown when writing the wire logs. When
using the blob store apis you might see binary data in the logs, as the
"jclouds.wire" logger logs the response/request payloads which might be
huge for some blobs and can cause this kind of exceptions.

Could you try disabling the wire logs? (I recommend doing this for
production environments).

Perhaps for your use case the "jclouds.headers" are enough; that will log
all request/reponse path and headers but skip the bodies.

More on this here:
https://issues.apache.org/jira/browse/JCLOUDS-1187
https://issues.apache.org/jira/browse/JCLOUDS-932


HTH!

I.

On Feb 3, 2017 06:22, "Archana C"  wrote:

> Hi
>
> *I have written a sample code for multipart upload using jClouds-2.0*
>
> Properties overrides = new Properties();
> BlobStoreContext context = ContextBuilder.newBuilder("
> openstack-swift")
> .endpoint("http://x.xxx.xx.xx:5000/v2.0;)
> .credentials("xx:xx", "xx")
> .overrides(overrides)
> .modules(modules)
> .buildView(BlobStoreContext.class);
> BlobStore blobStore = context.getBlobStore();
> blobStore.createContainerInLocation(null, CONTAINER_NAME);
> Path path = Paths.get("test2");
> File f = new File("test2");
> byte []byteArray =  Files.readAllBytes(path);
> Payload payload = newByteSourcePayload(wrap(byteArray));
> PutOptions opt = new PutOptions();
> opt.multipart();
> Blob blob = blobStore.blobBuilder(OBJECT_NAME)
> .payload(payload).contentLength(f.length())
> .build();
> String etag =  blobStore.putBlob(CONTAINER_NAME, blob, opt);
>
> *test2 is the file I am trying to upload which is of size 36MB and I am
> getting the following exception*
>
> 10:21:52.355 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService -
> Sending request 1344471693: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> PUT
> http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP
> HTTP/1.1
> 10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
> 10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token:
> fd72b74db90c46cabcca3f317d5a09d4
> 10:21:53.129 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService -
> Receiving response 1344471693: HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
> 10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017
> 04:51:53 GMT
> 10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id:
> tx83ba6249347c43c99bb41-0058940c68
> 10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html;
> charset=UTF-8
> 10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0
> ---> *Container Creation Successful*
> 10:21:53.373 [user thread 1] DEBUG o.j.rest.internal.InvokeHttpMethod -
> >> invoking object:put
> 10:21:53.373 [user thread 0] DEBUG o.j.rest.internal.InvokeHttpMethod -
> >> invoking object:put
> 10:21:53.374 [user thread 1] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
> - Sending request 823625484: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/
> 1486097513.327000/0/33554432/0001 HTTP/1.1
> 10:21:53.376 [user thread 0] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService
> - Sending request -1220101806: PUT http://x.x.x.x:8091/v1/AUTH_
> 0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/
> 1486097513.327000/0/33554432/ HTTP/1.1
> 10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal.HttpWire -
> over limit* 3145728/262144*: wrote temp file
> 10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal.HttpWire -
> over limit* 33554432/262144*: wrote temp file
> Exception in thread "main" com.google.common.util.concurrent.ExecutionError:
> java.lang.OutOfMemoryError: Java heap space
> at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1380)
> at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1373)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:650)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.
> java:628)
> at org.jclouds.openstack.swift.v1.blobstore.
> RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
> at jclouds20.App.main(App.java:83)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
> at java.lang.StringBuilder.append(StringBuilder.java:208)
> at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
> at

jClouds 2.0 MultiPart Upload

2017-02-02 Thread Archana C 
Hi 

I have written a sample code for multipart upload using jClouds-2.0
    Properties overrides = new Properties();
        BlobStoreContext context = ContextBuilder.newBuilder("openstack-swift")
                .endpoint("http://x.xxx.xx.xx:5000/v2.0;)
                .credentials("xx:xx", "xx")
                .overrides(overrides)
                .modules(modules)
                .buildView(BlobStoreContext.class);
        BlobStore blobStore = context.getBlobStore();
        blobStore.createContainerInLocation(null, CONTAINER_NAME);
        Path path = Paths.get("test2");
        File f = new File("test2");
        byte []byteArray =  Files.readAllBytes(path);
        Payload payload = newByteSourcePayload(wrap(byteArray));
        PutOptions opt = new PutOptions();
        opt.multipart();
        Blob blob = blobStore.blobBuilder(OBJECT_NAME)
                .payload(payload).contentLength(f.length())
                .build();
        String etag =  blobStore.putBlob(CONTAINER_NAME, blob, opt);
test2 is the file I am trying to upload which is of size 36MB and I am getting 
the following exception
10:21:52.355 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Sending 
request 1344471693: PUT 
http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> PUT 
http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP HTTP/1.1
10:21:52.356 [main] DEBUG jclouds.headers - >> Accept: application/json
10:21:52.357 [main] DEBUG jclouds.headers - >> X-Auth-Token: 
fd72b74db90c46cabcca3f317d5a09d4
10:21:53.129 [main] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - Receiving 
response 1344471693: HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << HTTP/1.1 201 Created
10:21:53.129 [main] DEBUG jclouds.headers - << Date: Fri, 03 Feb 2017 04:51:53 
GMT
10:21:53.129 [main] DEBUG jclouds.headers - << X-Trans-Id: 
tx83ba6249347c43c99bb41-0058940c68
10:21:53.129 [main] DEBUG jclouds.headers - << Connection: keep-alive
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Type: text/html; 
charset=UTF-8
10:21:53.129 [main] DEBUG jclouds.headers - << Content-Length: 0    
---> Container Creation Successful
10:21:53.373 [user thread 1] DEBUG o.j.rest.internal.InvokeHttpMethod - >> 
invoking object:put
10:21:53.373 [user thread 0] DEBUG o.j.rest.internal.InvokeHttpMethod - >> 
invoking object:put
10:21:53.374 [user thread 1] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - 
Sending request 823625484: PUT 
http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/1486097513.327000/0/33554432/0001
 HTTP/1.1
10:21:53.376 [user thread 0] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService - 
Sending request -1220101806: PUT 
http://x.x.x.x:8091/v1/AUTH_0909ac10e7024847b1a9fe9787c7de8f/arctestMP/arc/slo/1486097513.327000/0/33554432/
 HTTP/1.1
10:21:53.396 [user thread 1] DEBUG org.jclouds.http.internal.HttpWire - over 
limit 3145728/262144: wrote temp file
10:21:53.552 [user thread 0] DEBUG org.jclouds.http.internal.HttpWire - over 
limit 33554432/262144: wrote temp fileException in thread "main" 
com.google.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: 
Java heap space
    at 
com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1380)
    at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1373)
    at 
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:650)
    at 
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putMultipartBlob(RegionScopedSwiftBlobStore.java:628)
    at 
org.jclouds.openstack.swift.v1.blobstore.RegionScopedSwiftBlobStore.putBlob(RegionScopedSwiftBlobStore.java:274)
    at jclouds20.App.main(App.java:83)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.lang.StringBuilder.ensureCapacityImpl(StringBuilder.java:342)
    at java.lang.StringBuilder.append(StringBuilder.java:208)
    at org.jclouds.logging.internal.Wire.wire(Wire.java:68)
    at org.jclouds.logging.internal.Wire.copy(Wire.java:99)
    at org.jclouds.logging.internal.Wire.output(Wire.java:176)
    at org.jclouds.logging.internal.Wire.output(Wire.java:143)
    at org.jclouds.http.HttpUtils.wirePayloadIfEnabled(HttpUtils.java:296)
    at 
org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
    at 
org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
    at 
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
    at 
org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
    at 
org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
    at 
com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
    at

Re: Re:Is there any way to add custom headers on HTTP headers to Rackspace Cloudfiles?

2017-02-02 Thread Ignasi Barrera 
Hi Jeff,

There is no default way to add arbitrary headers to the requests generated
by jclouds. The approach I'd recommend is to use the OkHttp driver and its
interceptor mechanism. Here is how you can set it up:

* First, configure the OkHttp driver when creating the context by adding
its module to the list of modules, as explained in the driver's README [1].
* Then, create your OkHttp interceptor class that will add the required
headers. You will find good examples here [2].
* Finally, initialize the OkHttp client used by the driver to use your
interceptor. You can do this by adding a custom module to the list of
modules used to create the context. It could be something like:













*Module customInterceptorModule = new AbstractModule() {   @Override
 protected void configure() {
bind(OkHttpClientSupplier.class).toInstance(new OkHttpClientSupplier() {
 @Override public OkHttpClient get() {OkHttpClient
client = new OkHttpClient();client.interceptors().add(new
MyInterceptor());return client; }  });   }};*




*ContextBuilder.newBuilder("provider")   .endpoint("endpoint")
 .credentials("identity", "credential")   .modules(ImmutableSet.of(new
OkHttpCommandExecutorServiceModule(), customInterceptorModule))*
*   .build();*

HTH!

I.



[1] https://github.com/jclouds/jclouds/tree/master/drivers/okhttp
[2] https://github.com/square/okhttp/wiki/Interceptors

On 2 February 2017 at 22:35, Andrew Phillips  wrote:

> Er, that should have been "user@..." - sorry!
>
> ap
>
>  Original Message 
> Subject: Re:Is there any way to add custom headers on HTTP headers to
> Rackspace Cloudfiles?
> Date: 2017-02-02 15:59
> From: Andrew Phillips 
> To: us...@jclouds.apache.org
> Copy: jwolf...@gmail.com
>
> [Forwarding Jeff's email to users@. @Jeff: notifications@ is just for
> automated emails.]
>
> "In order to troubleshoot some issues downloading objects from Rackspace
> Cloudfiles, they are requesting that I add a "x-trans-id" header to my API
> requests.  Is there any way to instruct the jclouds client library to do
> this for me?  Seems like it should be straightforward to insert a custom
> HTTP header, but I can't find any way to do it."
>

Fwd: Re:Is there any way to add custom headers on HTTP headers to Rackspace Cloudfiles?

2017-02-02 Thread Andrew Phillips 

Er, that should have been "user@..." - sorry!

ap

 Original Message 
Subject: Re:Is there any way to add custom headers on HTTP headers to 
Rackspace Cloudfiles?

Date: 2017-02-02 15:59
From: Andrew Phillips 
To: us...@jclouds.apache.org
Copy: jwolf...@gmail.com

[Forwarding Jeff's email to users@. @Jeff: notifications@ is just for 
automated emails.]


"In order to troubleshoot some issues downloading objects from Rackspace 
Cloudfiles, they are requesting that I add a "x-trans-id" header to my 
API requests.  Is there any way to instruct the jclouds client library 
to do this for me?  Seems like it should be straightforward to insert a 
custom HTTP header, but I can't find any way to do it."