Re: Signature v4 support for non amazon S3

[Ranjith R]

With sigv4, we see that we are doing single chunk upload with signed
payload.  We also noticed that the data is read twice (once for calculating
the hash and once for actual transfer).  While reading the sig v4
documentation at
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
I saw that there is a unsigned payload option.   Is there a way to use
unsigned payload from jclouds to avoid this double read?

Thanks,
Ranjith

On Thu, Jun 21, 2018 at 8:33 PM Ignasi Barrera  wrote:

> Yeah! :)
>
>
> On 21 June 2018 at 16:41, Ranjith R  wrote:
>
>> Thanks Ignasi.  That worked.
>>
>> Thanks,
>> Ranjith
>>
>> On Thu, Jun 21, 2018 at 4:03 PM Ignasi Barrera  wrote:
>>
>>> I haven't tried it, but you should be able to define a Guice module that
>>> extends the default S3 module and overrides the request signer
>>> configuration. Then you can pass that one to the list of modules you pass
>>> when creating the context:
>>>
>>> @ConfiguresHttpApipublic static class S3V4SignerModule extends 
>>> S3HttpApiModule {
>>>@Override
>>>protected void bindRequestSigner() {
>>>   
>>> bind(RequestAuthorizeSignature.class).to(RequestAuthorizeSignatureV4.class).in(Scopes.SINGLETON);
>>>}
>>> }
>>> public static void main(String[] args) {
>>>ContextBuilder.newBuilder("s3")
>>>   ...
>>>   .modules(ImmutableSet.of(new S3V4SignerModule(), ...))
>>>   ...
>>> }
>>>
>>> ​
>>>
>>> Make sure you annotate the custom module with "@ConfiguresHttpApi".
>>> Can you try this?
>>>
>>>
>>>
>>> I.
>>>
>>>
>>> On 21 June 2018 at 11:58, Ranjith R  wrote:
>>>
>>>> I was looking at https://issues.apache.org/jira/browse/JCLOUDS-480 and
>>>> it talks about the default signing for AWS being v4 and other s3 clones
>>>> being v2.  I just want to know if I can use v4 for a s3 clone?  Is there
>>>> any example that I can look at?
>>>>
>>>> Thanks,
>>>> Ranjith
>>>>
>>>> On Mon, Jun 18, 2018 at 7:21 PM Ranjith R  wrote:
>>>>
>>>>> Hi All - I know signature v4 signing is implemented for Amazon S3
>>>>> (aws-s3). Just wanted to know if I can use v4 signing for a non amazon
>>>>> cloud which supports S3 API and sigV4  (s3).  If it does, what changes
>>>>> should be done from the client side?
>>>>>
>>>>> Thanks,
>>>>> Ranjith
>>>>>
>>>>
>>>
>

Re: Signature v4 support for non amazon S3

[Ranjith R]

Thanks Ignasi.  That worked.

Thanks,
Ranjith

On Thu, Jun 21, 2018 at 4:03 PM Ignasi Barrera  wrote:

> I haven't tried it, but you should be able to define a Guice module that
> extends the default S3 module and overrides the request signer
> configuration. Then you can pass that one to the list of modules you pass
> when creating the context:
>
> @ConfiguresHttpApipublic static class S3V4SignerModule extends 
> S3HttpApiModule {
>@Override
>protected void bindRequestSigner() {
>   
> bind(RequestAuthorizeSignature.class).to(RequestAuthorizeSignatureV4.class).in(Scopes.SINGLETON);
>}
> }
> public static void main(String[] args) {
>ContextBuilder.newBuilder("s3")
>   ...
>   .modules(ImmutableSet.of(new S3V4SignerModule(), ...))
>   ...
> }
>
> ​
>
> Make sure you annotate the custom module with "@ConfiguresHttpApi".
> Can you try this?
>
>
>
> I.
>
>
> On 21 June 2018 at 11:58, Ranjith R  wrote:
>
>> I was looking at https://issues.apache.org/jira/browse/JCLOUDS-480 and
>> it talks about the default signing for AWS being v4 and other s3 clones
>> being v2.  I just want to know if I can use v4 for a s3 clone?  Is there
>> any example that I can look at?
>>
>> Thanks,
>> Ranjith
>>
>> On Mon, Jun 18, 2018 at 7:21 PM Ranjith R  wrote:
>>
>>> Hi All - I know signature v4 signing is implemented for Amazon S3
>>> (aws-s3). Just wanted to know if I can use v4 signing for a non amazon
>>> cloud which supports S3 API and sigV4  (s3).  If it does, what changes
>>> should be done from the client side?
>>>
>>> Thanks,
>>> Ranjith
>>>
>>
>

Re: Signature v4 support for non amazon S3

[Ranjith R]

I was looking at https://issues.apache.org/jira/browse/JCLOUDS-480 and it
talks about the default signing for AWS being v4 and other s3 clones being
v2.  I just want to know if I can use v4 for a s3 clone?  Is there any
example that I can look at?

Thanks,
Ranjith

On Mon, Jun 18, 2018 at 7:21 PM Ranjith R  wrote:

> Hi All - I know signature v4 signing is implemented for Amazon S3
> (aws-s3). Just wanted to know if I can use v4 signing for a non amazon
> cloud which supports S3 API and sigV4  (s3).  If it does, what changes
> should be done from the client side?
>
> Thanks,
> Ranjith
>

Signature v4 support for non amazon S3

[Ranjith R]

Hi All - I know signature v4 signing is implemented for Amazon S3 (aws-s3).
Just wanted to know if I can use v4 signing for a non amazon cloud which
supports S3 API and sigV4  (s3).  If it does, what changes should be done
from the client side?

Thanks,
Ranjith

Re: Need to reset an http header in Jcloud.

[Ranjith R]

That sounds like the problem in

https://mail-archives.apache.org/mod_mbox/jclouds-user/201601.mbox/%3c7fb39cd6f35fba4a84437e6707bd6f59150d5...@g9w0757.americas.hpqcorp.net%3E

Check the responses, there was suggestion to use a different http client
and that worked.

On Thu, Mar 1, 2018 at 8:00 PM, Pratheesh 
wrote:

> Hi,
>
>
>
> I am using Jcloud 1.9.1 to connect to a custom S3 server. But when Itry to
> upload a file to this server, I am getting following error - –“ HTTP/1.1
> 500 Internal Server Error”.
>
> I found issue  because sending the Http header – “Accept: text/html,
> image/gif, image/jpeg, *; q=.2, */*; q=.2” ,server throwing this
> error.  If I modify this to – “Accept: text/plain”, it is working perfectly.
>
> So I would like to know, If there is a way to remove this HttpHeader in
> Jcloud? Please provide some sample, if it has..
>
>
>
> Regards
>
> Pratheesh
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: requested location eu-central-1, which is not in the configured locations

[Ranjith R]

Hi,
   I would also like to know how to use AWS S3 v4 signature for
authentication.  Is there way to specify which signature version to use
while constructing a BlobStoreContext?

Thanks in advance,
Ranjith

On Tue, Mar 8, 2016 at 3:24 PM, Archana C  wrote:

> From the amazon documentation, we understand that the region eu-central-1
> (Frankfurt) supports only AWS signature version 4, and from
> https://issues.apache.org/jira/browse/JCLOUDS-480 we understand the
> jclouds already has the support added for version 4 signature.
>
> What we are trying to figure out is *if* we need to do anything from the
> client side to change the authorization signature.  We were hoping the
> BlobStore APIs might work it used to with out changing the client code,
> however, we see the requests sent are using V2 signature, and the server is
> rejecting them.
>
> 2016-03-08 14:57:57,044 DEBUG [jclouds.wire] [main] >> "Test[\n]"
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> PUT
> https://testcontainer3.s3-eu-central-1.amazonaws.com/file1 HTTP/1.1
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Expect:
> 100-continue
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Host: 
> *testcontainer3.s3-eu-central-1.amazonaws.com
> *
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Date: Tue, 08
> Mar 2016 09:27:50 GMT
> *2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Authorization:
> AWS AKIAISCW6DRRITWR6IWQ:6AndVHQV2w75OXQDq/9sWt37KN0=*
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Content-Type:
> application/unknown
> 2016-03-08 14:57:57,045 DEBUG [jclouds.headers] [main] >> Content-Length: 5
>
> *org.jclouds.http.HttpResponseException: Server rejected operation
> connecting to PUT
> https://testcontainer3.s3-eu-central-1.amazonaws.com/file1
>  HTTP/1.1*
> * at
> org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:118)*
> * at
> org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)*
> * at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)*
> * at
> org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)*
>
> *Any help on how to use the AWS version 4 signature for the blobstore APIs
> will be much appreciated. *
>
> *NB: With the last commit mentioned in the above reply, we do not see the
> region not supported exception anymore. *
>
> *Thanks!*
>
> *Regards*
>
> *Archana*
> 
>
>
> On Tuesday, 8 March 2016, 10:54, Andrew Gaul  wrote:
>
>
> Please test again with the latest master which includes a fix:
>
> https://git-wip-us.apache.org/repos/asf?p=jclouds.git;a=commit;h=c18371a7
>
> On Mon, Mar 07, 2016 at 12:29:59PM +, Archana C wrote:
> > public class App
> > {
> > public static void main( String[] args ) throws IOException
> > {
> > // TODO Auto-generated method stub
> > // TODO Auto-generated method stub
> > String containername = "archanatrial12";
> > String objectname = "object1";
> > String tempFile =
> "/home/archana/Eclipse/trialV42/src/main/java/trialV41/trialV42/result.txt";
> > //int length;
> >
> > // s3.amazonaws.com   s3.eu-central-1.amazonaws.com
> s3-external-1.amazonaws.com
> >
> > BlobStoreContext context =
> ContextBuilder.newBuilder("aws-s3")
> > .credentials("XXX",
> "YYY")
> > .buildView(BlobStoreContext.class);
> >
> > // Access the BlobStore
> > BlobStore blobStore = context.getBlobStore();
> > //Location loc = "us-east-1";
> > Location loc = new
> LocationBuilder().scope(LocationScope.REGION)
> > .id("eu-central-1")
> > .description("region")
> > .build();
> >
> > // Create a Container
> > blobStore.createContainerInLocation(loc, containername);
> >
> > // Create a Blob
> > File input = new
> File("/home/archana/Eclipse/jclouds1/src/main/java/jclouds1/sample.txt");
> > long length = input.length();
> > // Add a Blob
> > Blob blob =
> blobStore.blobBuilder(objectname).payload(Files.asByteSource(input)).contentLength(length)
> > .contentDisposition(objectname).build();
> >
> > // Upload the Blob
> > String eTag = blobStore.putBlob(containername, blob);
> > System.out.println(eTag);}}
> > Error  : requested location eu-central-1, which is not in the configured
> locations
> > Solution to rectify the 

Re: Setting LocationConstraint in createContainerInLocation()

[Ranjith R]

Found a work around, I just had to set some random string to
jclouds.regions property, and after that the location constraint was being
sent to the object store. Only thing is I am seeing a warning that

"WARNING: region xyz not in [abc] " but the object server is getting the
location parameter.


I am not sure if there is a better way.

On Tue, Jan 19, 2016 at 5:12 PM, Ranjith R <ranji...@gmail.com> wrote:

> Hi,
>  I am trying out jclouds against a cloud object store which is
> supporting the S3 APIs.  for the putBucket request, the object store
> accepts the LocationConstraint and creates the bucket based on value
> provided by the user.  However, I see that jclouds is not sending the
> LocationConstraint parameter when the request is against this S3 type
> provider.
>
> For eg, when I try to create a bucket in amazon s3 after setting a
> location, I see this in the logs.
>
> 00:18:21.884 [pc01-00] DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService -
> Sending request 188071658: PUT
> https://testprovisioningcode.s3.amazonaws.com/ HTTP/1.1
> 00:18:21.884 [pc01-00] DEBUG jclouds.wire - >>
> "us-west-1"
> 00:18:21.885 [pc01-00] DEBUG jclouds.headers - >> PUT
> https://testprovisioningcode.s3.amazonaws.com/ HTTP/1.1
> 00:18:21.885 [pc01-00] DEBUG jclouds.headers - >> Host:
> testprovisioningcode.s3.amazonaws.com
>
> When I send the same request against my S3 emulated object store, this is
> the request.
>
> 00:22:10.916 [pc01-00] DEBUG o.j.h.a.ApacheHCHttpCommandExecutorService -
> Sending request 1737215775: PUT http://x.x.98.187/testprovisioningcode
> HTTP/1.1
> 00:22:10.917 [pc01-00] DEBUG jclouds.headers - >> PUT
> http://9.114.98.187/testprovisioningcode HTTP/1.1
> 00:22:10.917 [pc01-00] DEBUG jclouds.headers - >> Date: Tue, 19 Jan 2016
> 05:22:10 GMT
> 00:22:10.917 [pc01-00] DEBUG jclouds.headers - >> Authorization: AWS
> iQU39EfkxHBHGhZWFINP:fhoDEfo8UB5EgSyf1KKeMtkrWXw=
>
>
> Any pointers on how can I specify a location when the provider type is s3
> and not aws-s3?
>
> Thanks,
> Ranjith
>
>
>