Re: Example of setting Blob ACL using S3 provider?
although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used One is the API-specific client, the other the provider-specific client (see [1]). I.e. S3Client should work with all S3-compatible providers, whereas the AWSS3Client could expose extra functionality that *only* Amazon offers. Hope that helps! ap PS: PRs to update outdated docs much appreciated...thanks! [1] http://jclouds.apache.org/start/concepts/
Re: Example of setting Blob ACL using S3 provider?
Thanks, now I understand the difference between APIs and providers better. I had previously thought that I needed to create an S3Client instance via contextBuilder.buildApi(S3Client.class). But it looks like I can get a reference to it from the basic BlobStoreContext instead: BlobStoreContext context = ContextBuilder.newBuilder(s3)buildView(BlobStoreContext.class); S3Client s3Client = context.RestContextS3Client, S3AsyncClientunwrap().getApi(); The only problem is that RestContext is deprecated (in Jclouds 1.6.3), and I'm not sure how to get the the S3Client object using ApiContext instead. *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Fri, Sep 26, 2014 at 2:17 AM, Andrew Phillips andr...@apache.org wrote: although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used One is the API-specific client, the other the provider-specific client (see [1]). I.e. S3Client should work with all S3-compatible providers, whereas the AWSS3Client could expose extra functionality that *only* Amazon offers. Hope that helps! ap PS: PRs to update outdated docs much appreciated...thanks! [1] http://jclouds.apache.org/start/concepts/
Re: Example of setting Blob ACL using S3 provider?
You should be able to create the portable BlobStoreContext and use the unwrapApi method to get the provider specific api you want to use (S3Client or AWSS3Client). El 26/09/2014 17:38, Steve Kingsland steve.kingsl...@opower.com escribió: Thanks, now I understand the difference between APIs and providers better. I had previously thought that I needed to create an S3Client instance via contextBuilder.buildApi(S3Client.class). But it looks like I can get a reference to it from the basic BlobStoreContext instead: BlobStoreContext context = ContextBuilder.newBuilder(s3)buildView(BlobStoreContext.class); S3Client s3Client = context.RestContextS3Client, S3AsyncClientunwrap().getApi(); The only problem is that RestContext is deprecated (in Jclouds 1.6.3), and I'm not sure how to get the the S3Client object using ApiContext instead. *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Fri, Sep 26, 2014 at 2:17 AM, Andrew Phillips andr...@apache.org wrote: although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used One is the API-specific client, the other the provider-specific client (see [1]). I.e. S3Client should work with all S3-compatible providers, whereas the AWSS3Client could expose extra functionality that *only* Amazon offers. Hope that helps! ap PS: PRs to update outdated docs much appreciated...thanks! [1] http://jclouds.apache.org/start/concepts/
Re: Example of setting Blob ACL using S3 provider?
Ah, you mean this: http://jclouds.apache.org/reference/javadoc/1.7.x/org/jclouds/View.html#unwrapApi(java.lang.Class) Looks like it was just added in Jclouds 1.7, which explains why I'm not seeing it locally. :( *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Fri, Sep 26, 2014 at 11:59 AM, Ignasi Barrera n...@apache.org wrote: You should be able to create the portable BlobStoreContext and use the unwrapApi method to get the provider specific api you want to use (S3Client or AWSS3Client). El 26/09/2014 17:38, Steve Kingsland steve.kingsl...@opower.com escribió: Thanks, now I understand the difference between APIs and providers better. I had previously thought that I needed to create an S3Client instance via contextBuilder.buildApi(S3Client.class). But it looks like I can get a reference to it from the basic BlobStoreContext instead: BlobStoreContext context = ContextBuilder.newBuilder(s3)buildView(BlobStoreContext.class); S3Client s3Client = context.RestContextS3Client, S3AsyncClientunwrap().getApi(); The only problem is that RestContext is deprecated (in Jclouds 1.6.3), and I'm not sure how to get the the S3Client object using ApiContext instead. *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Fri, Sep 26, 2014 at 2:17 AM, Andrew Phillips andr...@apache.org wrote: although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used One is the API-specific client, the other the provider-specific client (see [1]). I.e. S3Client should work with all S3-compatible providers, whereas the AWSS3Client could expose extra functionality that *only* Amazon offers. Hope that helps! ap PS: PRs to update outdated docs much appreciated...thanks! [1] http://jclouds.apache.org/start/concepts/
Re: Example of setting Blob ACL using S3 provider?
Ah yes, sorry. IIRC it is deprecated to indicate that it will disappear in future releases. 1.7 introduces the new ApiContext and most providers have been migrated to use that. It should be safe to use it in 1.6 even if it is deprecated. On 26 September 2014 18:12, Steve Kingsland steve.kingsl...@opower.com wrote: Ah, you mean this: http://jclouds.apache.org/reference/javadoc/1.7.x/org/jclouds/View.html#unwrapApi(java.lang.Class) Looks like it was just added in Jclouds 1.7, which explains why I'm not seeing it locally. :( Steve Kingsland Senior Software Engineer Opower We’re hiring! See jobs here On Fri, Sep 26, 2014 at 11:59 AM, Ignasi Barrera n...@apache.org wrote: You should be able to create the portable BlobStoreContext and use the unwrapApi method to get the provider specific api you want to use (S3Client or AWSS3Client). El 26/09/2014 17:38, Steve Kingsland steve.kingsl...@opower.com escribió: Thanks, now I understand the difference between APIs and providers better. I had previously thought that I needed to create an S3Client instance via contextBuilder.buildApi(S3Client.class). But it looks like I can get a reference to it from the basic BlobStoreContext instead: BlobStoreContext context = ContextBuilder.newBuilder(s3)buildView(BlobStoreContext.class); S3Client s3Client = context.RestContextS3Client, S3AsyncClientunwrap().getApi(); The only problem is that RestContext is deprecated (in Jclouds 1.6.3), and I'm not sure how to get the the S3Client object using ApiContext instead. Steve Kingsland Senior Software Engineer Opower We’re hiring! See jobs here On Fri, Sep 26, 2014 at 2:17 AM, Andrew Phillips andr...@apache.org wrote: although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used One is the API-specific client, the other the provider-specific client (see [1]). I.e. S3Client should work with all S3-compatible providers, whereas the AWSS3Client could expose extra functionality that *only* Amazon offers. Hope that helps! ap PS: PRs to update outdated docs much appreciated...thanks! [1] http://jclouds.apache.org/start/concepts/
Re: Example of setting Blob ACL using S3 provider?
Andrew, I'd be happy to create a ticket for using the BlobStore API to set object-level ACLs on the putBlob() call, if you think it's achievable across all providers that Jclouds supports. I won't be able to implement this myself, though, because like I said Jclouds has advanced beyond the point where I'm able to use the latest version, because of the Guava version it depends on (see JCLOUDS-427). I don't totally follow what you're talking about with signed URLs; is this explained somewhere on jclouds.apache.org that I could read more about it, and see if it fits my needs? Also, perhaps I made too big a deal of the ACL issue; it's actually only 1 line of code, and it's working quite nicely for me locally: S3Object s3Object = s3Client.newS3Object(); s3Object.setPayload(payload); s3Object.getMetadata().setKey(objectName); PutObjectOptions putObjectOptions = *PutObjectOptions.Builder.withAcl(CannedAccessPolicy.PUBLIC_READ);* s3Client.putObject(bucketName, s3Object, putObjectOptions); The only catch I struggled with was how to get a handle to the S3Client instance; the docs on the site were out-of date. (I'll submit a pull request to fix them up, although I'm not entirely sure what the difference is between the AWSS3Client the docs are using, and the S3Client that I've used. OK if I change them to use S3Client instead?) Thanks for your help! *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Wed, Sep 24, 2014 at 7:03 PM, Andrew Gaul g...@apache.org wrote: Many providers support ACLs and the jclouds portable abstraction could provide support for a subset of them, mostly the public read and write variants. I opened a JIRA issue for container ACLs: https://issues.apache.org/jira/browse/JCLOUDS-660 Could you open an issue for object ACLs? Could you also research this issue further and potentially implement the desired functionality itself? The portable abstraction is growing and it should provide support for copying blobs in the next major release. In the mean-time, you might consider signed URLs instead of object ACLs. Signed URLs allow your application to vend a time-limited read or write token to a client which then interacts with the object store directly. This approach offers finer-grained permissions and most jclouds providers support this feature. On Wed, Sep 24, 2014 at 02:28:53PM -0400, Steve Kingsland wrote: Thanks Andrew, it now makes sense that ACLs would be a provider-specific feature. Maybe my confusion arose simply because the S3-specific docs [1] were a bit out-of-date, with regards to getting the provider-specific API? (referencing a method which no longer exists) AFA supporting ACLs on the AWSS3PutOptions, I'm afraid that adding this to the latest version of jclouds wouldn't help me at all. I'm stuck on 1.6.3 because of a Guava incompatibility between Jclouds (see JCLOUDS-427) and the version of HBase we're using (see HBASE-9667). But that's fine; instead of passing around a BlobStoreContext in my code, it's a straightforward change to use an S3Client object instead, which gives me access to all the S3-specific features (like ACLs) that I need. [1] http://jclouds.apache.org/guides/aws/ *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Wed, Sep 24, 2014 at 1:10 PM, Andrew Phillips andr...@apache.org wrote: was apparently removed in Jclouds 1.6. I think I can work around this using contextBuilder.buildApi(S3Client.class) and the S3Object class to set the ACL. But then if I have to use an S3-specific API to set an ACL, why not just use the com.amazonaws.services.s3.AmazonS3Client client directly? How is Jclouds actually benefitting me, if I'm using it to code directly to the S3 provider? The challenge here is that ACLs are not something that is supported by all the blobstores jclouds supports, so is not included in the BlobStore abstraction. In other words, coding with ACLs on blobs is implicitly making your code provider-dependent already. In order to support provider-specific features like that, jclouds indeed supports access to the underlying API, as you describe (see [1] for more details). This is indeed not all that different from using the provider client directly, but if the number of provider-specific calls you need to make are small, using jclouds will allow you to move to another provider relatively easily if you can re-implement the provider-specific calls or remove them. A middle-ground approach that jclouds takes for some options is the ability to pass provider-specific options to the abstract interface. E.g. something like: Blob myBlob = ... PutOptions options = AWSS3PutOptions.Builder.
Re: Example of setting Blob ACL using S3 provider?
It always depends on your needs. jclouds provides a portable abstraction layer that allows you to talk to different clouds with the same code. It provides portable Compute, Blobstore and Load Balancing models and APIs you can use. Each cloud, however, has its own semantics and specific features, and although jclouds does a good job in putting them together behind the portable abstractions, some provider specific things have to be done with the provider specific APIs. IT wouldn't make sense to promote every specific feature of every provider to the portable layer. That said, jclouds gives you the freedom to use the portable abstractions or the provider specific APIs. Is up to you. But even when you use the provider specific APIs to do concrete things, jclouds provides menu benefits: * You still have a consistent pattern to use APIs. All apis are constructed the same way and used in a similar fashion. You don't need to learn how a new library works if you want to use the specific API of several providers. * You keep your dependencies clean, as provider specific APIs don't bring new players. * You also benefit from the jclouds built-in HTTP features: smart retry policies, consistent error handling (guess how different APIs can populate similar errors? jclouds does a good job standardizing these behaviors), transparent pagination (you don't have to worry about getting the N page or even if the list is paginated, jclouds does it for you transparently), and more. * It deals with authentication, sessions, expired tokens, etc. In the end it is up to your use case. If you only need to talk with one cloud provider and there exist a library for it, perhaps it is better to just use that library, but jclouds is more than an api aggregator and does a great job also when using the provider specific APIs. HTH! I. On 24 September 2014 17:00, Steve Kingsland steve.kingsl...@opower.com wrote: I'm trying to get code working which sets an ACL on an object when I upload it to the BlobStore. I'm using the Jclouds S3 provider, and the closest documentation I can find is from the Using S3 example code at http://jclouds.apache.org/guides/aws/: // when you need access to s3-specific features, // use the provider-specific context AWSS3Client s3Client = AWSS3Client.class.cast(context.getProviderSpecificContext().getApi()); // make the object world readable String publicReadWriteObjectKey = public-read-write-acl; S3Object object = s3Client.newS3Object(); object.getMetadata().setKey(publicReadWriteObjectKey); object.setPayload(hello world); s3Client.putObject(bucket, object, withAcl(CannedAccessPolicy.PUBLIC_READ)); context.close(); Problem is, the getProviderSpecificContext() method was apparently removed in Jclouds 1.6. I think I can work around this using contextBuilder.buildApi(S3Client.class) and the S3Object class to set the ACL. But then if I have to use an S3-specific API to set an ACL, why not just use the com.amazonaws.services.s3.AmazonS3Client client directly? How is Jclouds actually benefitting me, if I'm using it to code directly to the S3 provider? Steve Kingsland Senior Software Engineer Opower We’re hiring! See jobs here
Re: Example of setting Blob ACL using S3 provider?
was apparently removed in Jclouds 1.6. I think I can work around this using contextBuilder.buildApi(S3Client.class) and the S3Object class to set the ACL. But then if I have to use an S3-specific API to set an ACL, why not just use the com.amazonaws.services.s3.AmazonS3Client client directly? How is Jclouds actually benefitting me, if I'm using it to code directly to the S3 provider? The challenge here is that ACLs are not something that is supported by all the blobstores jclouds supports, so is not included in the BlobStore abstraction. In other words, coding with ACLs on blobs is implicitly making your code provider-dependent already. In order to support provider-specific features like that, jclouds indeed supports access to the underlying API, as you describe (see [1] for more details). This is indeed not all that different from using the provider client directly, but if the number of provider-specific calls you need to make are small, using jclouds will allow you to move to another provider relatively easily if you can re-implement the provider-specific calls or remove them. A middle-ground approach that jclouds takes for some options is the ability to pass provider-specific options to the abstract interface. E.g. something like: Blob myBlob = ... PutOptions options = AWSS3PutOptions.Builder.storageClass(...).otherOption...; blobstore.putBlob(myContainer, myBlob, options); // [2] This is *also* AWS-specific code, but not quite as tied to the specific underlying API. Unfortunately, withAcl is not an option currently supported on AWSS3PutOptions [3] (it *is* supported on PutObjectOptions [4], but that's the options class for the S3-specific call). That should be a relatively easy fix, though - would you be interested in submitting a PR for that? Hope that helps! ap [1] http://jclouds.apache.org/start/concepts/ [2] http://javadocs.jclouds.cloudbees.net/org/jclouds/blobstore/BlobStore.html#putBlob(java.lang.String, org.jclouds.blobstore.domain.Blob, org.jclouds.blobstore.options.PutOptions) [3] http://javadocs.jclouds.cloudbees.net/org/jclouds/aws/s3/blobstore/options/AWSS3PutOptions.html [4] http://javadocs.jclouds.cloudbees.net/org/jclouds/s3/options/PutObjectOptions.html
Re: Example of setting Blob ACL using S3 provider?
Thanks Andrew, it now makes sense that ACLs would be a provider-specific feature. Maybe my confusion arose simply because the S3-specific docs [1] were a bit out-of-date, with regards to getting the provider-specific API? (referencing a method which no longer exists) AFA supporting ACLs on the AWSS3PutOptions, I'm afraid that adding this to the latest version of jclouds wouldn't help me at all. I'm stuck on 1.6.3 because of a Guava incompatibility between Jclouds (see JCLOUDS-427) and the version of HBase we're using (see HBASE-9667). But that's fine; instead of passing around a BlobStoreContext in my code, it's a straightforward change to use an S3Client object instead, which gives me access to all the S3-specific features (like ACLs) that I need. [1] http://jclouds.apache.org/guides/aws/ *Steve Kingsland* Senior Software Engineer *Opower* http://www.opower.com/ *We’re hiring! See jobs here http://www.opower.com/careers* On Wed, Sep 24, 2014 at 1:10 PM, Andrew Phillips andr...@apache.org wrote: was apparently removed in Jclouds 1.6. I think I can work around this using contextBuilder.buildApi(S3Client.class) and the S3Object class to set the ACL. But then if I have to use an S3-specific API to set an ACL, why not just use the com.amazonaws.services.s3.AmazonS3Client client directly? How is Jclouds actually benefitting me, if I'm using it to code directly to the S3 provider? The challenge here is that ACLs are not something that is supported by all the blobstores jclouds supports, so is not included in the BlobStore abstraction. In other words, coding with ACLs on blobs is implicitly making your code provider-dependent already. In order to support provider-specific features like that, jclouds indeed supports access to the underlying API, as you describe (see [1] for more details). This is indeed not all that different from using the provider client directly, but if the number of provider-specific calls you need to make are small, using jclouds will allow you to move to another provider relatively easily if you can re-implement the provider-specific calls or remove them. A middle-ground approach that jclouds takes for some options is the ability to pass provider-specific options to the abstract interface. E.g. something like: Blob myBlob = ... PutOptions options = AWSS3PutOptions.Builder. storageClass(...).otherOption...; blobstore.putBlob(myContainer, myBlob, options); // [2] This is *also* AWS-specific code, but not quite as tied to the specific underlying API. Unfortunately, withAcl is not an option currently supported on AWSS3PutOptions [3] (it *is* supported on PutObjectOptions [4], but that's the options class for the S3-specific call). That should be a relatively easy fix, though - would you be interested in submitting a PR for that? Hope that helps! ap [1] http://jclouds.apache.org/start/concepts/ [2] http://javadocs.jclouds.cloudbees.net/org/jclouds/ blobstore/BlobStore.html#putBlob(java.lang.String, org.jclouds.blobstore.domain.Blob, org.jclouds.blobstore.options. PutOptions) [3] http://javadocs.jclouds.cloudbees.net/org/jclouds/aws/ s3/blobstore/options/AWSS3PutOptions.html [4] http://javadocs.jclouds.cloudbees.net/org/jclouds/s3/ options/PutObjectOptions.html
