Back from 10 messages ago, yes, cephs rados gateway also provides S3 compatible presigned URL capability. On Jun 27, 2013 9:04 PM, "Thomas O'Dowd" <tpod...@cloudian.com> wrote:
> Hi Min/Jessica, > > I attached an image to that issue to show what what my browser is > showing. > > https://issues.apache.org/jira/browse/CLOUDSTACK-3220 > > Tom. > > On Fri, 2013-06-28 at 09:45 +0900, Thomas O'Dowd wrote: > > Hi Min, > > > > Yes. I'll try it again today to check again but when I added Amazon S3 > > as the S3 secondary storage and uploaded a template, I was not shown the > > "download template" link. However - for Cloudian S3, I am shown it so > > I'm wondering why. > > > > Tom. > > > > On Fri, 2013-06-28 at 00:26 +0000, Min Chen wrote: > > > Hi Tom, > > > > > > Are you saying that you cannot see a Download Template button from > UI > > > when Amazon S3 is added as secondary storage? I only tested with RiakCS > > > and Cloudian, so didn't see this issue. But I am CC Jessica her to > confirm > > > what special handling is done in UI to enable/disable a button from UI. > > > > > > Thanks > > > -min > > > > > > On 6/27/13 5:23 PM, "Thomas O'Dowd" <tpod...@cloudian.com> wrote: > > > > > > >Hi Min, > > > > > > > >Can you check this bug? I'm trying to test this feature for Amazon but > > > >having no luck getting the Download template link/button to appear. > > > > > > > >https://issues.apache.org/jira/browse/CLOUDSTACK-3220 > > > > > > > >Thanks, > > > > > > > >Tom. > > > > > > > >On Fri, 2013-06-21 at 17:21 +0000, Min Chen wrote: > > > >> John, > > > >> > > > >> For S3, the api call createEntityExtractUrl is done on management > > > >>server > > > >> side; while for NFS secondary storage, if the implementation of > > > >> createEntityExtractUrl will involve some code be executed in ssvm to > > > >>copy > > > >> template from the install location to a public accessible web server > > > >> location. > > > >> I don't quite understand some of your comments below. This API is > not > > > >> used to write any information to S3 bucket/directory. This is used > for > > > >> object already existed on S3, and we just provide a URL for user to > > > >> download a template from S3, just like how Amazon provided user a > way to > > > >> user to extract a S3 object through generatePresignedUrl. We can > discuss > > > >> more on this on collaboration conference. > > > >> > > > >> Thanks > > > >> -min > > > >> > > > >> > > > >> > > > >> On 6/21/13 7:25 AM, "John Burwell" <jburw...@basho.com> wrote: > > > >> > > > >> >Min, > > > >> > > > > >> >(I apologize for my belated reply -- I lost track of this draft in > the > > > >> >chaos of the last couple of days.) > > > >> > > > > >> >Upon further review, I think I feel into the confusion between > > > >>management > > > >> >server and ssvm. This code is executing on the management server > side, > > > >> >correct? Based on my "corrected" understanding is correct, I would > > > >>like > > > >> >to amend my thoughts. Namely, I would like to see the driver > > > >>operations > > > >> >pushed out to the SSVM where we can use the stream. As I think > about > > > >>it, > > > >> >the management server should not need to interact with the driver. > > > >> >Simply yard up the DataStore attributes + details map and other > extract > > > >> >parameters, and send them to the SSVM. Using this information, > the S3 > > > >> >driver could open a stream to write the template out to the > > > >> >bucket/directory. I recognize it changes the protocol between the > > > >> >management server and SSVM, but it simply both sides of the > operation > > > >>by > > > >> >allowing the DataStore information to be treated opaquely until it > is > > > >> >consumed by the driver to execute the write operation. I also > > > >>recognize > > > >> >that we may a little late in the cycle to address it for 4.2, and > it > > > >>may > > > >> >need to be part of the 4.3 enhancements. > > > >> > > > > >> >Thanks, > > > >> >-John > > > >> > > > > >> >On Jun 18, 2013, at 3:55 PM, Min Chen <min.c...@citrix.com> wrote: > > > >> > > > > >> >> John, > > > >> >> In that case, how do we keep backward compatibility of > > > >>extractTemplate > > > >> >> api, which requires a URL in the response? > > > >> >> > > > >> >> Thanks > > > >> >> -min > > > >> >> > > > >> >> On 6/18/13 11:53 AM, "John Burwell" <jburw...@basho.com> wrote: > > > >> >> > > > >> >>> Min, > > > >> >>> > > > >> >>> Looking through the code, I think we can simplify driver > operation > > > >>and > > > >> >>> increase robustness by changing > > > >> >>>ImageStoreDriver#createEntityExtractUrl() > > > >> >>> : String to ImageStoreDriver#readEntity(Š) : InputStream. My > first > > > >> >>> concern with the current implementation is that it circumvents > any > > > >> >>> connection pooling/resource management underlying client > libraries > > > >> >>> provide. I/O streams provide a higher-level abstraction that > allows > > > >> >>> drivers to provide the orchestration components with actual > > > >>resources > > > >> >>> rather String references. Second, the current interface seems > to > > > >> >>>appears > > > >> >>> to assume that an http/https URL will be returned. With I/O > > > >>streams, > > > >> >>>we > > > >> >>> can support any client library capable of using the standard I/O > > > >> >>> framework -- enabling us to support other protocols for > downloading > > > >> >>> templates in the future (e.g. RBD, local filesystem, NBD, etc). > > > >> >>> > > > >> >>> Thanks, > > > >> >>> -John > > > >> >>> > > > >> >>> On Jun 18, 2013, at 1:11 PM, Min Chen <min.c...@citrix.com> > wrote: > > > >> >>> > > > >> >>>> A new version of using generatePresignedUrl in > > > >>S3ImageStoreDriverImpl > > > >> >>>>is > > > >> >>>> checked into object_store. > > > >> >>>> > > > >> >>>> THanks > > > >> >>>> -min > > > >> >>>> > > > >> >>>> On 6/18/13 8:29 AM, "Min Chen" <min.c...@citrix.com> wrote: > > > >> >>>> > > > >> >>>>> Yes, current code is in > > > >> >>>>>S3ImageStoreDriverImpl.createEntityExtractUrl, > > > >> >>>>> which has a security issue mentioned in CLOUDSTACK-3030. I am > > > >>going > > > >> >>>>>to > > > >> >>>>> change it to use generatePresignedUrl api from AWS S3 api. > > > >> >>>>> > > > >> >>>>> Thanks > > > >> >>>>> -min > > > >> >>>>> > > > >> >>>>> From: John Burwell <jburw...@basho.com<mailto: > jburw...@basho.com>> > > > >> >>>>> Date: Tuesday, June 18, 2013 8:07 AM > > > >> >>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com > >> > > > >> >>>>> Cc: Thomas O'Dowd > > > >> >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, > > > >> >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" > > > >> >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > > > >> >>>>> Subject: Re: Query String Request Authentication(QSRA) > support by > > > >>S3 > > > >> >>>>> providers > > > >> >>>>> > > > >> >>>>> Min, > > > >> >>>>> > > > >> >>>>> Is the code checked into the object_store branch? If so, > which > > > >>lines > > > >> >>>>> in > > > >> >>>>> S3TemplateDownloader? > > > >> >>>>> > > > >> >>>>> Thanks, > > > >> >>>>> -John > > > >> >>>>> > > > >> >>>>> On Jun 18, 2013, at 12:39 AM, Min Chen > > > >> >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: > > > >> >>>>> > > > >> >>>>> Hi John, > > > >> >>>>> > > > >> >>>>> This is regarding extractTemplate api, where for extractable > > > >> >>>>>template, > > > >> >>>>> users can click "Download Template" button from UI to get a > http > > > >>url > > > >> >>>>>to > > > >> >>>>> download the template already stored at S3 without providing > S3 > > > >> >>>>> credentials. In 4.1, we don't have this issue, since the URL > > > >>returned > > > >> >>>>> is > > > >> >>>>> the public web server location hosted in ssvm, and in 4.2, we > are > > > >> >>>>> returning URL pointing to s3 object. Without setting ACL to > the S3 > > > >> >>>>> object, user cannot directly click the URL returned from > > > >> >>>>> extractTemplate > > > >> >>>>> api to download the template without providing credentials. By > > > >> >>>>>reading > > > >> >>>>> the AWS SDK doc today, I ran across the following API that I > may > > > >>be > > > >> >>>>> able > > > >> >>>>> to use for this purpose: > > > >> >>>>> > > > >> >>>>> > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>URL< > http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-ext > > > >>>>>>>er > > > >> >>>>>na > > > >> >>>>> l= > > > >> >>>>> true> > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>generatePresignedUrl< > http://docs.aws.amazon.com/AWSJavaSDK/latest/ja > > > >>>>>>>va > > > >> >>>>>do > > > >> >>>>> c/ > > > >> >>>>> > > > >> >>>>> > > > >> > > > > >>>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%2 > > > >>>>>>>8j > > > >> >>>>>av > > > >> >>>>> a. > > > >> >>>>> > > > >> >>>>> > > > >> > > > > >>>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.H > > > >>>>>>>tt > > > >> >>>>>pM > > > >> >>>>> et > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>hod%29>(String< > http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Str > > > >>>>>>>in > > > >> >>>>>g. > > > >> >>>>> ht > > > >> >>>>> ml?is-external=true> bucketName, > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>String< > http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html > > > >>>>>>>?i > > > >> >>>>>s- > > > >> >>>>> ex > > > >> >>>>> ternal=true> key, > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>Date< > http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is- > > > >>>>>>>ex > > > >> >>>>>te > > > >> >>>>> rn > > > >> >>>>> al=true> expiration, > > > >> >>>>> > > > >> >>>>> > > > >> > > > >>>>>>>HttpMethod< > http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/ > > > >>>>>>>am > > > >> >>>>>az > > > >> >>>>> on > > > >> >>>>> aws/HttpMethod.html> method) > > > >> >>>>> Returns a pre-signed URL for accessing an Amazon S3 > > > >>resource. > > > >> >>>>> > > > >> >>>>> This is along the same line as QSRA mentioned by Tom, by > wrapped > > > >>in > > > >> >>>>> AmazonS3Client for easy consumption. By using this method, I > think > > > >> >>>>> that I > > > >> >>>>> don't need to change ACL of S3 object to open a security hole. > > > >> >>>>> > > > >> >>>>> Thanks > > > >> >>>>> -min > > > >> >>>>> > > > >> >>>>> From: John Burwell <jburw...@basho.com<mailto: > jburw...@basho.com>> > > > >> >>>>> Date: Monday, June 17, 2013 7:38 PM > > > >> >>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com > >> > > > >> >>>>> Cc: Thomas O'Dowd > > > >> >>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>, > > > >> >>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>" > > > >> >>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>> > > > >> >>>>> Subject: Re: Query String Request Authentication(QSRA) > support by > > > >>S3 > > > >> >>>>> providers > > > >> >>>>> > > > >> >>>>> Min, > > > >> >>>>> > > > >> >>>>> Why are we mucking with ACLs at all? The best security > practice > > > >> >>>>>would > > > >> >>>>> be > > > >> >>>>> to create a bucket for CloudStack's use and assign it a > dedicated > > > >> >>>>> access > > > >> >>>>> key and secret key pair with read/write access only to that > > > >>bucket. > > > >> >>>>> Requiring an administrative account to an object store opens > an > > > >> >>>>> unnecessarily large attack surface. Therefore, as > implemented in > > > >> >>>>>4.1, > > > >> >>>>> we > > > >> >>>>> should defer bucket creation, ACL assignment, and credential > > > >>creation > > > >> >>>>> to > > > >> >>>>> the administrator/operator. > > > >> >>>>> > > > >> >>>>> Thanks, > > > >> >>>>> -John > > > >> >>>>> > > > >> >>>>> On Jun 17, 2013, at 1:15 PM, Min Chen > > > >> >>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote: > > > >> >>>>> > > > >> >>>>> Tom filed a very good bug for ACL setting change on S3 object > when > > > >> >>>>> users > > > >> >>>>> issue extractTemplate API > > > >> >>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and > his > > > >> >>>>> recommendation of using Query String Request Authentication > (QSRA) > > > >> >>>>> alternative sounds like a right approach to fix this bug. > Before > > > >> >>>>> implementing it, I would like to confirm if QSRA should be > > > >>supported > > > >> >>>>>by > > > >> >>>>> all S3 providers if they claim that they are AWS s3 > compatible. If > > > >> >>>>>so, > > > >> >>>>> we > > > >> >>>>> will make this assumption in our code. Based on Tom, Cloudian > is > > > >> >>>>> supporting it. How about RiakCS, John? > > > >> >>>>> > > > >> >>>>> Thanks > > > >> >>>>> -min > > > >> >>>>> > > > >> >>>>> > > > >> >>>> > > > >> >>> > > > >> >> > > > >> > > > > >> > > > > > > > >-- > > > >Cloudian KK - http://www.cloudian.com/get-started.html > > > >Fancy 100TB of full featured S3 Storage? > > > >Checkout the Cloudian(R) Community Edition! > > > > > > > > > > > -- > Cloudian KK - http://www.cloudian.com/get-started.html > Fancy 100TB of full featured S3 Storage? > Checkout the Cloudian® Community Edition! > >
