John,
For S3, the api call createEntityExtractUrl is done on management server
side; while for NFS secondary storage, if the implementation of
createEntityExtractUrl will involve some code be executed in ssvm to copy
template from the install location to a public accessible web server
location.
I don't quite understand some of your comments below. This API is not
used to write any information to S3 bucket/directory. This is used for
object already existed on S3, and we just provide a URL for user to
download a template from S3, just like how Amazon provided user a way to
user to extract a S3 object through generatePresignedUrl. We can discuss
more on this on collaboration conference.
Thanks
-min
On 6/21/13 7:25 AM, "John Burwell" <jburw...@basho.com> wrote:
>Min,
>
>(I apologize for my belated reply -- I lost track of this draft in the
>chaos of the last couple of days.)
>
>Upon further review, I think I feel into the confusion between management
>server and ssvm. This code is executing on the management server side,
>correct? Based on my "corrected" understanding is correct, I would like
>to amend my thoughts. Namely, I would like to see the driver operations
>pushed out to the SSVM where we can use the stream. As I think about it,
>the management server should not need to interact with the driver.
>Simply yard up the DataStore attributes + details map and other extract
>parameters, and send them to the SSVM. Using this information, the S3
>driver could open a stream to write the template out to the
>bucket/directory. I recognize it changes the protocol between the
>management server and SSVM, but it simply both sides of the operation by
>allowing the DataStore information to be treated opaquely until it is
>consumed by the driver to execute the write operation. I also recognize
>that we may a little late in the cycle to address it for 4.2, and it may
>need to be part of the 4.3 enhancements.
>
>Thanks,
>-John
>
>On Jun 18, 2013, at 3:55 PM, Min Chen <min.c...@citrix.com> wrote:
>
>> John,
>> In that case, how do we keep backward compatibility of extractTemplate
>> api, which requires a URL in the response?
>>
>> Thanks
>> -min
>>
>> On 6/18/13 11:53 AM, "John Burwell" <jburw...@basho.com> wrote:
>>
>>> Min,
>>>
>>> Looking through the code, I think we can simplify driver operation and
>>> increase robustness by changing
>>>ImageStoreDriver#createEntityExtractUrl()
>>> : String to ImageStoreDriver#readEntity(Š) : InputStream. My first
>>> concern with the current implementation is that it circumvents any
>>> connection pooling/resource management underlying client libraries
>>> provide. I/O streams provide a higher-level abstraction that allows
>>> drivers to provide the orchestration components with actual resources
>>> rather String references. Second, the current interface seems to
>>>appears
>>> to assume that an http/https URL will be returned. With I/O streams,
>>>we
>>> can support any client library capable of using the standard I/O
>>> framework -- enabling us to support other protocols for downloading
>>> templates in the future (e.g. RBD, local filesystem, NBD, etc).
>>>
>>> Thanks,
>>> -John
>>>
>>> On Jun 18, 2013, at 1:11 PM, Min Chen <min.c...@citrix.com> wrote:
>>>
>>>> A new version of using generatePresignedUrl in S3ImageStoreDriverImpl
>>>>is
>>>> checked into object_store.
>>>>
>>>> THanks
>>>> -min
>>>>
>>>> On 6/18/13 8:29 AM, "Min Chen" <min.c...@citrix.com> wrote:
>>>>
>>>>> Yes, current code is in
>>>>>S3ImageStoreDriverImpl.createEntityExtractUrl,
>>>>> which has a security issue mentioned in CLOUDSTACK-3030. I am going
>>>>>to
>>>>> change it to use generatePresignedUrl api from AWS S3 api.
>>>>>
>>>>> Thanks
>>>>> -min
>>>>>
>>>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>>
>>>>> Date: Tuesday, June 18, 2013 8:07 AM
>>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>>
>>>>> Cc: Thomas O'Dowd
>>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>,
>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>>>>> Subject: Re: Query String Request Authentication(QSRA) support by S3
>>>>> providers
>>>>>
>>>>> Min,
>>>>>
>>>>> Is the code checked into the object_store branch? If so, which lines
>>>>> in
>>>>> S3TemplateDownloader?
>>>>>
>>>>> Thanks,
>>>>> -John
>>>>>
>>>>> On Jun 18, 2013, at 12:39 AM, Min Chen
>>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote:
>>>>>
>>>>> Hi John,
>>>>>
>>>>> This is regarding extractTemplate api, where for extractable
>>>>>template,
>>>>> users can click "Download Template" button from UI to get a http url
>>>>>to
>>>>> download the template already stored at S3 without providing S3
>>>>> credentials. In 4.1, we don't have this issue, since the URL returned
>>>>> is
>>>>> the public web server location hosted in ssvm, and in 4.2, we are
>>>>> returning URL pointing to s3 object. Without setting ACL to the S3
>>>>> object, user cannot directly click the URL returned from
>>>>> extractTemplate
>>>>> api to download the template without providing credentials. By
>>>>>reading
>>>>> the AWS SDK doc today, I ran across the following API that I may be
>>>>> able
>>>>> to use for this purpose:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>URL<http://java.sun.com/j2se/1.5.0/docs/api/java/net/URL.html?is-exter
>>>>>na
>>>>> l=
>>>>> true>
>>>>>
>>>>>
>>>>>generatePresignedUrl<http://docs.aws.amazon.com/AWSJavaSDK/latest/java
>>>>>do
>>>>> c/
>>>>>
>>>>>
>>>>>com/amazonaws/services/s3/AmazonS3Client.html#generatePresignedUrl%28j
>>>>>av
>>>>> a.
>>>>>
>>>>>
>>>>>lang.String,%20java.lang.String,%20java.util.Date,%20com.amazonaws.Htt
>>>>>pM
>>>>> et
>>>>>
>>>>>
>>>>>hod%29>(String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Strin
>>>>>g.
>>>>> ht
>>>>> ml?is-external=true> bucketName,
>>>>>
>>>>>
>>>>>String<http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html?i
>>>>>s-
>>>>> ex
>>>>> ternal=true> key,
>>>>>
>>>>>
>>>>>Date<http://java.sun.com/j2se/1.5.0/docs/api/java/util/Date.html?is-ex
>>>>>te
>>>>> rn
>>>>> al=true> expiration,
>>>>>
>>>>>
>>>>>HttpMethod<http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/am
>>>>>az
>>>>> on
>>>>> aws/HttpMethod.html> method)
>>>>> Returns a pre-signed URL for accessing an Amazon S3 resource.
>>>>>
>>>>> This is along the same line as QSRA mentioned by Tom, by wrapped in
>>>>> AmazonS3Client for easy consumption. By using this method, I think
>>>>> that I
>>>>> don't need to change ACL of S3 object to open a security hole.
>>>>>
>>>>> Thanks
>>>>> -min
>>>>>
>>>>> From: John Burwell <jburw...@basho.com<mailto:jburw...@basho.com>>
>>>>> Date: Monday, June 17, 2013 7:38 PM
>>>>> To: Min Chen <min.c...@citrix.com<mailto:min.c...@citrix.com>>
>>>>> Cc: Thomas O'Dowd
>>>>><tpod...@cloudian.com<mailto:tpod...@cloudian.com>>,
>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>"
>>>>> <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>>>>> Subject: Re: Query String Request Authentication(QSRA) support by S3
>>>>> providers
>>>>>
>>>>> Min,
>>>>>
>>>>> Why are we mucking with ACLs at all? The best security practice
>>>>>would
>>>>> be
>>>>> to create a bucket for CloudStack's use and assign it a dedicated
>>>>> access
>>>>> key and secret key pair with read/write access only to that bucket.
>>>>> Requiring an administrative account to an object store opens an
>>>>> unnecessarily large attack surface. Therefore, as implemented in
>>>>>4.1,
>>>>> we
>>>>> should defer bucket creation, ACL assignment, and credential creation
>>>>> to
>>>>> the administrator/operator.
>>>>>
>>>>> Thanks,
>>>>> -John
>>>>>
>>>>> On Jun 17, 2013, at 1:15 PM, Min Chen
>>>>> <min.c...@citrix.com<mailto:min.c...@citrix.com>> wrote:
>>>>>
>>>>> Tom filed a very good bug for ACL setting change on S3 object when
>>>>> users
>>>>> issue extractTemplate API
>>>>> (https://issues.apache.org/jira/browse/CLOUDSTACK-3030), and his
>>>>> recommendation of using Query String Request Authentication (QSRA)
>>>>> alternative sounds like a right approach to fix this bug. Before
>>>>> implementing it, I would like to confirm if QSRA should be supported
>>>>>by
>>>>> all S3 providers if they claim that they are AWS s3 compatible. If
>>>>>so,
>>>>> we
>>>>> will make this assumption in our code. Based on Tom, Cloudian is
>>>>> supporting it. How about RiakCS, John?
>>>>>
>>>>> Thanks
>>>>> -min
>>>>>
>>>>>
>>>>
>>>
>>
>
