Hi folks, I have finished the basic functionality of the Storage app[1]. Below are the next set of functions I'll be working on.
- Support h2 db (support h2 database) - Support content-type storage (Store the content type of the file for better rendering) - Make engines and storage-providers different (engine stores UUIDs, tokens etc where as storage-provider is the actual storage) - Support consumer tokens and validation for consumers - Route with given name - requirement that came from ES team (storage/resource/:uuid/:filename) Another question I have is whether we have a component that deals with token generation. I want to generate tokens for file requests. [1] - https://github.com/dulichan/storage Cheers~ On Fri, Feb 28, 2014 at 9:03 PM, Shani Ranasinghe <[email protected]> wrote: > Hi Chan, > > In SS we have the HDFS browser facility. This is a feature where you can > upload/download files to the HDFS. Please refer > https://docs.wso2.org/display/SS110/HDFS+Browser. Please check if this > fits your requirement. > > Also, as far as UES is concerned, implementing the HDFS UI as a jaggery > app is a roadmap item https://redmine.wso2.com/issues/1744. > > Mutitenancy is yet to be completed and in the current release is not > production recommended. With the introduction of Hadoop 2.x in SS (in a > future release) we are planning to look into possibilities of federating > the file system and thereby address multitenancy. It is yet to be decided. > > > > On Fri, Feb 28, 2014 at 6:33 AM, Chan <[email protected]> wrote: > >> *Simple spec for the Unified Storage proposal * >> The goal of the Unified Storage proposal is to provide a REST API and >> jaggery module wrapper which will provide below features - >> >> - Persist files >> - Get files >> >> >> *REST API* >> * Persist file* >> >> URL - storage/resource/ >> VERB - PUT >> >> Headers >> >> consumer token - Get the token of the provider >> >> consumer user - User that's authenticated to the service provider >> >> Body >> >> File data >> >> Response >> >> Returns id >> >> >> *Get file* >> >> URL - storage/ >> resource >> /:id >> >> VERB - GET >> Attributes >> >> id - The ID obtained from the PUT call >> >> Headers >> >> consumer token - Get the token of the provider >> >> consumer user - User that's authenticated to the service provider >> >> Response >> >> Returns the file >> >> *Jaggery Module* >> A jaggery module will be implemented which wrap the REST API and provide >> a developer friendly javascript api. >> >> var storage = require("storage").storage; >> /* >> To be configured with >> { >> provider-name: sdf >> provider-secret: sdf >> } >> */ >> storage.configure({}); >> var file = new File(); >> >> /* >> persist a file >> meta will contain >> { >> id = id to access the file >> } >> */ >> var meta = storage.put(file); >> >> /* >> request a file by sending the file id >> */ >> var file = storage.get(id); >> >> >> We can improve on the spec by providing security, multi-tenancy >> *Buckets* >> A bucket is a namespace for files or objects. This will be useful for a >> use-case where namespaces for files are required. >> >> *Multi-tenancy* >> There are some limitations in multi-tenancy in HDFS (@Deep or @Shani can >> better explain this). We can implement the tenant isolation in the storage >> app layer. >> >> *Security* >> A token can be requested from storage for a resource before issuing the >> url to a client. A use-case is where store takes a token and append it to >> the url before sending it to the android device. There will be a governance >> model on the time period and invalidation to secure the executable >> resource. Also another layer of security will be implemented so that >> Storage app can trust the Service-Provider (which is Publisher or Store). >> >> Any ideas are welcome. >> >> Cheers~ >> >> >> On Mon, Feb 24, 2014 at 1:23 PM, Chan <[email protected]> wrote: >> >>> @Dilshan as I have mentioned - SS only handles provisioning of storage. >>> The proposed solution is for File Storage (not for provisioning). >>> >>> >>> On Mon, Feb 24, 2014 at 12:51 PM, Dilshan Edirisuriya >>> <[email protected]>wrote: >>> >>>> Hi Chan, >>>> >>>> I think we discussed this few months ago with SS team (Prabath, Deep). >>>> What was the final conclusion? >>>> >>>> Regards, >>>> >>>> Dilshan >>>> >>>> >>>> On Thu, Feb 20, 2014 at 7:49 PM, Chan <[email protected]> wrote: >>>> >>>>> Hi folks, >>>>> In our platform - we don't yet have a solution for storing static >>>>> files in the cloud. A solution that provides a simple service of >>>>> persisting >>>>> a file and getting back a URI. The Enterprise Store team built a storage >>>>> module in the Publisher app that provides persisting of files to a >>>>> database >>>>> and getting back URIs. My proposal is to build a jaggery-app that >>>>> specifically handles file storage with a REST interface extending what the >>>>> ES team has done. >>>>> >>>>> *Use case* >>>>> I will be using the Enterprise Store as an example for the Storage >>>>> App. >>>>> [image: Inline image 1] >>>>> 1:- Developer uploads a file to the publisher app using an entry form >>>>> 2:- Publisher app calls the storage module to contact the storage >>>>> provider >>>>> 3:- Storage module contacts the Storage app with a HTTP PUT call and >>>>> returns the URI of the file persisted. Publisher app will persist this URI >>>>> to the registry along with other information. >>>>> 4:- A client contacts the server requesting an asset page. >>>>> 5:- The server will contact the storage module to request a token. It >>>>> will in turn call the Storage app requesting a token. After retrieving the >>>>> token It will append the token to the URI when rendering the pages server >>>>> side. >>>>> 6:- Browser will contact the Storage app directly when requesting the >>>>> file using HTTP GET. Since the browser is passing a valid token - Storage >>>>> app will return the file in the response. >>>>> >>>>> *Architecture overview* >>>>> Storage app will have service providers. These service providers will >>>>> be apps that require the Storage service from the Storage app. First let's >>>>> start off with the API. Storage app will provide an HTTP interface as >>>>> below >>>>> GET >>>>> >>>>> GET interface is used to request a file. The request can be >>>>> for permanent links or token based links. >>>>> >>>>> PUT >>>>> >>>>> PUT can only called by a registered service provider. A token >>>>> (Provider token) has to be passed to validate the service provider. It >>>>> will >>>>> also allow options of - security, permanent links etc. >>>>> >>>>> DELETE >>>>> >>>>> URI is passed to the DELETE API as well as the provider token. Used to >>>>> delete files. >>>>> >>>>> >>>>> Next is what the real storage options are. For the first cut we can >>>>> have the storage app connected to a database. Afterwards we can move to >>>>> HDFS type storage or Cassandra perhaps. >>>>> >>>>> As with Multi-tenancy - we can handle multi-tenancy the same way we >>>>> handle it for jaggery apps (SaaS multi-tenancy). >>>>> >>>>> Ultimate objective of this proposal is to build a cloud storage >>>>> service like Amazon S3, Google Cloud storage [1]. WDYT guys? >>>>> >>>>> [1] - >>>>> http://www.quora.com/Amazon-S3/What-are-the-best-alternatives-to-S3-and-what-are-the-pros-and-cons >>>>> >>>>> Cheers~ >>>>> -- >>>>> Chan (Dulitha Wijewantha) >>>>> Software Engineer - Mobile Development >>>>> WSO2Mobile >>>>> Lean.Enterprise.Mobileware >>>>> * ~Email [email protected] <[email protected]>* >>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>> >>>>> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >>>>> >>>>> * ~Blog blog.dulithawijewantha.com >>>>> <http://dulichan.github.io/chan/>* >>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dilshan Edirisuriya >>>> Senior Software Engineer - WSO2Mobile >>>> Mob: + 94 772245502 >>>> http://wso2mobile.com/ >>>> >>> >>> >>> >>> -- >>> Chan (Dulitha Wijewantha) >>> Software Engineer - Mobile Development >>> WSO2Mobile >>> Lean.Enterprise.Mobileware >>> * ~Email [email protected] <[email protected]>* >>> * ~Mobile +94712112165 <%2B94712112165>* >>> >>> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >>> >>> * ~Blog blog.dulithawijewantha.com >>> <http://dulichan.github.io/chan/>* >>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>> >> >> >> >> -- >> Chan (Dulitha Wijewantha) >> Software Engineer - Mobile Development >> WSO2Mobile >> Lean.Enterprise.Mobileware >> * ~Email [email protected] <[email protected]>* >> * ~Mobile +94712112165 <%2B94712112165>* >> >> * ~Website dulithawijewantha.com <http://dulithawijewantha.com/> * >> >> * ~Blog blog.dulithawijewantha.com >> <http://dulichan.github.io/chan/>* >> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >> > > > > -- > Thanks and Regards > *,Shani Ranasinghe* > Software Engineer > WSO2 Inc.; http://wso2.com > lean.enterprise.middleware > > mobile: +94 77 2273555 > linked in: lk.linkedin.com/pub/shani-ranasinghe/34/111/ab > -- Chan (Dulitha Wijewantha) Software Engineer - Mobile Development WSO2Mobile Lean.Enterprise.Mobileware * ~Email [email protected] <[email protected]>* * ~Mobile +94712112165* * ~Website dulitha.me <http://dulitha.me>* * ~Twitter @dulitharw <https://twitter.com/dulitharw>* *~SO @chan <http://stackoverflow.com/users/813471/chan>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
