Here's a summary of the decisions we have made so far and the implementation status.
As Martin probably mentioned, we're repurposing a good chunk of the infrastructure used by Ubuntu One for HTTP Uploads and Downloads. This service is relatively and intentionally simple and has no knowledge about the other parts of the Click Packages infrastructure. The core of this service is a custom Django/Twisted hybrid that efficiently streams down files from a storage backend without tying up a thread, but apart from that is pretty plain Django. [DONE] For Ubuntu One, the storage backend used is Amazon S3, and files are historically stored deflated, which means they can be served with gzip or deflate compression pretty trivially. For the Click Packages Upload/Download service (a.k.a. Click Up/Down), we've made the decision to not compress those files and instead store them as-is. Our understanding is that the files will already be compressed with zip or similar, so compressing them for storage and transfer would yield pretty minimal savings. Additionally, we're switching the storage backend to OpenStack's Swift. We wrote a fairly minimal Swift/Keystone wrapper around our core library that talks to S3, such that we can trivially switch between S3 and Swift if needed. [DONE] The download service will support Range requests, so resumable downloads should be possible to implement on the client side without much effort. [DONE] Public and Private downloads will be served by different subdomains, public.apps.ubuntu.com and private.apps.ubuntu.com respectively. Since all packages will be stored in the same place, there will be some state kept in Click Up/Down to prevent private packages from being fetched via the public endpoint. [IN PROGRESS] Public/Free downloads will be heavily cached at various levels, and will reuse pretty much the same setup as Ubuntu One public downloads. This includes a caching service in front of the application server, using consistent hashing such that the files are not cached on more than one caching server, and another caching service behind the application server, for caching downloads from the storage backend. [DONE] At the moment our plan for authenticated downloads is to use the SSO API for OAuth validation, which means downloads will require a valid SSO OAuth token. [DONE] The validation of access to restricted downloads is still to be defined and may end up involving an API call to the Software Center service. [IN PROGRESS] For uploads, the flow involves a developer registering the package with the Software Center service, and subsequently providing the actual packages to the Click Up/Down service via a POST request. [IN PROGRESS] Since the Software Center service and the Click Up/Down service are completely separate and don't share any data, we need a way to restrict uploads to authenticated users of the Software Center service. We've settled on using HMAC-SHA256 signed requests for this, with a shared private key between Software Center and Click Up/Down. All POST fields but the packages themselves will be part of the HMAC digest and thus validated upon reception by the Click Up/Down service. [IN PROGRESS] After being received by the Click Up/Down service, the packages will default to being in an 'Unpublished' state. [DONE] Upon (automated) review in the Software Center service, an API call will be made from the Software Center service to flip the state to 'Published', at which point the package will be immediately available for download. This API call will be HMAC-signed too. [IN PROGRESS] HTTPS will be required for all requests, both for uploads and downloads. HTTP requests will be unconditionally redirected to HTTPS. [DONE] We do not plan to encrypt internal requests (eg, between the frontend and the caching servers, and between caching servers and the application server) for the time being, though this could be revisited if necessary to ensure end-to-end integrity of the package. [DONE] Traffic between the backend caching server and Swift should use HTTPS. [IN PROGRESS] We do not plan to employ any encryption or validation of the package when stored in Swift and in the disk cache of caching servers, but this could also be revisited if necessary to ensure the integrity of the package. [DONE] During uploads from Click Up/Down to Swift we provide an MD5 checksum to Swift, which as part of it's API ensures that the upload request fails if the received file does not match the provided MD5 checksum. [DONE] A Juju stack comprising the whole infrastructure, including the Postgres database for keeping Published/Unpublished state and the mapping between filenames and Swift object ids will be created. [DONE] This infrastructure will be deployed to Canonical's private OpenStack infrastructure (a.k.a. 'prodstack'). [IN PROGRESS] We expect a staging deployment of this to be completed in the next few days. A demo environment of the items marked as DONE can be provided upon request. It takes around 40min to deploy it from scratch into Canonical's private OpenStack infrastructure for developers (a.k.a. 'canonistack'). Feel free to ask any questions and suggest improvements where you see fit. I'm particularly interested in feedback about security and integrity issues. -- Sidnei Make the most of Ubuntu with Ubuntu One http://one.ubuntu.com
-- Mailing list: https://launchpad.net/~ubuntu-appstore-developers Post to : [email protected] Unsubscribe : https://launchpad.net/~ubuntu-appstore-developers More help : https://help.launchpad.net/ListHelp
