Hi Tomek, Some time ago (November 2016 Oakathon IIRC) some people explored a similar concept using AWS (S3) instead of Azure. If you haven’t discussed with them already it may be worth doing so. IIRC Stefan Egli and I believe Michael Duerig were involved and probably some others as well.
-MR On March 1, 2018 at 5:42:07 AM, Tomek Rekawek ([email protected]) wrote: Hi Tommaso, so, the goal is to run the Oak in a cloud, in this case Azure. In order to do this in a scalable way (eg. multiple instances on a single VM, containerized), we need to take care of provisioning the sufficient amount of space for the segmentstore. Mounting the physical SSD/HDD disks (in Azure they’re called “Managed Disks” aka EBS in Amazon) has two drawbacks: * it’s expensive, * it’s complex (each disk is a separate /dev/sdX that has to be formatted, mounted, etc.) The point of the Azure Segment Store is to deal with these two issues, by replacing the need for a local file system space with a remote service, that will be (a) cheaper and (b) easier to provision (as it’ll be configured on the application layer rather than VM layer). Another option would be using the Azure File Storage (which mounts the SMB file system, not the “physical” disk). However, in this case we’d have a remote storage that emulates a local one and SegmentMK doesn’t really expect this. Rather than that it’s better to create a full-fledged remote storage implementation, so we can work out the issues caused by the higher latency, etc. Regards, Tomek -- Tomek Rękawek | Adobe Research | www.adobe.com [email protected] > On 1 Mar 2018, at 11:16, Tommaso Teofili <[email protected]> wrote: > > Hi Tomek, > > While I think it's an interesting feature, I'd be also interested to hear > about the user story behind your prototype. > > Regards, > Tommaso > > > Il giorno gio 1 mar 2018 alle ore 10:31 Tomek Rękawek <[email protected]> > ha scritto: > >> Hello, >> >> I prepared a prototype for the Azure-based Segment Store, which allows to >> persist all the SegmentMK-related resources (segments, journal, manifest, >> etc.) on a remote service, namely the Azure Blob Storage [1]. The whole >> description of the approach, data structure, etc. as well as the patch can >> be found in OAK-6922. It uses the extension points introduced in the >> OAK-6921. >> >> While it’s still an experimental code, I’d like to commit it to trunk >> rather sooner than later. The patch is already pretty big and I’d like to >> avoid developing it “privately” on my own branch. It’s a new, optional >> Maven module, which doesn’t change any existing behaviour of Oak or >> SegmentMK. The only change it makes externally is adding a few exports to >> the oak-segment-tar, so it can use the SPI introduced in the OAK-6921. We >> may narrow these exports to a single package if you think it’d be good for >> the encapsulation. >> >> There’s a related issue OAK-7297, which introduces the new fixture for >> benchmark and ITs. After merging it, all the Oak integration tests pass on >> the Azure Segment Store. >> >> Looking forward for the feedback. >> >> Regards, >> Tomek >> >> [1] https://azure.microsoft.com/en-us/services/storage/blobs/ >> >> -- >> Tomek Rękawek | Adobe Research | www.adobe.com >> [email protected] >> >>
