Hi all, So shall we go with a simple DB solution for now? Ideally we have an > interface so we can plug in an alternate store later but this is a small > fix we can do later.
+1. For now we can follow a simple interface in the usage so that we can later implement on the best possible way. IMO this should become a platform feature for MbaaS as well as Stratos since blob storage is a necessary requirement for file uploading. (A good example would be the AWS offering S3). Thanks On Sun, Aug 4, 2013 at 10:42 AM, Prabath Abeysekera <[email protected]>wrote: > Hi, > > On Sun, Aug 4, 2013 at 10:20 AM, Sanjiva Weerawarana <[email protected]>wrote: > >> Honestly for the size of blobs we're talking about (1-2MB) this is >> overkill. >> >> Also, even Cassandra is not a good fit - that's good for fast write >> scenarios. This is write-once, read-many scenario (APK is written only once >> and then each install reads it). Also fairly low TPS .. not really >> expecting 100 users to install at the same time .. that's a lot of users!!! >> > > Oh okay. Got it. > > >> >> So shall we go with a simple DB solution for now? Ideally we have an >> interface so we can plug in an alternate store later but this is a small >> fix we can do later. >> > > +1. > > >> >> Sanjiva. >> >> >> On Sun, Aug 4, 2013 at 1:50 AM, Hasitha Hiranya <[email protected]>wrote: >> >>> Hi, >>> >>> Came across this thread in which some ideas are discussed. >>> >>> >>> http://www.quora.com/What-is-a-good-choice-for-storing-blob-like-files-in-a-distributed-environment# >>> >>> >>> Some quote from the thread >>> >>> There are two major classes of solutions for this. One is a distributed >>> filesystem - but do yourself a favor and avoid Lustre. It has the same >>> SPOF/bottleneck you mention for HDFS, and the general effort to keep it >>> running is probably higher than you want. I'd suggest GlusterFS, OrangeFS, >>> ExtreemFS or MooseFS as better alternatives, depending on your specific >>> needs. (Disclaimer: I've been working with the Gluster team for years, and >>> my company recently acquired theirs.) >>> >>> The other main type of solution is a dedicated "blob store" like >>> OpenStack Storage (a.k.a. Swift) or S3. Multiple implementations of the S3 >>> API exist, including "Walrus" in Eucalyptus, "tabled" in Project Hail, and >>> "UFO" in GlusterFS. There are also in-between solutions that you have more >>> of a filesystem interface (rather than HTTP-based) but implement blob-store >>> semantics underneath. Examples of this include MogileFS and KosmosFS. >>> Any of these would probably work for you. Since you're also looking for >>> a webserver, though, I think I'd lean toward the Swift/S3 solutions that >>> are already basically webservers themselves and don't require a separate >>> one. >>> >>> Thanks. >>> >>> >>> >>> On Saturday, August 3, 2013, Prabath Abeysekera wrote: >>> >>>> Hi Sanjiva, >>>> >>>> On Sat, Aug 3, 2013 at 11:32 PM, Sanjiva Weerawarana >>>> <[email protected]>wrote: >>>> >>>>> On Sat, Aug 3, 2013 at 11:06 PM, Prabath Abeysekera <[email protected] >>>>> > wrote: >>>>> >>>>>> >>>>>>> I'm wondering whether using a relational DB blob store is good >>>>>>> enough for a start. >>>>>>> >>>>>> >>>>>> IMO, HDFS too doesn't have any added advantage over a Relational/Any >>>>>> other blob store with respect to this particular use case as it's just >>>>>> going to be used as a container for those media, etc files. However, >>>>>> considering the OOTB capabilities of scalabilities and support for fault >>>>>> tolerance, etc maybe HDFS/Cassandra would be a good choice? >>>>>> >>>>> >>>>> Yes what they need is a scalable blob store. If Cassandra is good for >>>>> that we can go with that since that's older and better understood. >>>>> >>>>> The main thing is that storing the file in the regular file system >>>>> doesn't work in an (elastic) cluster environment. >>>>> >>>> >>>> Understood. I believe, additionally, we'll also have to consider how >>>> the data access is optimized (availability of streaming, specially since >>>> the files can be quite larger), whether the storage option itself is >>>> optimized for storing a few GBs of data (if that is part of the >>>> requirement), etc as well. >>>> >>>> So, first, we'd have to check whether Cassandra is capable of handling >>>> data contents of few hundreds of MBs (or maybe more than that). I believe, >>>> that would/wouldn't make it a valid reason to use HDFS or any other >>>> scalable blob store. However, it would be good to check on this a bit more >>>> in detail, considering the aspects such as the availability of >>>> multitenancy, etc so we can come up with the best suited solution for the >>>> problem at hand. Will do a bit of researching and update the thread. >>>> >>>> >>>> Cheers, >>>> Prabath >>>> >>>> >>>>> >>>>> Sanjiva. >>>>> -- >>>>> Sanjiva Weerawarana, Ph.D. >>>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>>> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880| +1 >>>>> 650 265 8311 >>>>> blog: http://sanjiva.weerawarana.org/ >>>>> >>>>> Lean . Enterprise . Middleware >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Prabath Abeysekara >>>> Associate Technical Lead, Data TG. >>>> WSO2 Inc. >>>> Email: [email protected] >>>> Mobile: +94774171471 >>>> >>>> <http://harshana05.blogspot.com/> >>>> >>> >>> >>> -- >>> *Hasitha Abeykoon* >>> Software Engineer; WSO2, Inc.; http://wso2.com >>> *cell:* *+94 719363063* >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>* * >>> * >>> * >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Sanjiva Weerawarana, Ph.D. >> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 >> 650 265 8311 >> blog: http://sanjiva.weerawarana.org/ >> >> Lean . Enterprise . Middleware >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Prabath Abeysekara > Associate Technical Lead, Data TG. > WSO2 Inc. > Email: [email protected] <[email protected]> > Mobile: +94774171471 > > <http://harshana05.blogspot.com/> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Chan (Dulitha Wijewantha) Software Engineer - Mobile Development WSO2Mobile Lean.Enterprise.Mobileware * ~Email [email protected]* * ~Mobile +94712112165* * ~Website dulithawijewantha.com * * ~Blog blog.dulithawijewantha.com<http://dulichan.github.io/chan/> * * ~Twitter @dulitharw <https://twitter.com/dulitharw>*
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
