Hi, On Sun, Aug 4, 2013 at 10:20 AM, Sanjiva Weerawarana <[email protected]>wrote:
> Honestly for the size of blobs we're talking about (1-2MB) this is > overkill. > > Also, even Cassandra is not a good fit - that's good for fast write > scenarios. This is write-once, read-many scenario (APK is written only once > and then each install reads it). Also fairly low TPS .. not really > expecting 100 users to install at the same time .. that's a lot of users!!! > Oh okay. Got it. > > So shall we go with a simple DB solution for now? Ideally we have an > interface so we can plug in an alternate store later but this is a small > fix we can do later. > +1. > > Sanjiva. > > > On Sun, Aug 4, 2013 at 1:50 AM, Hasitha Hiranya <[email protected]> wrote: > >> Hi, >> >> Came across this thread in which some ideas are discussed. >> >> >> http://www.quora.com/What-is-a-good-choice-for-storing-blob-like-files-in-a-distributed-environment# >> >> >> Some quote from the thread >> >> There are two major classes of solutions for this. One is a distributed >> filesystem - but do yourself a favor and avoid Lustre. It has the same >> SPOF/bottleneck you mention for HDFS, and the general effort to keep it >> running is probably higher than you want. I'd suggest GlusterFS, OrangeFS, >> ExtreemFS or MooseFS as better alternatives, depending on your specific >> needs. (Disclaimer: I've been working with the Gluster team for years, and >> my company recently acquired theirs.) >> >> The other main type of solution is a dedicated "blob store" like >> OpenStack Storage (a.k.a. Swift) or S3. Multiple implementations of the S3 >> API exist, including "Walrus" in Eucalyptus, "tabled" in Project Hail, and >> "UFO" in GlusterFS. There are also in-between solutions that you have more >> of a filesystem interface (rather than HTTP-based) but implement blob-store >> semantics underneath. Examples of this include MogileFS and KosmosFS. >> Any of these would probably work for you. Since you're also looking for >> a webserver, though, I think I'd lean toward the Swift/S3 solutions that >> are already basically webservers themselves and don't require a separate >> one. >> >> Thanks. >> >> >> >> On Saturday, August 3, 2013, Prabath Abeysekera wrote: >> >>> Hi Sanjiva, >>> >>> On Sat, Aug 3, 2013 at 11:32 PM, Sanjiva Weerawarana >>> <[email protected]>wrote: >>> >>>> On Sat, Aug 3, 2013 at 11:06 PM, Prabath Abeysekera >>>> <[email protected]>wrote: >>>> >>>>> >>>>>> I'm wondering whether using a relational DB blob store is good enough >>>>>> for a start. >>>>>> >>>>> >>>>> IMO, HDFS too doesn't have any added advantage over a Relational/Any >>>>> other blob store with respect to this particular use case as it's just >>>>> going to be used as a container for those media, etc files. However, >>>>> considering the OOTB capabilities of scalabilities and support for fault >>>>> tolerance, etc maybe HDFS/Cassandra would be a good choice? >>>>> >>>> >>>> Yes what they need is a scalable blob store. If Cassandra is good for >>>> that we can go with that since that's older and better understood. >>>> >>>> The main thing is that storing the file in the regular file system >>>> doesn't work in an (elastic) cluster environment. >>>> >>> >>> Understood. I believe, additionally, we'll also have to consider how the >>> data access is optimized (availability of streaming, specially since the >>> files can be quite larger), whether the storage option itself is optimized >>> for storing a few GBs of data (if that is part of the requirement), etc as >>> well. >>> >>> So, first, we'd have to check whether Cassandra is capable of handling >>> data contents of few hundreds of MBs (or maybe more than that). I believe, >>> that would/wouldn't make it a valid reason to use HDFS or any other >>> scalable blob store. However, it would be good to check on this a bit more >>> in detail, considering the aspects such as the availability of >>> multitenancy, etc so we can come up with the best suited solution for the >>> problem at hand. Will do a bit of researching and update the thread. >>> >>> >>> Cheers, >>> Prabath >>> >>> >>>> >>>> Sanjiva. >>>> -- >>>> Sanjiva Weerawarana, Ph.D. >>>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>>> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880| +1 >>>> 650 265 8311 >>>> blog: http://sanjiva.weerawarana.org/ >>>> >>>> Lean . Enterprise . Middleware >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Prabath Abeysekara >>> Associate Technical Lead, Data TG. >>> WSO2 Inc. >>> Email: [email protected] >>> Mobile: +94774171471 >>> >>> <http://harshana05.blogspot.com/> >>> >> >> >> -- >> *Hasitha Abeykoon* >> Software Engineer; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>* * >> * >> * >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Sanjiva Weerawarana, Ph.D. > Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ > email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 > 650 265 8311 > blog: http://sanjiva.weerawarana.org/ > > Lean . Enterprise . Middleware > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Prabath Abeysekara Associate Technical Lead, Data TG. WSO2 Inc. Email: [email protected] <[email protected]> Mobile: +94774171471 <http://harshana05.blogspot.com/>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
