Honestly for the size of blobs we're talking about (1-2MB) this is overkill.
Also, even Cassandra is not a good fit - that's good for fast write scenarios. This is write-once, read-many scenario (APK is written only once and then each install reads it). Also fairly low TPS .. not really expecting 100 users to install at the same time .. that's a lot of users!!! So shall we go with a simple DB solution for now? Ideally we have an interface so we can plug in an alternate store later but this is a small fix we can do later. Sanjiva. On Sun, Aug 4, 2013 at 1:50 AM, Hasitha Hiranya <[email protected]> wrote: > Hi, > > Came across this thread in which some ideas are discussed. > > > http://www.quora.com/What-is-a-good-choice-for-storing-blob-like-files-in-a-distributed-environment# > > > Some quote from the thread > > There are two major classes of solutions for this. One is a distributed > filesystem - but do yourself a favor and avoid Lustre. It has the same > SPOF/bottleneck you mention for HDFS, and the general effort to keep it > running is probably higher than you want. I'd suggest GlusterFS, OrangeFS, > ExtreemFS or MooseFS as better alternatives, depending on your specific > needs. (Disclaimer: I've been working with the Gluster team for years, and > my company recently acquired theirs.) > > The other main type of solution is a dedicated "blob store" like OpenStack > Storage (a.k.a. Swift) or S3. Multiple implementations of the S3 API > exist, including "Walrus" in Eucalyptus, "tabled" in Project Hail, and > "UFO" in GlusterFS. There are also in-between solutions that you have more > of a filesystem interface (rather than HTTP-based) but implement blob-store > semantics underneath. Examples of this include MogileFS and KosmosFS. > Any of these would probably work for you. Since you're also looking for a > webserver, though, I think I'd lean toward the Swift/S3 solutions that are > already basically webservers themselves and don't require a separate one. > > Thanks. > > > > On Saturday, August 3, 2013, Prabath Abeysekera wrote: > >> Hi Sanjiva, >> >> On Sat, Aug 3, 2013 at 11:32 PM, Sanjiva Weerawarana <[email protected]>wrote: >> >>> On Sat, Aug 3, 2013 at 11:06 PM, Prabath Abeysekera >>> <[email protected]>wrote: >>> >>>> >>>>> I'm wondering whether using a relational DB blob store is good enough >>>>> for a start. >>>>> >>>> >>>> IMO, HDFS too doesn't have any added advantage over a Relational/Any >>>> other blob store with respect to this particular use case as it's just >>>> going to be used as a container for those media, etc files. However, >>>> considering the OOTB capabilities of scalabilities and support for fault >>>> tolerance, etc maybe HDFS/Cassandra would be a good choice? >>>> >>> >>> Yes what they need is a scalable blob store. If Cassandra is good for >>> that we can go with that since that's older and better understood. >>> >>> The main thing is that storing the file in the regular file system >>> doesn't work in an (elastic) cluster environment. >>> >> >> Understood. I believe, additionally, we'll also have to consider how the >> data access is optimized (availability of streaming, specially since the >> files can be quite larger), whether the storage option itself is optimized >> for storing a few GBs of data (if that is part of the requirement), etc as >> well. >> >> So, first, we'd have to check whether Cassandra is capable of handling >> data contents of few hundreds of MBs (or maybe more than that). I believe, >> that would/wouldn't make it a valid reason to use HDFS or any other >> scalable blob store. However, it would be good to check on this a bit more >> in detail, considering the aspects such as the availability of >> multitenancy, etc so we can come up with the best suited solution for the >> problem at hand. Will do a bit of researching and update the thread. >> >> >> Cheers, >> Prabath >> >> >>> >>> Sanjiva. >>> -- >>> Sanjiva Weerawarana, Ph.D. >>> Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ >>> email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880| +1 >>> 650 265 8311 >>> blog: http://sanjiva.weerawarana.org/ >>> >>> Lean . Enterprise . Middleware >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Prabath Abeysekara >> Associate Technical Lead, Data TG. >> WSO2 Inc. >> Email: [email protected] >> Mobile: +94774171471 >> >> <http://harshana05.blogspot.com/> >> > > > -- > *Hasitha Abeykoon* > Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>* * > * > * > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Sanjiva Weerawarana, Ph.D. Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 650 265 8311 blog: http://sanjiva.weerawarana.org/ Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
