might be off-topic but how would you compare GlusterFS to HDFS and
MogileFS for such an application? Did you look at that at all and
decided against it?
Ted Dunning wrote:
We evaluated several options for just this problem and eventually settled on
MogileFS. That said, Mogile needed several weeks of work to get it ready
for prime time. It will work pretty well for modest sized collections, but
for our stuff (many hundreds of millions of files, approaching PB of
storage), it just wasn't ready. The fixes had to do with sharding the name
database across many mySQL instances and improving the handling of storage
system up-state.
On 3/27/08 2:13 AM, "Robert Krüger" <[EMAIL PROTECTED]> wrote:
Hi,
we're looking for options for creating a scalable storage solution based
on commodity hardware for media files (spacewise dominated video files
of a few hundred MB but also to store up to a few million smaller files
such as thumbnails). The system will start with a few TB and should be
able to scale to about a PB.
Is anyone using HDFS for native storage for critical files or is it just
common to use HDFS for large amounts of temporary more or less
non-critical data? What would be the trade-offs to decide whether to use
HDFS or something like GlusterFS? Note that we'r ecurrently not planning
on using MapReduce.
Thanks in advance,
Robert
--
(-) Robert Krüger
(-) SIGNAL 7 Gesellschaft für Informationstechnologie mbH
(-) Landwehrstraße 4 - 64293 Darmstadt,
(-) Tel: +49 (0) 6151 969 96 11, Fax: +49 (0) 6151 969 96 29
(-) [EMAIL PROTECTED], www.signal7.de
(-) Amtsgericht Darmstadt, HRB 6833
(-) Geschäftsführer: Robert Krüger, Frank Peters, Jochen Strunk
