Thanks a lot for your reply. I'm curious if anyone else has experience using mogilefs in this way? (To store millions of medium-sized files.) Would something like a replicated Reiser3 filesystem be a better choice?
-Andrew On 9/18/07, dormando <[EMAIL PROTECTED]> wrote: > Andrew Cantino wrote: > > I'm interested in using mogilefs to store very large numbers of files > > (tens to hundreds of millions). Will this be a problem? > > Depends on how often you add, delete, re-replicate, read etc. mogilefs' > choke point is currently the database. Which a nice dual cpu quadcore > machine with 32G+ of RAM will likely hold hundreds of millions of files > okay. > > It also wouldn't be too huge of an investment to add database > partitioning support. > > I guess it's worth noting that the more actively your dataset is > changing, the more load the system presents to itself in general. If you > load up a hundred million files then mostly read on them most of > mogilefs will be pretty bored. Doing more adds some extra DB load. > > > Could anyone > > point me to benchmark documentation about how mogilefs scales as the > > number of stored files grows? > > Wish there was some :) There's the one database, the rest is dependent > on how many spindles you add to the cluster. Disks slow? Add more disks, > rebalance, or drain overloaded devices, move on. Trackers overloaded? > Add more trackers. Database slow? Possibly a bigger issue. > > -Dormando > -- Andrew Cantino CastTV
