Re: Storing hundreds of millions of files in HAMMER (1 or 2)

[tautolog]

Hi Mehmet, I think an OS having a seekable file interface pretty much means that it is suited best for seekable data, which is going to have some length to it. The interface is designed for that kind of consumption, so I would hope the implementation is optimized for it. Otherwise the interface overhead is wasted.  Some databases are very good for relational data. Some are good at being a key-value store. Some are good at storing XML. A filesystem is good at storing seekable data of significant length, and that is what I want my filesystem optimized for. Sure, a filesystem should be more general-purpose than a more specialized database, but to expect it to be the better solution for a completely small-file problem is, I think, asking too much. At some point, using the seekability has to be more appropriate. I could see wanting to use a filesystem for its other features, like the folder hierachy, and still wanting small files, like the MH email format. However, even in that extreme case, there will often be large files, like PDF files, and ZIP files. Of course, hammer still suits that case just fine. Maybe XFS would be faster.  Think of building a fuse filesystem on top of postgres. It would be cool for compatibility, but it wouldn't be an appropriate thing to do besides that. For example, you could search an MH folder and store the results in a postgres database, and do it with grep and some shell generating results in the fuse mount. But that is an extreme case.  Ben From: Mehmet Erol Sanliturk Sent: Wednesday, July 15, 2015 6:43 PM To: Sepherosa Ziehau Cc: Matthew Dillon; users Subject: Re: Storing hundreds of millions of files in HAMMER (1 or 2) On Wed, Jul 15, 2015 at 6:30 PM, Sepherosa Ziehau <sepher...@gmail.com> wrote: Maybe just use large file, and sub-index the chunks of a large file and add open/read/write/lseek/close like APIs for users.  You will have more control than using database. On Wed, Jul 15, 2015 at 11:58 PM, Michael Neumann <mneum...@ntecs.de> wrote: > Hi, > > Lets say I want to store 100 million small files (each one about 1k in size) > in a HAMMER file system. > Files are only written once, then kept unmodified and accessed randomly > (older files will be access less often). > It is basically a simple file based key/value store, but accessible by > multiple processes. > > a) What is the overhead in size for HAMMER1? For HAMMER2 I expect each file > to take exactly 1k when the file > is below 512 bytes. > > b) Can I store all files in one huge directory? Or is it better to fan out > the files into several sub-directories? > > c) What other issues I should expect to run into? For sure I should enable > swapcache :) > > I probably should use a "real" database like LMDB, but I like the > versatility of files. > > Regards, > >   Michael -- Tomorrow Will Never Die In reality , an operating system is one of the "best" data base management system . Question by Michael Neumann is very important with respect to this fact . I was thinking to use DragonFly BSD for such a task , but it seems that it is not useful on that issue , because assumption about that files should be large is not so much suitable for an operating system . Then XFS seems to be a good alternative in Fedora Server edition . Thank you very much . Mehmet Erol Sanliturk