Hi Kevin If you can find a way to talk to this database, it could provide just what you're looking for in terms of speed:
http://hsqldb.org/ It's a SQL database, but it can be run entirely in RAM. I've used it before (although I don't recall using it from Gambas...yet!) and it's an amazing piece of work. 4M files sounds like a LOT of files, so it worries me that it'll prove too slow. I'm not convinced that using the file system would be any faster than using a 'proper' database. If you do go the fs route, there's some discussion here: http://fixunix.com/ubuntu/356538-filesystem-lots-small-files.html which may help. Regards, Caveat On Fri, 2011-04-15 at 03:22 -0400, Kevin Fishburne wrote: > I'm in the early phases of creating a "database" that uses the file > system for data organization rather than a traditional software database > such as MySQL, etc. I'm hoping that this could be faster since my > requirements are very specific and (I think) don't need a general > purpose database. > > I will have 4,194,304 "cells", each of which has about three datafiles > that will need to be opened, read from, written to and closed regularly. > I could consolidate the three files into one, reducing the number of > files, but this would increase the amount of time taken to parse the > files and slow the program significantly. > > I'm considering dividing them into hierarchies of directories to avoid > having four to 16 million data files in the same directory. Initial > tests hit file system (or file space, not sure yet) limits. > > Does anyone have any insights into the ext3/4 filesystem, possible > alternate file systems, and databases to know what the best solution > would be for this type of problem? Any insights into the most efficient > way to create hierarchies of directories and files would be appreciated > as well. > ------------------------------------------------------------------------------ Benefiting from Server Virtualization: Beyond Initial Workload Consolidation -- Increasing the use of server virtualization is a top priority.Virtualization can reduce costs, simplify management, and improve application availability and disaster protection. Learn more about boosting the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev _______________________________________________ Gambas-user mailing list Gambas-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/gambas-user
