Hello! I've been thinking about it.
I am using Postgres and it seems to be bottleneck. But using flat files is likely to be slower and more danderous of data loss. I think, the most fast and safe method is to use Database, but copy objects to RAM. If you have enough memory to copy the entire database to it. When something is changed, it is added to queue and saved to Database asynchronously (some data may be lost). I think, this can be implemented as a SqlConnection sub-class. It would work as a front-end of any other backend (postgres, firebird, sqlite3 or mysql). Retriving of objects will be very fast, but I do not know what to do with advanced SQL queries (e.g., WHERE and ORDER BY). Maybe, they also can be implemented in memory, using in-memory indices. Another approach is to use sqlite3 database located in memory or in tmpfs. The file is copied to disk from time to time. On Sun, Oct 7, 2012 at 5:44 PM, Mohammed Rashad <mohammedrasha...@gmail.com> wrote: > All, > > Due to large data used in a crowd source mapping project. I had decided to > completely eliminate the use of database and use Flat files for storage, > retrieval and query. > > I thought of storing each record in db as individual files. so this will > help in the retrieval speed and no search is needed in the entire db or a > file. > > but if a table have more than 10000's of records and users accessing > (same/different) records from different places will result in N number of > File I/O > > Will this be a bottleneck in the application. consider each file of size <= > 15KB.? > > The main reason to eliminate db is because of performance bottleneck in > database I/O. > > So moving to new model will help in anyway as the number of users and data > will be much more than expected? > > > -- > Regards, > Rashad > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > witty-interest mailing list > witty-interest@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/witty-interest > ------------------------------------------------------------------------------ Don't let slow site performance ruin your business. Deploy New Relic APM Deploy New Relic app performance management and know exactly what is happening inside your Ruby, Python, PHP, Java, and .NET app Try New Relic at no cost today and get our sweet Data Nerd shirt too! http://p.sf.net/sfu/newrelic-dev2dev _______________________________________________ witty-interest mailing list witty-interest@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/witty-interest
