Hello! On Saturday 28 February 2009 00:31:37 pyt...@bdurham.com wrote: > Your approach of splitting large data sets sounds similar to what other > SQLite users with large data sets seem to be doing. At a high level, > this sounds like how one would partition data using Oracle? I'm going to > start a new thread on this topic.
Oracle manage partitions automatically but SQLite databases are manages by your application. So you must write more code. But good result is that SQLite databases may be packed and removed to archive and your application can easy resolv this situation. And a lot of external programs (as example demon for network monitoring or tacacs/radius server) can store data in single databases without working with full dataset. And Oracle is using some buffers for modified data and can perform a lot of write/read transactions in parallels. SQLIte- driven application must plan write operation and use some hashes/in-memory databases as buffers and perform saves sets of records to on-disk database because write operation with single record is not effective. So I have tcl data collectors which regular save chunks of data to on-disk SQLite database. Of cource users operations by web-interface performs immediately and it's work fine because in web-applications there are a lot of read operations and a few write operations. > > With your hardware I think 100Gb dataset is not limit. > > Good news. I'm looking forward to verifying this over the next month or > so. Please mail to list or to me directly about you problems and results. I do some big systems based on SQLite now and the information about similar projects is important for me. I support debian repository with my own SQLite build with extra modules. For big projects is needed some additional functions (data compression, md5 functions, extended csv import, etc.). Debian etch: deb http://mobigroup.ru/debian/ etch main contrib non-free deb-src http://mobigroup.ru/debian/ etch main contrib non-free Debian lenny: deb http://mobigroup.ru/debian/ lenny main contrib non-free deb-src http://mobigroup.ru/debian/ lenny main contrib non-free Keyring may be installed as sudo aptitude install debian-mobigroup-keyring Best regards. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
