-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jérôme Magnin wrote: > SQLite provides decent indexing times for such tables with up to 1M > rows,
As another data point, my data set is 10M records and the 6 indices I need are created in under two minutes. (I also analyze.) The database is 2GB but the least amount of memory I have is 4GB. I also use a 32kb page size, larger cache etc. > The only > workaround I could imagine to ease the problem would be to duplicate the > auxiliary table and load pre-sorted rows in it, with sort key being the > column I intend to index on. This is unfortunately too costly in terms > of disk space used. What numbers are you getting? One man's unacceptable is another's ok. Surely the index generation is a one time thing being done at upgrade time. Couldn't that be done late at night so taking several hours would be ok? > I therefore had to develop an alternate datasource type (based on flat > files) in order for my system to be able to efficiently handle big > files. Which is a pity since SQLite provides great features I still > would like to be able to rely upon when dealing with large files. You can also make virtual tables which will let you have the appearance of everything being in SQLite. Note that SQLite is unlikely to beat a custom storage format specifically formatted for a particular data set, as it is general code. > Can I nourish reasonable hopes that the situation will improve on this aspect > within the next 2 years ? This > really has become a critical factor for me to decide on my future > development strategy with this product. Unless you post a representative schema and data set somewhere, no one else can make it better for what you are doing. Alternatively see the first section of http://sqlite.org/support.html Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkt2+7UACgkQmOOfHg372QRC/wCeLu5+vffdSfenbi+vM5qG4OnU iFUAn3cJPbZRbSNjNyXTXhtixRaGBgOJ =KCyU -----END PGP SIGNATURE----- _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
