I have an interesting problem. I need to generate a large table periodically. The table contains a unique SHA1 hash code and 6 integer values and has about 6 million rows. Generating this table is fast. I can set it up in less than 90 seconds on a slow iMac (2.16 Ghz and slow disk). The thing that takes a (relatively) long time is the index creation on the unique hash code .. 720 seconds.
(I'm importing within transactions with chunks of 25.000 records) The question is, is there any way to speed up the indexing proces? Couple of things on my mind: Doing this all on an in-memory database takes about 150 seconds in total. Is it possible to build a database in memory and then dump it to disk? Would it make sense to sort the records that i import? Could that result in a quicker index operation? This certainly helps for Berkeley DB. But it is lower level and you can use it's internal hashing functions on your data to pre-sort it. Maybe SQLite has something similar? Are there any other tunable options that can improve indexing speed? S. _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
