Re: [sqlite] creating unique data takes many hours, help

mrobi...@cs.fiu.edu wrote: > this is how I create the db, table and index. > > > rc = sqlite3_exec(db, "CREATE TABLE probes (probe STRING unique);", > > rc = sqlite3_exec(db, "CREATE INDEX probe on probes (probe); > ", NULL, > In addition to what the others have said you should also

Re: [sqlite] creating unique data takes many hours, help

recompiling the sqlite library disabling threading would likely help some. --- On Mon, 3/30/09, mrobi...@cs.fiu.edu <mrobi...@cs.fiu.edu> wrote: > From: mrobi...@cs.fiu.edu <mrobi...@cs.fiu.edu> > Subject: Re: [sqlite] creating unique data takes many hours, help > To

Re: [sqlite] creating unique data takes many hours, help

using > *two* hashes may not be much of a gain. > > (Of course, since this is not a cryptographic application, you don't > need to worry about whether your chosen hash algorithm is "secure" or > not.) > > Regards, >Donald > > > > -Original Messa

Re: [sqlite] creating unique data takes many hours, help

nald -Original Message- From: sqlite-users-boun...@sqlite.org [mailto:sqlite-users-boun...@sqlite.org] On Behalf Of Vinnie Sent: Sunday, March 29, 2009 6:14 PM To: sqlite-users@sqlite.org Subject: Re: [sqlite] creating unique data takes many hours, help Michael: While my answer isn't

Re: [sqlite] creating unique data takes many hours, help

Michael: While my answer isn't strictly limited to sqlite, the performance nut in me can't help myself. You have a a table with only one column, a string. I could be wrong here but it seems like you just want to keep a list of values that you have already tried. After you insert a bunch of

Re: [sqlite] creating unique data takes many hours, help

Yes, you're right, but if the data is already in index order, you'll do less I/O when creating the index. Whether the sort + create DB time is less than "create DB from random input" time is another question. Jim On 3/29/09, mrobi...@cs.fiu.edu wrote: > question: > When

Re: [sqlite] creating unique data takes many hours, help

Thanks very much. I will try different values for the loops until a get an optimal number, and I will try the "PRAGMA cache_size = <#pages>". command. question: When you say > 2)Sort your input file on the PRIMARY KEY > or on some other INDEX I thought that while sqlite inserts the

Re: [sqlite] creating unique data takes many hours, help

On Sun, 29 Mar 2009 15:19:00 -0400 (EDT), mrobi...@cs.fiu.edu wrote: >Hi, > >I am new with sqlite, and I create a program that reads several mllion >records and puts them into a sqlite db using. > >The table has one column ONLY indexed and unique, but it takes many hours. > >Is it necessary to