Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Kit
2011/8/9 David Garfield : > Having said that, let me present a database for consideration: Any > filesystem.  Split the hex of the MD5 into directory levels and make > what you need.  Might be slower, particularly with some OSes, but the > tools are easy. > --David

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread David Garfield
Richard Hipp writes: > This is a locality of reference problem. The caching mechanisms (both in > SQLite and in the filesystem of your computer) begins to break down when the > size of the database exceeds available RAM. And when the cache stops > working well, you have to wait on physical I/O

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Kit
2011/8/9 Jaco Breitenbach : > I am building an application that filters out duplicate input data by > generating an MD5 hash of each input, and implicitly comparing that against > a set of keys already stored in the SQLite database by doing an insert into > a

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Eric Scouten
If non-GPL is a firm requirement, you might also look at CouchDB ( http://couchdb.org). On Tue, Aug 9, 2011 at 07:02, gabriel.b...@gmail.com wrote: > i would sugest Mongo db > > just use it from its binaries packages and don't worrie > > like it says > If you are using

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Simon Slavin
On 9 Aug 2011, at 2:27pm, Jaco Breitenbach wrote: > The problem that I'm facing, is that I would ultimately need to process > 1,000,000,000 records a day, with history to be kept for up to 128 days. I > am currently creating a new data file per day, with hourly tables. However, > that will

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Richard Hipp
On Tue, Aug 9, 2011 at 9:27 AM, Jaco Breitenbach wrote: > Unfortunately the performance rate of the inserts > into the indexed tables decreases significantly as the number of records in > the tables increases. This seems to be because of a CPU bottleneck rather > than

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread gabriel.b...@gmail.com
i would sugest Mongo db just use it from its binaries packages and don't worrie like it says If you are using a vanilla MongoDB server from either source or binary packages you have NO obligations. You can ignore the rest of this page. http://www.mongodb.org/display/DOCS/Licensing 2011/8/9

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Black, Michael (IS)
enbach [jjbreitenb...@gmail.com] Sent: Tuesday, August 09, 2011 8:51 AM To: General Discussion of SQLite Database Subject: EXT :Re: [sqlite] Maximum number of tables in a data file Hi Gabriel, Is there such a database that is both free and non-GPL that you can recommend? Jaco On 9 August 2011

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Jaco Breitenbach
Hi Gabriel, Is there such a database that is both free and non-GPL that you can recommend? Jaco On 9 August 2011 14:38, gabriel.b...@gmail.com wrote: > Heve you ever considere using a NOSQL database I think it would serve you > better > > 2011/8/9 Jaco Breitenbach

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Jaco Breitenbach
Yes, but each input record also contains a timestamp that can be used to identify the relevant table. On 9 August 2011 14:43, Igor Tandetnik wrote: > Jaco Breitenbach wrote: > > I am building an application that filters out duplicate input data by

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Igor Tandetnik
Jaco Breitenbach wrote: > I am building an application that filters out duplicate input data by > generating an MD5 hash of each input, and implicitly comparing that against > a set of keys already stored in the SQLite database by doing an insert into > a unique-indexed

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread gabriel.b...@gmail.com
Heve you ever considere using a NOSQL database I think it would serve you better 2011/8/9 Jaco Breitenbach > Hi Igor and Michael, > > Yes, of course, 1440 minutes in a day. :-) > > I am building an application that filters out duplicate input data by > generating an MD5

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Jaco Breitenbach
Hi Igor and Michael, Yes, of course, 1440 minutes in a day. :-) I am building an application that filters out duplicate input data by generating an MD5 hash of each input, and implicitly comparing that against a set of keys already stored in the SQLite database by doing an insert into a

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Igor Tandetnik
Jaco Breitenbach wrote: > Can anyone please tell me if there is a limit to the number of tables that > can be held in a single data file? I am considering an application that > will require a table for every minute in a day, i.e. 3600+ tables in a > single database or

Re: [sqlite] Maximum number of tables in a data file

2011-08-09 Thread Black, Michael (IS)
09, 2011 8:05 AM To: General Discussion of SQLite Database Subject: EXT :[sqlite] Maximum number of tables in a data file Dear experts, Can anyone please tell me if there is a limit to the number of tables that can be held in a single data file? I am considering an application that will require

[sqlite] Maximum number of tables in a data file

2011-08-09 Thread Jaco Breitenbach
Dear experts, Can anyone please tell me if there is a limit to the number of tables that can be held in a single data file? I am considering an application that will require a table for every minute in a day, i.e. 3600+ tables in a single database or data file. Regards, Jaco