With only 20,000 rows it is my intuitive feel that a linear search on a
memory resident table (like a mmap'd shared file) would be faster than
having a B-Tree setup like Sqlite which would often be doing straight
row searches anyway. Joins are not more complex, and perhaps less, than
using SQL and your search options using regular expressions are immense.
I have had some success using that technique, but made sure that I
used a fast search algorithm like Boyer-Moore.
If you are performing run time insertions rather than just updates, then
my suggestion is not so good.
I guess you have noticed that Sqlite does have a memory resident option.
JS
Radomyselskiy, Mikhail (IT) wrote:
That's not a bad idea and we thought about it,
but...
With large number of attributes to be compared (over 100 in some
cases),
possible numeric comparison (value between 10 and 50),
joins to other tables (comparisons might include indirect
comparisons like ratings Aaa, Bbb that require a separate value look up
table),
several pass updates (outputs will have to be determined in
several passes through update statements since many rules need to be
applied before an output is made with each rule altering a different
component of the output)
and a relatively large data set (100s of attributes, dozens of
output columns, and over 20000 rows to which rules need to be applied)
it feels like we'll just end up rewriting a db engine in parts.
Do you still think your suggestion would still work? If not, would
SQLite support the memory-only lock-free mode?
MR
-----Original Message-----
From: John Stanton [mailto:[EMAIL PROTECTED]
Sent: Monday, September 26, 2005 3:00 PM
To: sqlite-users@sqlite.org
Subject: Re: [sqlite] Running SQLite in memory-only mode
You might do better not to use any database manager and to place your
rules in a flat file which you memory map into your program, then use s
REGEX regular expression search (or simpler string search) to pop out a
list of pointers to the rules. That would be blindingly fast and memory
resident (or at least virtual memory resident).
Simpler can be better.
JS
Radomyselskiy, Mikhail (IT) wrote:
Hi,
I would like to use SQLite for processing of rules that look like
WHERE clauses (so if i fill a table with attributes and outputs i can
run "SELECT <OUTPUTS> WHERE <ATTRIBUTE MATCHING RULES>")
Looking at the speed comparison (and i realize its not uptodate but
for the lack of better info) it seems a lot of time is spent on the
file access and synchronization. We need an engine that runs
completely in memory without disk use and that doesn't have to be
thread-safe (FastDB seems to do this but I wanted to see if there is a
more main-stream alternative).
From various comments it looks like this is possible and I just
wanted to make sure and maybe get some idea of how to do this.
Thank You,
Mike Radomyselskiy
[EMAIL PROTECTED]
--------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender.
Sender does not waive confidentiality or privilege, and use is
prohibited.
--------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender. Sender does
not waive confidentiality or privilege, and use is prohibited.
