On Tue, 2011-03-08 at 12:12 +0000, Martyn Russell wrote:
> On 08/03/11 11:58, [email protected] wrote:
> > Hi,
>
> Hi Igor,
>
> > From: ext Martyn Russell [[email protected]]
> > Sent: 08 March 2011 13:53
> >
> >> As you can imagine, if you want to be able to query, insert and merge
> >> 120k of contracts, you really can't have your cake and eat it so to
> >> speak without some sacrifices one way or another.
> >
> > That's not true.
> >
> > Nothing prevents from having 2 DB, where the first one is the usual
> > (could be considered as cache) and the other is much larger and used
> > only in case the first search fails. Maybe with the explicit consent
> > from the user to proceed with such expensive operation.
[cut]
> Also, we've had > 1 DB before in Tracker and it was a nightmare. There
> are problems with joins, security, speed, etc. and we've (as a team)
> discussed this so much already. We have also done testing with n
> databases vs 1 database to make sure that our approach makes the most
> sense with SQLite as it also depends on which database you use.
Some technical chitchat on this:
One of the big problems with 'two or more databases' for one class of
data (in a decomposed schema, where each class gets its own table) is
that each (sub)query done on the tables must become a UNION with the two
tables.
UNION in SQLite internally means ~ executing two queries; twice as
slow. Mind that SQLite does relatively few magic behind the scenes. As a
DB engine it's usually / relatively straightforward.
And in the end is or should splitting up a table yield similar results
as adding an index on a single table, at least for querying. So just add
an index?
Having a lot of tables with some of the tables having a lot of rows
should not slow down queries on the smaller tables in a sqlite's .db. So
for that there is also no real reason to have multiple .db files. If
SQLite does get slowed down (a lot) because of that, then that sounds to
me like a (serious) bug in SQLite.
As for read access without IPC, and the claim that with multiple .db
files you can avoid some locks when multiple processes access the .db
file, can you open SQLite databases in WAL journaling mode and avoid
most of any such lock altogether (for reading). It's not yet true MVCC
but for just reading it's quite useful that way.
For updating this doesn't work, so we have fd-passing over DBus to
tracker-store for updating (for that reason).
Also note that in SQLite a transaction locks each .db that is ATTACHed
to the connection where the transaction started.
ps. I'm not saying that SQLite is the best DBMS on all architectures and
platforms. Especially not when you have a Gig of RAM (just a few more
years and phone-sized devices have that, you'll see). Over the years we
learned to deal with what it is. It has quite a few problems idd.
Most of those popular DBMS's do use too much RAM, though. And if I can
use gigabytes of RAM and waste massive amounts of I/O and battery then
no, it's not difficult to make things perform well. Also if I can choose
the storage hardware, would help a lot. Etc.
For all the numbers that have (not) been posted here: I wonder about all
those. What RAM usage? What storage hardware? What amount of data? What
kind of data? How flexible can it be queried afterward? And how fast is
that query post data entry?
But I have the feeling that we won't see *any* numbers whatsoever.
Cheers,
Philip
--
Philip Van Hoof
freelance software developer
Codeminded BVBA - http://codeminded.be
_______________________________________________
MeeGo-dev mailing list
[email protected]
http://lists.meego.com/listinfo/meego-dev
http://wiki.meego.com/Mailing_list_guidelines
