On Tue, Nov 4, 2008 at 6:47 PM, Antony Blakey
<[EMAIL PROTECTED]> wrote:
My solution is to use the notification mechanism to maintain
(multiple)
SQLite databases containing the document keys I need to search over.
In the
URP example, I store (db, src, dest) records. I also store the last
seqno,
so that I can do incremental updates to SQLite.
Then, using _external, I can make queries of SQLite to get me either
(user,
permission) pairs using a self-join, or in the case of the arbitrary
metadata queries, a list of document ids.
The primary difficulties I have with this simple model are:
1) The notification mechanism doesn't give me enough information.
Currently
I have to do a _all_docs_by_seq and check for deletions by
attempting to get
each document, which I have to do for every document in any case
(unless I
use transparent id's) to determine if I'm interested in it, and then
get the
data. I presume this technique works because deletion is actually a
form of
update until compaction (I copied it from GeoCouch).
** SUGGESTION ** I would prefer an update interface that gave me a
stream of
(old, new) document pairs, which covers add/update/delete, plus a
(from, to)
seqno pair. Have a 'from' seqno lets me know when I have to trigger
a full
rescan, which I need to do in a variety of circumstances such as
configuration change.
In retrospect this is not a good idea. I think notification handlers
should do nothing more than mark a view or query source as dirty,
invalidate a cache such as memcached, and possibly check for mods to a
config document to enable/disable the query service. The _external/
plugin query handler should do the subsequence processing and update
any private data structures or on-disk indexes, just as the map/reduce
views do, and for the same reason. So I don't think the notification
mechanism should be changed.
However, that raises a question about external view/query server
updating: should a view update to the seqnum that is current when the
request is received, or should it keep looping in an update cycle
until the seqnum has reached a steady state?
The former would make sense if you just wanted the ensure that the
view was up-to-date with records a client might have just written in
the requesting thread, whilst the later would seem to potentially
block forever depending on the amount of processing required to update
the external view and the update rate.
Finally, does anyone have advice about the merits of mnesia vs. sqlite
(or another sql db) for this kind of auxiliary indexing?
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Lack of will power has caused more failure than lack of intelligence
or ability.
-- Flower A. Newhouse
