Yonik Seeley wrote:
On 10/18/06, Doug Cutting <[EMAIL PROTECTED]> wrote:
We assume that, within an index, a file with a given name is written
only once.

Is this necessary, and will we need the lockless patch (that avoids
renaming or rewriting *any* files), or is Lucene's current index
behavior sufficient?

It's not strictly required, but it would make index synchronization a lot simpler. Yes, I was assuming the lockless patch would be committed to Lucene before this project gets very far. Something more than that would be required in order to keep old versions, but this could be as simple as a Directory subclass that refuses to remove files for a time.

The search side seems straightforward enough, but I haven't totally
figured out how the update side should work.

The master should be out of the loop as much as possible. One approach is that clients randomly assign documents to indexes and send the updates directly to the indexing node. Alternately, clients might index locally, then ship the updates to a node packaged as an index. That was the intent of the addIndex method.

One potental problem is a document overwrite implemented as a delete
then an add.
More than one client doing this for the same document could result in
0 or 2 documents, instead of 1.  I guess clients will just need to be
relatively coordinated in their activities.

Good point. Either the two clients must coordinate, to make sure that they're not updating the same document at the same time, or use a strategy where updates are routed to the slave that contained the old version of the document. That would require a broadcast query to figure out which slave that is.

It's unfortunate the master needs to be involved on every document add.

That should not normally be the case. Clients can cache the set of writable index locations and directly submit new documents without involving the master.

If deletes were broadcast, and documents could go to any partition,
that would be one way around it (with the downside of a less powerful
master that could implement certain distribution policies).
Another way to lessen the master-in-the-middle cost is to make sure
one can aggregate small requests:
   IndexLocation[] getUpdateableIndex(String[] id);

I'd assumed that the updateable version of an index does not move around very often. Perhaps a lease mechanism is required. For example, a call to getUpdateableIndex might be valid for ten minutes.

We might consider a delete() on the master interface too. That way it could
 3) hide the delete policy (broadcast or directl-to-server-that-has-doc)
2) potentially do some batching of deletes
1) simply do the delete locally if there is a single index partition
and this is a combination master/searcher

I'm reticent to put any frequently-made call on the master. I'd prefer to keep the master only involved at an executive level, with all per-document and per-query traffic going directly from client to slave.

It seems like the master might want to be involved in commits too, or
maybe we just rely on the slave to master heartbeat to kick of
immediately after a commit so that index replication can be initiated?

I like the latter approach. New versions are only published as frequently as clients poll the master for updated IndexLocations. Clients keep a cache of both readable and updatable index locations that are periodically refreshed.

I was not imagining a real-time system, where the next query after a document is added would always include that document. Is that a requirement? That's harder.

At this point I'm mostly trying to see if this functionality would meet the needs of Solr, Nutch and others.

Must we include a notion of document identity and/or document version in the mechanism? Would that facillitate updates and coherency?

In Nutch a typical case is that you have a bunch of URLs with content that may-or-may-not have been previously indexed. The approach I'm currently leaning towards is that we'd broadcast the deletions of all of these to all slaves, then add index them to randomly assigned indexes. In Nutch multiple clients would naturally be coordinated, since each url is represented only once in each update cycle.


Reply via email to