On 10/18/06, Doug Cutting <[EMAIL PROTECTED]> wrote:
We assume that, within an index, a file with a given name is written
only once.

Is this necessary, and will we need the lockless patch (that avoids
renaming or rewriting *any* files), or is Lucene's current index
behavior sufficient?

I like the explicit index version and keeping the last few version
around.  The whole idea of a master seems to lessen the amount of
manual configuration in large clusters too.
The search side seems straightforward enough, but I haven't totally
figured out how the update side should work.

Deletions could be broadcast to all slaves.  That would probably be fast

Hmmm, that does allow one to move documents around the cluster and
more easily resize things.

One potental problem is a document overwrite implemented as a delete
then an add.
More than one client doing this for the same document could result in
0 or 2 documents, instead of 1.  I guess clients will just need to be
relatively coordinated in their activities.

 Alternately, indexes could be partitioned by a hash of each
document's unique id, permitting deletions to be routed to the
appropriate slave.

A hash is nice, but then you can't resize the number of partitions
your index is split into.

It's unfortunate the master needs to be involved on every document add.
If deletes were broadcast, and documents could go to any partition,
that would be one way around it (with the downside of a less powerful
master that could implement certain distribution policies).
Another way to lessen the master-in-the-middle cost is to make sure
one can aggregate small requests:
   IndexLocation[] getUpdateableIndex(String[] id);

We might consider a delete() on the master interface too.  That way it could
 3) hide the delete policy (broadcast or directl-to-server-that-has-doc)
2) potentially do some batching of deletes
1) simply do the delete locally if there is a single index partition
and this is a combination master/searcher

It seems like the master might want to be involved in commits too, or
maybe we just rely on the slave to master heartbeat to kick of
immediately after a commit so that index replication can be initiated?

Does this make sense?  Does it sound like it would be useful to Solr?
To Nutch?  To others?  Who would be interested and able to work on it?

Still interested, and able :-)


Reply via email to