On Thu, 5 Apr 2007, David Pratt wrote:
Hi Andi. I was thinking of using twisted's pb to retreive the objects from
remote servers over the wire - sorry for not being clear. I should be able to
load balance pb servers as to which server handles aggregation easily enough
and hopefully merge results. I thought hits objects might be able to be
merged.
To get better performance, each remote server could use index in RAM as
opposed to filesystem. As far as writing to the distributed index, I could
keep track of which index an object is writing to in the same way the load
balancing is done for searching but a db is needed to keep track. A
replication strategy could use the db to rebuild index in case a machine goes
down (or you would temporarily loose results from all servers until the index
was rebuilt).
Does this type of approach have any merit? What sort of strategy do you
envision? Many thanks.
It's getting the objects over the wire that is going to be tricky. Once you
have that working the rest is the usual web serving scaling replicating
yadayada...
Andi..
_______________________________________________
pylucene-dev mailing list
[email protected]
http://lists.osafoundation.org/mailman/listinfo/pylucene-dev
