Hi
I'm going to develop a manifold connector and one requirements is that
it should be able to handle document insertion and deletion in order (details
coming).
Actually I've already built such crawler as a standalone application and the
design was conceptually this
instead of a Document Queue I have a CommandQueue
commands can be delete (specifying the docId) or add (specifying the
doc to be added)
when a worker thread takes a delete no other worker is allowed to pick other
commands from the queue until the delete has been committed
Ex. suppose I have the following chunk of CommandQueue:
add{doc1}, delete{doc1}, add{doc1}
I need to avoid the situation where commands are processed in this order:
add{doc1}, add{doc1}, delete{doc1}
I think the EventSequence could help me implement this synchronization in
Manifold
when seeding the identifiers I could embed in the identifier the command
Ex.
instead of stuffing the identifier "hd-samsing-500GB"
I could stuff "add hd-samsung-500GB"
The question is: Am I running into huge troubles trying to implement this
requirement or not?
--
Matteo Grolla
Sourcesense - making sense of Open Source
http://www.sourcesense.com
