Considerations that you may want to think about when sanitizing your
clustered indecies:
1) Number of documents available vs. number of documents in the
persistent store.
2) Are all the document up to date (involves comparing the existence and
the last date updated of Lucene documents to persistent store)
3) Have all the documents that should have been deleted actually been
deleted from the index, if you delete the documents from the persistent
store this is not trivial, we're using an "is_deleted" flag so we can
query the index for deleted documents, if we get any results, then
there's a problem.
Sync issues will happen, welcome to the wonderful work of NDC, not by
any fault of programming, but simply because this is the nature of
network communication, and computing in demanding environments. Failing
to provide for contingency and fail safes will give you some of the most
intense headaches, ultimately, you want the system to fix itself (assume
failures for day one) I've been monitoring the system from a distance
for the past year with little or no interference (4 years in total),
there is nothing better than knowing that I could take a tire iron to a
few of our machines and that would not affect the site performance, I
get to sleep at night.
My two cents.
Nader Henein
Paul Smith wrote:
If you use ActiveMQ for JMS, you can take advantage of it's
Composite Destination feature and have a virtual Queue/Topic that
is actually several Queues/Topics. This is what we use to keep a
mirror index server completely in sync. The application sends an
update message to a queue named "queue://index1, queue://index2",
which becomes 2 separate queues for the 2 servers, allowing them
to process the same message whenever they can get around to it.
Ah, the composite topic, is indeed a good nice. But out of
curiosity...did you put your 2 nodes (consumers) as embedded brokers
or is the producer as the main broker ?
Neither in our case, a central broker (albeit we plan to have a
backup plan with ActiveMQ installed on all servers with the same
config ready to roll, using zeroconf for discovery).
We then place Apache in front of these 2 mirrored Index/Search
nodes so the application can use web-services to query the search
node without actually being aware that there is 2 of them behind
the scenes, leaving Apache to do the load-balancing and fail-over
as the index/search nodes come up/down without the main
application knowing anything about it.
Ideally, the 2 nodes have the same state when running.
Ideally, reality is different. We're going to be monitoring
consistency of the pair closely, and should the fall out of sync with
regularity, then one of them is just going to be the hot-spare for
failure purposes only.
What happens when a node fails and that you put it back online and
that it needs to catch up with all missing messages in its queue ?
Is it considered 'offline' until it catches up ? If yes how do you
do it ? If no, I guess you don't mind that a search request may not
give the same result depending on the node it is load-balanced,
correct ?
In this case we will manually mark the node via Apache worker configs
to be be disabled until it has caught up.
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---
--
Nader S. Henein
Senior Applications Architect
Bayt.com
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
