On 03/08/2010 10:53 AM, Mark Fletcher wrote:
Hi Shalin,
Thank you for the mail.
My main purpose of having 2 identical cores
COREX - always serves user request
COREY - every day once, takes the updates/latest data and passess it on to
COREX.
is:-
Suppose say I have only one COREY and suppose a request comes to COREY while
the update of the latest data is happening on to it. Wouldn't it degrade
performance of the requests at that point of time?
Yes - but your not going to help anything by using two indexes - best
you can do it use two boxes. 2 indexes on the same box will actually
be worse than one if they are identical and you are swapping between
them. Writes on an index will not affect reads in the way you are
thinking - only in that its uses IO and CPU that the read process cant.
Thats going to happen with 2 indexes on the same box too - except now
you have way more data to cache and flip between, and you can't take any
advantage of things just being written possibly being in the cache for
reads.
Lucene indexes use a write once strategy - when writing new segments,
you are not touching the segments being read from. Lucene is already
doing the index juggling for you at the segment level.
So I was planning to keep COREX and COREY always identical. Once COREY has
the latest it should somehow sync with COREX so that COREX also now has the
latest. COREY keeps on getting the updates at a particular time of day and
it will again pass it on to COREX. This process continues everyday.
What is the best possible way to implement this?
Thanks,
Mark.
On Mon, Mar 8, 2010 at 9:53 AM, Shalin Shekhar Mangar<
shalinman...@gmail.com> wrote:
Hi Mark,
On Mon, Mar 8, 2010 at 7:38 PM, Mark Fletcher<
mark.fletcher2...@gmail.com> wrote:
I ran the SWAP command. Now:-
COREX has the dataDir pointing to the updated dataDir of COREY. So COREX
has the latest.
Again, COREY (on which the update regularly runs) is pointing to the old
index of COREX. So this now doesnt have the most updated index.
Now shouldn't I update the index of COREY (now pointing to the old COREX)
so that it has the latest footprint as in COREX (having the latest COREY
index)so that when the update again happens to COREY, it has the latest and
I again do the SWAP.
Is a physical copying of the index named COREY (the latest and now datDir
of COREX after SWAP) to the index COREX (now the dataDir of COREY.. the
orginal non-updated index of COREX) the best way for this or is there any
other better option.
Once again, later when COREY is again updated with the latest, I will run
the SWAP again and it will be fine with COREX again pointing to its original
dataDir (now the updated one).So every even SWAP command run will point
COREX back to its original dataDir. (same case with COREY).
My only concern is after the SWAP is done, updating the old index (which
was serving previously and now replaced by the new index). What is the best
way to do that? Physically copy the latest index to the old one and make it
in sync with the latest one so that by the time it is to get the latest
updates it has the latest in it so that the new ones can be added to this
and it becomes the latest and is again swapped?
Perhaps it is best if we take a step back and understand why you need two
identical cores?
--
Regards,
Shalin Shekhar Mangar.
--
- Mark
http://www.lucidimagination.com
