Yes, I consciously let my slaves run away from the master in order to
reduce update latency, but every now and then they sync up with master
that is doing heavy lifting.
The price you pay is that slaves do not see the same documents as the
master, but this is the case anyhow with replication, in my setup
slave may go ahead of master with updates, this delta gets zeroed
after replication and the game starts again.
What you have to take into account with this is very small time window
where you may "go back in time" on slaves (not seeing documents that
were already there), but we are talking about seconds and a couple out
of 200Mio documents (only those documents that were softComited on
slave during replication, since commit ond master and postCommit on
slave).
Why do you think something is strange here?
> What are you expecting a BeforeCommitListener could do for you, if one
> would exist?
Why should I be expecting something?
I just need to read userCommit Data as soon as replication is done,
and I am looking for proper/easy way to do it. (postCommitListener is
what I use now).
What makes me slightly nervous are those life cycle questions, e.g.
when I issue update command before and after postCommit event, which
index gets updated, the one just replicated or the one that was there
just before replication.
There are definitely ways to optimize this, for example to force
replication handler to copy only delta files if index gets updated on
slave and master (there is already todo somewhere on solr replication
Wiki I think). Now replicationHandler copies complete index if this
gets detected ...
I am all ears if there are better proposals to have low latency
updates in multi server setup...
On Tue, Feb 21, 2012 at 11:53 PM, Em <mailformailingli...@yahoo.de> wrote:
> Eks,
>
> that sounds strange!
>
> Am I getting you right?
> You have a master which indexes batch-updates from time to time.
> Furthermore you got some slaves, pulling data from that master to keep
> them up-to-date with the newest batch-updates.
> Additionally your slaves index own content in soft-commit mode that
> needs to be available as soon as possible.
> In consequence the slavesare not in sync with the master.
>
> I am not 100% certain, but chances are good that Solr's
> replication-mechanism only changes those segments that are not in sync
> with the master.
>
> What are you expecting a BeforeCommitListener could do for you, if one
> would exist?
>
> Kind regards,
> Em
>
> Am 21.02.2012 21:10, schrieb eks dev:
>> Thanks Mark,
>> Hmm, I would like to have this information asap, not to wait until the
>> first search gets executed (depends on user) . Is solr going to create
>> new searcher as a part of "replication transaction"...
>>
>> Just to make it clear why I need it...
>> I have simple master, many slaves config where master does "batch"
>> updates in big chunks (things user can wait longer to see on search
>> side) but slaves work in soft commit mode internally where I permit
>> them to run away slightly from master.... in order to know where
>> "incremental update" should start, I read it from UserData ....
>>
>> Basically, ideally, before commit (after successful replication is
>> finished) ends, I would like to read in these counters to let
>> "incremental update" run from the right point...
>>
>> I need to prevent updating "replicated index" before I read this
>> information (duplicates can appear).... are there any "IndexWriter"
>> listeners around?
>>
>>
>> Thanks again,
>> eks.
>>
>>
>>
>> On Tue, Feb 21, 2012 at 8:03 PM, Mark Miller <markrmil...@gmail.com> wrote:
>>> Post commit calls are made before a new searcher is opened.
>>>
>>> Might be easier to try to hook in with a new searcher listener?
>>>
>>> On Feb 21, 2012, at 8:23 AM, eks dev wrote:
>>>
>>>> Hi all,
>>>> I am a bit confused with IndexSearcher refresh lifecycles...
>>>> In a master slave setup, I override postCommit listener on slave
>>>> (solr trunk version) to read some user information stored in
>>>> userCommitData on master
>>>>
>>>> ----------
>>>> @Override
>>>> public final void postCommit() {
>>>> // This returnes "stale" information that was present before
>>>> replication finished
>>>> RefCounted<SolrIndexSearcher> refC = core.getNewestSearcher(true);
>>>> Map<String, String> userData =
>>>> refC.get().getIndexReader().getIndexCommit().getUserData();
>>>> }
>>>> ------------
>>>> I expected core.getNewestSearcher(true); to return refreshed
>>>> SolrIndexSearcher, but it didn't
>>>>
>>>> When is this information going to be refreshed to the status from the
>>>> replicated index, I repeat this is postCommit listener?
>>>>
>>>> What is the way to get the information from the last commit point?
>>>>
>>>> Maybe like this?
>>>> core.getDeletionPolicy().getLatestCommit().getUserData();
>>>>
>>>> Or I need to explicitly open new searcher (isn't solr does this behind
>>>> the scenes?)
>>>> core.openNewSearcher(false, false)
>>>>
>>>> Not critical, reopening new searcher works, but I would like to
>>>> understand these lifecycles, when solr loads latest commit point...
>>>>
>>>> Thanks, eks
>>>
>>> - Mark Miller
>>> lucidimagination.com
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
