Erick,
> You'll *really like* the SolrCloud stuff going into trunk when it's baked
> for a while....
How stable is SolrCloud at the moment?
I can not wait to try it out.
Kind regards,
Em
Am 22.02.2012 14:45, schrieb Erick Erickson:
> You'll *really like* the SolrCloud stuff going into trunk when it's baked
> for a while....
>
> Best
> Erick
>
> On Wed, Feb 22, 2012 at 3:25 AM, eks dev <eks...@googlemail.com> wrote:
>> Yes, I consciously let my slaves run away from the master in order to
>> reduce update latency, but every now and then they sync up with master
>> that is doing heavy lifting.
>>
>> The price you pay is that slaves do not see the same documents as the
>> master, but this is the case anyhow with replication, in my setup
>> slave may go ahead of master with updates, this delta gets zeroed
>> after replication and the game starts again.
>>
>> What you have to take into account with this is very small time window
>> where you may "go back in time" on slaves (not seeing documents that
>> were already there), but we are talking about seconds and a couple out
>> of 200Mio documents (only those documents that were softComited on
>> slave during replication, since commit ond master and postCommit on
>> slave).
>>
>> Why do you think something is strange here?
>>
>>> What are you expecting a BeforeCommitListener could do for you, if one
>>> would exist?
>> Why should I be expecting something?
>>
>> I just need to read userCommit Data as soon as replication is done,
>> and I am looking for proper/easy way to do it. (postCommitListener is
>> what I use now).
>>
>> What makes me slightly nervous are those life cycle questions, e.g.
>> when I issue update command before and after postCommit event, which
>> index gets updated, the one just replicated or the one that was there
>> just before replication.
>>
>> There are definitely ways to optimize this, for example to force
>> replication handler to copy only delta files if index gets updated on
>> slave and master (there is already todo somewhere on solr replication
>> Wiki I think). Now replicationHandler copies complete index if this
>> gets detected ...
>>
>> I am all ears if there are better proposals to have low latency
>> updates in multi server setup...
>>
>>
>> On Tue, Feb 21, 2012 at 11:53 PM, Em <mailformailingli...@yahoo.de> wrote:
>>> Eks,
>>>
>>> that sounds strange!
>>>
>>> Am I getting you right?
>>> You have a master which indexes batch-updates from time to time.
>>> Furthermore you got some slaves, pulling data from that master to keep
>>> them up-to-date with the newest batch-updates.
>>> Additionally your slaves index own content in soft-commit mode that
>>> needs to be available as soon as possible.
>>> In consequence the slavesare not in sync with the master.
>>>
>>> I am not 100% certain, but chances are good that Solr's
>>> replication-mechanism only changes those segments that are not in sync
>>> with the master.
>>>
>>> What are you expecting a BeforeCommitListener could do for you, if one
>>> would exist?
>>>
>>> Kind regards,
>>> Em
>>>
>>> Am 21.02.2012 21:10, schrieb eks dev:
>>>> Thanks Mark,
>>>> Hmm, I would like to have this information asap, not to wait until the
>>>> first search gets executed (depends on user) . Is solr going to create
>>>> new searcher as a part of "replication transaction"...
>>>>
>>>> Just to make it clear why I need it...
>>>> I have simple master, many slaves config where master does "batch"
>>>> updates in big chunks (things user can wait longer to see on search
>>>> side) but slaves work in soft commit mode internally where I permit
>>>> them to run away slightly from master.... in order to know where
>>>> "incremental update" should start, I read it from UserData ....
>>>>
>>>> Basically, ideally, before commit (after successful replication is
>>>> finished) ends, I would like to read in these counters to let
>>>> "incremental update" run from the right point...
>>>>
>>>> I need to prevent updating "replicated index" before I read this
>>>> information (duplicates can appear).... are there any "IndexWriter"
>>>> listeners around?
>>>>
>>>>
>>>> Thanks again,
>>>> eks.
>>>>
>>>>
>>>>
>>>> On Tue, Feb 21, 2012 at 8:03 PM, Mark Miller <markrmil...@gmail.com> wrote:
>>>>> Post commit calls are made before a new searcher is opened.
>>>>>
>>>>> Might be easier to try to hook in with a new searcher listener?
>>>>>
>>>>> On Feb 21, 2012, at 8:23 AM, eks dev wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I am a bit confused with IndexSearcher refresh lifecycles...
>>>>>> In a master slave setup, I override postCommit listener on slave
>>>>>> (solr trunk version) to read some user information stored in
>>>>>> userCommitData on master
>>>>>>
>>>>>> ----------
>>>>>> @Override
>>>>>> public final void postCommit() {
>>>>>> // This returnes "stale" information that was present before
>>>>>> replication finished
>>>>>> RefCounted<SolrIndexSearcher> refC = core.getNewestSearcher(true);
>>>>>> Map<String, String> userData =
>>>>>> refC.get().getIndexReader().getIndexCommit().getUserData();
>>>>>> }
>>>>>> ------------
>>>>>> I expected core.getNewestSearcher(true); to return refreshed
>>>>>> SolrIndexSearcher, but it didn't
>>>>>>
>>>>>> When is this information going to be refreshed to the status from the
>>>>>> replicated index, I repeat this is postCommit listener?
>>>>>>
>>>>>> What is the way to get the information from the last commit point?
>>>>>>
>>>>>> Maybe like this?
>>>>>> core.getDeletionPolicy().getLatestCommit().getUserData();
>>>>>>
>>>>>> Or I need to explicitly open new searcher (isn't solr does this behind
>>>>>> the scenes?)
>>>>>> core.openNewSearcher(false, false)
>>>>>>
>>>>>> Not critical, reopening new searcher works, but I would like to
>>>>>> understand these lifecycles, when solr loads latest commit point...
>>>>>>
>>>>>> Thanks, eks
>>>>>
>>>>> - Mark Miller
>>>>> lucidimagination.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>
