Re: SolrCloud - leader updates not updating followers

[Erick Erickson]

How did you set up your cluster? I hope you just used the collections
create API.
If you tried to do this by adding individual cores via the core admin
APIs, I'd advise
you to delete them all and use the collections API. In fact you should use the
collections API for everything; the "core admin API" is more for stand-alone and
internal use.

OK, a couple of other random thoughts:

1> in the admin/cloud view you should have a
tree>>collections>>your_collection>>state.json
node. What's in it? If that node isn't there it's in:
tree>>clusterstate.json

2> Are you totally sure you're referencing the external ZooKeeper ensemble
in all your Solr JVMs? This one's tricky. It's vaguely possible that
you're referencing
the embedded ZK in some of your Solr nodes. If you didn't provide the ensemble's
address to your Solr instances correctly it _might_ account for this.
Wild stab in the dark here.

Best,
Erick

On Tue, Aug 8, 2017 at 12:45 PM, Peter Lancaster
<peter.lancas...@findmypast.com> wrote:
> Hi Erik,
>
> Thanks for your quick reply. It's given me a few things to research and work 
> on tomorrow.
>
> In the meantime, in case it triggers any other thoughts,  just to say that 
> our AutoCommit settings are
> 
> 180
> 30
> 
> 
> 1
> 
>
> When I ingest data I don't see data appearing on the follower in the log.
>
> It really seems like the data isn't being sent from the leader. As I said it 
> could easily be something stupid that I've done along the way but I can't see 
> what it is.
>
> Thanks again,
> Peter.
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: 08 August 2017 18:23
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: SolrCloud - leader updates not updating followers
>
> This _better_ be a problem with your configuration or all my assumptions are 
> false ;)
>
> What are you autocommit settings? The documents should be forwarded to each 
> replica from the leader during ingestion. However, they are not visible on 
> the follower until a hard commit(openSearcher=true) or soft commit is 
> triggered. Long blog on all this here:
>
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> An easy way to check that docs are being sent to the follower is to tail 
> solr.log and send a doc to the leader. You should see the doc arrive at the 
> follower. If you do see that then it's your autocommit settings.
>
> NOTE: Solr promises eventual consistency. Due to the fact that autocommits 
> will execute at different wall-clock times on the leaders and followers you 
> can be out of sync by up to your autocommit interval.
>
> You can force a commit by a URL like
> "http://solr:port/solr/collection/update?commit=true; that will commit on all 
> replicas in a collection that may also be useful for seeing if it's an 
> autocommit issue.
>
> Best,
> Erick
>
> On Tue, Aug 8, 2017 at 9:49 AM, lancasp22 <peter.lancas...@findmypast.com> 
> wrote:
>> Hi,
>>
>> I've recently created a solr cloud on solr 5.5.2 with a separate
>> zookeeper cluster. I write to the cloud by posting to update/json and
>> the documents appear fine in the leader.
>>
>> The problem I have is that new documents added to the cloud aren't
>> then being automatically applied to the followers of each leader. If I
>> query a core then I can get different counts depending on which
>> version of the core the query ran over and the solr admin statistics
>> page confirms that the followers have fewer documents and are behind the 
>> leader.
>>
>> If I restart the solr core for a follower, it does recover quickly and
>> brings itself up-to-date with the leader. Looking at the logs for the
>> follower you see that on re-start it identifies the leader and gets
>> the changes from that leader.
>>
>> I think when documents are added to the leader these should be pushed
>> to the followers so maybe it's this push process that isn't being triggered.
>>
>> It’s quite possible that I’ve made a simple error in the set-up but
>> I’m baffled as to what it is. Please can anyone advise on any
>> configuration that I need to check that might be causing these symptoms.
>>
>> Thanks in anticipation,
>> Peter Lancaster.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updati
>> ng-followers-tp4349618.html Sent from the Solr - User mailing list
>> archive at Nabble.com.
> __

RE: SolrCloud - leader updates not updating followers

[Peter Lancaster]

Hi Erik,

Thanks for your quick reply. It's given me a few things to research and work on 
tomorrow.

In the meantime, in case it triggers any other thoughts,  just to say that our 
AutoCommit settings are

180
30


1


When I ingest data I don't see data appearing on the follower in the log.

It really seems like the data isn't being sent from the leader. As I said it 
could easily be something stupid that I've done along the way but I can't see 
what it is.

Thanks again,
Peter.

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: 08 August 2017 18:23
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: SolrCloud - leader updates not updating followers

This _better_ be a problem with your configuration or all my assumptions are 
false ;)

What are you autocommit settings? The documents should be forwarded to each 
replica from the leader during ingestion. However, they are not visible on the 
follower until a hard commit(openSearcher=true) or soft commit is triggered. 
Long blog on all this here:

https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

An easy way to check that docs are being sent to the follower is to tail 
solr.log and send a doc to the leader. You should see the doc arrive at the 
follower. If you do see that then it's your autocommit settings.

NOTE: Solr promises eventual consistency. Due to the fact that autocommits will 
execute at different wall-clock times on the leaders and followers you can be 
out of sync by up to your autocommit interval.

You can force a commit by a URL like
"http://solr:port/solr/collection/update?commit=true; that will commit on all 
replicas in a collection that may also be useful for seeing if it's an 
autocommit issue.

Best,
Erick

On Tue, Aug 8, 2017 at 9:49 AM, lancasp22 <peter.lancas...@findmypast.com> 
wrote:
> Hi,
>
> I've recently created a solr cloud on solr 5.5.2 with a separate
> zookeeper cluster. I write to the cloud by posting to update/json and
> the documents appear fine in the leader.
>
> The problem I have is that new documents added to the cloud aren't
> then being automatically applied to the followers of each leader. If I
> query a core then I can get different counts depending on which
> version of the core the query ran over and the solr admin statistics
> page confirms that the followers have fewer documents and are behind the 
> leader.
>
> If I restart the solr core for a follower, it does recover quickly and
> brings itself up-to-date with the leader. Looking at the logs for the
> follower you see that on re-start it identifies the leader and gets
> the changes from that leader.
>
> I think when documents are added to the leader these should be pushed
> to the followers so maybe it's this push process that isn't being triggered.
>
> It’s quite possible that I’ve made a simple error in the set-up but
> I’m baffled as to what it is. Please can anyone advise on any
> configuration that I need to check that might be causing these symptoms.
>
> Thanks in anticipation,
> Peter Lancaster.
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updati
> ng-followers-tp4349618.html Sent from the Solr - User mailing list
> archive at Nabble.com.


This message is confidential and may contain privileged information. You should 
not disclose its contents to any other person. If you are not the intended 
recipient, please notify the sender named above immediately. It is expressly 
declared that this e-mail does not constitute nor form part of a contract or 
unilateral obligation. Opinions, conclusions and other information in this 
message that do not relate to the official business of findmypast shall be 
understood as neither given nor endorsed by it.


__

This email has been checked for virus and other malicious content prior to 
leaving our network.
__

Re: SolrCloud - leader updates not updating followers

[Erick Erickson]

This _better_ be a problem with your configuration or all my
assumptions are false ;)

What are you autocommit settings? The documents should be forwarded to
each replica from the leader during ingestion. However, they are not
visible on the follower until a hard commit(openSearcher=true) or soft
commit is triggered. Long blog on all this here:

https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

An easy way to check that docs are being sent to the follower is to
tail solr.log and send a doc to the leader. You should see the doc
arrive at the follower. If you do see that then it's your autocommit
settings.

NOTE: Solr promises eventual consistency. Due to the fact that
autocommits will execute at different wall-clock times on the leaders
and followers you can be out of sync by up to your autocommit
interval.

You can force a commit by a URL like
"http://solr:port/solr/collection/update?commit=true; that will commit
on all replicas in a collection that may also be useful for seeing if
it's an autocommit issue.

Best,
Erick

On Tue, Aug 8, 2017 at 9:49 AM, lancasp22
 wrote:
> Hi,
>
> I've recently created a solr cloud on solr 5.5.2 with a separate zookeeper
> cluster. I write to the cloud by posting to update/json and the documents
> appear fine in the leader.
>
> The problem I have is that new documents added to the cloud aren't then
> being automatically applied to the followers of each leader. If I query a
> core then I can get different counts depending on which version of the core
> the query ran over and the solr admin statistics page confirms that the
> followers have fewer documents and are behind the leader.
>
> If I restart the solr core for a follower, it does recover quickly and
> brings itself up-to-date with the leader. Looking at the logs for the
> follower you see that on re-start it identifies the leader and gets the
> changes from that leader.
>
> I think when documents are added to the leader these should be pushed to the
> followers so maybe it's this push process that isn't being triggered.
>
> It’s quite possible that I’ve made a simple error in the set-up but I’m
> baffled as to what it is. Please can anyone advise on any configuration that
> I need to check that might be causing these symptoms.
>
> Thanks in anticipation,
> Peter Lancaster.
>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/SolrCloud-leader-updates-not-updating-followers-tp4349618.html
> Sent from the Solr - User mailing list archive at Nabble.com.