Re: Master/Slave

[Eric Pugh]

I’ve met folks who’ve actually used the streaming expressions to move data 
around if you are looking for a “all Solr” approach.  If you go down that 
route, I’d love to hear how it works.

> On Oct 8, 2020, at 7:10 AM, Erick Erickson  wrote:
> 
> What Jan said. I wanted to add that the replication API also makes use of it. 
> A little-known fact: you can use the replication API in SolrCloud _without_ 
> configuring anything in solrconfig.xml. You can specify the URL to pull from 
> on the fly in the command….
> 
> Best,
> Erick
> 
>> On Oct 8, 2020, at 2:54 AM, Jan Høydahl  wrote:
>> 
>> The API that enables master/slave is the ReplicationHandler, where the 
>> follower (slave) pulls index files from leader (master).
>> This same API is used in SolrCloud for the PULL replica type, and also as a 
>> fallback for full recovery if transaction log is not enough. 
>> So I don’t see it going away anytime soon, even if the non-cloud deployment 
>> style is less promoted in the documentation.
>> 
>> Jan
>> 
>>> 6. okt. 2020 kl. 16:25 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>>> :
>>> 
>>>> it better not ever be depreciated.  it has been the most reliable 
>>>> mechanism for its purpose
>>> 
>>> I would like to know whether that is the consensus of Solr developers.
>>> 
>>> We had been scrambling to move from Master/Slave to CDCR based on the 
>>> assertion that CDCR support would last far longer than Master/Slave support.
>>> 
>>> Can we now assume safely that this assertion is now completely moot? Can we 
>>> now assume safely that Master/Slave is likely to be supported for the 
>>> foreseeable future? Or are we forced to assume that Master/Slave support 
>>> will evaporate shortly after the now-evaporated CDCR support?
>>> 
>>> -Original Message-
>>> From: David Hastings  
>>> Sent: Wednesday, September 30, 2020 3:10 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Master/Slave
>>> 
>>>> whether we should expect Master/Slave replication also to be deprecated
>>> 
>>> it better not ever be depreciated.  it has been the most reliable mechanism
>>> for its purpose, solr cloud isnt going to replace standalone, if it does,
>>> thats when I guess I stop upgrading or move to elastic
>>> 
>>> On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
>>>  wrote:
>>> 
>>>> Based on the thread below (reading "legacy" as meaning "likely to be
>>>> deprecated in later versions"), we have been working to extract ourselves
>>>> from Master/Slave replication
>>>> 
>>>> Most of our collections need to be in two data centers (a read/write copy
>>>> in one local data center: the disaster-recovery-site SolrCloud could be
>>>> read-only). We also need redundancy within each data center for when one
>>>> host or another is unavailable. We implemented this by having different
>>>> SolrClouds in the different data centers; with Master/Slave replication
>>>> pulling data from one of the read/write replicas to each of the Slave
>>>> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
>>>> for some collections, there is a desire to have local read-only replicas
>>>> remain unchanged for querying during the loading process: for these
>>>> collections, there is a local read/write loading SolrCloud, a local
>>>> read-only querying SolrCloud (normally configured for Master/Slave
>>>> replication from one of the replicas of the loader SolrCloud to both
>>>> replicas of the query SolrCloud, but with Master/Slave disabled when the
>>>> load was in progress on the loader SolrCloud, and with Master/Slave resumed
>>>> after the loaded data passes QA checks).
>>>> 
>>>> Based on the thread below, we made an attempt to switch to CDCR. The main
>>>> reason for wanting to change was that CDCR was said to be the supported
>>>> mechanism, and the replacement for Master/Slave replication.
>>>> 
>>>> After multiple unsuccessful attempts to get CDCR to work, we ended up with
>>>> reproducible cases of CDCR loosing data in transit. In June, I initiated a
>>>> thread in this group asking for clarification of how/whether CDCR could be
>>>> made reliable. This seemed to me to be met with deafening silence until the
>>>> announcement in July of the release of Solr8.6 a

Re: Master/Slave

[Erick Erickson]

What Jan said. I wanted to add that the replication API also makes use of it. A 
little-known fact: you can use the replication API in SolrCloud _without_ 
configuring anything in solrconfig.xml. You can specify the URL to pull from on 
the fly in the command….

Best,
Erick

> On Oct 8, 2020, at 2:54 AM, Jan Høydahl  wrote:
> 
> The API that enables master/slave is the ReplicationHandler, where the 
> follower (slave) pulls index files from leader (master).
> This same API is used in SolrCloud for the PULL replica type, and also as a 
> fallback for full recovery if transaction log is not enough. 
> So I don’t see it going away anytime soon, even if the non-cloud deployment 
> style is less promoted in the documentation.
> 
> Jan
> 
>> 6. okt. 2020 kl. 16:25 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
>> :
>> 
>>> it better not ever be depreciated.  it has been the most reliable mechanism 
>>> for its purpose
>> 
>> I would like to know whether that is the consensus of Solr developers.
>> 
>> We had been scrambling to move from Master/Slave to CDCR based on the 
>> assertion that CDCR support would last far longer than Master/Slave support.
>> 
>> Can we now assume safely that this assertion is now completely moot? Can we 
>> now assume safely that Master/Slave is likely to be supported for the 
>> foreseeable future? Or are we forced to assume that Master/Slave support 
>> will evaporate shortly after the now-evaporated CDCR support?
>> 
>> -Original Message-
>> From: David Hastings  
>> Sent: Wednesday, September 30, 2020 3:10 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Master/Slave
>> 
>>> whether we should expect Master/Slave replication also to be deprecated
>> 
>> it better not ever be depreciated.  it has been the most reliable mechanism
>> for its purpose, solr cloud isnt going to replace standalone, if it does,
>> thats when I guess I stop upgrading or move to elastic
>> 
>> On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
>>  wrote:
>> 
>>> Based on the thread below (reading "legacy" as meaning "likely to be
>>> deprecated in later versions"), we have been working to extract ourselves
>>> from Master/Slave replication
>>> 
>>> Most of our collections need to be in two data centers (a read/write copy
>>> in one local data center: the disaster-recovery-site SolrCloud could be
>>> read-only). We also need redundancy within each data center for when one
>>> host or another is unavailable. We implemented this by having different
>>> SolrClouds in the different data centers; with Master/Slave replication
>>> pulling data from one of the read/write replicas to each of the Slave
>>> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
>>> for some collections, there is a desire to have local read-only replicas
>>> remain unchanged for querying during the loading process: for these
>>> collections, there is a local read/write loading SolrCloud, a local
>>> read-only querying SolrCloud (normally configured for Master/Slave
>>> replication from one of the replicas of the loader SolrCloud to both
>>> replicas of the query SolrCloud, but with Master/Slave disabled when the
>>> load was in progress on the loader SolrCloud, and with Master/Slave resumed
>>> after the loaded data passes QA checks).
>>> 
>>> Based on the thread below, we made an attempt to switch to CDCR. The main
>>> reason for wanting to change was that CDCR was said to be the supported
>>> mechanism, and the replacement for Master/Slave replication.
>>> 
>>> After multiple unsuccessful attempts to get CDCR to work, we ended up with
>>> reproducible cases of CDCR loosing data in transit. In June, I initiated a
>>> thread in this group asking for clarification of how/whether CDCR could be
>>> made reliable. This seemed to me to be met with deafening silence until the
>>> announcement in July of the release of Solr8.6 and the deprecation of CDCR.
>>> 
>>> So we are left with the question whether we should expect Master/Slave
>>> replication also to be deprecated; and if so, with what is it expected to
>>> be replaced (since not with CDCR)? Or is it now sufficiently safe to assume
>>> that Master/Slave replication will continue to be supported after all
>>> (since the assertion that it would be replaced by CDCR has been
>>> discredited)? In either case, are there other suggested implementations of
>>> having a read-onl

Re: Master/Slave

[Jan Høydahl]

The API that enables master/slave is the ReplicationHandler, where the follower 
(slave) pulls index files from leader (master).
This same API is used in SolrCloud for the PULL replica type, and also as a 
fallback for full recovery if transaction log is not enough. 
So I don’t see it going away anytime soon, even if the non-cloud deployment 
style is less promoted in the documentation.

Jan

> 6. okt. 2020 kl. 16:25 skrev Oakley, Craig (NIH/NLM/NCBI) [C] 
> :
> 
>> it better not ever be depreciated.  it has been the most reliable mechanism 
>> for its purpose
> 
> I would like to know whether that is the consensus of Solr developers.
> 
> We had been scrambling to move from Master/Slave to CDCR based on the 
> assertion that CDCR support would last far longer than Master/Slave support.
> 
> Can we now assume safely that this assertion is now completely moot? Can we 
> now assume safely that Master/Slave is likely to be supported for the 
> foreseeable future? Or are we forced to assume that Master/Slave support will 
> evaporate shortly after the now-evaporated CDCR support?
> 
> -Original Message-
> From: David Hastings  
> Sent: Wednesday, September 30, 2020 3:10 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Master/Slave
> 
>> whether we should expect Master/Slave replication also to be deprecated
> 
> it better not ever be depreciated.  it has been the most reliable mechanism
> for its purpose, solr cloud isnt going to replace standalone, if it does,
> thats when I guess I stop upgrading or move to elastic
> 
> On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
>  wrote:
> 
>> Based on the thread below (reading "legacy" as meaning "likely to be
>> deprecated in later versions"), we have been working to extract ourselves
>> from Master/Slave replication
>> 
>> Most of our collections need to be in two data centers (a read/write copy
>> in one local data center: the disaster-recovery-site SolrCloud could be
>> read-only). We also need redundancy within each data center for when one
>> host or another is unavailable. We implemented this by having different
>> SolrClouds in the different data centers; with Master/Slave replication
>> pulling data from one of the read/write replicas to each of the Slave
>> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
>> for some collections, there is a desire to have local read-only replicas
>> remain unchanged for querying during the loading process: for these
>> collections, there is a local read/write loading SolrCloud, a local
>> read-only querying SolrCloud (normally configured for Master/Slave
>> replication from one of the replicas of the loader SolrCloud to both
>> replicas of the query SolrCloud, but with Master/Slave disabled when the
>> load was in progress on the loader SolrCloud, and with Master/Slave resumed
>> after the loaded data passes QA checks).
>> 
>> Based on the thread below, we made an attempt to switch to CDCR. The main
>> reason for wanting to change was that CDCR was said to be the supported
>> mechanism, and the replacement for Master/Slave replication.
>> 
>> After multiple unsuccessful attempts to get CDCR to work, we ended up with
>> reproducible cases of CDCR loosing data in transit. In June, I initiated a
>> thread in this group asking for clarification of how/whether CDCR could be
>> made reliable. This seemed to me to be met with deafening silence until the
>> announcement in July of the release of Solr8.6 and the deprecation of CDCR.
>> 
>> So we are left with the question whether we should expect Master/Slave
>> replication also to be deprecated; and if so, with what is it expected to
>> be replaced (since not with CDCR)? Or is it now sufficiently safe to assume
>> that Master/Slave replication will continue to be supported after all
>> (since the assertion that it would be replaced by CDCR has been
>> discredited)? In either case, are there other suggested implementations of
>> having a read-only SolrCloud receive data from a read/write SolrCloud?
>> 
>> 
>> Thanks
>> 
>> -Original Message-
>> From: Shawn Heisey 
>> Sent: Tuesday, May 21, 2019 11:15 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: SolrCloud (7.3) and Legacy replication slaves
>> 
>> On 5/21/2019 8:48 AM, Michael Tracey wrote:
>>> Is it possible set up an existing SolrCloud cluster as the master for
>>> legacy replication to a slave server or two?   It looks like another
>> option
>>> is to use Uni-direction CDCR, but not sure what is the best option in
>> this
>>> case.
>> 
>> You're asking for problems if you try to combine legacy replication with
>> SolrCloud.  The two features are not guaranteed to work together.
>> 
>> CDCR is your best bet.  This replicates from one SolrCloud cluster to
>> another.
>> 
>> Thanks,
>> Shawn
>> 

RE: Master/Slave

[Oakley, Craig (NIH/NLM/NCBI) [C]]

> it better not ever be depreciated.  it has been the most reliable mechanism 
> for its purpose

I would like to know whether that is the consensus of Solr developers.

We had been scrambling to move from Master/Slave to CDCR based on the assertion 
that CDCR support would last far longer than Master/Slave support.

Can we now assume safely that this assertion is now completely moot? Can we now 
assume safely that Master/Slave is likely to be supported for the foreseeable 
future? Or are we forced to assume that Master/Slave support will evaporate 
shortly after the now-evaporated CDCR support?

-Original Message-
From: David Hastings  
Sent: Wednesday, September 30, 2020 3:10 PM
To: solr-user@lucene.apache.org
Subject: Re: Master/Slave

>whether we should expect Master/Slave replication also to be deprecated

it better not ever be depreciated.  it has been the most reliable mechanism
for its purpose, solr cloud isnt going to replace standalone, if it does,
thats when I guess I stop upgrading or move to elastic

On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
 wrote:

> Based on the thread below (reading "legacy" as meaning "likely to be
> deprecated in later versions"), we have been working to extract ourselves
> from Master/Slave replication
>
> Most of our collections need to be in two data centers (a read/write copy
> in one local data center: the disaster-recovery-site SolrCloud could be
> read-only). We also need redundancy within each data center for when one
> host or another is unavailable. We implemented this by having different
> SolrClouds in the different data centers; with Master/Slave replication
> pulling data from one of the read/write replicas to each of the Slave
> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
> for some collections, there is a desire to have local read-only replicas
> remain unchanged for querying during the loading process: for these
> collections, there is a local read/write loading SolrCloud, a local
> read-only querying SolrCloud (normally configured for Master/Slave
> replication from one of the replicas of the loader SolrCloud to both
> replicas of the query SolrCloud, but with Master/Slave disabled when the
> load was in progress on the loader SolrCloud, and with Master/Slave resumed
> after the loaded data passes QA checks).
>
> Based on the thread below, we made an attempt to switch to CDCR. The main
> reason for wanting to change was that CDCR was said to be the supported
> mechanism, and the replacement for Master/Slave replication.
>
> After multiple unsuccessful attempts to get CDCR to work, we ended up with
> reproducible cases of CDCR loosing data in transit. In June, I initiated a
> thread in this group asking for clarification of how/whether CDCR could be
> made reliable. This seemed to me to be met with deafening silence until the
> announcement in July of the release of Solr8.6 and the deprecation of CDCR.
>
> So we are left with the question whether we should expect Master/Slave
> replication also to be deprecated; and if so, with what is it expected to
> be replaced (since not with CDCR)? Or is it now sufficiently safe to assume
> that Master/Slave replication will continue to be supported after all
> (since the assertion that it would be replaced by CDCR has been
> discredited)? In either case, are there other suggested implementations of
> having a read-only SolrCloud receive data from a read/write SolrCloud?
>
>
> Thanks
>
> -Original Message-
> From: Shawn Heisey 
> Sent: Tuesday, May 21, 2019 11:15 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud (7.3) and Legacy replication slaves
>
> On 5/21/2019 8:48 AM, Michael Tracey wrote:
> > Is it possible set up an existing SolrCloud cluster as the master for
> > legacy replication to a slave server or two?   It looks like another
> option
> > is to use Uni-direction CDCR, but not sure what is the best option in
> this
> > case.
>
> You're asking for problems if you try to combine legacy replication with
> SolrCloud.  The two features are not guaranteed to work together.
>
> CDCR is your best bet.  This replicates from one SolrCloud cluster to
> another.
>
> Thanks,
> Shawn
>

Re: Master/Slave

[Walter Underwood]

We do this sort of thing outside of Solr. The indexing process includes creating
a feed file with one JSON object per line. The feed files are stored in S3 with
names that are ISO 8601 timestamps. Those files are picked up and loaded into
Solr. Because S3 is cross-region in AWS, those files are also our disaster
recovery method for indexing. And of course, two clusters could be loaded
from the same file.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Sep 30, 2020, at 12:09 PM, David Hastings  
> wrote:
> 
>> whether we should expect Master/Slave replication also to be deprecated
> 
> it better not ever be depreciated.  it has been the most reliable mechanism
> for its purpose, solr cloud isnt going to replace standalone, if it does,
> thats when I guess I stop upgrading or move to elastic
> 
> On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
>  wrote:
> 
>> Based on the thread below (reading "legacy" as meaning "likely to be
>> deprecated in later versions"), we have been working to extract ourselves
>> from Master/Slave replication
>> 
>> Most of our collections need to be in two data centers (a read/write copy
>> in one local data center: the disaster-recovery-site SolrCloud could be
>> read-only). We also need redundancy within each data center for when one
>> host or another is unavailable. We implemented this by having different
>> SolrClouds in the different data centers; with Master/Slave replication
>> pulling data from one of the read/write replicas to each of the Slave
>> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
>> for some collections, there is a desire to have local read-only replicas
>> remain unchanged for querying during the loading process: for these
>> collections, there is a local read/write loading SolrCloud, a local
>> read-only querying SolrCloud (normally configured for Master/Slave
>> replication from one of the replicas of the loader SolrCloud to both
>> replicas of the query SolrCloud, but with Master/Slave disabled when the
>> load was in progress on the loader SolrCloud, and with Master/Slave resumed
>> after the loaded data passes QA checks).
>> 
>> Based on the thread below, we made an attempt to switch to CDCR. The main
>> reason for wanting to change was that CDCR was said to be the supported
>> mechanism, and the replacement for Master/Slave replication.
>> 
>> After multiple unsuccessful attempts to get CDCR to work, we ended up with
>> reproducible cases of CDCR loosing data in transit. In June, I initiated a
>> thread in this group asking for clarification of how/whether CDCR could be
>> made reliable. This seemed to me to be met with deafening silence until the
>> announcement in July of the release of Solr8.6 and the deprecation of CDCR.
>> 
>> So we are left with the question whether we should expect Master/Slave
>> replication also to be deprecated; and if so, with what is it expected to
>> be replaced (since not with CDCR)? Or is it now sufficiently safe to assume
>> that Master/Slave replication will continue to be supported after all
>> (since the assertion that it would be replaced by CDCR has been
>> discredited)? In either case, are there other suggested implementations of
>> having a read-only SolrCloud receive data from a read/write SolrCloud?
>> 
>> 
>> Thanks
>> 
>> -Original Message-
>> From: Shawn Heisey 
>> Sent: Tuesday, May 21, 2019 11:15 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: SolrCloud (7.3) and Legacy replication slaves
>> 
>> On 5/21/2019 8:48 AM, Michael Tracey wrote:
>>> Is it possible set up an existing SolrCloud cluster as the master for
>>> legacy replication to a slave server or two?   It looks like another
>> option
>>> is to use Uni-direction CDCR, but not sure what is the best option in
>> this
>>> case.
>> 
>> You're asking for problems if you try to combine legacy replication with
>> SolrCloud.  The two features are not guaranteed to work together.
>> 
>> CDCR is your best bet.  This replicates from one SolrCloud cluster to
>> another.
>> 
>> Thanks,
>> Shawn
>> 

Re: Master/Slave

[David Hastings]

>whether we should expect Master/Slave replication also to be deprecated

it better not ever be depreciated.  it has been the most reliable mechanism
for its purpose, solr cloud isnt going to replace standalone, if it does,
thats when I guess I stop upgrading or move to elastic

On Wed, Sep 30, 2020 at 2:58 PM Oakley, Craig (NIH/NLM/NCBI) [C]
 wrote:

> Based on the thread below (reading "legacy" as meaning "likely to be
> deprecated in later versions"), we have been working to extract ourselves
> from Master/Slave replication
>
> Most of our collections need to be in two data centers (a read/write copy
> in one local data center: the disaster-recovery-site SolrCloud could be
> read-only). We also need redundancy within each data center for when one
> host or another is unavailable. We implemented this by having different
> SolrClouds in the different data centers; with Master/Slave replication
> pulling data from one of the read/write replicas to each of the Slave
> replicas in the disaster-recovery-site read-only SolrCloud. Additionally,
> for some collections, there is a desire to have local read-only replicas
> remain unchanged for querying during the loading process: for these
> collections, there is a local read/write loading SolrCloud, a local
> read-only querying SolrCloud (normally configured for Master/Slave
> replication from one of the replicas of the loader SolrCloud to both
> replicas of the query SolrCloud, but with Master/Slave disabled when the
> load was in progress on the loader SolrCloud, and with Master/Slave resumed
> after the loaded data passes QA checks).
>
> Based on the thread below, we made an attempt to switch to CDCR. The main
> reason for wanting to change was that CDCR was said to be the supported
> mechanism, and the replacement for Master/Slave replication.
>
> After multiple unsuccessful attempts to get CDCR to work, we ended up with
> reproducible cases of CDCR loosing data in transit. In June, I initiated a
> thread in this group asking for clarification of how/whether CDCR could be
> made reliable. This seemed to me to be met with deafening silence until the
> announcement in July of the release of Solr8.6 and the deprecation of CDCR.
>
> So we are left with the question whether we should expect Master/Slave
> replication also to be deprecated; and if so, with what is it expected to
> be replaced (since not with CDCR)? Or is it now sufficiently safe to assume
> that Master/Slave replication will continue to be supported after all
> (since the assertion that it would be replaced by CDCR has been
> discredited)? In either case, are there other suggested implementations of
> having a read-only SolrCloud receive data from a read/write SolrCloud?
>
>
> Thanks
>
> -Original Message-
> From: Shawn Heisey 
> Sent: Tuesday, May 21, 2019 11:15 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SolrCloud (7.3) and Legacy replication slaves
>
> On 5/21/2019 8:48 AM, Michael Tracey wrote:
> > Is it possible set up an existing SolrCloud cluster as the master for
> > legacy replication to a slave server or two?   It looks like another
> option
> > is to use Uni-direction CDCR, but not sure what is the best option in
> this
> > case.
>
> You're asking for problems if you try to combine legacy replication with
> SolrCloud.  The two features are not guaranteed to work together.
>
> CDCR is your best bet.  This replicates from one SolrCloud cluster to
> another.
>
> Thanks,
> Shawn
>

Re: Master Slave Terminology

[Walter Underwood]

I’ve long thought that master/slave was not the right metaphor for a pull model 
anyway.

We probably should not use “replica” since that already has a use in Solr Cloud.

Where is the discussion?

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Jun 16, 2020, at 11:51 PM, Jan Høydahl  wrote:
> 
> Hi Kaya,
> 
> Thanks for bringing it up. The topic is already being discussed by 
> developers, so expect to see some change in this area; Not over-night, but 
> incremental.
> Also, if you want to lend a helping hand, patches are more than welcome as 
> always.
> 
> Jan
> 
>> 17. jun. 2020 kl. 04:22 skrev Kayak28 :
>> 
>> Hello, Community:
>> 
>> As the Github and Python will replace terminologies that relative to
>> slavery,
>> why don't we replace master-slave for Solr as well?
>> 
>> https://developers.srad.jp/story/18/09/14/0935201/
>> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
>> 
>> -- 
>> 
>> Sincerely,
>> Kaya
>> github: https://github.com/28kayak
> 

Re: Master Slave Terminology

[Doug Turnbull]

+1 to name change. Also 'overseer' which doesn't go well with Master/Slave!

On Wed, Jun 17, 2020 at 11:16 AM David Smiley 
wrote:

> priv...@lucene.apache.org but it should have been public and expect it to
> spill out to the dev list today.
>
> ~ David
>
>
> On Wed, Jun 17, 2020 at 11:14 AM Mike Drob  wrote:
>
> > Hi Jan,
> >
> > Can you link to the discussion? I searched the dev list and didn’t see
> > anything, is it on slack or a jira or somewhere else?
> >
> > Mike
> >
> > On Wed, Jun 17, 2020 at 1:51 AM Jan Høydahl 
> wrote:
> >
> > > Hi Kaya,
> > >
> > > Thanks for bringing it up. The topic is already being discussed by
> > > developers, so expect to see some change in this area; Not over-night,
> > but
> > > incremental.
> > > Also, if you want to lend a helping hand, patches are more than welcome
> > as
> > > always.
> > >
> > > Jan
> > >
> > > > 17. jun. 2020 kl. 04:22 skrev Kayak28 :
> > > >
> > > > Hello, Community:
> > > >
> > > > As the Github and Python will replace terminologies that relative to
> > > > slavery,
> > > > why don't we replace master-slave for Solr as well?
> > > >
> > > > https://developers.srad.jp/story/18/09/14/0935201/
> > > >
> > >
> >
> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
> > > >
> > > > --
> > > >
> > > > Sincerely,
> > > > Kaya
> > > > github: https://github.com/28kayak
> > >
> > >
> >
>


-- 
*Doug Turnbull **| CTO* | OpenSource Connections
, LLC | 240.476.9983
Author: Relevant Search ; Contributor: *AI
Powered Search *
This e-mail and all contents, including attachments, is considered to be
Company Confidential unless explicitly stated otherwise, regardless
of whether attachments are marked as such.

Re: Master Slave Terminology

[David Smiley]

priv...@lucene.apache.org but it should have been public and expect it to
spill out to the dev list today.

~ David


On Wed, Jun 17, 2020 at 11:14 AM Mike Drob  wrote:

> Hi Jan,
>
> Can you link to the discussion? I searched the dev list and didn’t see
> anything, is it on slack or a jira or somewhere else?
>
> Mike
>
> On Wed, Jun 17, 2020 at 1:51 AM Jan Høydahl  wrote:
>
> > Hi Kaya,
> >
> > Thanks for bringing it up. The topic is already being discussed by
> > developers, so expect to see some change in this area; Not over-night,
> but
> > incremental.
> > Also, if you want to lend a helping hand, patches are more than welcome
> as
> > always.
> >
> > Jan
> >
> > > 17. jun. 2020 kl. 04:22 skrev Kayak28 :
> > >
> > > Hello, Community:
> > >
> > > As the Github and Python will replace terminologies that relative to
> > > slavery,
> > > why don't we replace master-slave for Solr as well?
> > >
> > > https://developers.srad.jp/story/18/09/14/0935201/
> > >
> >
> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
> > >
> > > --
> > >
> > > Sincerely,
> > > Kaya
> > > github: https://github.com/28kayak
> >
> >
>

Re: Master Slave Terminology

[Mike Drob]

Hi Jan,

Can you link to the discussion? I searched the dev list and didn’t see
anything, is it on slack or a jira or somewhere else?

Mike

On Wed, Jun 17, 2020 at 1:51 AM Jan Høydahl  wrote:

> Hi Kaya,
>
> Thanks for bringing it up. The topic is already being discussed by
> developers, so expect to see some change in this area; Not over-night, but
> incremental.
> Also, if you want to lend a helping hand, patches are more than welcome as
> always.
>
> Jan
>
> > 17. jun. 2020 kl. 04:22 skrev Kayak28 :
> >
> > Hello, Community:
> >
> > As the Github and Python will replace terminologies that relative to
> > slavery,
> > why don't we replace master-slave for Solr as well?
> >
> > https://developers.srad.jp/story/18/09/14/0935201/
> >
> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
> >
> > --
> >
> > Sincerely,
> > Kaya
> > github: https://github.com/28kayak
>
>

Re: Master Slave Terminology

[Jan Høydahl]

Hi Kaya,

Thanks for bringing it up. The topic is already being discussed by developers, 
so expect to see some change in this area; Not over-night, but incremental.
Also, if you want to lend a helping hand, patches are more than welcome as 
always.

Jan

> 17. jun. 2020 kl. 04:22 skrev Kayak28 :
> 
> Hello, Community:
> 
> As the Github and Python will replace terminologies that relative to
> slavery,
> why don't we replace master-slave for Solr as well?
> 
> https://developers.srad.jp/story/18/09/14/0935201/
> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
> 
> -- 
> 
> Sincerely,
> Kaya
> github: https://github.com/28kayak

Re: Master Slave Replication Issue

[Erick Erickson]

Hmmm, afraid I'm out of my depth, perhaps some of the Lucene
folks can chime in.

 Sorry I can't be more  help.
Erick
On Mon, Nov 12, 2018 at 12:27 AM damian.pawski  wrote:
>
> Hi
>
> I had to re-create the index as some tokenizers are not longer supported on
> the 7.x version.
> I have a fresh 7.x index.
>
> Thank you
> Damian
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication Issue

[damian.pawski]

Hi

I had to re-create the index as some tokenizers are not longer supported on
the 7.x version.
I have a fresh 7.x index.

Thank you
Damian



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication Issue

[damian.pawski]

Hi

I had to re-create the index, as some Tokenizers are no longer supported on
7.X, so I have a fresh 7.x index, but still having issues with the
replication.

Thank you
Damian



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication Issue

[Erick Erickson]

Damian:

You say you've switched from 5x to 7x. Did you try to use an index
created with 5x or did you index fresh with 7x? Solr/Lucene do not
guarantee backward compatibility across more than one major version.

Best,
Erick
On Fri, Nov 9, 2018 at 2:34 AM damian.pawski  wrote:
>
> Hi,
> We have switched from 5.4 to 7.2.1 and we have started to see more issues
> with the replication.
> I think it may be related to the fact that a delta import was started during
> a full import (not the case for the Solr 5.4).
>
> I am getting below error:
>
> XXX: java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
> Directory MMapDirectory@XXX\index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@21ff4974 still has
> pending deleted files; cannot initialize IndexWriter
>
> Are there more known issues with Solr 7.X and the replication?
> Based on https://issues.apache.org/jira/browse/SOLR-11938 I can not trust
> Solr 7.X anymore.
>
> How can I fix the
> "XXX: java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
> Directory MMapDirectory@XXX\index
> lockFactory=org.apache.lucene.store.NativeFSLockFactory@21ff4974 still has
> pending deleted files; cannot initialize IndexWriter
> "
> issue?
>
> Thank you
> Damian
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication Issue

[damian.pawski]

Hi, 
We have switched from 5.4 to 7.2.1 and we have started to see more issues
with the replication.
I think it may be related to the fact that a delta import was started during
a full import (not the case for the Solr 5.4).

I am getting below error:

XXX: java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
Directory MMapDirectory@XXX\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@21ff4974 still has
pending deleted files; cannot initialize IndexWriter

Are there more known issues with Solr 7.X and the replication?
Based on https://issues.apache.org/jira/browse/SOLR-11938 I can not trust
Solr 7.X anymore.

How can I fix the 
"XXX: java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
Directory MMapDirectory@XXX\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@21ff4974 still has
pending deleted files; cannot initialize IndexWriter
"
issue?

Thank you
Damian



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication of Solr with Basic Authentication

[Shawn Heisey]

On 3/25/2018 1:56 AM, Basheeruddin Ahmed (syedbahm) wrote:

Seems even when we use Secuirty.json with BasicAuthentication Plugin as 
documented here -- 
https://lucene.apache.org/solr/guide/7_2/basic-authentication-plugin.html
, which nicely encrypts the user password using SHA256 encryption,  when it 
comes to configuring



Please let me know how I can use the same encrypted password as in 
Security.json when setting up Master/Slave Replication for Solr.


At the moment, the cleartext password is the only way it can be configured.

It is not possible to use the same string that goes in security.json for 
a feature like replication.  That string is a one-way hash of the 
password, so it cannot be decrypted.  The replication handler must be 
able to obtain the cleartext password.


The DIH feature offers password encryption for database passwords.  
Scroll down a little bit on the following page to the description 
numbered "2":


https://lucene.apache.org/solr/guide/6_6/uploading-structured-data-store-data-with-the-data-import-handler.html#configuring-the-dih-configuration-file

The replication handler CAN be enhanced to use a the same kind of 
encryption.  Note that this is merely security through obscurity.  If 
whoever is looking at the configuration also has access to the key file, 
then they will be able to decrypt the password.


Can you file an enhancement issue in Jira to add this capability to 
other handlers like replication?


https://issues.apache.org/jira/browse/SOLR

You'll need an account to create an issue.  Anyone can create an account.

Thanks,
Shawn

Re: Master Slave Replication Issue

[den_nal]

Done https://issues.apache.org/jira/browse/SOLR-11938.

Thanks!



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Master Slave Replication Issue

[Tomas Fernandez Lobbe]

This seems pretty serious. Please create a Jira issue

Sent from my iPhone

> On Feb 1, 2018, at 12:15 AM, dennis nalog  
> wrote:
> 
> Hi,
> We are using Solr 7.1 and are solr setup is master-slave replication.
> We encounter this issue that when we disable the replication in master via UI 
> or URL (server/solr/solr_core/replication?command=disablereplication), the 
> data in our slave servers suddenly becomes 0.
> Just wanna know if this is a known issue or this is the expected behavior. 
> Thanks in advance.
> Best regards,Dennis

Re: master slave replication taking time

[Erick Erickson]

So you say that master/slave replication takes up to 20 seconds but it
takes 8-10 minutes just to copy the entire index?

Solr is _already_ speeding up your copy enormously by just copying
changed segments. There's nothing magic you can do to make it faster.
And if your master happens to merge many segments into a large one
your replication will take up to the time it takes to copy your entire
index (8-10 minutes).

Going to SolrCloud would eliminate that kind of lag at the expense of
indexing on every node. Otherwise it's the best you can do.

Best,
Erick

On Thu, Jun 29, 2017 at 12:21 AM, Midas A  wrote:
> Erick,
>
> when we copy entire index it takes 8- 10 mins .
>
>
> On Wed, Jun 28, 2017 at 9:22 PM, Erick Erickson 
> wrote:
>
>> How long it takes to copy the entire index from one machine to another
>> over your network. Solr can't go any faster than your network can
>> support. Consider SolrCloud if you need something closer to NRT.
>>
>> Best,
>> Erick
>>
>> On Tue, Jun 27, 2017 at 11:31 PM, Midas A  wrote:
>> > Hi,
>> >
>> > we have around 2000 documents and our master to slave replication is
>> > taking time  upto 20 second.
>> >
>> > What should i check ?
>>

Re: master slave replication taking time

[Midas A]

Erick,

when we copy entire index it takes 8- 10 mins .


On Wed, Jun 28, 2017 at 9:22 PM, Erick Erickson 
wrote:

> How long it takes to copy the entire index from one machine to another
> over your network. Solr can't go any faster than your network can
> support. Consider SolrCloud if you need something closer to NRT.
>
> Best,
> Erick
>
> On Tue, Jun 27, 2017 at 11:31 PM, Midas A  wrote:
> > Hi,
> >
> > we have around 2000 documents and our master to slave replication is
> > taking time  upto 20 second.
> >
> > What should i check ?
>

Re: master slave replication taking time

[Erick Erickson]

How long it takes to copy the entire index from one machine to another
over your network. Solr can't go any faster than your network can
support. Consider SolrCloud if you need something closer to NRT.

Best,
Erick

On Tue, Jun 27, 2017 at 11:31 PM, Midas A  wrote:
> Hi,
>
> we have around 2000 documents and our master to slave replication is
> taking time  upto 20 second.
>
> What should i check ?

RE: Master/Slave out of sync

[Pouliot, Scott]

I figured the attachments would get stripped, but it was worth a shot!  It was 
just a screenshot showing the version numbers off from each other.

Here are the Master/Slave commit settings:

 
   18 
   false 


 
   6 


-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Tuesday, June 27, 2017 11:17 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Master/Slave out of sync

First, attachments are almost always stripped by the mail program, so we can't 
see anything.

Hmmm, does look odd. What happens if you issue a commit against the slave via a 
url? I.e.
http://server:port/solr/core/update?commit=true?

And what are the autocommit settings on the slave?

Best,
Erick

On Tue, Jun 27, 2017 at 7:22 AM, Pouliot, Scott < 
scott.poul...@peoplefluent.com> wrote:

> Hey guys…
>
>
>
> Does anyone else have a problem with the master/slave setup getting 
> out of sync and staying that way until I either optimize the core or 
> restart SOLR?  It seems to be happening more and more frequently these 
> days and I’m looking for a solution here.  Running SOLR 6.2 on these 
> instances using jetty.
>
>
>
> I do see some log entries like the following at the moment, but it has 
> happened WITHOUT these errors in the past as well.  This error just 
> looks like the core is being loaded, so it can’t replicate (as far as I can 
> tell):
>
>
>
> 2017-06-23 00:44:08.624 ERROR (indexFetcher-677-thread-1) [   x:Client1]
> o.a.s.h.IndexFetcher Master at: http://master:8080/solr/Client1 is not 
> available. Index fetch failed. Exception: Error from server at
> http://master:8080/solr/Client1: Expected mime type 
> application/octet-stream but got text/html. 
>
> 
>
> 
>
> Error 503 {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},m
> sg=SolrCore
> is loading,code=503}
>
> 
>
> HTTP ERROR 503
>
> Problem accessing /solr/Client1/replication. Reason:
>
> {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},m
> sg=SolrCore
> is loading,code=503}
>
> 
>
> 
>
>
>
> Our setup looks something like this:
>
>
>
> Master
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Slave
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Master Config
>
>  class="solr.ReplicationHandler"
> >
>
> 
>
>   
>
>   startup
>
>   commit
>
>
>
>   
>
>   00:00:10
>
> 
>
> 
>
> 
>
> 1
>
>   
>
>
>
>
>
> Slave Config
>
>  class="solr.ReplicationHandler"
> >
>
> 
>
>
>
>   
>
>   http://master:8080/solr/${solr.core.name}
> 
>
>
>
>   
>
>   00:00:45
>
> 
>
>   
>
>
>
> Master screenshot
>
>
>
>
>
> Slave Screenshot
>
>

Re: Master/Slave out of sync

[Erick Erickson]

First, attachments are almost always stripped by the mail program, so we
can't see anything.

Hmmm, does look odd. What happens if you issue a commit against the slave
via a url? I.e.
http://server:port/solr/core/update?commit=true?

And what are the autocommit settings on the slave?

Best,
Erick

On Tue, Jun 27, 2017 at 7:22 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:

> Hey guys…
>
>
>
> Does anyone else have a problem with the master/slave setup getting out of
> sync and staying that way until I either optimize the core or restart
> SOLR?  It seems to be happening more and more frequently these days and I’m
> looking for a solution here.  Running SOLR 6.2 on these instances using
> jetty.
>
>
>
> I do see some log entries like the following at the moment, but it has
> happened WITHOUT these errors in the past as well.  This error just looks
> like the core is being loaded, so it can’t replicate (as far as I can tell):
>
>
>
> 2017-06-23 00:44:08.624 ERROR (indexFetcher-677-thread-1) [   x:Client1]
> o.a.s.h.IndexFetcher Master at: http://master:8080/solr/Client1 is not
> available. Index fetch failed. Exception: Error from server at
> http://master:8080/solr/Client1: Expected mime type
> application/octet-stream but got text/html. 
>
> 
>
> 
>
> Error 503 {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore
> is loading,code=503}
>
> 
>
> HTTP ERROR 503
>
> Problem accessing /solr/Client1/replication. Reason:
>
> {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore
> is loading,code=503}
>
> 
>
> 
>
>
>
> Our setup looks something like this:
>
>
>
> Master
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Slave
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Master Config
>
>  class="solr.ReplicationHandler"
> >
>
> 
>
>   
>
>   startup
>
>   commit
>
>
>
>   
>
>   00:00:10
>
> 
>
> 
>
> 
>
> 1
>
>   
>
>
>
>
>
> Slave Config
>
>  class="solr.ReplicationHandler"
> >
>
> 
>
>
>
>   
>
>   http://master:8080/solr/${solr.core.name}
> 
>
>
>
>   
>
>   00:00:45
>
> 
>
>   
>
>
>
> Master screenshot
>
>
>
>
>
> Slave Screenshot
>
>

Re: Master Slave set up in Solr Cloud

[S.L]

Resending this  as I might have not been clear in my earlier query.

I want to use SolrCloud for everything except the replication , is it
possible to set up the master-slave configuration using different Solr
instances and still be able to use the sharding feature provided by
SolrCloud ?

On Thu, Oct 30, 2014 at 6:18 PM, S.L simpleliving...@gmail.com wrote:
 Hi All,

 As I previously reported due to no overlap in terms of the documets in the
 SolrCloud replicas of the index shards , I have turned off the replication
 and basically have there shards with a replication factor of 1.

 It obviously seems will not be scalable due to the fact that the same core
 will be indexed and queried at the same time as this is a long running
 indexing task.

 My questions is what options do I have to set up the replicas of the single
 per shard core outside of the SolrCloud replication factor mechanism because
 that does not seem to work for me ?


 Thanks.

Re: Master Slave set up in Solr Cloud

[Erick Erickson]

I _strongly_ urge you to try it before assuming it won't work.

bq: It obviously seems will not be scalable due to the fact that the same core
will be indexed and queried at the same time as this is a long running
indexing task.

What evidence do you have for this? SolrCloud is designed for this scenario.
You have to have a huge ingestion rate for this to be a problem, essentially you
need to be pushing Solr's CPU utilization through the roof.

I don't know of configurations where you can essentially have each of
your shards
act as a master in the old-style master/slave situation. But a lot of
people with
very high indexing and query rates seem to make SolrCloud work in
their situations,
before recommending anything I'd like to see some evidence that the stock
solution won't work.

Best,
Erick

On Sun, Nov 2, 2014 at 7:06 PM, S.L simpleliving...@gmail.com wrote:
 Resending this  as I might have not been clear in my earlier query.

 I want to use SolrCloud for everything except the replication , is it
 possible to set up the master-slave configuration using different Solr
 instances and still be able to use the sharding feature provided by
 SolrCloud ?

 On Thu, Oct 30, 2014 at 6:18 PM, S.L simpleliving...@gmail.com wrote:
 Hi All,

 As I previously reported due to no overlap in terms of the documets in the
 SolrCloud replicas of the index shards , I have turned off the replication
 and basically have there shards with a replication factor of 1.

 It obviously seems will not be scalable due to the fact that the same core
 will be indexed and queried at the same time as this is a long running
 indexing task.

 My questions is what options do I have to set up the replicas of the single
 per shard core outside of the SolrCloud replication factor mechanism because
 that does not seem to work for me ?


 Thanks.

Re: Master Slave set up in Solr Cloud

[Meraj A. Khan]

Eirck,

I meant having a replication factor of 1 in SolrCloud would not be
scalable, I ended up using a replication factor of 1 , because of the
disjoint replicas issue for replication factor greater than 1 , that I
mentioned previously in the other email thread.

Are you saying that a SolrCloud cluster with a replication factor of 1
would be performant as well ? I had read in SolrCloud documentation
that the required number of replicas is directly proportional to the
number of queries(concurrent).

Thanks

On Sun, Nov 2, 2014 at 11:59 PM, Erick Erickson erickerick...@gmail.com wrote:
 I _strongly_ urge you to try it before assuming it won't work.

 bq: It obviously seems will not be scalable due to the fact that the same core
 will be indexed and queried at the same time as this is a long running
 indexing task.

 What evidence do you have for this? SolrCloud is designed for this scenario.
 You have to have a huge ingestion rate for this to be a problem, essentially 
 you
 need to be pushing Solr's CPU utilization through the roof.

 I don't know of configurations where you can essentially have each of
 your shards
 act as a master in the old-style master/slave situation. But a lot of
 people with
 very high indexing and query rates seem to make SolrCloud work in
 their situations,
 before recommending anything I'd like to see some evidence that the stock
 solution won't work.

 Best,
 Erick

 On Sun, Nov 2, 2014 at 7:06 PM, S.L simpleliving...@gmail.com wrote:
 Resending this  as I might have not been clear in my earlier query.

 I want to use SolrCloud for everything except the replication , is it
 possible to set up the master-slave configuration using different Solr
 instances and still be able to use the sharding feature provided by
 SolrCloud ?

 On Thu, Oct 30, 2014 at 6:18 PM, S.L simpleliving...@gmail.com wrote:
 Hi All,

 As I previously reported due to no overlap in terms of the documets in the
 SolrCloud replicas of the index shards , I have turned off the replication
 and basically have there shards with a replication factor of 1.

 It obviously seems will not be scalable due to the fact that the same core
 will be indexed and queried at the same time as this is a long running
 indexing task.

 My questions is what options do I have to set up the replicas of the single
 per shard core outside of the SolrCloud replication factor mechanism because
 that does not seem to work for me ?


 Thanks.

Re: Master-Slave setup using SolrCloud

[Sachin Kale]

Apparently, there is a bug in Solr 4.10.0 which was causing the
NullPointerExceptions. SOLR-6501
https://issues.apache.org/jira/browse/SOLR-6501
We have updated our production SOLR to 4.10.1


On Thu, Oct 2, 2014 at 8:13 PM, Sachin Kale sachinpk...@gmail.com wrote:

 If I look into the logs, many times I get only following line without any
 stacktrace:

 *ERROR - 2014-10-02 19:35:25.516; org.apache.solr.common.SolrException;
 java.lang.NullPointerException*

 These exceptions are not coming continuously. Once in every 10-15 minutes.
 But once it starts, there are continuous 800-1000 such exceptions one after
 another. Is it related to cache warmup?

 I can provide following information regarding the setup:
 We are now on using Solr 4.10.0
 Memory allocated to each SOLR instance is 7GB. I guess it is more than
 sufficient for 1 GB index, right?
 Indexes are stored as normal, local filesystem.
 I am using three caches:
 Query Cache: Size 4096, autoWarmCount 2048
 Filter cache: size 8192, autoWarmCount 4096
 Document cache: size 4096

 I am experimenting with commitMaxTime for both soft and hard commits

 After referring following:

 http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

 Hence, I set following:

 autoCommit
 maxTime${solr.autoCommit.maxTime:6}/maxTime
 openSearcherfalse/openSearcher
 /autoCommit

 autoSoftCommit
 maxTime${solr.autoSoftCommit.maxTime:90}/maxTime
 /autoSoftCommit

 Also, we are getting following warnings many times:

 *java.lang.NumberFormatException: For input string: 5193.0*

 Earlier we were on SOLR 4.4.0 and when we are upgraded to 4.10.0, we
 pointed it to the same index we were using for 4.4.0

 On Thu, Oct 2, 2014 at 7:11 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/2/2014 6:58 AM, Sachin Kale wrote:
  We are trying to move our traditional master-slave Solr configuration to
  SolrCloud. As our index size is very small (around 1 GB), we are having
  only one shard.
  So basically, we are having same master-slave configuration with one
 leader
  and 6 replicas.
  We are experimenting with maxTime of both AutoCommit and AutoSoftCommit.
  Currently, autoCommit maxTime is 15 minutes and autoSoftCommit is 1
 minute
  (Let me know if these values does not make sense).
 
  Caches are set such that warmup time is at most 20 seconds.
 
  We are having continuous indexing requests mostly for updating the
 existing
  documents. Few requests are for deleting/adding the documents.
 
  The problem we are facing is that we are getting very frequent
  NullPointerExceptions.
  We get continuous 200-300 such exceptions within a period of 30 seconds
 and
  for next few minutes, it works fine.
 
  Stacktrace of NullPointerException:
 
  *ERROR - 2014-10-02 18:09:38.464; org.apache.solr.common.SolrException;
  null:java.lang.NullPointerException*
  *at
 
 org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:1257)*
  *at
 
 org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:720)*
  *at
 
 org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)*
 
  ​
  I am not sure what would be causing it. My guess, whenever, it is
 trying to
  replay tlog, we are getting these exceptions. Is anything wrong in my
  configuration?

 Your automatic commit settings are fine.  If you had tried to use a very
 small maxTime like 1000 (1 second), I would tell you that it's probably
 too short.

 The tlogs only get replayed when a core is first started or reloaded.
 These appear to be errors during queries, having nothing at all to do
 with indexing.

 I can't be sure with the available information (no Solr version,
 incomplete stacktrace, no info about what request caused and received
 the error), but if I had to guess, I'd say you probably changed your
 schema so that certain fields are now required that weren't required
 before, and didn't reindex, so those fields are not present on every
 document.  Or it might be that you added a uniqueKey and didn't reindex,
 and that field is not present on every document.

 http://wiki.apache.org/solr/HowToReindex

 Thanks,
 Shawn

Re: Master-Slave setup using SolrCloud

[Shawn Heisey]

On 10/2/2014 6:58 AM, Sachin Kale wrote:
 We are trying to move our traditional master-slave Solr configuration to
 SolrCloud. As our index size is very small (around 1 GB), we are having
 only one shard.
 So basically, we are having same master-slave configuration with one leader
 and 6 replicas.
 We are experimenting with maxTime of both AutoCommit and AutoSoftCommit.
 Currently, autoCommit maxTime is 15 minutes and autoSoftCommit is 1 minute
 (Let me know if these values does not make sense).
 
 Caches are set such that warmup time is at most 20 seconds.
 
 We are having continuous indexing requests mostly for updating the existing
 documents. Few requests are for deleting/adding the documents.
 
 The problem we are facing is that we are getting very frequent
 NullPointerExceptions.
 We get continuous 200-300 such exceptions within a period of 30 seconds and
 for next few minutes, it works fine.
 
 Stacktrace of NullPointerException:
 
 *ERROR - 2014-10-02 18:09:38.464; org.apache.solr.common.SolrException;
 null:java.lang.NullPointerException*
 *at
 org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:1257)*
 *at
 org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:720)*
 *at
 org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)*
 
 ​
 I am not sure what would be causing it. My guess, whenever, it is trying to
 replay tlog, we are getting these exceptions. Is anything wrong in my
 configuration?

Your automatic commit settings are fine.  If you had tried to use a very
small maxTime like 1000 (1 second), I would tell you that it's probably
too short.

The tlogs only get replayed when a core is first started or reloaded.
These appear to be errors during queries, having nothing at all to do
with indexing.

I can't be sure with the available information (no Solr version,
incomplete stacktrace, no info about what request caused and received
the error), but if I had to guess, I'd say you probably changed your
schema so that certain fields are now required that weren't required
before, and didn't reindex, so those fields are not present on every
document.  Or it might be that you added a uniqueKey and didn't reindex,
and that field is not present on every document.

http://wiki.apache.org/solr/HowToReindex

Thanks,
Shawn

Re: Master-Slave setup using SolrCloud

[Sachin Kale]

If I look into the logs, many times I get only following line without any
stacktrace:

*ERROR - 2014-10-02 19:35:25.516; org.apache.solr.common.SolrException;
java.lang.NullPointerException*

These exceptions are not coming continuously. Once in every 10-15 minutes.
But once it starts, there are continuous 800-1000 such exceptions one after
another. Is it related to cache warmup?

I can provide following information regarding the setup:
We are now on using Solr 4.10.0
Memory allocated to each SOLR instance is 7GB. I guess it is more than
sufficient for 1 GB index, right?
Indexes are stored as normal, local filesystem.
I am using three caches:
Query Cache: Size 4096, autoWarmCount 2048
Filter cache: size 8192, autoWarmCount 4096
Document cache: size 4096

I am experimenting with commitMaxTime for both soft and hard commits

After referring following:
http://lucidworks.com/blog/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

Hence, I set following:

autoCommit
maxTime${solr.autoCommit.maxTime:6}/maxTime
openSearcherfalse/openSearcher
/autoCommit

autoSoftCommit
maxTime${solr.autoSoftCommit.maxTime:90}/maxTime
/autoSoftCommit

Also, we are getting following warnings many times:

*java.lang.NumberFormatException: For input string: 5193.0*

Earlier we were on SOLR 4.4.0 and when we are upgraded to 4.10.0, we
pointed it to the same index we were using for 4.4.0

On Thu, Oct 2, 2014 at 7:11 PM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/2/2014 6:58 AM, Sachin Kale wrote:
  We are trying to move our traditional master-slave Solr configuration to
  SolrCloud. As our index size is very small (around 1 GB), we are having
  only one shard.
  So basically, we are having same master-slave configuration with one
 leader
  and 6 replicas.
  We are experimenting with maxTime of both AutoCommit and AutoSoftCommit.
  Currently, autoCommit maxTime is 15 minutes and autoSoftCommit is 1
 minute
  (Let me know if these values does not make sense).
 
  Caches are set such that warmup time is at most 20 seconds.
 
  We are having continuous indexing requests mostly for updating the
 existing
  documents. Few requests are for deleting/adding the documents.
 
  The problem we are facing is that we are getting very frequent
  NullPointerExceptions.
  We get continuous 200-300 such exceptions within a period of 30 seconds
 and
  for next few minutes, it works fine.
 
  Stacktrace of NullPointerException:
 
  *ERROR - 2014-10-02 18:09:38.464; org.apache.solr.common.SolrException;
  null:java.lang.NullPointerException*
  *at
 
 org.apache.solr.handler.component.QueryComponent.returnFields(QueryComponent.java:1257)*
  *at
 
 org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:720)*
  *at
 
 org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:695)*
 
  ​
  I am not sure what would be causing it. My guess, whenever, it is trying
 to
  replay tlog, we are getting these exceptions. Is anything wrong in my
  configuration?

 Your automatic commit settings are fine.  If you had tried to use a very
 small maxTime like 1000 (1 second), I would tell you that it's probably
 too short.

 The tlogs only get replayed when a core is first started or reloaded.
 These appear to be errors during queries, having nothing at all to do
 with indexing.

 I can't be sure with the available information (no Solr version,
 incomplete stacktrace, no info about what request caused and received
 the error), but if I had to guess, I'd say you probably changed your
 schema so that certain fields are now required that weren't required
 before, and didn't reindex, so those fields are not present on every
 document.  Or it might be that you added a uniqueKey and didn't reindex,
 and that field is not present on every document.

 http://wiki.apache.org/solr/HowToReindex

 Thanks,
 Shawn

Re: Master / Slave Set Up Documentation

[Andrea Gazzarini]

You mean this

http://wiki.apache.org/solr/SolrReplication

?

What's wrong with this page? It seems clear.
I'm widely using replication and the first time I set up a 1 master + 2
slaves by simply following that page
On 26 Aug 2013 18:54, Jared Griffith jgriff...@picsauditing.com wrote:

 Hello,
 I'm new to this Solr thing, and I was wondering if there is any good /
 solid documentation on setting up and running replication.  I'm going
 through the Wiki but I am not seeing anything that is obvious there.

 --

 Jared Griffith
 Linux Administrator, PICS Auditing, LLC
 P: (949) 936-4574
 C: (909) 653-7814

 http://www.picsauditing.com

 17701 Cowan #140 | Irvine, CA | 92614

 Join PICS on LinkedIn and Twitter!

 https://twitter.com/PICSAuditingLLC

Re: Master / Slave Set Up Documentation

[Jared Griffith]

Ha, I guess I didn't see that page listed in the Table of contents 
it's definitely Monday.  Thanks.


On Mon, Aug 26, 2013 at 10:36 AM, Andrea Gazzarini 
andrea.gazzar...@gmail.com wrote:

 You mean this

 http://wiki.apache.org/solr/SolrReplication

 ?

 What's wrong with this page? It seems clear.
 I'm widely using replication and the first time I set up a 1 master + 2
 slaves by simply following that page
 On 26 Aug 2013 18:54, Jared Griffith jgriff...@picsauditing.com wrote:

  Hello,
  I'm new to this Solr thing, and I was wondering if there is any good /
  solid documentation on setting up and running replication.  I'm going
  through the Wiki but I am not seeing anything that is obvious there.
 
  --
 
  Jared Griffith
  Linux Administrator, PICS Auditing, LLC
  P: (949) 936-4574
  C: (909) 653-7814
 
  http://www.picsauditing.com
 
  17701 Cowan #140 | Irvine, CA | 92614
 
  Join PICS on LinkedIn and Twitter!
 
  https://twitter.com/PICSAuditingLLC
 




-- 

Jared Griffith
Linux Administrator, PICS Auditing, LLC
P: (949) 936-4574
C: (909) 653-7814

http://www.picsauditing.com

17701 Cowan #140 | Irvine, CA | 92614

Join PICS on LinkedIn and Twitter!

https://twitter.com/PICSAuditingLLC

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

Shawn thanks for the detailed answer. I have 5 shards and 1 leader - 1
replica for each. I mean I have 10 Solr nodes. When I look at admin gui of
one of the shards leader I see that its replica has less MB of index than
leader. I don't update the data, I don't index new ones. I think that after
a time later it will sync its replica to itself but nothing has changed.

2013/5/1 Shawn Heisey s...@elyograg.org

 On 4/30/2013 8:33 AM, Furkan KAMACI wrote:

 I think that replication occurs after commit by default. It has been long
 time however there is still mismatch between leader and replica
 (approximately 5 MB). I tried to pull indexes from leader but it is still
 same.


 My mail server has been down most of the day, and the Apache mail
 infrastructure hasn't noticed yet that I'm back up.  I don't have copies of
 the newest messages on this thread.  I checked the web archive to see what
 else has been said.  I'll be repeating some of what has been said before.

 On SolrCloud terminology: SolrCloud divides your index into one or more
 shards, each of which has a different piece of the index.  Each shard is
 made up of replicas.  One replica in each shard is designated leader. Note:
 a leader is still a replica, it is just the winner of the latest leader
 election.  Summary: shards, replicas, leader.

 One term that you are using is follower ... this is not a valid
 SolrCloud term.  It might make sense to use this term for a replica that is
 not a leader, but I have never seen it used in anything official. Any
 replica can become leader, if the conditions are just right.

 There are only two times that the leader replica has special significance
 - when you are indexing and when a replica starts operation, either as an
 existing replica that went down or as a new replica.

 In SolrCloud, replication is *NOT* used when you index new data.  The
 *ONLY* time that replication happens in SolrCloud is when a replica is
 starts up, and even then it will only happen if the leader cannot figure
 out how to use its transaction log to sync the replica.

 SolrCloud does distributed indexing.  This means that when an update comes
 in, SolrCloud determines which shard needs that update.  If the core that
 received the request is not the leader of that shard, the request is
 forwarded to the correct leader.  That leader will index the update and
 send it to all of the replicas for that shard, each of which will index the
 update independently.

 Because each replica indexes independently, you can end up with different
 sizes.  The actual search results should be the same, although scoring can
 sometimes be a little bit different between replicas because deleted
 documents that exist in one replica but not another will contribute to the
 score.  SolrCloud does not attempt to keep the replicas absolutely
 identical, as long as they contain the same non-deleted documents.

 Thanks,
 Shawn

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Shawn Heisey]

On 4/30/2013 6:13 AM, Furkan KAMACI wrote:
 I use SolrCloud, 4.2.1 of Solr.
 
 Here is a detail from my admin page:
 
 Replication (Slave) Version Gen Size
 Master: 1367309548534 84 779.87 MB
 Slave: 1367307649512 82 784.44 MB
 
 When I use command=abortfetch still file size are not same. Any idea?

I won't know what abortfetch means unless I look it up, but I think this
is normal.

I have a 4.2.1 SolrCloud where things are working perfectly.  My two
replicas are not identical in size, but I have not seen any evidence of
a difference in contents:

Version Gen Size
Master: 1367280043801   15331   1,013.07 MB
Slave:  1367280083570   15331   1 GB

Version Gen Size
Master: 1367280083570   15331   1 GB
Slave:  1367280043801   15331   1,013.07 MB

For anyone who is wondering why these numbers are displayed differently,
these are powers of 2, not powers of 10.  Solr won't switch to 1GB until
you hit 1024MB, and each MB is 1048576 bytes, not 100.

Thanks,
Shawn

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Michael Della Bitta]

In Solr Cloud, commits can happen at different times across replicas. Which
means merges also may happen at different times. So there's no expectation
of the cores of different replicas being totally similar.


Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI furkankam...@gmail.comwrote:

 I use SolrCloud, 4.2.1 of Solr.

 Here is a detail from my admin page:

 Replication (Slave) Version Gen Size
 Master: 1367309548534 84 779.87 MB
 Slave: 1367307649512 82 784.44 MB

 When I use command=abortfetch still file size are not same. Any idea?

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

I think that replication occurs after commit by default. It has been long
time however there is still mismatch between leader and replica
(approximately 5 MB). I tried to pull indexes from leader but it is still
same.

2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 In Solr Cloud, commits can happen at different times across replicas. Which
 means merges also may happen at different times. So there's no expectation
 of the cores of different replicas being totally similar.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  I use SolrCloud, 4.2.1 of Solr.
 
  Here is a detail from my admin page:
 
  Replication (Slave) Version Gen Size
  Master: 1367309548534 84 779.87 MB
  Slave: 1367307649512 82 784.44 MB
 
  When I use command=abortfetch still file size are not same. Any idea?
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Michael Della Bitta]

I'm a little confused. Are you using Solr Cloud, or ordinary replication?


Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI furkankam...@gmail.comwrote:

 I think that replication occurs after commit by default. It has been long
 time however there is still mismatch between leader and replica
 (approximately 5 MB). I tried to pull indexes from leader but it is still
 same.

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  In Solr Cloud, commits can happen at different times across replicas.
 Which
  means merges also may happen at different times. So there's no
 expectation
  of the cores of different replicas being totally similar.
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI furkankam...@gmail.com
  wrote:
 
   I use SolrCloud, 4.2.1 of Solr.
  
   Here is a detail from my admin page:
  
   Replication (Slave) Version Gen Size
   Master: 1367309548534 84 779.87 MB
   Slave: 1367307649512 82 784.44 MB
  
   When I use command=abortfetch still file size are not same. Any idea?
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

I use Solr 4.2.1 as SolrCloud

2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 I'm a little confused. Are you using Solr Cloud, or ordinary replication?


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  I think that replication occurs after commit by default. It has been long
  time however there is still mismatch between leader and replica
  (approximately 5 MB). I tried to pull indexes from leader but it is still
  same.
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
   In Solr Cloud, commits can happen at different times across replicas.
  Which
   means merges also may happen at different times. So there's no
  expectation
   of the cores of different replicas being totally similar.
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI furkankam...@gmail.com
   wrote:
  
I use SolrCloud, 4.2.1 of Solr.
   
Here is a detail from my admin page:
   
Replication (Slave) Version Gen Size
Master: 1367309548534 84 779.87 MB
Slave: 1367307649512 82 784.44 MB
   
When I use command=abortfetch still file size are not same. Any idea?
   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

Shawn, why they don't have same data byte per byte? Can I force slave to
pull them, I tried but didn't work.

2013/4/30 Furkan KAMACI furkankam...@gmail.com

 I use Solr 4.2.1 as SolrCloud


 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 I'm a little confused. Are you using Solr Cloud, or ordinary replication?


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  I think that replication occurs after commit by default. It has been
 long
  time however there is still mismatch between leader and replica
  (approximately 5 MB). I tried to pull indexes from leader but it is
 still
  same.
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
   In Solr Cloud, commits can happen at different times across replicas.
  Which
   means merges also may happen at different times. So there's no
  expectation
   of the cores of different replicas being totally similar.
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
 furkankam...@gmail.com
   wrote:
  
I use SolrCloud, 4.2.1 of Solr.
   
Here is a detail from my admin page:
   
Replication (Slave) Version Gen Size
Master: 1367309548534 84 779.87 MB
Slave: 1367307649512 82 784.44 MB
   
When I use command=abortfetch still file size are not same. Any
 idea?
   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Michael Della Bitta]

Then there is no replication, and no slaves nor masters. There's a leader
and followers. Documents themselves are sent from the leader to followers,
not cores or segments. You should not expect the bits on the disk across
leaders and followers to be the same because of the reasons I mentioned
already.


Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI furkankam...@gmail.comwrote:

 I use Solr 4.2.1 as SolrCloud

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  I'm a little confused. Are you using Solr Cloud, or ordinary replication?
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI furkankam...@gmail.com
  wrote:
 
   I think that replication occurs after commit by default. It has been
 long
   time however there is still mismatch between leader and replica
   (approximately 5 MB). I tried to pull indexes from leader but it is
 still
   same.
  
   2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
  
In Solr Cloud, commits can happen at different times across replicas.
   Which
means merges also may happen at different times. So there's no
   expectation
of the cores of different replicas being totally similar.
   
   
Michael Della Bitta
   

Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
   
www.appinions.com
   
Where Influence Isn’t a Game
   
   
On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
 furkankam...@gmail.com
wrote:
   
 I use SolrCloud, 4.2.1 of Solr.

 Here is a detail from my admin page:

 Replication (Slave) Version Gen Size
 Master: 1367309548534 84 779.87 MB
 Slave: 1367307649512 82 784.44 MB

 When I use command=abortfetch still file size are not same. Any
 idea?

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

However I am using SolrCloud with 5 shards. Every leader has a replica.
What do you mean with followers?

2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 Then there is no replication, and no slaves nor masters. There's a leader
 and followers. Documents themselves are sent from the leader to followers,
 not cores or segments. You should not expect the bits on the disk across
 leaders and followers to be the same because of the reasons I mentioned
 already.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  I use Solr 4.2.1 as SolrCloud
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
   I'm a little confused. Are you using Solr Cloud, or ordinary
 replication?
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI 
 furkankam...@gmail.com
   wrote:
  
I think that replication occurs after commit by default. It has been
  long
time however there is still mismatch between leader and replica
(approximately 5 MB). I tried to pull indexes from leader but it is
  still
same.
   
2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
   
 In Solr Cloud, commits can happen at different times across
 replicas.
Which
 means merges also may happen at different times. So there's no
expectation
 of the cores of different replicas being totally similar.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
  furkankam...@gmail.com
 wrote:

  I use SolrCloud, 4.2.1 of Solr.
 
  Here is a detail from my admin page:
 
  Replication (Slave) Version Gen Size
  Master: 1367309548534 84 779.87 MB
  Slave: 1367307649512 82 784.44 MB
 
  When I use command=abortfetch still file size are not same. Any
  idea?
 

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Michael Della Bitta]

I'd say a follower is a participant in a shard that's not the leader.


Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Apr 30, 2013 at 1:27 PM, Furkan KAMACI furkankam...@gmail.comwrote:

 However I am using SolrCloud with 5 shards. Every leader has a replica.
 What do you mean with followers?

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  Then there is no replication, and no slaves nor masters. There's a leader
  and followers. Documents themselves are sent from the leader to
 followers,
  not cores or segments. You should not expect the bits on the disk across
  leaders and followers to be the same because of the reasons I mentioned
  already.
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI furkankam...@gmail.com
  wrote:
 
   I use Solr 4.2.1 as SolrCloud
  
   2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
  
I'm a little confused. Are you using Solr Cloud, or ordinary
  replication?
   
   
Michael Della Bitta
   

Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
   
www.appinions.com
   
Where Influence Isn’t a Game
   
   
On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI 
  furkankam...@gmail.com
wrote:
   
 I think that replication occurs after commit by default. It has
 been
   long
 time however there is still mismatch between leader and replica
 (approximately 5 MB). I tried to pull indexes from leader but it is
   still
 same.

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  In Solr Cloud, commits can happen at different times across
  replicas.
 Which
  means merges also may happen at different times. So there's no
 expectation
  of the cores of different replicas being totally similar.
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
   furkankam...@gmail.com
  wrote:
 
   I use SolrCloud, 4.2.1 of Solr.
  
   Here is a detail from my admin page:
  
   Replication (Slave) Version Gen Size
   Master: 1367309548534 84 779.87 MB
   Slave: 1367307649512 82 784.44 MB
  
   When I use command=abortfetch still file size are not same. Any
   idea?
  
 

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

If we talk about SolrCloud terminology does follower and replica means
same? Is there any documentation at wiki for that?

2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 I'd say a follower is a participant in a shard that's not the leader.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 1:27 PM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  However I am using SolrCloud with 5 shards. Every leader has a replica.
  What do you mean with followers?
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
   Then there is no replication, and no slaves nor masters. There's a
 leader
   and followers. Documents themselves are sent from the leader to
  followers,
   not cores or segments. You should not expect the bits on the disk
 across
   leaders and followers to be the same because of the reasons I mentioned
   already.
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI 
 furkankam...@gmail.com
   wrote:
  
I use Solr 4.2.1 as SolrCloud
   
2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
   
 I'm a little confused. Are you using Solr Cloud, or ordinary
   replication?


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI 
   furkankam...@gmail.com
 wrote:

  I think that replication occurs after commit by default. It has
  been
long
  time however there is still mismatch between leader and replica
  (approximately 5 MB). I tried to pull indexes from leader but it
 is
still
  same.
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
 
   In Solr Cloud, commits can happen at different times across
   replicas.
  Which
   means merges also may happen at different times. So there's no
  expectation
   of the cores of different replicas being totally similar.
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
furkankam...@gmail.com
   wrote:
  
I use SolrCloud, 4.2.1 of Solr.
   
Here is a detail from my admin page:
   
Replication (Slave) Version Gen Size
Master: 1367309548534 84 779.87 MB
Slave: 1367307649512 82 784.44 MB
   
When I use command=abortfetch still file size are not same.
 Any
idea?
   
  
 

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Michael Della Bitta]

I could be getting this wrong, and the wiki is down at the moment, but I
think a replica can be a leader, whereas a follower is definitely not.


Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Apr 30, 2013 at 2:19 PM, Furkan KAMACI furkankam...@gmail.comwrote:

 If we talk about SolrCloud terminology does follower and replica means
 same? Is there any documentation at wiki for that?

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  I'd say a follower is a participant in a shard that's not the leader.
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 1:27 PM, Furkan KAMACI furkankam...@gmail.com
  wrote:
 
   However I am using SolrCloud with 5 shards. Every leader has a replica.
   What do you mean with followers?
  
   2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
  
Then there is no replication, and no slaves nor masters. There's a
  leader
and followers. Documents themselves are sent from the leader to
   followers,
not cores or segments. You should not expect the bits on the disk
  across
leaders and followers to be the same because of the reasons I
 mentioned
already.
   
   
Michael Della Bitta
   

Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
   
www.appinions.com
   
Where Influence Isn’t a Game
   
   
On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI 
  furkankam...@gmail.com
wrote:
   
 I use Solr 4.2.1 as SolrCloud

 2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

  I'm a little confused. Are you using Solr Cloud, or ordinary
replication?
 
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI 
furkankam...@gmail.com
  wrote:
 
   I think that replication occurs after commit by default. It has
   been
 long
   time however there is still mismatch between leader and replica
   (approximately 5 MB). I tried to pull indexes from leader but
 it
  is
 still
   same.
  
   2013/4/30 Michael Della Bitta 
 michael.della.bi...@appinions.com
  
  
In Solr Cloud, commits can happen at different times across
replicas.
   Which
means merges also may happen at different times. So there's
 no
   expectation
of the cores of different replicas being totally similar.
   
   
Michael Della Bitta
   

Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
   
www.appinions.com
   
Where Influence Isn’t a Game
   
   
On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
 furkankam...@gmail.com
wrote:
   
 I use SolrCloud, 4.2.1 of Solr.

 Here is a detail from my admin page:

 Replication (Slave) Version Gen Size
 Master: 1367309548534 84 779.87 MB
 Slave: 1367307649512 82 784.44 MB

 When I use command=abortfetch still file size are not same.
  Any
 idea?

   
  
 

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Furkan KAMACI]

It would be nice if I learn what is a follower means and how to define them
(I know the example of replica but didn't see an example of follower yet)

2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com

 I could be getting this wrong, and the wiki is down at the moment, but I
 think a replica can be a leader, whereas a follower is definitely not.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 2:19 PM, Furkan KAMACI furkankam...@gmail.com
 wrote:

  If we talk about SolrCloud terminology does follower and replica means
  same? Is there any documentation at wiki for that?
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
   I'd say a follower is a participant in a shard that's not the leader.
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 1:27 PM, Furkan KAMACI furkankam...@gmail.com
   wrote:
  
However I am using SolrCloud with 5 shards. Every leader has a
 replica.
What do you mean with followers?
   
2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
   
 Then there is no replication, and no slaves nor masters. There's a
   leader
 and followers. Documents themselves are sent from the leader to
followers,
 not cores or segments. You should not expect the bits on the disk
   across
 leaders and followers to be the same because of the reasons I
  mentioned
 already.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 11:04 AM, Furkan KAMACI 
   furkankam...@gmail.com
 wrote:

  I use Solr 4.2.1 as SolrCloud
 
  2013/4/30 Michael Della Bitta michael.della.bi...@appinions.com
 
 
   I'm a little confused. Are you using Solr Cloud, or ordinary
 replication?
  
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Apr 30, 2013 at 10:33 AM, Furkan KAMACI 
 furkankam...@gmail.com
   wrote:
  
I think that replication occurs after commit by default. It
 has
been
  long
time however there is still mismatch between leader and
 replica
(approximately 5 MB). I tried to pull indexes from leader but
  it
   is
  still
same.
   
2013/4/30 Michael Della Bitta 
  michael.della.bi...@appinions.com
   
   
 In Solr Cloud, commits can happen at different times across
 replicas.
Which
 means merges also may happen at different times. So there's
  no
expectation
 of the cores of different replicas being totally similar.


 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Apr 30, 2013 at 8:13 AM, Furkan KAMACI 
  furkankam...@gmail.com
 wrote:

  I use SolrCloud, 4.2.1 of Solr.
 
  Here is a detail from my admin page:
 
  Replication (Slave) Version Gen Size
  Master: 1367309548534 84 779.87 MB
  Slave: 1367307649512 82 784.44 MB
 
  When I use command=abortfetch still file size are not
 same.
   Any
  idea?
 

   
  
 

   
  
 

Re: Master - Slave File Sizes are not Same even after command=abortfetch

[Shawn Heisey]

On 4/30/2013 8:33 AM, Furkan KAMACI wrote:

I think that replication occurs after commit by default. It has been long
time however there is still mismatch between leader and replica
(approximately 5 MB). I tried to pull indexes from leader but it is still
same.


My mail server has been down most of the day, and the Apache mail 
infrastructure hasn't noticed yet that I'm back up.  I don't have copies 
of the newest messages on this thread.  I checked the web archive to see 
what else has been said.  I'll be repeating some of what has been said 
before.


On SolrCloud terminology: SolrCloud divides your index into one or more 
shards, each of which has a different piece of the index.  Each shard is 
made up of replicas.  One replica in each shard is designated leader. 
Note: a leader is still a replica, it is just the winner of the latest 
leader election.  Summary: shards, replicas, leader.


One term that you are using is follower ... this is not a valid 
SolrCloud term.  It might make sense to use this term for a replica that 
is not a leader, but I have never seen it used in anything official. 
Any replica can become leader, if the conditions are just right.


There are only two times that the leader replica has special 
significance - when you are indexing and when a replica starts 
operation, either as an existing replica that went down or as a new replica.


In SolrCloud, replication is *NOT* used when you index new data.  The 
*ONLY* time that replication happens in SolrCloud is when a replica is 
starts up, and even then it will only happen if the leader cannot figure 
out how to use its transaction log to sync the replica.


SolrCloud does distributed indexing.  This means that when an update 
comes in, SolrCloud determines which shard needs that update.  If the 
core that received the request is not the leader of that shard, the 
request is forwarded to the correct leader.  That leader will index the 
update and send it to all of the replicas for that shard, each of which 
will index the update independently.


Because each replica indexes independently, you can end up with 
different sizes.  The actual search results should be the same, although 
scoring can sometimes be a little bit different between replicas because 
deleted documents that exist in one replica but not another will 
contribute to the score.  SolrCloud does not attempt to keep the 
replicas absolutely identical, as long as they contain the same 
non-deleted documents.


Thanks,
Shawn

Re: Master slave replication with digest authentication

[Shawn Heisey]

On 4/17/2013 1:20 AM, Maciej Pestka wrote:

Hi,

I've configured basic authentication on tomcat  my slave solr instance and it 
works.

Any idea how to configure slave to replicate properly with digest 
authentication?
on Solr WIKI I could find only basic authentication example:
http://wiki.apache.org/solr/SolrSecurity

solrconfig.xml (slave config):
str name=httpBasicAuthUserusername/str
str name=httpBasicAuthPasswordpassword/str


I'm pretty sure that basic auth is all that's supported.  Right now, 
replication is the only internal solr communication that supports 
authentication.  There is an issue to add it for all internal solr 
communication.  The hope is to get it into 4.3.  I don't know how likely 
that is to happen.


https://issues.apache.org/jira/browse/SOLR-4470

As implemented, this will only support basic authentication, but the 
issue notes say that adding digest wouldn't be a huge change.  When this 
makes it into Solr (4.3 or 4.4) I wouldn't count on seeing digest until 
the next release, and only if there is enough desire in the community 
for the feature.


Thanks,
Shawn

Re: Master /Slave Set up on AWS - 3.6.1

[Otis Gospodnetic]

Hi Sujatha,

If I understand correctly, you will have only 1 slave (and 1 master), so
that's not really a HA architecture.  You could manually turn master into
slave, but that's going to mean some down time...

Otis
--
Solr  ElasticSearch Support
http://sematext.com/





On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi,

 We are planning to set up *2* *High-Memory Quadruple Extra Large Instance
  *as
 master and slave for our multicore solr setup  which has more than 200
 cores spread between a couple of webapps on a single JVM on *AWS*

 All indexing [via a queue will go to master ]  . One Slave  Server will
 replicate all the core level indexes from the master , slave Configurations
 are defined in the solr.xml  at the webapp level  with a different poll
 interval for each webapp.

 We are planning to LB the search requests by fronting the master and slave
 with an *AWS ELB *. The master configuration will not enable the slave
 properties as master is not replicating from any other machine. The master
 and slave have similar hardware configurations [*High-Memory Quadruple
 Extra Large Instance] .*This is mainly for HA if the slave goes down.
 *
 *
 Any issue with the above set up ,please advice.

 Regards,
 Sujatha




 *
 *

Re: Master /Slave Set up on AWS - 3.6.1

[Sujatha Arun]

Hi Otis,

Since currently we are planning for only one slave  due to cost
considerations, can we have an ELB fronting the master and slave for HA.

   1. All index requests will go to the master .
   2. Slave replicates  from master .
   3. Search request can go either to master /slave via ELB.

is that resonable   HA for search ?

Regards
Sujatha



On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic otis.gospodne...@gmail.com
 wrote:

 Hi Sujatha,

 If I understand correctly, you will have only 1 slave (and 1 master), so
 that's not really a HA architecture.  You could manually turn master into
 slave, but that's going to mean some down time...

 Otis
 --
 Solr  ElasticSearch Support
 http://sematext.com/





 On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com wrote:

  Hi,
 
  We are planning to set up *2* *High-Memory Quadruple Extra Large Instance
   *as
  master and slave for our multicore solr setup  which has more than 200
  cores spread between a couple of webapps on a single JVM on *AWS*
 
  All indexing [via a queue will go to master ]  . One Slave  Server will
  replicate all the core level indexes from the master , slave
 Configurations
  are defined in the solr.xml  at the webapp level  with a different poll
  interval for each webapp.
 
  We are planning to LB the search requests by fronting the master and
 slave
  with an *AWS ELB *. The master configuration will not enable the slave
  properties as master is not replicating from any other machine. The
 master
  and slave have similar hardware configurations [*High-Memory Quadruple
  Extra Large Instance] .*This is mainly for HA if the slave goes down.
  *
  *
  Any issue with the above set up ,please advice.
 
  Regards,
  Sujatha
 
 
 
 
  *
  *
 

Re: Master /Slave Set up on AWS - 3.6.1

[Sujatha Arun]

Is there anything wrong with set up?

On Tue, Mar 5, 2013 at 5:43 PM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi Otis,

 Since currently we are planning for only one slave  due to cost
 considerations, can we have an ELB fronting the master and slave for HA.

1. All index requests will go to the master .
2. Slave replicates  from master .
3. Search request can go either to master /slave via ELB.

 is that resonable   HA for search ?

 Regards
 Sujatha



 On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
 otis.gospodne...@gmail.com wrote:

 Hi Sujatha,

 If I understand correctly, you will have only 1 slave (and 1 master), so
 that's not really a HA architecture.  You could manually turn master into
 slave, but that's going to mean some down time...

 Otis
 --
 Solr  ElasticSearch Support
 http://sematext.com/





 On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com wrote:

  Hi,
 
  We are planning to set up *2* *High-Memory Quadruple Extra Large
 Instance
   *as
  master and slave for our multicore solr setup  which has more than 200
  cores spread between a couple of webapps on a single JVM on *AWS*
 
  All indexing [via a queue will go to master ]  . One Slave  Server will
  replicate all the core level indexes from the master , slave
 Configurations
  are defined in the solr.xml  at the webapp level  with a different poll
  interval for each webapp.
 
  We are planning to LB the search requests by fronting the master and
 slave
  with an *AWS ELB *. The master configuration will not enable the slave
  properties as master is not replicating from any other machine. The
 master
  and slave have similar hardware configurations [*High-Memory Quadruple
  Extra Large Instance] .*This is mainly for HA if the slave goes down.
  *
  *
  Any issue with the above set up ,please advice.
 
  Regards,
  Sujatha
 
 
 
 
  *
  *
 

Re: Master /Slave Set up on AWS - 3.6.1

[Otis Gospodnetic]

Hi,

That my be fine.  I'd use the sticky session setting in ELB to avoid
having the same user's query hit both master and slave, say while paging
through results, and risking seeing inconsistent results.  THis will also
help with cache utilization.  This said, this is not a recommended setup.
 Note that AWS *just* lowered the prices for Reserved Instances today.

Otis
--
Solr  ElasticSearch Support
http://sematext.com/





On Tue, Mar 5, 2013 at 7:13 AM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi Otis,

 Since currently we are planning for only one slave  due to cost
 considerations, can we have an ELB fronting the master and slave for HA.

1. All index requests will go to the master .
2. Slave replicates  from master .
3. Search request can go either to master /slave via ELB.

 is that resonable   HA for search ?

 Regards
 Sujatha



 On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
 otis.gospodne...@gmail.com
  wrote:

  Hi Sujatha,
 
  If I understand correctly, you will have only 1 slave (and 1 master), so
  that's not really a HA architecture.  You could manually turn master into
  slave, but that's going to mean some down time...
 
  Otis
  --
  Solr  ElasticSearch Support
  http://sematext.com/
 
 
 
 
 
  On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com
 wrote:
 
   Hi,
  
   We are planning to set up *2* *High-Memory Quadruple Extra Large
 Instance
*as
   master and slave for our multicore solr setup  which has more than 200
   cores spread between a couple of webapps on a single JVM on *AWS*
  
   All indexing [via a queue will go to master ]  . One Slave  Server will
   replicate all the core level indexes from the master , slave
  Configurations
   are defined in the solr.xml  at the webapp level  with a different poll
   interval for each webapp.
  
   We are planning to LB the search requests by fronting the master and
  slave
   with an *AWS ELB *. The master configuration will not enable the slave
   properties as master is not replicating from any other machine. The
  master
   and slave have similar hardware configurations [*High-Memory Quadruple
   Extra Large Instance] .*This is mainly for HA if the slave goes down.
   *
   *
   Any issue with the above set up ,please advice.
  
   Regards,
   Sujatha
  
  
  
  
   *
   *
  
 

Re: Master /Slave Set up on AWS - 3.6.1

[Michael Della Bitta]

If your index is on EBS, you'll see big iowait percentages when merges happen.

I'm not sure what that's going to do to your master's ability to
service requests. You should test.

Alternatively, you might figure out the size of machine you need to
index vs. the size of machine you need to service queries. They're
very likely not the same, in which case, that may afford you the
ability to have 2 slaves and 1 master in a similar budget.

Also, once you've settled on an infrastructure, you should investigate
buying reserved instances for a year. It will greatly reduce your
costs.

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Tue, Mar 5, 2013 at 8:59 AM, Sujatha Arun suja.a...@gmail.com wrote:
 Is there anything wrong with set up?

 On Tue, Mar 5, 2013 at 5:43 PM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi Otis,

 Since currently we are planning for only one slave  due to cost
 considerations, can we have an ELB fronting the master and slave for HA.

1. All index requests will go to the master .
2. Slave replicates  from master .
3. Search request can go either to master /slave via ELB.

 is that resonable   HA for search ?

 Regards
 Sujatha



 On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
 otis.gospodne...@gmail.com wrote:

 Hi Sujatha,

 If I understand correctly, you will have only 1 slave (and 1 master), so
 that's not really a HA architecture.  You could manually turn master into
 slave, but that's going to mean some down time...

 Otis
 --
 Solr  ElasticSearch Support
 http://sematext.com/





 On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com wrote:

  Hi,
 
  We are planning to set up *2* *High-Memory Quadruple Extra Large
 Instance
   *as
  master and slave for our multicore solr setup  which has more than 200
  cores spread between a couple of webapps on a single JVM on *AWS*
 
  All indexing [via a queue will go to master ]  . One Slave  Server will
  replicate all the core level indexes from the master , slave
 Configurations
  are defined in the solr.xml  at the webapp level  with a different poll
  interval for each webapp.
 
  We are planning to LB the search requests by fronting the master and
 slave
  with an *AWS ELB *. The master configuration will not enable the slave
  properties as master is not replicating from any other machine. The
 master
  and slave have similar hardware configurations [*High-Memory Quadruple
  Extra Large Instance] .*This is mainly for HA if the slave goes down.
  *
  *
  Any issue with the above set up ,please advice.
 
  Regards,
  Sujatha
 
 
 
 
  *
  *
 

Re: Master /Slave Set up on AWS - 3.6.1

[Sujatha Arun]

Hi Otis,Michael,

Thanks for your input and suggestions .

Yes, we were considering the sticky session for pagination  and we are not
planning for having index on EBS

I would like to understand  why its not the recommended approach,  can you
please explain?

Till now we were having a single server for both Indexing and search
,though this was on dedicated server and not on cloud. Indexing would
happen sequentially via  a queue due to which commits would happen for only
one or at most 2 cores simultaneously.

With Master /Slave approach  I see that when a slave replicates form master
based on the poll time which I have defined in solr.xml file  depending on
the number of cores/webapp  ,the replication and hence commit is going to
happen simultaneously for many cores increasing  load on slave server.

By having  2 slaves and one master -  whenever we create a core,which
happens quite frequently  we need to create this on 3 servers instead of
two , which has to done manually by running a script on each server.We have
requirement for adding cores per each customer.

I do understand that hardware requirements for master can be quite
different [Lower memory /higher CPU/Cache setting in config /autowarming
etc  ] from slave.But given that we will be Indexing sequentially and
having the same configuration in terms of memory and CPU/cache  for both
master and slave,would this be a reasonable approach?

ThanksRegards,
Sujatha



On Tue, Mar 5, 2013 at 9:10 PM, Michael Della Bitta 
michael.della.bi...@appinions.com wrote:

 If your index is on EBS, you'll see big iowait percentages when merges
 happen.

 I'm not sure what that's going to do to your master's ability to
 service requests. You should test.

 Alternatively, you might figure out the size of machine you need to
 index vs. the size of machine you need to service queries. They're
 very likely not the same, in which case, that may afford you the
 ability to have 2 slaves and 1 master in a similar budget.

 Also, once you've settled on an infrastructure, you should investigate
 buying reserved instances for a year. It will greatly reduce your
 costs.

 Michael Della Bitta

 
 Appinions
 18 East 41st Street, 2nd Floor
 New York, NY 10017-6271

 www.appinions.com

 Where Influence Isn’t a Game


 On Tue, Mar 5, 2013 at 8:59 AM, Sujatha Arun suja.a...@gmail.com wrote:
  Is there anything wrong with set up?
 
  On Tue, Mar 5, 2013 at 5:43 PM, Sujatha Arun suja.a...@gmail.com
 wrote:
 
  Hi Otis,
 
  Since currently we are planning for only one slave  due to cost
  considerations, can we have an ELB fronting the master and slave for HA.
 
 1. All index requests will go to the master .
 2. Slave replicates  from master .
 3. Search request can go either to master /slave via ELB.
 
  is that resonable   HA for search ?
 
  Regards
  Sujatha
 
 
 
  On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
  otis.gospodne...@gmail.com wrote:
 
  Hi Sujatha,
 
  If I understand correctly, you will have only 1 slave (and 1 master),
 so
  that's not really a HA architecture.  You could manually turn master
 into
  slave, but that's going to mean some down time...
 
  Otis
  --
  Solr  ElasticSearch Support
  http://sematext.com/
 
 
 
 
 
  On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com
 wrote:
 
   Hi,
  
   We are planning to set up *2* *High-Memory Quadruple Extra Large
  Instance
*as
   master and slave for our multicore solr setup  which has more than
 200
   cores spread between a couple of webapps on a single JVM on *AWS*
  
   All indexing [via a queue will go to master ]  . One Slave  Server
 will
   replicate all the core level indexes from the master , slave
  Configurations
   are defined in the solr.xml  at the webapp level  with a different
 poll
   interval for each webapp.
  
   We are planning to LB the search requests by fronting the master and
  slave
   with an *AWS ELB *. The master configuration will not enable the
 slave
   properties as master is not replicating from any other machine. The
  master
   and slave have similar hardware configurations [*High-Memory
 Quadruple
   Extra Large Instance] .*This is mainly for HA if the slave goes down.
   *
   *
   Any issue with the above set up ,please advice.
  
   Regards,
   Sujatha
  
  
  
  
   *
   *
  
 
 
 

Re: Master /Slave Set up on AWS - 3.6.1

[Otis Gospodnetic]

Hello,

This is not recommended because people typically don't want the load from
indexing affect queries/user experience.  If your numbers are low, then
this may not be a big deal. If you already need to create a core on 2
machines, creating it on 3 doesn't seem a big deal.  There is a slight
conflict in what you wrote, which is that cores will be created frequently,
which implies you will quickly have lots of cores vs. desire to have
minimal hardware.

If it were up to me, I'd consider a weaker/cheaper master, and more slaves.

Otis
--
Solr  ElasticSearch Support
http://sematext.com/





On Tue, Mar 5, 2013 at 12:46 PM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi Otis,Michael,

 Thanks for your input and suggestions .

 Yes, we were considering the sticky session for pagination  and we are not
 planning for having index on EBS

 I would like to understand  why its not the recommended approach,  can you
 please explain?

 Till now we were having a single server for both Indexing and search
 ,though this was on dedicated server and not on cloud. Indexing would
 happen sequentially via  a queue due to which commits would happen for only
 one or at most 2 cores simultaneously.

 With Master /Slave approach  I see that when a slave replicates form master
 based on the poll time which I have defined in solr.xml file  depending on
 the number of cores/webapp  ,the replication and hence commit is going to
 happen simultaneously for many cores increasing  load on slave server.

 By having  2 slaves and one master -  whenever we create a core,which
 happens quite frequently  we need to create this on 3 servers instead of
 two , which has to done manually by running a script on each server.We have
 requirement for adding cores per each customer.

 I do understand that hardware requirements for master can be quite
 different [Lower memory /higher CPU/Cache setting in config /autowarming
 etc  ] from slave.But given that we will be Indexing sequentially and
 having the same configuration in terms of memory and CPU/cache  for both
 master and slave,would this be a reasonable approach?

 ThanksRegards,
 Sujatha



 On Tue, Mar 5, 2013 at 9:10 PM, Michael Della Bitta 
 michael.della.bi...@appinions.com wrote:

  If your index is on EBS, you'll see big iowait percentages when merges
  happen.
 
  I'm not sure what that's going to do to your master's ability to
  service requests. You should test.
 
  Alternatively, you might figure out the size of machine you need to
  index vs. the size of machine you need to service queries. They're
  very likely not the same, in which case, that may afford you the
  ability to have 2 slaves and 1 master in a similar budget.
 
  Also, once you've settled on an infrastructure, you should investigate
  buying reserved instances for a year. It will greatly reduce your
  costs.
 
  Michael Della Bitta
 
  
  Appinions
  18 East 41st Street, 2nd Floor
  New York, NY 10017-6271
 
  www.appinions.com
 
  Where Influence Isn’t a Game
 
 
  On Tue, Mar 5, 2013 at 8:59 AM, Sujatha Arun suja.a...@gmail.com
 wrote:
   Is there anything wrong with set up?
  
   On Tue, Mar 5, 2013 at 5:43 PM, Sujatha Arun suja.a...@gmail.com
  wrote:
  
   Hi Otis,
  
   Since currently we are planning for only one slave  due to cost
   considerations, can we have an ELB fronting the master and slave for
 HA.
  
  1. All index requests will go to the master .
  2. Slave replicates  from master .
  3. Search request can go either to master /slave via ELB.
  
   is that resonable   HA for search ?
  
   Regards
   Sujatha
  
  
  
   On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
   otis.gospodne...@gmail.com wrote:
  
   Hi Sujatha,
  
   If I understand correctly, you will have only 1 slave (and 1 master),
  so
   that's not really a HA architecture.  You could manually turn master
  into
   slave, but that's going to mean some down time...
  
   Otis
   --
   Solr  ElasticSearch Support
   http://sematext.com/
  
  
  
  
  
   On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com
  wrote:
  
Hi,
   
We are planning to set up *2* *High-Memory Quadruple Extra Large
   Instance
 *as
master and slave for our multicore solr setup  which has more than
  200
cores spread between a couple of webapps on a single JVM on *AWS*
   
All indexing [via a queue will go to master ]  . One Slave  Server
  will
replicate all the core level indexes from the master , slave
   Configurations
are defined in the solr.xml  at the webapp level  with a different
  poll
interval for each webapp.
   
We are planning to LB the search requests by fronting the master
 and
   slave
with an *AWS ELB *. The master configuration will not enable the
  slave
properties as master is not replicating from any other machine. The
   master
and slave have similar hardware configurations [*High-Memory
  Quadruple
Extra Large 

Re: Master /Slave Set up on AWS - 3.6.1

[Sujatha Arun]

Thanks Otis . Yes  true but considering that the Indexing is via a queue,
there would actually be minimal load on the machine. And we are planning to
replicate this set up via adding more machines when the server reaches
about 80% capacity for adding more cores.

Regards,
Sujatha

On Wed, Mar 6, 2013 at 9:26 AM, Otis Gospodnetic otis.gospodne...@gmail.com
 wrote:

 Hello,

 This is not recommended because people typically don't want the load from
 indexing affect queries/user experience.  If your numbers are low, then
 this may not be a big deal. If you already need to create a core on 2
 machines, creating it on 3 doesn't seem a big deal.  There is a slight
 conflict in what you wrote, which is that cores will be created frequently,
 which implies you will quickly have lots of cores vs. desire to have
 minimal hardware.

 If it were up to me, I'd consider a weaker/cheaper master, and more slaves.

 Otis
 --
 Solr  ElasticSearch Support
 http://sematext.com/





 On Tue, Mar 5, 2013 at 12:46 PM, Sujatha Arun suja.a...@gmail.com wrote:

  Hi Otis,Michael,
 
  Thanks for your input and suggestions .
 
  Yes, we were considering the sticky session for pagination  and we are
 not
  planning for having index on EBS
 
  I would like to understand  why its not the recommended approach,  can
 you
  please explain?
 
  Till now we were having a single server for both Indexing and search
  ,though this was on dedicated server and not on cloud. Indexing would
  happen sequentially via  a queue due to which commits would happen for
 only
  one or at most 2 cores simultaneously.
 
  With Master /Slave approach  I see that when a slave replicates form
 master
  based on the poll time which I have defined in solr.xml file  depending
 on
  the number of cores/webapp  ,the replication and hence commit is going to
  happen simultaneously for many cores increasing  load on slave server.
 
  By having  2 slaves and one master -  whenever we create a core,which
  happens quite frequently  we need to create this on 3 servers instead of
  two , which has to done manually by running a script on each server.We
 have
  requirement for adding cores per each customer.
 
  I do understand that hardware requirements for master can be quite
  different [Lower memory /higher CPU/Cache setting in config /autowarming
  etc  ] from slave.But given that we will be Indexing sequentially and
  having the same configuration in terms of memory and CPU/cache  for both
  master and slave,would this be a reasonable approach?
 
  ThanksRegards,
  Sujatha
 
 
 
  On Tue, Mar 5, 2013 at 9:10 PM, Michael Della Bitta 
  michael.della.bi...@appinions.com wrote:
 
   If your index is on EBS, you'll see big iowait percentages when merges
   happen.
  
   I'm not sure what that's going to do to your master's ability to
   service requests. You should test.
  
   Alternatively, you might figure out the size of machine you need to
   index vs. the size of machine you need to service queries. They're
   very likely not the same, in which case, that may afford you the
   ability to have 2 slaves and 1 master in a similar budget.
  
   Also, once you've settled on an infrastructure, you should investigate
   buying reserved instances for a year. It will greatly reduce your
   costs.
  
   Michael Della Bitta
  
   
   Appinions
   18 East 41st Street, 2nd Floor
   New York, NY 10017-6271
  
   www.appinions.com
  
   Where Influence Isn’t a Game
  
  
   On Tue, Mar 5, 2013 at 8:59 AM, Sujatha Arun suja.a...@gmail.com
  wrote:
Is there anything wrong with set up?
   
On Tue, Mar 5, 2013 at 5:43 PM, Sujatha Arun suja.a...@gmail.com
   wrote:
   
Hi Otis,
   
Since currently we are planning for only one slave  due to cost
considerations, can we have an ELB fronting the master and slave for
  HA.
   
   1. All index requests will go to the master .
   2. Slave replicates  from master .
   3. Search request can go either to master /slave via ELB.
   
is that resonable   HA for search ?
   
Regards
Sujatha
   
   
   
On Tue, Mar 5, 2013 at 5:12 PM, Otis Gospodnetic 
otis.gospodne...@gmail.com wrote:
   
Hi Sujatha,
   
If I understand correctly, you will have only 1 slave (and 1
 master),
   so
that's not really a HA architecture.  You could manually turn
 master
   into
slave, but that's going to mean some down time...
   
Otis
--
Solr  ElasticSearch Support
http://sematext.com/
   
   
   
   
   
On Tue, Mar 5, 2013 at 3:05 AM, Sujatha Arun suja.a...@gmail.com
   wrote:
   
 Hi,

 We are planning to set up *2* *High-Memory Quadruple Extra Large
Instance
  *as
 master and slave for our multicore solr setup  which has more
 than
   200
 cores spread between a couple of webapps on a single JVM on *AWS*

 All indexing [via a queue will go to master ]  . One Slave
  Server
   will
 replicate 

Re: Master /Slave Architecture3.6.1

[Otis Gospodnetic]

Hi,

You are going in the right direction and your assumptions are correct. In
short, if the performance hit is too big then you simply need more ec2
instances (some have high cpu, some memory, some disk IO ... pick wisely).

Otis
Solr  ElasticSearch Support
http://sematext.com/
On Jan 10, 2013 4:44 AM, Sujatha Arun suja.a...@gmail.com wrote:

 Hi,

 Our current architecture is as follows ,

- Single server  [ On which we do both Indexing and Searching]
- Solr version 3.6.1  Multicores
- We have several small  big indexes as cores within a webapp
- Our Indexing to the individual cores happen via an index queue ,due to
which at any given time ,we are indexing only to one or at most 2 cores
- Also we processing our pdf's and html files externally to text files
before feeding it to solr


 We are planning to move to the AWS using 3.6.1 and  would want to

-  Separate the  Indexing and  Searching to separate servers as master
/slave .This is mainly   so that the both the activities are not
 competing
for resources
- Also to use  Tika to process pdf and also to process html files
directly via solr ,which might increase the CPU load.
-  But ,if I set up so that all Indexing request are going to one server
sequentially and each core in slave polls the master core for index
 changes
,and then issues a commit to load a new index reader,then all this
 activity
might happen in parallel which will actually spike the CPU activity on
slave and hence will degrade the search performance?

 Is this assumption correct?Then is there any advantage other
 than availability to this architecture ,any advice on this?

 Regards
 Sujatha

Re: Master /Slave Architecture3.6.1

[Sujatha Arun]

Thanks,Otis..

But then what exactly is the advantage  for a master slave architecture
 for  multicore  ,when  replication has the same effect as that of a commit
and if I am going to have worse performance by moving to master/ slave over
a single server with sequential indexing?Am I missing anything?

Would it make sense to have each server act as both master and slave and
 LB the indexing and  searching requests to both servers?

Regards,
Sujatha
On Thu, Jan 10, 2013 at 8:41 PM, Otis Gospodnetic 
otis.gospodne...@gmail.com wrote:

 Hi,

 You are going in the right direction and your assumptions are correct. In
 short, if the performance hit is too big then you simply need more ec2
 instances (some have high cpu, some memory, some disk IO ... pick wisely).

 Otis
 Solr  ElasticSearch Support
 http://sematext.com/
 On Jan 10, 2013 4:44 AM, Sujatha Arun suja.a...@gmail.com wrote:

  Hi,
 
  Our current architecture is as follows ,
 
 - Single server  [ On which we do both Indexing and Searching]
 - Solr version 3.6.1  Multicores
 - We have several small  big indexes as cores within a webapp
 - Our Indexing to the individual cores happen via an index queue ,due
 to
 which at any given time ,we are indexing only to one or at most 2
 cores
 - Also we processing our pdf's and html files externally to text files
 before feeding it to solr
 
 
  We are planning to move to the AWS using 3.6.1 and  would want to
 
 -  Separate the  Indexing and  Searching to separate servers as master
 /slave .This is mainly   so that the both the activities are not
  competing
 for resources
 - Also to use  Tika to process pdf and also to process html files
 directly via solr ,which might increase the CPU load.
 -  But ,if I set up so that all Indexing request are going to one
 server
 sequentially and each core in slave polls the master core for index
  changes
 ,and then issues a commit to load a new index reader,then all this
  activity
 might happen in parallel which will actually spike the CPU activity on
 slave and hence will degrade the search performance?
 
  Is this assumption correct?Then is there any advantage other
  than availability to this architecture ,any advice on this?
 
  Regards
  Sujatha
 

Re: Master /Slave Architecture3.6.1

[Upayavira]

In the end, the best advice is try it. 

You'll save the effort of indexing with this master/slave setup, but
you'll still need to warm your caches on each slave, which is a
reasonable portion of the work done on a commit. However, with a
master/slave setup, you get the option to go to two slaves, or three,
etc as demand increases - and you can put them all behind an elastic
load balancer, and scale easily.

You may have multiple cores on your Solr system, but note that servers
have multiple CPUs, so two simultaneous replication requests needn't be
a disaster.

Upayavira

On Thu, Jan 10, 2013, at 05:50 PM, Sujatha Arun wrote:
 Thanks,Otis..
 
 But then what exactly is the advantage  for a master slave architecture
  for  multicore  ,when  replication has the same effect as that of a
  commit
 and if I am going to have worse performance by moving to master/ slave
 over
 a single server with sequential indexing?Am I missing anything?
 
 Would it make sense to have each server act as both master and slave and
  LB the indexing and  searching requests to both servers?
 
 Regards,
 Sujatha
 On Thu, Jan 10, 2013 at 8:41 PM, Otis Gospodnetic 
 otis.gospodne...@gmail.com wrote:
 
  Hi,
 
  You are going in the right direction and your assumptions are correct. In
  short, if the performance hit is too big then you simply need more ec2
  instances (some have high cpu, some memory, some disk IO ... pick wisely).
 
  Otis
  Solr  ElasticSearch Support
  http://sematext.com/
  On Jan 10, 2013 4:44 AM, Sujatha Arun suja.a...@gmail.com wrote:
 
   Hi,
  
   Our current architecture is as follows ,
  
  - Single server  [ On which we do both Indexing and Searching]
  - Solr version 3.6.1  Multicores
  - We have several small  big indexes as cores within a webapp
  - Our Indexing to the individual cores happen via an index queue ,due
  to
  which at any given time ,we are indexing only to one or at most 2
  cores
  - Also we processing our pdf's and html files externally to text files
  before feeding it to solr
  
  
   We are planning to move to the AWS using 3.6.1 and  would want to
  
  -  Separate the  Indexing and  Searching to separate servers as master
  /slave .This is mainly   so that the both the activities are not
   competing
  for resources
  - Also to use  Tika to process pdf and also to process html files
  directly via solr ,which might increase the CPU load.
  -  But ,if I set up so that all Indexing request are going to one
  server
  sequentially and each core in slave polls the master core for index
   changes
  ,and then issues a commit to load a new index reader,then all this
   activity
  might happen in parallel which will actually spike the CPU activity on
  slave and hence will degrade the search performance?
  
   Is this assumption correct?Then is there any advantage other
   than availability to this architecture ,any advice on this?
  
   Regards
   Sujatha
  
 

Re: Master/Slave switch on teh fly. Replication

[Erick Erickson]

What's the use-case? Presumably you have different configs...

I'm actually not sure if you can do a reload
see: http://wiki.apache.org/solr/CoreAdmin#RELOAD
without a core, but you could try.

Best
Erick

On Thu, Mar 15, 2012 at 4:59 AM, stockii stock.jo...@googlemail.com wrote:
 Hello.

 Is it possible to switch master/slave on the fly without restarting the
 server?

 -
 --- System 
 

 One Server, 12 GB RAM, 2 Solr Instances, 8 Cores,
 1 Core with 45 Million Documents other Cores  200.000

 - Solr1 for Search-Requests - commit every Minute  - 5GB Xmx
 - Solr2 for Update-Request  - delta every Minute - 4GB Xmx
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Master-Slave-switch-on-teh-fly-Replication-tp3828313p3828313.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Master/Slave switch on teh fly. Replication

[stockii]

i have 8 cores ;-)

i thought that replication is defined in solrconfig.xml and this file is
only load on startup and i cannot change master to slave and slave to master
without restarting the servlet-container ?!?!?!

-
--- System 

One Server, 12 GB RAM, 2 Solr Instances, 8 Cores, 
1 Core with 45 Million Documents other Cores  200.000

- Solr1 for Search-Requests - commit every Minute  - 5GB Xmx
- Solr2 for Update-Request  - delta every Minute - 4GB Xmx
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Master-Slave-switch-on-the-fly-Replication-tp3828313p3831948.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Master/Slave switch on teh fly. Replication

[Michael Kuhlmann]

Am 16.03.2012 15:05, schrieb stockii:

i have 8 cores ;-)

i thought that replication is defined in solrconfig.xml and this file is
only load on startup and i cannot change master to slave and slave to master
without restarting the servlet-container ?!?!?!


No, you can reload the whole core at any time, without interruption. 
Even with a new solrconfig.xml.


You can even add a new core at runtime, fill it with data and switch 
cores afterwards.


See http://wiki.apache.org/solr/CoreAdmin for details.

-Kuli

RE: Master Slave Question

[Jaeger, Jay - DOT]

You could prevent queries to the master by limiting what IP addresses are 
allowed to communicate with it, or by modifying web.xml to put different 
security on /update vs. /select .

We took a simplistic approach.  We did some load testing, and discovered that 
we could handle our expected update load and our query load on the master.  We 
use replication just for failover in case the master dies (in which case 
updates would be held up until the master was fixed).  (We also have security 
on both /update and /select -- just different security).

JRJ

-Original Message-
From: Patrick Sauts [mailto:patrick.via...@gmail.com] 
Sent: Saturday, September 10, 2011 11:26 AM
To: solr-user@lucene.apache.org
Subject: Re: Master Slave Question

Real Time indexing (solr 4) or decrease replication poll and auto commit
time.

2011/9/10 Jamie Johnson jej2...@gmail.com

 Is it appropriate to query the master servers when replicating?  I ask
 because there could be a case where we index say 50 documents to the
 master, they have not yet been replicated and a user asks for page 2,
 when they ask for page 2 the request could be sent to a slave and get
 0.  Is there a way to avoid this?  My thought was to not allow
 querying of the master but I'm not sure that this could be configured
 in solr

Re: Master Slave Question

[Patrick Sauts]

Real Time indexing (solr 4) or decrease replication poll and auto commit
time.

2011/9/10 Jamie Johnson jej2...@gmail.com

 Is it appropriate to query the master servers when replicating?  I ask
 because there could be a case where we index say 50 documents to the
 master, they have not yet been replicated and a user asks for page 2,
 when they ask for page 2 the request could be sent to a slave and get
 0.  Is there a way to avoid this?  My thought was to not allow
 querying of the master but I'm not sure that this could be configured
 in solr

Re: Master Slave Question

[Jamie Johnson]

Is this feature on Trunk currently?

On Sat, Sep 10, 2011 at 12:26 PM, Patrick Sauts
patrick.via...@gmail.com wrote:
 Real Time indexing (solr 4) or decrease replication poll and auto commit
 time.

 2011/9/10 Jamie Johnson jej2...@gmail.com

 Is it appropriate to query the master servers when replicating?  I ask
 because there could be a case where we index say 50 documents to the
 master, they have not yet been replicated and a user asks for page 2,
 when they ask for page 2 the request could be sent to a slave and get
 0.  Is there a way to avoid this?  My thought was to not allow
 querying of the master but I'm not sure that this could be configured
 in solr

Re: Master Slave Replication in Solr Cloud - What happens if the master is not available?

[Erick Erickson]

No, there's nothing built into Solr to automatically promote a slave
to a master.

You have several choices here. One is to build a new master and
reindex from scratch.

Another is to configure your slave as a new master and then
bring up a new machine and have it replicate. Now make that new machine
your master (you'll have to re-configure both).

The fun part is continuing to serve requests while all this is going
on. It's easier
if you have more than one slave so you can move things around while the
remaining slave is reconfigured (or whatever)

Best
Erick

On Mon, Jun 20, 2011 at 8:28 AM, Mark Schoy hei...@gmx.de wrote:
 Hi,

 if I use a master slave replication in Solr Cloud and the master
 crashes, can the slave automatically switch to master mode?

 Or is there another way to index documents after the master is down?

 Thanks.

Re: Master Slave Replication in Solr Cloud - What happens if the master is not available?

[Mark Schoy]

Thanks for your answer Erick.

So the easiest way will be to set up 2 shard cluster with shard replicas ;)

2011/6/20 Erick Erickson erickerick...@gmail.com:
 No, there's nothing built into Solr to automatically promote a slave
 to a master.

 You have several choices here. One is to build a new master and
 reindex from scratch.

 Another is to configure your slave as a new master and then
 bring up a new machine and have it replicate. Now make that new machine
 your master (you'll have to re-configure both).

 The fun part is continuing to serve requests while all this is going
 on. It's easier
 if you have more than one slave so you can move things around while the
 remaining slave is reconfigured (or whatever)

 Best
 Erick

 On Mon, Jun 20, 2011 at 8:28 AM, Mark Schoy hei...@gmx.de wrote:
 Hi,

 if I use a master slave replication in Solr Cloud and the master
 crashes, can the slave automatically switch to master mode?

 Or is there another way to index documents after the master is down?

 Thanks.

Re: Master Slave Replication in Solr Cloud - What happens if the master is not available?

[Erick Erickson]

Hmmm, be a little careful here with terminology.
Shards may be unnecessary if you  can put your whole index
on a single searcher. It's preferable to   simply have each
slave hold a complete copy of the index, no sharding necessary.

Best
Erick

On Mon, Jun 20, 2011 at 9:45 AM, Mark Schoy hei...@gmx.de wrote:
 Thanks for your answer Erick.

 So the easiest way will be to set up 2 shard cluster with shard replicas ;)

 2011/6/20 Erick Erickson erickerick...@gmail.com:
 No, there's nothing built into Solr to automatically promote a slave
 to a master.

 You have several choices here. One is to build a new master and
 reindex from scratch.

 Another is to configure your slave as a new master and then
 bring up a new machine and have it replicate. Now make that new machine
 your master (you'll have to re-configure both).

 The fun part is continuing to serve requests while all this is going
 on. It's easier
 if you have more than one slave so you can move things around while the
 remaining slave is reconfigured (or whatever)

 Best
 Erick

 On Mon, Jun 20, 2011 at 8:28 AM, Mark Schoy hei...@gmx.de wrote:
 Hi,

 if I use a master slave replication in Solr Cloud and the master
 crashes, can the slave automatically switch to master mode?

 Or is there another way to index documents after the master is down?

 Thanks.

Re: Master Slave Replication in Solr Cloud - What happens if the master is not available?

[Mark Schoy]

You're right, thanks!

2011/6/20 Erick Erickson erickerick...@gmail.com:
 Hmmm, be a little careful here with terminology.
 Shards may be unnecessary if you  can put your whole index
 on a single searcher. It's preferable to   simply have each
 slave hold a complete copy of the index, no sharding necessary.

 Best
 Erick

 On Mon, Jun 20, 2011 at 9:45 AM, Mark Schoy hei...@gmx.de wrote:
 Thanks for your answer Erick.

 So the easiest way will be to set up 2 shard cluster with shard replicas ;)

 2011/6/20 Erick Erickson erickerick...@gmail.com:
 No, there's nothing built into Solr to automatically promote a slave
 to a master.

 You have several choices here. One is to build a new master and
 reindex from scratch.

 Another is to configure your slave as a new master and then
 bring up a new machine and have it replicate. Now make that new machine
 your master (you'll have to re-configure both).

 The fun part is continuing to serve requests while all this is going
 on. It's easier
 if you have more than one slave so you can move things around while the
 remaining slave is reconfigured (or whatever)

 Best
 Erick

 On Mon, Jun 20, 2011 at 8:28 AM, Mark Schoy hei...@gmx.de wrote:
 Hi,

 if I use a master slave replication in Solr Cloud and the master
 crashes, can the slave automatically switch to master mode?

 Or is there another way to index documents after the master is down?

 Thanks.

Re: Master Slave help

[Rohit Gupta]

thanks Jayendra..






From: Jayendra Patil jayendra.patil@gmail.com
To: solr-user@lucene.apache.org
Sent: Tue, 7 June, 2011 6:55:58 AM
Subject: Re: Master Slave help

Do you mean the replication happens everytime you restart the server ?
If so, you would need to modify the events you want the replication to happen.

Check for the replicateAfter tag and remove the startup option, if you
don't need it.

requestHandler name=/replication class=solr.ReplicationHandler 
lst name=master
!--Replicate on 'startup' and 'commit'. 'optimize' is also a
valid value for replicateAfter. --
str name=replicateAfterstartup/str
str name=replicateAftercommit/str

!--Create a backup after 'optimize'. Other values can be
'commit', 'startup'. It is possible to have multiple entries of this
config string.  Note that this is just for backup, replication does
not require this. --
!-- str name=backupAfteroptimize/str --

!--If configuration files need to be replicated give the
names here, separated by comma --
str name=confFilesschema.xml,stopwords.txt,elevate.xml/str
   !--The default value of reservation is 10 secs.See the
documentation below . Normally , you should not need to specify this
--
str name=commitReserveDuration00:00:10/str
/lst
/requestHandler

Regards,
Jayendra

On Mon, Jun 6, 2011 at 11:24 AM, Rohit Gupta ro...@in-rev.com wrote:
 Hi,

 I have configured my master slave server and everything seems to be running
 fine, the replication completed the firsttime it ran. But everytime I go the 
the
 replication link in the admin panel after restarting the server or server
 startup I notice the replication starting from scratch or at least the stats
 show that.

 What could be wrong?

 Thanks,
 Rohit

Re: Master Slave help

[Jayendra Patil]

Do you mean the replication happens everytime you restart the server ?
If so, you would need to modify the events you want the replication to happen.

Check for the replicateAfter tag and remove the startup option, if you
don't need it.

requestHandler name=/replication class=solr.ReplicationHandler 
lst name=master
!--Replicate on 'startup' and 'commit'. 'optimize' is also a
valid value for replicateAfter. --
str name=replicateAfterstartup/str
str name=replicateAftercommit/str

!--Create a backup after 'optimize'. Other values can be
'commit', 'startup'. It is possible to have multiple entries of this
config string.  Note that this is just for backup, replication does
not require this. --
!-- str name=backupAfteroptimize/str --

!--If configuration files need to be replicated give the
names here, separated by comma --
str name=confFilesschema.xml,stopwords.txt,elevate.xml/str
   !--The default value of reservation is 10 secs.See the
documentation below . Normally , you should not need to specify this
--
str name=commitReserveDuration00:00:10/str
/lst
/requestHandler

Regards,
Jayendra

On Mon, Jun 6, 2011 at 11:24 AM, Rohit Gupta ro...@in-rev.com wrote:
 Hi,

 I have configured my master slave server and everything seems to be running
 fine, the replication completed the firsttime it ran. But everytime I go the 
 the
 replication link in the admin panel after restarting the server or server
 startup I notice the replication starting from scratch or at least the stats
 show that.

 What could be wrong?

 Thanks,
 Rohit

Re: Master/Slave High CPU Usage

[Ofer Fort]

ok, we ran some tests and doing the commit for the slave as a post commit
event of the master reloaded the index and allowed us to achieve a master
slave configuration, without replication
This is useful only if your master and slave are on the same machine, and it
helps reducing the resources needed, as you don't have 2 indexes and you
don't need to copy the data from one to the other.

Thanks  Lance for that proposal.


On Sun, Nov 21, 2010 at 2:12 PM, Ofer Fort ofer...@gmail.com wrote:

 do i really need a commit? or can i use the 
 *readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript? since i 
 don't need to comit anything, just reopen the reader.
 thanks


 On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote:

 Yes, the Solr commit operations always reloads the index. And it
 always throws away the Solr caches: queryresult, document, filter
 query.

 If you do this, please post your results.

 On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote:
  OK,
  so to make sure i understand, even though the slave doesn't do any
  indexing, i will call commit and it will do nothing to the index itself,
 but
  will reload it?
  thanks
 
  On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com
 wrote:
 
  Ah! If the program doing the indexing has manual commits, the program
  could send a commit to the slave. If the indexer uses automatic
  commits, there is a trick: you can add a program as a postCommit event
  in solrconfig.xml. This can just be a shell script or a curl command
  that sends a commit to the slave Solr.
 
  Be sure to make all of the wait options false to this command; you
  don't want the master to block while the slave loads up the new index.
  Or, to control the maximum load on your server, you might actually
  want to make the master wait while the slave loads up
 
  Lance
 
  On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
   thanks Erick,
   but my question was regard the configuration Lance suggested, a
   configuration where i have two servers, set set logical master and
 slave,
   not as a true replication. Since both are running on the same
 machine,
  just
   have one only doing updates, and the other only queries, but both are
  using
   the same index files.
  
   Ofer
  
  
   On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
  
   The slave polls. See: http://wiki.apache.org/solr/SolrReplication
  
   Best
   Erick
  
   On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
Another question on that configuration, when the master commits,
 how
   does
the slave knows that the index has changed? Does it check the
 index
  and
finds out that it has a newer version?
Thanks again for the help,
Ofer
   
   
   
ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com
 כתב/ה:
   
If they are on the same server, you do not need to replicate.
   
If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the
 same
LockPolicy in solrconfig.xml. For security reasons, I would run
 the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.
   
On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
anybody?
   
   
On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
   
Hi, I'm working with Erez,
   
we experienced this again, and this time the slave index folder
 didn't
contain the index.XXX folder, only one index folder.
   
if we shutdown the slave, the CPU on the master was normal, as
 soon as
  we
started the slave again, the CPU went up to 100% again.
   
thanks for any help
   
ofer
   
   
On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
  wrote:
   
   
Hi all,
   
We've been seeing this for the second time already.
   
I have a solr (1.4.1) master and a slave. both are located on the
 same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the
 master)
   
All our updates are going towards the master, and all the queries
 are
towards the slave.
   
Once in a while the slave gets OutOfMemoryError. This is not the
 big
problem
(i have a about 100M documents)
   
The problem is that from that moment the CPU of the slave AND the
  master
   is
almost 100%.
   
If i shutdown the slave, the CPU of the master drops.
   
If i start the slave again, the CPU is 100% again.
   
I have the replication set on commit and startup.
   
I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ
   
   
The only way i was able to get pass it (worked two times already),
 is
  to
shutdown the two servers, and to copy all the index of the master
 to
  the
slave, and start them again.
   

Re: Master/Slave High CPU Usage

[Lance Norskog]

You're welcome.

On Tue, Nov 23, 2010 at 3:55 PM, Ofer Fort ofer...@gmail.com wrote:
 ok, we ran some tests and doing the commit for the slave as a post commit
 event of the master reloaded the index and allowed us to achieve a master
 slave configuration, without replication
 This is useful only if your master and slave are on the same machine, and it
 helps reducing the resources needed, as you don't have 2 indexes and you
 don't need to copy the data from one to the other.

 Thanks  Lance for that proposal.


 On Sun, Nov 21, 2010 at 2:12 PM, Ofer Fort ofer...@gmail.com wrote:

 do i really need a commit? or can i use the 
 *readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript? since 
 i don't need to comit anything, just reopen the reader.
 thanks


 On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote:

 Yes, the Solr commit operations always reloads the index. And it
 always throws away the Solr caches: queryresult, document, filter
 query.

 If you do this, please post your results.

 On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote:
  OK,
  so to make sure i understand, even though the slave doesn't do any
  indexing, i will call commit and it will do nothing to the index itself,
 but
  will reload it?
  thanks
 
  On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com
 wrote:
 
  Ah! If the program doing the indexing has manual commits, the program
  could send a commit to the slave. If the indexer uses automatic
  commits, there is a trick: you can add a program as a postCommit event
  in solrconfig.xml. This can just be a shell script or a curl command
  that sends a commit to the slave Solr.
 
  Be sure to make all of the wait options false to this command; you
  don't want the master to block while the slave loads up the new index.
  Or, to control the maximum load on your server, you might actually
  want to make the master wait while the slave loads up
 
  Lance
 
  On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
   thanks Erick,
   but my question was regard the configuration Lance suggested, a
   configuration where i have two servers, set set logical master and
 slave,
   not as a true replication. Since both are running on the same
 machine,
  just
   have one only doing updates, and the other only queries, but both are
  using
   the same index files.
  
   Ofer
  
  
   On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
  
   The slave polls. See: http://wiki.apache.org/solr/SolrReplication
  
   Best
   Erick
  
   On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
Another question on that configuration, when the master commits,
 how
   does
the slave knows that the index has changed? Does it check the
 index
  and
finds out that it has a newer version?
Thanks again for the help,
Ofer
   
   
   
ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com
 כתב/ה:
   
If they are on the same server, you do not need to replicate.
   
If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the
 same
LockPolicy in solrconfig.xml. For security reasons, I would run
 the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.
   
On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
anybody?
   
   
On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
   
Hi, I'm working with Erez,
   
we experienced this again, and this time the slave index folder
 didn't
contain the index.XXX folder, only one index folder.
   
if we shutdown the slave, the CPU on the master was normal, as
 soon as
  we
started the slave again, the CPU went up to 100% again.
   
thanks for any help
   
ofer
   
   
On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
  wrote:
   
   
Hi all,
   
We've been seeing this for the second time already.
   
I have a solr (1.4.1) master and a slave. both are located on the
 same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the
 master)
   
All our updates are going towards the master, and all the queries
 are
towards the slave.
   
Once in a while the slave gets OutOfMemoryError. This is not the
 big
problem
(i have a about 100M documents)
   
The problem is that from that moment the CPU of the slave AND the
  master
   is
almost 100%.
   
If i shutdown the slave, the CPU of the master drops.
   
If i start the slave again, the CPU is 100% again.
   
I have the replication set on commit and startup.
   
I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ
   
   
The only way i was able to get pass it (worked two times already),
 is
  to
shutdown the two 

Re: Master/Slave High CPU Usage

[Lance Norskog]

Yes, the Solr commit operations always reloads the index. And it
always throws away the Solr caches: queryresult, document, filter
query.

If you do this, please post your results.

On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote:
 OK,
 so to make sure i understand, even though the slave doesn't do any
 indexing, i will call commit and it will do nothing to the index itself, but
 will reload it?
 thanks

 On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote:

 Ah! If the program doing the indexing has manual commits, the program
 could send a commit to the slave. If the indexer uses automatic
 commits, there is a trick: you can add a program as a postCommit event
 in solrconfig.xml. This can just be a shell script or a curl command
 that sends a commit to the slave Solr.

 Be sure to make all of the wait options false to this command; you
 don't want the master to block while the slave loads up the new index.
 Or, to control the maximum load on your server, you might actually
 want to make the master wait while the slave loads up

 Lance

 On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
  thanks Erick,
  but my question was regard the configuration Lance suggested, a
  configuration where i have two servers, set set logical master and slave,
  not as a true replication. Since both are running on the same machine,
 just
  have one only doing updates, and the other only queries, but both are
 using
  the same index files.
 
  Ofer
 
 
  On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com
 wrote:
 
  The slave polls. See: http://wiki.apache.org/solr/SolrReplication
 
  Best
  Erick
 
  On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote:
 
   Another question on that configuration, when the master commits, how
  does
   the slave knows that the index has changed? Does it check the index
 and
   finds out that it has a newer version?
   Thanks again for the help,
   Ofer
  
  
  
   ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:
  
   If they are on the same server, you do not need to replicate.
  
   If you only do queries, the query server can use the same index
   directory as the master. Works quite well. Both have to have the same
   LockPolicy in solrconfig.xml. For security reasons, I would run the
   query server as a different user who has read-only access to the
   index; that way it cannot touch the index.
  
   On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
   anybody?
  
  
   On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
  
   Hi, I'm working with Erez,
  
   we experienced this again, and this time the slave index folder didn't
   contain the index.XXX folder, only one index folder.
  
   if we shutdown the slave, the CPU on the master was normal, as soon as
 we
   started the slave again, the CPU went up to 100% again.
  
   thanks for any help
  
   ofer
  
  
   On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
 wrote:
  
  
   Hi all,
  
   We've been seeing this for the second time already.
  
   I have a solr (1.4.1) master and a slave. both are located on the same
   machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
  
   All our updates are going towards the master, and all the queries are
   towards the slave.
  
   Once in a while the slave gets OutOfMemoryError. This is not the big
   problem
   (i have a about 100M documents)
  
   The problem is that from that moment the CPU of the slave AND the
 master
  is
   almost 100%.
  
   If i shutdown the slave, the CPU of the master drops.
  
   If i start the slave again, the CPU is 100% again.
  
   I have the replication set on commit and startup.
  
   I see that in the data folder contains three index folders: index,
   index.XXXYYY and  index.XXXYYY.ZZZ
  
  
   The only way i was able to get pass it (worked two times already), is
 to
   shutdown the two servers, and to copy all the index of the master to
 the
   slave, and start them again.
  
   From that moment and on, they continue to work and replicate with a
 very
   reasonable CPU usage.
  
  
   Our guess is that it failed to replicate due to the OOM and since then
   tries
   to do a full replication again and again?
  
   but why is the CPU of the master so high?
  
  
  
  
  
  
   --
   Lance Norskog
   goks...@gmail.com
  
 
 



 --
 Lance Norskog
 goks...@gmail.com





-- 
Lance Norskog
goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

ok, i'll try that and update the group
thanks

On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote:

 Yes, the Solr commit operations always reloads the index. And it
 always throws away the Solr caches: queryresult, document, filter
 query.

 If you do this, please post your results.

 On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote:
  OK,
  so to make sure i understand, even though the slave doesn't do any
  indexing, i will call commit and it will do nothing to the index itself,
 but
  will reload it?
  thanks
 
  On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com
 wrote:
 
  Ah! If the program doing the indexing has manual commits, the program
  could send a commit to the slave. If the indexer uses automatic
  commits, there is a trick: you can add a program as a postCommit event
  in solrconfig.xml. This can just be a shell script or a curl command
  that sends a commit to the slave Solr.
 
  Be sure to make all of the wait options false to this command; you
  don't want the master to block while the slave loads up the new index.
  Or, to control the maximum load on your server, you might actually
  want to make the master wait while the slave loads up
 
  Lance
 
  On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
   thanks Erick,
   but my question was regard the configuration Lance suggested, a
   configuration where i have two servers, set set logical master and
 slave,
   not as a true replication. Since both are running on the same machine,
  just
   have one only doing updates, and the other only queries, but both are
  using
   the same index files.
  
   Ofer
  
  
   On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
  
   The slave polls. See: http://wiki.apache.org/solr/SolrReplication
  
   Best
   Erick
  
   On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
Another question on that configuration, when the master commits,
 how
   does
the slave knows that the index has changed? Does it check the
 index
  and
finds out that it has a newer version?
Thanks again for the help,
Ofer
   
   
   
ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com
 כתב/ה:
   
If they are on the same server, you do not need to replicate.
   
If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the
 same
LockPolicy in solrconfig.xml. For security reasons, I would run the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.
   
On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
anybody?
   
   
On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
   
Hi, I'm working with Erez,
   
we experienced this again, and this time the slave index folder
 didn't
contain the index.XXX folder, only one index folder.
   
if we shutdown the slave, the CPU on the master was normal, as soon
 as
  we
started the slave again, the CPU went up to 100% again.
   
thanks for any help
   
ofer
   
   
On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
  wrote:
   
   
Hi all,
   
We've been seeing this for the second time already.
   
I have a solr (1.4.1) master and a slave. both are located on the
 same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the
 master)
   
All our updates are going towards the master, and all the queries
 are
towards the slave.
   
Once in a while the slave gets OutOfMemoryError. This is not the
 big
problem
(i have a about 100M documents)
   
The problem is that from that moment the CPU of the slave AND the
  master
   is
almost 100%.
   
If i shutdown the slave, the CPU of the master drops.
   
If i start the slave again, the CPU is 100% again.
   
I have the replication set on commit and startup.
   
I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ
   
   
The only way i was able to get pass it (worked two times already),
 is
  to
shutdown the two servers, and to copy all the index of the master
 to
  the
slave, and start them again.
   
From that moment and on, they continue to work and replicate with a
  very
reasonable CPU usage.
   
   
Our guess is that it failed to replicate due to the OOM and since
 then
tries
to do a full replication again and again?
   
but why is the CPU of the master so high?
   
   
   
   
   
   
--
Lance Norskog
goks...@gmail.com
   
  
  
 
 
 
  --
  Lance Norskog
  goks...@gmail.com
 
 



 --
 Lance Norskog
 goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

do i really need a commit? or can i use the
*readercycle*http://wiki.apache.org/solr/SolrOperationsToolsscript?
since i don't need to comit anything, just reopen the reader.
thanks

On Sun, Nov 21, 2010 at 12:17 PM, Lance Norskog goks...@gmail.com wrote:

 Yes, the Solr commit operations always reloads the index. And it
 always throws away the Solr caches: queryresult, document, filter
 query.

 If you do this, please post your results.

 On Sat, Nov 20, 2010 at 11:16 PM, Ofer Fort ofer...@gmail.com wrote:
  OK,
  so to make sure i understand, even though the slave doesn't do any
  indexing, i will call commit and it will do nothing to the index itself,
 but
  will reload it?
  thanks
 
  On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com
 wrote:
 
  Ah! If the program doing the indexing has manual commits, the program
  could send a commit to the slave. If the indexer uses automatic
  commits, there is a trick: you can add a program as a postCommit event
  in solrconfig.xml. This can just be a shell script or a curl command
  that sends a commit to the slave Solr.
 
  Be sure to make all of the wait options false to this command; you
  don't want the master to block while the slave loads up the new index.
  Or, to control the maximum load on your server, you might actually
  want to make the master wait while the slave loads up
 
  Lance
 
  On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
   thanks Erick,
   but my question was regard the configuration Lance suggested, a
   configuration where i have two servers, set set logical master and
 slave,
   not as a true replication. Since both are running on the same machine,
  just
   have one only doing updates, and the other only queries, but both are
  using
   the same index files.
  
   Ofer
  
  
   On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson 
 erickerick...@gmail.com
  wrote:
  
   The slave polls. See: http://wiki.apache.org/solr/SolrReplication
  
   Best
   Erick
  
   On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
Another question on that configuration, when the master commits,
 how
   does
the slave knows that the index has changed? Does it check the
 index
  and
finds out that it has a newer version?
Thanks again for the help,
Ofer
   
   
   
ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com
 כתב/ה:
   
If they are on the same server, you do not need to replicate.
   
If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the
 same
LockPolicy in solrconfig.xml. For security reasons, I would run the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.
   
On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
anybody?
   
   
On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
  wrote:
   
   
Hi, I'm working with Erez,
   
we experienced this again, and this time the slave index folder
 didn't
contain the index.XXX folder, only one index folder.
   
if we shutdown the slave, the CPU on the master was normal, as soon
 as
  we
started the slave again, the CPU went up to 100% again.
   
thanks for any help
   
ofer
   
   
On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
  wrote:
   
   
Hi all,
   
We've been seeing this for the second time already.
   
I have a solr (1.4.1) master and a slave. both are located on the
 same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the
 master)
   
All our updates are going towards the master, and all the queries
 are
towards the slave.
   
Once in a while the slave gets OutOfMemoryError. This is not the
 big
problem
(i have a about 100M documents)
   
The problem is that from that moment the CPU of the slave AND the
  master
   is
almost 100%.
   
If i shutdown the slave, the CPU of the master drops.
   
If i start the slave again, the CPU is 100% again.
   
I have the replication set on commit and startup.
   
I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ
   
   
The only way i was able to get pass it (worked two times already),
 is
  to
shutdown the two servers, and to copy all the index of the master
 to
  the
slave, and start them again.
   
From that moment and on, they continue to work and replicate with a
  very
reasonable CPU usage.
   
   
Our guess is that it failed to replicate due to the OOM and since
 then
tries
to do a full replication again and again?
   
but why is the CPU of the master so high?
   
   
   
   
   
   
--
Lance Norskog
goks...@gmail.com
   
  
  
 
 
 
  --
  Lance Norskog
  goks...@gmail.com
 
 



 --
 Lance Norskog
 goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

Another question on that configuration, when the master commits, how does
the slave knows that the index has changed? Does it check the index and
finds out that it has a newer version?
Thanks again for the help,
Ofer



ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:

If they are on the same server, you do not need to replicate.

If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the same
LockPolicy in solrconfig.xml. For security reasons, I would run the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.

On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:

anybody?


On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:


Hi, I'm working with Erez,

we experienced this again, and this time the slave index folder didn't
contain the index.XXX folder, only one index folder.

if we shutdown the slave, the CPU on the master was normal, as soon as we
started the slave again, the CPU went up to 100% again.

thanks for any help

ofer


On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:


Hi all,

We've been seeing this for the second time already.

I have a solr (1.4.1) master and a slave. both are located on the same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)

All our updates are going towards the master, and all the queries are
towards the slave.

Once in a while the slave gets OutOfMemoryError. This is not the big problem
(i have a about 100M documents)

The problem is that from that moment the CPU of the slave AND the master is
almost 100%.

If i shutdown the slave, the CPU of the master drops.

If i start the slave again, the CPU is 100% again.

I have the replication set on commit and startup.

I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ


The only way i was able to get pass it (worked two times already), is to
shutdown the two servers, and to copy all the index of the master to the
slave, and start them again.

From that moment and on, they continue to work and replicate with a very
reasonable CPU usage.


Our guess is that it failed to replicate due to the OOM and since then tries
to do a full replication again and again?

but why is the CPU of the master so high?






-- 
Lance Norskog
goks...@gmail.com

Re: Master/Slave High CPU Usage

[Erick Erickson]

The slave polls. See: http://wiki.apache.org/solr/SolrReplication

Best
Erick

On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote:

 Another question on that configuration, when the master commits, how does
 the slave knows that the index has changed? Does it check the index and
 finds out that it has a newer version?
 Thanks again for the help,
 Ofer



 ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:

 If they are on the same server, you do not need to replicate.

 If you only do queries, the query server can use the same index
 directory as the master. Works quite well. Both have to have the same
 LockPolicy in solrconfig.xml. For security reasons, I would run the
 query server as a different user who has read-only access to the
 index; that way it cannot touch the index.

 On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:

 anybody?


 On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:


 Hi, I'm working with Erez,

 we experienced this again, and this time the slave index folder didn't
 contain the index.XXX folder, only one index folder.

 if we shutdown the slave, the CPU on the master was normal, as soon as we
 started the slave again, the CPU went up to 100% again.

 thanks for any help

 ofer


 On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:


 Hi all,

 We've been seeing this for the second time already.

 I have a solr (1.4.1) master and a slave. both are located on the same
 machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)

 All our updates are going towards the master, and all the queries are
 towards the slave.

 Once in a while the slave gets OutOfMemoryError. This is not the big
 problem
 (i have a about 100M documents)

 The problem is that from that moment the CPU of the slave AND the master is
 almost 100%.

 If i shutdown the slave, the CPU of the master drops.

 If i start the slave again, the CPU is 100% again.

 I have the replication set on commit and startup.

 I see that in the data folder contains three index folders: index,
 index.XXXYYY and  index.XXXYYY.ZZZ


 The only way i was able to get pass it (worked two times already), is to
 shutdown the two servers, and to copy all the index of the master to the
 slave, and start them again.

 From that moment and on, they continue to work and replicate with a very
 reasonable CPU usage.


 Our guess is that it failed to replicate due to the OOM and since then
 tries
 to do a full replication again and again?

 but why is the CPU of the master so high?






 --
 Lance Norskog
 goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

thanks Erick,
but my question was regard the configuration Lance suggested, a
configuration where i have two servers, set set logical master and slave,
not as a true replication. Since both are running on the same machine, just
have one only doing updates, and the other only queries, but both are using
the same index files.

Ofer


On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.comwrote:

 The slave polls. See: http://wiki.apache.org/solr/SolrReplication

 Best
 Erick

 On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote:

  Another question on that configuration, when the master commits, how
 does
  the slave knows that the index has changed? Does it check the index and
  finds out that it has a newer version?
  Thanks again for the help,
  Ofer
 
 
 
  ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:
 
  If they are on the same server, you do not need to replicate.
 
  If you only do queries, the query server can use the same index
  directory as the master. Works quite well. Both have to have the same
  LockPolicy in solrconfig.xml. For security reasons, I would run the
  query server as a different user who has read-only access to the
  index; that way it cannot touch the index.
 
  On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:
 
  anybody?
 
 
  On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:
 
 
  Hi, I'm working with Erez,
 
  we experienced this again, and this time the slave index folder didn't
  contain the index.XXX folder, only one index folder.
 
  if we shutdown the slave, the CPU on the master was normal, as soon as we
  started the slave again, the CPU went up to 100% again.
 
  thanks for any help
 
  ofer
 
 
  On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:
 
 
  Hi all,
 
  We've been seeing this for the second time already.
 
  I have a solr (1.4.1) master and a slave. both are located on the same
  machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
 
  All our updates are going towards the master, and all the queries are
  towards the slave.
 
  Once in a while the slave gets OutOfMemoryError. This is not the big
  problem
  (i have a about 100M documents)
 
  The problem is that from that moment the CPU of the slave AND the master
 is
  almost 100%.
 
  If i shutdown the slave, the CPU of the master drops.
 
  If i start the slave again, the CPU is 100% again.
 
  I have the replication set on commit and startup.
 
  I see that in the data folder contains three index folders: index,
  index.XXXYYY and  index.XXXYYY.ZZZ
 
 
  The only way i was able to get pass it (worked two times already), is to
  shutdown the two servers, and to copy all the index of the master to the
  slave, and start them again.
 
  From that moment and on, they continue to work and replicate with a very
  reasonable CPU usage.
 
 
  Our guess is that it failed to replicate due to the OOM and since then
  tries
  to do a full replication again and again?
 
  but why is the CPU of the master so high?
 
 
 
 
 
 
  --
  Lance Norskog
  goks...@gmail.com
 

Re: Master/Slave High CPU Usage

[Lance Norskog]

Ah! If the program doing the indexing has manual commits, the program
could send a commit to the slave. If the indexer uses automatic
commits, there is a trick: you can add a program as a postCommit event
in solrconfig.xml. This can just be a shell script or a curl command
that sends a commit to the slave Solr.

Be sure to make all of the wait options false to this command; you
don't want the master to block while the slave loads up the new index.
Or, to control the maximum load on your server, you might actually
want to make the master wait while the slave loads up

Lance

On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
 thanks Erick,
 but my question was regard the configuration Lance suggested, a
 configuration where i have two servers, set set logical master and slave,
 not as a true replication. Since both are running on the same machine, just
 have one only doing updates, and the other only queries, but both are using
 the same index files.

 Ofer


 On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson 
 erickerick...@gmail.comwrote:

 The slave polls. See: http://wiki.apache.org/solr/SolrReplication

 Best
 Erick

 On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote:

  Another question on that configuration, when the master commits, how
 does
  the slave knows that the index has changed? Does it check the index and
  finds out that it has a newer version?
  Thanks again for the help,
  Ofer
 
 
 
  ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:
 
  If they are on the same server, you do not need to replicate.
 
  If you only do queries, the query server can use the same index
  directory as the master. Works quite well. Both have to have the same
  LockPolicy in solrconfig.xml. For security reasons, I would run the
  query server as a different user who has read-only access to the
  index; that way it cannot touch the index.
 
  On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:
 
  anybody?
 
 
  On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:
 
 
  Hi, I'm working with Erez,
 
  we experienced this again, and this time the slave index folder didn't
  contain the index.XXX folder, only one index folder.
 
  if we shutdown the slave, the CPU on the master was normal, as soon as we
  started the slave again, the CPU went up to 100% again.
 
  thanks for any help
 
  ofer
 
 
  On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:
 
 
  Hi all,
 
  We've been seeing this for the second time already.
 
  I have a solr (1.4.1) master and a slave. both are located on the same
  machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
 
  All our updates are going towards the master, and all the queries are
  towards the slave.
 
  Once in a while the slave gets OutOfMemoryError. This is not the big
  problem
  (i have a about 100M documents)
 
  The problem is that from that moment the CPU of the slave AND the master
 is
  almost 100%.
 
  If i shutdown the slave, the CPU of the master drops.
 
  If i start the slave again, the CPU is 100% again.
 
  I have the replication set on commit and startup.
 
  I see that in the data folder contains three index folders: index,
  index.XXXYYY and  index.XXXYYY.ZZZ
 
 
  The only way i was able to get pass it (worked two times already), is to
  shutdown the two servers, and to copy all the index of the master to the
  slave, and start them again.
 
  From that moment and on, they continue to work and replicate with a very
  reasonable CPU usage.
 
 
  Our guess is that it failed to replicate due to the OOM and since then
  tries
  to do a full replication again and again?
 
  but why is the CPU of the master so high?
 
 
 
 
 
 
  --
  Lance Norskog
  goks...@gmail.com
 





-- 
Lance Norskog
goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

OK,
so to make sure i understand, even though the slave doesn't do any
indexing, i will call commit and it will do nothing to the index itself, but
will reload it?
thanks

On Sun, Nov 21, 2010 at 8:26 AM, Lance Norskog goks...@gmail.com wrote:

 Ah! If the program doing the indexing has manual commits, the program
 could send a commit to the slave. If the indexer uses automatic
 commits, there is a trick: you can add a program as a postCommit event
 in solrconfig.xml. This can just be a shell script or a curl command
 that sends a commit to the slave Solr.

 Be sure to make all of the wait options false to this command; you
 don't want the master to block while the slave loads up the new index.
 Or, to control the maximum load on your server, you might actually
 want to make the master wait while the slave loads up

 Lance

 On Sat, Nov 20, 2010 at 2:13 PM, Ofer Fort ofer...@gmail.com wrote:
  thanks Erick,
  but my question was regard the configuration Lance suggested, a
  configuration where i have two servers, set set logical master and slave,
  not as a true replication. Since both are running on the same machine,
 just
  have one only doing updates, and the other only queries, but both are
 using
  the same index files.
 
  Ofer
 
 
  On Sat, Nov 20, 2010 at 8:52 PM, Erick Erickson erickerick...@gmail.com
 wrote:
 
  The slave polls. See: http://wiki.apache.org/solr/SolrReplication
 
  Best
  Erick
 
  On Sat, Nov 20, 2010 at 1:13 PM, Ofer Fort ofer...@gmail.com wrote:
 
   Another question on that configuration, when the master commits, how
  does
   the slave knows that the index has changed? Does it check the index
 and
   finds out that it has a newer version?
   Thanks again for the help,
   Ofer
  
  
  
   ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:
  
   If they are on the same server, you do not need to replicate.
  
   If you only do queries, the query server can use the same index
   directory as the master. Works quite well. Both have to have the same
   LockPolicy in solrconfig.xml. For security reasons, I would run the
   query server as a different user who has read-only access to the
   index; that way it cannot touch the index.
  
   On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
   anybody?
  
  
   On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com
 wrote:
  
  
   Hi, I'm working with Erez,
  
   we experienced this again, and this time the slave index folder didn't
   contain the index.XXX folder, only one index folder.
  
   if we shutdown the slave, the CPU on the master was normal, as soon as
 we
   started the slave again, the CPU went up to 100% again.
  
   thanks for any help
  
   ofer
  
  
   On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il
 wrote:
  
  
   Hi all,
  
   We've been seeing this for the second time already.
  
   I have a solr (1.4.1) master and a slave. both are located on the same
   machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
  
   All our updates are going towards the master, and all the queries are
   towards the slave.
  
   Once in a while the slave gets OutOfMemoryError. This is not the big
   problem
   (i have a about 100M documents)
  
   The problem is that from that moment the CPU of the slave AND the
 master
  is
   almost 100%.
  
   If i shutdown the slave, the CPU of the master drops.
  
   If i start the slave again, the CPU is 100% again.
  
   I have the replication set on commit and startup.
  
   I see that in the data folder contains three index folders: index,
   index.XXXYYY and  index.XXXYYY.ZZZ
  
  
   The only way i was able to get pass it (worked two times already), is
 to
   shutdown the two servers, and to copy all the index of the master to
 the
   slave, and start them again.
  
   From that moment and on, they continue to work and replicate with a
 very
   reasonable CPU usage.
  
  
   Our guess is that it failed to replicate due to the OOM and since then
   tries
   to do a full replication again and again?
  
   but why is the CPU of the master so high?
  
  
  
  
  
  
   --
   Lance Norskog
   goks...@gmail.com
  
 
 



 --
 Lance Norskog
 goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

That sounds like a great option, and it will also free some storage space on
that sever (now each index is about 130GB).
Other than the lock policy (we use single), any other things to look out to?
Thanks


ב-19 בנוב 2010, בשעה 05:30, Lance Norskog goks...@gmail.com כתב/ה:

If they are on the same server, you do not need to replicate.

If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the same
LockPolicy in solrconfig.xml. For security reasons, I would run the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.

On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:

anybody?


On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:


Hi, I'm working with Erez,

we experienced this again, and this time the slave index folder didn't
contain the index.XXX folder, only one index folder.

if we shutdown the slave, the CPU on the master was normal, as soon as we
started the slave again, the CPU went up to 100% again.

thanks for any help

ofer


On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:


Hi all,

We've been seeing this for the second time already.

I have a solr (1.4.1) master and a slave. both are located on the same
machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)

All our updates are going towards the master, and all the queries are
towards the slave.

Once in a while the slave gets OutOfMemoryError. This is not the big problem
(i have a about 100M documents)

The problem is that from that moment the CPU of the slave AND the master is
almost 100%.

If i shutdown the slave, the CPU of the master drops.

If i start the slave again, the CPU is 100% again.

I have the replication set on commit and startup.

I see that in the data folder contains three index folders: index,
index.XXXYYY and  index.XXXYYY.ZZZ


The only way i was able to get pass it (worked two times already), is to
shutdown the two servers, and to copy all the index of the master to the
slave, and start them again.

From that moment and on, they continue to work and replicate with a very
reasonable CPU usage.


Our guess is that it failed to replicate due to the OOM and since then tries
to do a full replication again and again?

but why is the CPU of the master so high?






-- 
Lance Norskog
goks...@gmail.com

Re: Master/Slave High CPU Usage

[Lance Norskog]

If they are on the same server, you do not need to replicate.

If you only do queries, the query server can use the same index
directory as the master. Works quite well. Both have to have the same
LockPolicy in solrconfig.xml. For security reasons, I would run the
query server as a different user who has read-only access to the
index; that way it cannot touch the index.

On Wed, Nov 17, 2010 at 11:28 PM, Ofer Fort ofer...@gmail.com wrote:
 anybody?

 On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:

 Hi, I'm working with Erez,
 we experienced this again, and this time the slave index folder didn't 
 contain the index.XXX folder, only one index folder.
 if we shutdown the slave, the CPU on the master was normal, as soon as we 
 started the slave again, the CPU went up to 100% again.
 thanks for any help
 ofer

 On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:

 Hi all,
 We've been seeing this for the second time already.
 I have a solr (1.4.1) master and a slave. both are located on the same 
 machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
 All our updates are going towards the master, and all the queries are 
 towards the slave.
 Once in a while the slave gets OutOfMemoryError. This is not the big 
 problem (i have a about 100M documents)
 The problem is that from that moment the CPU of the slave AND the master is 
 almost 100%.
 If i shutdown the slave, the CPU of the master drops.
 If i start the slave again, the CPU is 100% again.
 I have the replication set on commit and startup.
 I see that in the data folder contains three index folders: index, 
 index.XXXYYY and  index.XXXYYY.ZZZ

 The only way i was able to get pass it (worked two times already), is to 
 shutdown the two servers, and to copy all the index of the master to the 
 slave, and start them again.
 From that moment and on, they continue to work and replicate with a very 
 reasonable CPU usage.

 Our guess is that it failed to replicate due to the OOM and since then 
 tries to do a full replication again and again?
 but why is the CPU of the master so high?





-- 
Lance Norskog
goks...@gmail.com

Re: Master/Slave High CPU Usage

[Ofer Fort]

Hi, I'm working with Erez,
we experienced this again, and this time the slave index folder didn't
contain the index.XXX folder, only one index folder.
if we shutdown the slave, the CPU on the master was normal, as soon as we
started the slave again, the CPU went up to 100% again.
thanks for any help
ofer

On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:

 Hi all,
 We've been seeing this for the second time already.
 I have a solr (1.4.1) master and a slave. both are located on the same
 machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
 All our updates are going towards the master, and all the queries are
 towards the slave.
 Once in a while the slave gets OutOfMemoryError. This is not the big
 problem (i have a about 100M documents)
 The problem is that from that moment the CPU of the slave AND the master is
 almost 100%.
 If i shutdown the slave, the CPU of the master drops.
 If i start the slave again, the CPU is 100% again.
 I have the replication set on commit and startup.
 I see that in the data folder contains three index folders: index,
 index.XXXYYY and  index.XXXYYY.ZZZ

 The only way i was able to get pass it (worked two times already), is to
 shutdown the two servers, and to copy all the index of the master to the
 slave, and start them again.
 From that moment and on, they continue to work and replicate with a very
 reasonable CPU usage.

 Our guess is that it failed to replicate due to the OOM and since then
 tries to do a full replication again and again?
 but why is the CPU of the master so high?

Re: Master/Slave High CPU Usage

[Ofer Fort]

anybody?

On Wed, Nov 17, 2010 at 12:09 PM, Ofer Fort ofer...@gmail.com wrote:

 Hi, I'm working with Erez,
 we experienced this again, and this time the slave index folder didn't 
 contain the index.XXX folder, only one index folder.
 if we shutdown the slave, the CPU on the master was normal, as soon as we 
 started the slave again, the CPU went up to 100% again.
 thanks for any help
 ofer

 On Wed, Nov 17, 2010 at 11:15 AM, Erez Zarum e...@icinga.org.il wrote:

 Hi all,
 We've been seeing this for the second time already.
 I have a solr (1.4.1) master and a slave. both are located on the same 
 machine (16GB RAM, 4GB allocated to the slave and 3GB to the master)
 All our updates are going towards the master, and all the queries are 
 towards the slave.
 Once in a while the slave gets OutOfMemoryError. This is not the big problem 
 (i have a about 100M documents)
 The problem is that from that moment the CPU of the slave AND the master is 
 almost 100%.
 If i shutdown the slave, the CPU of the master drops.
 If i start the slave again, the CPU is 100% again.
 I have the replication set on commit and startup.
 I see that in the data folder contains three index folders: index, 
 index.XXXYYY and  index.XXXYYY.ZZZ

 The only way i was able to get pass it (worked two times already), is to 
 shutdown the two servers, and to copy all the index of the master to the 
 slave, and start them again.
 From that moment and on, they continue to work and replicate with a very 
 reasonable CPU usage.

 Our guess is that it failed to replicate due to the OOM and since then tries 
 to do a full replication again and again?
 but why is the CPU of the master so high?

Re: master/slave

[Peter Sturge]

The SolrEmbededServer doesn't have any http, and so you can't use the http
replication.
You can use the script-based replication if you're on LUNIX. See:
http://wiki.apache.org/solr/CollectionDistribution

It would be worth looking at using Solr in a Jetty container and using the
http replication, it is really awesome.



On Tue, Mar 9, 2010 at 5:27 PM, Dino Di Cola dinodic...@gmail.com wrote:

 Dear all, I am trying to setup a master/slave index replication
 with two slaves embedded in a tomcat cluster and a master kept in a
 separate
 machine.
 I would like to know if is it possible to configure slaves with a
 ReplicationHandler able to access master
 by starting an embedded server instead of using http communication.

 I understand that HTTP is the preferred way to work with solr,
 but for some annoying reasons I cannot startup another http server. Thus, I
 wonder to know if (and possibly how)
 this approach can be technically 'feasible', already conscious that it
 cannot be definitively 'reasonable'... :)

 Many thanks for the support,
 Dino.
 --

Re: master/slave

[Dino Di Cola]

Ok Peter for script-based replication; I forgot to mention I already
verified that mechanism.

When I configure the slave as follows

requestHandler name=/replication class=solr.ReplicationHandler 
lst name=slave
str name=masterUrlhttp://localhost:8983/solr/admin/replication/str

str name=pollInterval00:00:20/str
...
/lst
/requestHandler

SOLR uses the org.apache.solr.handler.ReplicationHandler to access every 20s
the masterUrl via http.
My question is: is it possible to use another ReplicationHandler that polls
the master by pointing directly to its SOLR home?
I mean, is technically feasible?

Please correct me if I am not clear.
Thanks.
--
2010/3/9 Peter Sturge peter.stu...@googlemail.com

 The SolrEmbededServer doesn't have any http, and so you can't use the http
 replication.
 You can use the script-based replication if you're on LUNIX. See:
http://wiki.apache.org/solr/CollectionDistribution

 It would be worth looking at using Solr in a Jetty container and using the
 http replication, it is really awesome.



 On Tue, Mar 9, 2010 at 5:27 PM, Dino Di Cola dinodic...@gmail.com wrote:

  Dear all, I am trying to setup a master/slave index replication
  with two slaves embedded in a tomcat cluster and a master kept in a
  separate
  machine.
  I would like to know if is it possible to configure slaves with a
  ReplicationHandler able to access master
  by starting an embedded server instead of using http communication.
 
  I understand that HTTP is the preferred way to work with solr,
  but for some annoying reasons I cannot startup another http server. Thus,
 I
  wonder to know if (and possibly how)
  this approach can be technically 'feasible', already conscious that it
  cannot be definitively 'reasonable'... :)
 
  Many thanks for the support,
  Dino.
  --
 

Re: master/slave

[Peter Sturge]

Hi Dino,

I suppose you could write your own ReplicationHandler to do the replication
yourself, but I should think the effort involved would be better spent
deploying the existing Solr http replication or using a Hadoop-based
solution, or UNIX scripting.

By far, the easiest path to replication is to use Solr within a Jetty or
similar container.
I've been down the EmbeddedServer route, and the embedded server is very
good and fast, but if you want replication, your practical choices are unix
scripting or http.

Hope this helps.


On Tue, Mar 9, 2010 at 6:02 PM, Dino Di Cola dinodic...@gmail.com wrote:

 Ok Peter for script-based replication; I forgot to mention I already
 verified that mechanism.

 When I configure the slave as follows

 requestHandler name=/replication class=solr.ReplicationHandler 
lst name=slave
str name=masterUrlhttp://localhost:8983/solr/admin/replication
 /str

str name=pollInterval00:00:20/str
...
/lst
 /requestHandler

 SOLR uses the org.apache.solr.handler.ReplicationHandler to access every
 20s
 the masterUrl via http.
 My question is: is it possible to use another ReplicationHandler that polls
 the master by pointing directly to its SOLR home?
 I mean, is technically feasible?

 Please correct me if I am not clear.
 Thanks.
 --
 2010/3/9 Peter Sturge peter.stu...@googlemail.com

  The SolrEmbededServer doesn't have any http, and so you can't use the
 http
  replication.
  You can use the script-based replication if you're on LUNIX. See:
 http://wiki.apache.org/solr/CollectionDistribution
 
  It would be worth looking at using Solr in a Jetty container and using
 the
  http replication, it is really awesome.
 
 
 
  On Tue, Mar 9, 2010 at 5:27 PM, Dino Di Cola dinodic...@gmail.com
 wrote:
 
   Dear all, I am trying to setup a master/slave index replication
   with two slaves embedded in a tomcat cluster and a master kept in a
   separate
   machine.
   I would like to know if is it possible to configure slaves with a
   ReplicationHandler able to access master
   by starting an embedded server instead of using http communication.
  
   I understand that HTTP is the preferred way to work with solr,
   but for some annoying reasons I cannot startup another http server.
 Thus,
  I
   wonder to know if (and possibly how)
   this approach can be technically 'feasible', already conscious that it
   cannot be definitively 'reasonable'... :)
  
   Many thanks for the support,
   Dino.
   --
  
 

Re: master/slave replication issue

[Noble Paul നോബിള്‍ नोब्ळ्]

The log messages are shown when you hit the admin page. So on't worry
about that. Keep a minimal configuration of Replication. All you need
is  masterUrl and pollInterval.


On Thu, Aug 27, 2009 at 5:52 AM, J Gskinny_joe...@hotmail.com wrote:







 Hello,

 I'm having an issue getting the master to replicate its index to the slave. 
 Below you will find my configuration settings. Here is what is happening: I 
 can access the replication dashboard for both the slave and master and I can 
 successfully execute HTTP commands against both of these urls through my 
 browser. Now, my slave is configured to use the same URL as the one I am 
 using in my browser when I query the master, yet when I do a tail -f tomcat 
 home/logs/catalina.out on the slave server all I see is :


 Master - server1.xyz.com Aug 27, 2009 12:13:29 AM 
 org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=8

 Aug 27, 2009 12:13:32 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=8

 Aug 27, 2009 12:13:34 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=4

 Aug 27, 2009 12:13:36 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=4

 Aug 27, 2009 12:13:39 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=4

 Aug 27, 2009 12:13:42 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=8

 Aug 27, 2009 12:13:44 AM org.apache.solr.core.SolrCore execute

 INFO: [] webapp=null path=null params={command=details} status=0 QTime=


 For some reason, the webapp and the path is being set to null and I think 
 this is affecting the replication?!? I am running Solr as the WAR file and 
 it's 1.4 from a few weeks ago.



 requestHandler name=/replication class=solr.ReplicationHandler 
    lst name=master
        !--Replicate on 'optimize'. Other values can be 'commit', 'startup'. 
 It is possible to have multiple entries of this config string--
        str name=replicateAfteroptimize/str

        !--Create a backup after 'optimize'. Other values can be 'commit', 
 'startup'. It is possible to have multiple entries of this config string 
 .note that this is just for backup. Replication does not require this --
        str name=backupAfteroptimize/str

        !--If configuration files need to be replicated give the names here, 
 separated by comma --
        !--str name=confFilesschema.xml,stopwords.txt,elevate.xml/str--
    /lst
 /requestHandler
 Notice that I commented out the replication of the configuration files. I 
 didn't think this is important for the attempt to try to get replication 
 working. However, is it good to have these files replicated?


 Slave - server2.xyz.com

 requestHandler name=/replication class=solr.ReplicationHandler 
    lst name=slave

        !--fully qualified url for the replication handler of master . It is 
 possible to pass on this as a request param for the fetchindex command--
        str 
 name=masterUrlhttp://server1.xyz.com:8080/jdoe/replication/str

        !--Interval in which the slave should poll master .Format is HH:mm:ss 
 . If this is absent slave does not poll automatically.
         But a fetchindex can be triggered from the admin or the http API --
        str name=pollInterval00:00:20/str
        !-- THE FOLLOWING PARAMETERS ARE USUALLY NOT REQUIRED--
        !--to use compression while transferring the index files. The 
 possible values are internal|external
         if the value is 'external' make sure that your master Solr has the 
 settings to honour the accept-encoding header.
         see here for details http://wiki.apache.org/solr/SolrHttpCompression
         If it is 'internal' everything will be taken care of automatically.
         USE THIS ONLY IF YOUR BANDWIDTH IS LOW . THIS CAN ACTUALLY SLOWDOWN 
 REPLICATION IN A LAN--
        str name=compressioninternal/str
        !--The following values are used when the slave connects to the 
 master to download the index files.
         Default values implicitly set as 5000ms and 1ms respectively. The 
 user DOES NOT need to specify
         these unless the bandwidth is extremely low or if there is an 
 extremely high latency--
        str name=httpConnTimeout5000/str
        str name=httpReadTimeout1/str

        !-- If HTTP Basic authentication is enabled on the master, then the 
 slave can be configured with the following --
        str name=httpBasicAuthUserusername/str
        str name=httpBasicAuthPasswordpassword/str

     /lst
 /requestHandler



 Thanks for your help!




 _
 Hotmail® is up to 70% faster. Now good news travels really fast.
 

Re: Master Slave data distribution | rsync fail issue

[Vicky_Dev]

Yes. Permissions are same across cores

~Vikrant


Bill Au wrote:
 
 Are the user/group/permissions on the snapshot files the same for both
 cases
 (manual vs postCommit/postOptimize events)?
 
 Bill
 
 On Tue, May 5, 2009 at 12:54 PM, tushar kapoor 
 tushar_kapoor...@rediffmail.com wrote:
 

 Hi,

 I am facing an issue while performing snapshot pulling thru Snappuller
 script from slave server :
 We have the setup of multicores on Master Solr and Slave Solr servers.
 Scenario , 2 cores are set :
 i)  CORE_WWW.ABCD.COM
 ii) CORE_WWW.XYZ.COM

 rsync-enable and rsync-start script run from CORE_WWW.ABCD.COM on master
 server. Thus rsyncd.commf file got generated on CORE_WWW.ABCD.COM  only ,
 but not on CORE_WWW.XYZ.COM.
 Rsyncd.conf of CORE_WWW.ABCD.COM :
  rsyncd.conf file 
 uid = webuser
 gid = webuser
 use chroot = no
 list = no
 pid file =
 /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.ABCD.COM/logs/rsyncd.pid
 log file =
 /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.ABCD.COM/logs/rsyncd.log
 [solr]
path =
 /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.ABCD.COM/data
comment = Solr

 rsync error used to get generated while doing the  pulling of master
 server
 snapshot of a particular core CORE_WWW.XYZ.COM from slave end, for core
 CORE_WWW.ABCD.COM snappuller occured without any error.

 Also, this issue is coming only when snapshot are generated at master end
 thru the way given below:
 A)  Snapshot are generated automatically by
 editing  “${SOLR_HOME}/solr/conf/solrconfig.xml” to let either commit
 index
 or optimize index trigger the snapshooter (search “postCommit” and
 “postOptimize” to find the configuration section).

 Sample of solrconfig.xml entry on Master server End:
 I)
 listener event=postCommit class=solr.RunExecutableListener
  str

 name=exe/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.ABCD.COM/bin/snapshooter/str
  str

 name=dir/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.ABCD.COM/bin/str
  bool name=waittrue/bool
  arr name=args strarg1/str strarg2/str /arr
  arr name=env strMYVAR=val1/str /arr
/listener

 same way done for core CORE_WWW.XYZ.COM solrConfig.xml.
 II) The  dataDir tag remains commented on both the cores .XML on master
 server.

 Log sample  for more clearity :
 rsyncd.log of the core CORE_WWW.XYZ.COM:
 2009/05/01 15:48:40 command: ./rsyncd-start
 2009/05/01 15:48:40 [15064] rsyncd version 2.6.3 starting, listening on
 port
 18983
 2009/05/01 15:48:40 rsyncd started with

 data_dir=/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.XYZ.COm/data
 and accepting requests
 2009/05/01 15:50:36 [15195] rsync on solr/snapshot.20090501153311/ from
 deltrialmac.mac1.com (10.210.7.191)
 2009/05/01 15:50:36 [15195] rsync: link_stat snapshot.20090501153311/.
 (in
 solr) failed: No such file or directory (2)
 2009/05/01 15:50:36 [15195] rsync error: some files could not be
 transferred
 (code 23) at main.c(442)
 2009/05/01 15:52:23 [15301] rsync on solr/snapshot.20090501155030/ from
 delpearsondm.sapient.com (10.210.7.191)
 2009/05/01 15:52:23 [15301] wrote 3438 bytes  read 290 bytes  total size
 2779
 2009/05/01 16:03:31 [15553] rsync on solr/snapshot.20090501160112/ from
 deltrialmac.mac1.com (10.210.7.191)
 2009/05/01 16:03:31 [15553] rsync: link_stat snapshot.20090501160112/.
 (in
 solr) failed: No such file or directory (2)
 2009/05/01 16:03:31 [15553] rsync error: some files could not be
 transferred
 (code 23) at main.c(442)
 2009/05/01 16:04:27 [15674] rsync on solr/snapshot.20090501160054/ from
 deltrialmac.mac1.com (10.210.7.191)
 2009/05/01 16:04:27 [15674] wrote 4173214 bytes  read 290 bytes  total
 size
 4174633

 I m unable to figure out that from where /. gets appeneded at the end
 snapshot.20090501153311/.
 Snappuller.log
 2009/05/04 16:55:43 started by solrUser
 2009/05/04 16:55:43 command:
 /opt/apache-solr-1.3.0/example/solr/multicore/
 CORE_WWW.PUFFINBOOKS.CA/bin/snappuller
 -u http://CORE_WWW.PUFFINBOOKS.CA/bin/snappuller%0A-u webuser
 2009/05/04 16:55:52 pulling snapshot snapshot.20090504164935
 2009/05/04 16:56:09 rsync failed
 2009/05/04 16:56:24 failed (elapsed time: 41 sec)

 Error shown on console :
 rsync: link_stat snapshot.20090504164935/. (in solr) failed: No such
 file
 or directory (2)
 client: nothing to do: perhaps you need to specify some filenames or the
 --recursive option?
 rsync error: some files could not be transferred (code 23) at main.c(723)

 B) The same issue is not coming while manually running the Snapshot
 script
 after reguler interval of time at Master server and then running
 Snappuller
 script at slave end for multiple cores. The postCommit/postOptimize part
 of
 solrConfig.xml has been commented.
 Here also rsync script run thru the core CORE_WWW.ABCD.COM. Snappuller
 and
 snapinstaller occurred 

Re: Master Slave data distribution | rsync fail issue

[Bill Au]

You can add the -V option to both your automatic and manual invocation of
snappuller and snapinstaller tor both core and compare the debug info.

Bill

On Thu, Jul 2, 2009 at 11:02 AM, Vicky_Dev
vikrantv_shirbh...@yahoo.co.inwrote:


 Yes. Permissions are same across cores

 ~Vikrant


 Bill Au wrote:
 
  Are the user/group/permissions on the snapshot files the same for both
  cases
  (manual vs postCommit/postOptimize events)?
 
  Bill
 
  On Tue, May 5, 2009 at 12:54 PM, tushar kapoor 
  tushar_kapoor...@rediffmail.com wrote:
 
 
  Hi,
 
  I am facing an issue while performing snapshot pulling thru Snappuller
  script from slave server :
  We have the setup of multicores on Master Solr and Slave Solr servers.
  Scenario , 2 cores are set :
  i)  CORE_WWW.ABCD.COM
  ii) CORE_WWW.XYZ.COM
 
  rsync-enable and rsync-start script run from CORE_WWW.ABCD.COM on
 master
  server. Thus rsyncd.commf file got generated on CORE_WWW.ABCD.COM  only
 ,
  but not on CORE_WWW.XYZ.COM.
  Rsyncd.conf of CORE_WWW.ABCD.COM :
   rsyncd.conf file 
  uid = webuser
  gid = webuser
  use chroot = no
  list = no
  pid file =
  /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.ABCD.COM/logs/rsyncd.pid
  log file =
  /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.ABCD.COM/logs/rsyncd.log
  [solr]
 path =
  /opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.ABCD.COM/data
 comment = Solr
 
  rsync error used to get generated while doing the  pulling of master
  server
  snapshot of a particular core CORE_WWW.XYZ.COM from slave end, for
 core
  CORE_WWW.ABCD.COM snappuller occured without any error.
 
  Also, this issue is coming only when snapshot are generated at master
 end
  thru the way given below:
  A)  Snapshot are generated automatically by
  editing  “${SOLR_HOME}/solr/conf/solrconfig.xml” to let either commit
  index
  or optimize index trigger the snapshooter (search “postCommit” and
  “postOptimize” to find the configuration section).
 
  Sample of solrconfig.xml entry on Master server End:
  I)
  listener event=postCommit class=solr.RunExecutableListener
   str
 
 
 name=exe/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.ABCD.COM/bin/snapshooter/str
   str
 
 
 name=dir/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.ABCD.COM/bin/str
   bool name=waittrue/bool
   arr name=args strarg1/str strarg2/str /arr
   arr name=env strMYVAR=val1/str /arr
 /listener
 
  same way done for core CORE_WWW.XYZ.COM solrConfig.xml.
  II) The  dataDir tag remains commented on both the cores .XML on
 master
  server.
 
  Log sample  for more clearity :
  rsyncd.log of the core CORE_WWW.XYZ.COM:
  2009/05/01 15:48:40 command: ./rsyncd-start
  2009/05/01 15:48:40 [15064] rsyncd version 2.6.3 starting, listening on
  port
  18983
  2009/05/01 15:48:40 rsyncd started with
 
 
 data_dir=/opt/apache-tomcat-6.0.18/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.XYZ.COm/data
  and accepting requests
  2009/05/01 15:50:36 [15195] rsync on solr/snapshot.20090501153311/ from
  deltrialmac.mac1.com (10.210.7.191)
  2009/05/01 15:50:36 [15195] rsync: link_stat snapshot.20090501153311/.
  (in
  solr) failed: No such file or directory (2)
  2009/05/01 15:50:36 [15195] rsync error: some files could not be
  transferred
  (code 23) at main.c(442)
  2009/05/01 15:52:23 [15301] rsync on solr/snapshot.20090501155030/ from
  delpearsondm.sapient.com (10.210.7.191)
  2009/05/01 15:52:23 [15301] wrote 3438 bytes  read 290 bytes  total size
  2779
  2009/05/01 16:03:31 [15553] rsync on solr/snapshot.20090501160112/ from
  deltrialmac.mac1.com (10.210.7.191)
  2009/05/01 16:03:31 [15553] rsync: link_stat snapshot.20090501160112/.
  (in
  solr) failed: No such file or directory (2)
  2009/05/01 16:03:31 [15553] rsync error: some files could not be
  transferred
  (code 23) at main.c(442)
  2009/05/01 16:04:27 [15674] rsync on solr/snapshot.20090501160054/ from
  deltrialmac.mac1.com (10.210.7.191)
  2009/05/01 16:04:27 [15674] wrote 4173214 bytes  read 290 bytes  total
  size
  4174633
 
  I m unable to figure out that from where /. gets appeneded at the end
  snapshot.20090501153311/.
  Snappuller.log
  2009/05/04 16:55:43 started by solrUser
  2009/05/04 16:55:43 command:
  /opt/apache-solr-1.3.0/example/solr/multicore/
  CORE_WWW.PUFFINBOOKS.CA/bin/snappuller
  -u http://CORE_WWW.PUFFINBOOKS.CA/bin/snappuller%0A-u webuser
  2009/05/04 16:55:52 pulling snapshot snapshot.20090504164935
  2009/05/04 16:56:09 rsync failed
  2009/05/04 16:56:24 failed (elapsed time: 41 sec)
 
  Error shown on console :
  rsync: link_stat snapshot.20090504164935/. (in solr) failed: No such
  file
  or directory (2)
  client: nothing to do: perhaps you need to specify some filenames or the
  --recursive option?
  rsync error: some files could not be transferred (code 23) at
 main.c(723)
 

Re: master/slave failure scenario

[nk 11]

Just curious. What would be the disadvantages of a no replication / multi
master (no slave) setup?
The client code should do the updates for evey master ofc, but if one
machine would fail then I can imediatly continue the indexing process and
also I can query the index on any machine for a valid result.
I might be missing something...
On Thu, May 14, 2009 at 4:19 PM, nk 11 nick.cass...@gmail.com wrote:

 wow! that was just a couple of days old!
 thanks as lot!
   2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 yeah there is a hack

 https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12708316

 On Thu, May 14, 2009 at 6:07 PM, nk 11 nick.cass...@gmail.com wrote:
  sorry for the mail. I wanted to hit reply :(
 
  On Thu, May 14, 2009 at 3:37 PM, nk 11 nick.cass...@gmail.com wrote:
 
  oh, so the configuration must be manualy changed?
  Can't something be passed at (re)start time?
 
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
 
  On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com
 wrote:
   Ok so the VIP will point to the new master. but what makes a slave
   promoted
   to a master? Only the fact that it will receive add/update requests?
   And I suppose that this hot promotion is possible only if the
 slave
   is
   convigured as master also...
  right.. By default you can setup all slaves to be master also. It does
  not cost anything if it is not serving any requests.
 
  so , if you have such a setting you will have to disable that slave to
  be a slave and restart it and you will have to make the VIP point to
  this new slave as master.
 
  so hot promotion is still not possible.
  
   2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
  
   ideally , we don't do that.
   you can just keep the master host behind a VIP so if you wish to
   change the master make the VIP point to the new host
  
   On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com
   wrote:
This is more interesting.Such a procedure would involve taking
 down
and
reconfiguring the slave?
   
On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
btal...@aeriagames.comwrote:
   
Or ...
   
1. Promote existing slave to new master
2. Add new slave to cluster
   
   
   
   
-Bryan
   
   
   
   
   
On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
   
 - Migrate configuration files from old master (or backup) to
 new
master.
- Replicate from a slave to the new master.
- Resume indexing to new master.
   
-Jay
   
On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com
 
wrote:
   
 Nice.
What if the master fails permanently (like a disk crash...)
 and
the
new
master is a clean machine?
2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
   
 On Wed, May 13, 2009 at 12:10 PM, nk 11 
 nick.cass...@gmail.com
wrote:
   
Hello
   
I'm kind of new to Solr and I've read about replication, and
the
fact
   
that a
   
node can act as both master and slave.
I a replica fails and then comes back on line I suppose that
 it
will
   
resyncs
   
with the master.
   
right
   
   
But what happnes if the master fails? A slave that is
configured as
   
master
   
will kick in? What if that slave is not yes fully sync'ed
 with
the
   
failed
   
master and has old data?
   
if the master fails you can't index the data. but the slaves
will
continue serving the requests with the last index. You an
 bring
back
the master up and resume indexing.
   
   
What happens when the original master comes back on line? He
will
   
remain
   
a
   
slave because there is another node with the master role?
   
Thank you!
   
   
   
   
--
-
Noble Paul | Principal Engineer| AOL | http://aol.com
   
   
   
   
   
  
  
  
   --
   -
   Noble Paul | Principal Engineer| AOL | http://aol.com
  
  
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
 
 



 --
  -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[Bryan Talbot]

Indexing is usually much more expensive that replication so it won't  
scale well as you add more servers.  Also, what would a client do if  
it was able to send the update to only some of the servers because  
others were down (for maintenance, etc)?




-Bryan




On May 21, 2009, at May 21, 6:04 AM, nk 11 wrote:

Just curious. What would be the disadvantages of a no replication /  
multi

master (no slave) setup?
The client code should do the updates for evey master ofc, but if one
machine would fail then I can imediatly continue the indexing  
process and

also I can query the index on any machine for a valid result.
I might be missing something...
On Thu, May 14, 2009 at 4:19 PM, nk 11 nick.cass...@gmail.com wrote:


wow! that was just a couple of days old!
thanks as lot!
 2009/5/14 Noble Paul നോബിള്‍ नोब्ळ्  
noble.p...@corp.aol.com



yeah there is a hack

https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel 
#action_12708316


On Thu, May 14, 2009 at 6:07 PM, nk 11 nick.cass...@gmail.com  
wrote:

sorry for the mail. I wanted to hit reply :(

On Thu, May 14, 2009 at 3:37 PM, nk 11 nick.cass...@gmail.com  
wrote:


oh, so the configuration must be manualy changed?
Can't something be passed at (re)start time?

2009/5/14 Noble Paul നോബിള്‍ नोब्ळ्  
noble.p...@corp.aol.com


On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com

wrote:
Ok so the VIP will point to the new master. but what makes a  
slave

promoted
to a master? Only the fact that it will receive add/update  
requests?

And I suppose that this hot promotion is possible only if the

slave

is
convigured as master also...
right.. By default you can setup all slaves to be master also.  
It does

not cost anything if it is not serving any requests.

so , if you have such a setting you will have to disable that  
slave to
be a slave and restart it and you will have to make the VIP  
point to

this new slave as master.

so hot promotion is still not possible.


2009/5/14 Noble Paul നോബിള്‍ नोब्ळ्  
noble.p...@corp.aol.com


ideally , we don't do that.
you can just keep the master host behind a VIP so if you wish  
to

change the master make the VIP point to the new host

On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass. 
1...@gmail.com

wrote:

This is more interesting.Such a procedure would involve taking

down

and
reconfiguring the slave?

On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
btal...@aeriagames.comwrote:


Or ...

1. Promote existing slave to new master
2. Add new slave to cluster




-Bryan





On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:

- Migrate configuration files from old master (or backup) to

new

master.

- Replicate from a slave to the new master.
- Resume indexing to new master.

-Jay

On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com



wrote:

Nice.

What if the master fails permanently (like a disk crash...)

and

the
new
master is a clean machine?
2009/5/13 Noble Paul നോബിള്‍ नो 
ब्ळ् noble.p...@corp.aol.com


On Wed, May 13, 2009 at 12:10 PM, nk 11 

nick.cass...@gmail.com

wrote:



Hello

I'm kind of new to Solr and I've read about  
replication, and

the
fact


that a


node can act as both master and slave.
I a replica fails and then comes back on line I suppose  
that

it

will


resyncs


with the master.


right



But what happnes if the master fails? A slave that is
configured as


master


will kick in? What if that slave is not yes fully sync'ed

with

the


failed



master and has old data?


if the master fails you can't index the data. but the  
slaves

will
continue serving the requests with the last index. You an

bring

back
the master up and resume indexing.


What happens when the original master comes back on  
line? He

will


remain



a


slave because there is another node with the master role?

Thank you!





--
-
Noble Paul | Principal Engineer| AOL | http://aol.com












--
-
Noble Paul | Principal Engineer| AOL | http://aol.com







--
-
Noble Paul | Principal Engineer| AOL | http://aol.com









--
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[nk 11]

You are right... I just don't like the idea of stopping the indexing process
if the master fails until a new one is started (more or less by hand).

On Thu, May 21, 2009 at 6:49 PM, Bryan Talbot btal...@aeriagames.comwrote:

 Indexing is usually much more expensive that replication so it won't scale
 well as you add more servers.  Also, what would a client do if it was able
 to send the update to only some of the servers because others were down (for
 maintenance, etc)?



 -Bryan





 On May 21, 2009, at May 21, 6:04 AM, nk 11 wrote:

  Just curious. What would be the disadvantages of a no replication / multi
 master (no slave) setup?
 The client code should do the updates for evey master ofc, but if one
 machine would fail then I can imediatly continue the indexing process and
 also I can query the index on any machine for a valid result.
 I might be missing something...
 On Thu, May 14, 2009 at 4:19 PM, nk 11 nick.cass...@gmail.com wrote:

  wow! that was just a couple of days old!
 thanks as lot!
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

  yeah there is a hack


 https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
 #action_12708316

 On Thu, May 14, 2009 at 6:07 PM, nk 11 nick.cass...@gmail.com wrote:

 sorry for the mail. I wanted to hit reply :(

 On Thu, May 14, 2009 at 3:37 PM, nk 11 nick.cass...@gmail.com wrote:


 oh, so the configuration must be manualy changed?
 Can't something be passed at (re)start time?

 2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com


 On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com

 wrote:

 Ok so the VIP will point to the new master. but what makes a slave
 promoted
 to a master? Only the fact that it will receive add/update requests?
 And I suppose that this hot promotion is possible only if the

 slave

 is
 convigured as master also...

 right.. By default you can setup all slaves to be master also. It
 does
 not cost anything if it is not serving any requests.

 so , if you have such a setting you will have to disable that slave
 to
 be a slave and restart it and you will have to make the VIP point to
 this new slave as master.

 so hot promotion is still not possible.


 2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com


 ideally , we don't do that.
 you can just keep the master host behind a VIP so if you wish to
 change the master make the VIP point to the new host

 On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com
 wrote:

 This is more interesting.Such a procedure would involve taking

 down

 and
 reconfiguring the slave?

 On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
 btal...@aeriagames.comwrote:

  Or ...

 1. Promote existing slave to new master
 2. Add new slave to cluster




 -Bryan





 On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:

 - Migrate configuration files from old master (or backup) to

 new

 master.

 - Replicate from a slave to the new master.
 - Resume indexing to new master.

 -Jay

 On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com


  wrote:

 Nice.

 What if the master fails permanently (like a disk crash...)

 and

 the
 new
 master is a clean machine?
 2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 On Wed, May 13, 2009 at 12:10 PM, nk 11 

 nick.cass...@gmail.com

 wrote:


  Hello

 I'm kind of new to Solr and I've read about replication, and
 the
 fact

  that a

  node can act as both master and slave.
 I a replica fails and then comes back on line I suppose that

 it

 will

  resyncs

  with the master.

  right


 But what happnes if the master fails? A slave that is
 configured as

  master

  will kick in? What if that slave is not yes fully sync'ed

 with

 the

  failed


  master and has old data?


  if the master fails you can't index the data. but the slaves
 will
 continue serving the requests with the last index. You an

 bring

 back
 the master up and resume indexing.


  What happens when the original master comes back on line? He
 will

  remain


  a

  slave because there is another node with the master role?

 Thank you!




 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com








 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com






 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com







 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[Otis Gospodnetic]

Hi,

You should be able to do the following.
Put masters behind a load balancer (LB).
Create a LB VIP and a pool with 2 masters, masterA  masterB with a rule that 
all requests always go to A unless A is down.  If If A is down they go to B.
Bring up master instances A and B on 2 servers and make them point to the 
shared storage.

masterA \
   \-- shared storage
   /
masterB /

Your indexing client doesn't talk to the servers directly. It talks through the 
VIP you created in LB.
At any one time only one of the masters is active.
If A goes down, LB detects it and makes B active.
Your indexer may have to reconnect if it detects a failure, maybe it would need 
to reindex some number of documents if they didn't make it to disk before A 
died, maybe even some lock file cleanup might be needed, but the above should 
be doable with little effort.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



- Original Message 
 From: nk 11 nick.cass...@gmail.com
 To: solr-user@lucene.apache.org
 Sent: Thursday, May 21, 2009 12:44:55 PM
 Subject: Re: master/slave failure scenario
 
 You are right... I just don't like the idea of stopping the indexing process
 if the master fails until a new one is started (more or less by hand).
 
 On Thu, May 21, 2009 at 6:49 PM, Bryan Talbot wrote:
 
  Indexing is usually much more expensive that replication so it won't scale
  well as you add more servers.  Also, what would a client do if it was able
  to send the update to only some of the servers because others were down (for
  maintenance, etc)?
 
 
 
  -Bryan
 
 
 
 
 
  On May 21, 2009, at May 21, 6:04 AM, nk 11 wrote:
 
   Just curious. What would be the disadvantages of a no replication / multi
  master (no slave) setup?
  The client code should do the updates for evey master ofc, but if one
  machine would fail then I can imediatly continue the indexing process and
  also I can query the index on any machine for a valid result.
  I might be missing something...
  On Thu, May 14, 2009 at 4:19 PM, nk 11 wrote:
 
   wow! that was just a couple of days old!
  thanks as lot!
   2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् 
 
   yeah there is a hack
 
 
  
 https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
  #action_12708316
 
  On Thu, May 14, 2009 at 6:07 PM, nk 11 wrote:
 
  sorry for the mail. I wanted to hit reply :(
 
  On Thu, May 14, 2009 at 3:37 PM, nk 11 wrote:
 
 
  oh, so the configuration must be manualy changed?
  Can't something be passed at (re)start time?
 
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् 
 
 
  On Thu, May 14, 2009 at 4:07 PM, nk 11 
 
  wrote:
 
  Ok so the VIP will point to the new master. but what makes a slave
  promoted
  to a master? Only the fact that it will receive add/update requests?
  And I suppose that this hot promotion is possible only if the
 
  slave
 
  is
  convigured as master also...
 
  right.. By default you can setup all slaves to be master also. It
  does
  not cost anything if it is not serving any requests.
 
  so , if you have such a setting you will have to disable that slave
  to
  be a slave and restart it and you will have to make the VIP point to
  this new slave as master.
 
  so hot promotion is still not possible.
 
 
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् 
 
 
  ideally , we don't do that.
  you can just keep the master host behind a VIP so if you wish to
  change the master make the VIP point to the new host
 
  On Wed, May 13, 2009 at 10:52 PM, nk 11 
  wrote:
 
  This is more interesting.Such a procedure would involve taking
 
  down
 
  and
  reconfiguring the slave?
 
  On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
  wrote:
 
   Or ...
 
  1. Promote existing slave to new master
  2. Add new slave to cluster
 
 
 
 
  -Bryan
 
 
 
 
 
  On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
 
  - Migrate configuration files from old master (or backup) to
 
  new
 
  master.
 
  - Replicate from a slave to the new master.
  - Resume indexing to new master.
 
  -Jay
 
  On Wed, May 13, 2009 at 4:26 AM, nk 11 
 
 
   wrote:
 
  Nice.
 
  What if the master fails permanently (like a disk crash...)
 
  and
 
  the
  new
  master is a clean machine?
  2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् 
 
  On Wed, May 13, 2009 at 12:10 PM, nk 11 
 
  nick.cass...@gmail.com
 
  wrote:
 
 
   Hello
 
  I'm kind of new to Solr and I've read about replication, and
  the
  fact
 
   that a
 
   node can act as both master and slave.
  I a replica fails and then comes back on line I suppose that
 
  it
 
  will
 
   resyncs
 
   with the master.
 
   right
 
 
  But what happnes if the master fails? A slave that is
  configured as
 
   master
 
   will kick in? What if that slave is not yes fully sync'ed
 
  with
 
  the
 
   failed
 
 
   master and has old data?
 
 
   if the master fails you can't index the data. but the slaves
  will
  continue

Re: master/slave failure scenario

[nk 11]

Ok so the VIP will point to the new master. but what makes a slave promoted
to a master? Only the fact that it will receive add/update requests?
And I suppose that this hot promotion is possible only if the slave is
convigured as master also...

2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 ideally , we don't do that.
 you can just keep the master host behind a VIP so if you wish to
 change the master make the VIP point to the new host

 On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com wrote:
  This is more interesting.Such a procedure would involve taking down and
  reconfiguring the slave?
 
  On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot btal...@aeriagames.com
 wrote:
 
  Or ...
 
  1. Promote existing slave to new master
  2. Add new slave to cluster
 
 
 
 
  -Bryan
 
 
 
 
 
  On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
 
   - Migrate configuration files from old master (or backup) to new
 master.
  - Replicate from a slave to the new master.
  - Resume indexing to new master.
 
  -Jay
 
  On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com wrote:
 
   Nice.
  What if the master fails permanently (like a disk crash...) and the
 new
  master is a clean machine?
  2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
 
   On Wed, May 13, 2009 at 12:10 PM, nk 11 nick.cass...@gmail.com
 wrote:
 
  Hello
 
  I'm kind of new to Solr and I've read about replication, and the
 fact
 
  that a
 
  node can act as both master and slave.
  I a replica fails and then comes back on line I suppose that it will
 
  resyncs
 
  with the master.
 
  right
 
 
  But what happnes if the master fails? A slave that is configured as
 
  master
 
  will kick in? What if that slave is not yes fully sync'ed with the
 
  failed
 
  master and has old data?
 
  if the master fails you can't index the data. but the slaves will
  continue serving the requests with the last index. You an bring back
  the master up and resume indexing.
 
 
  What happens when the original master comes back on line? He will
 
  remain
 
  a
 
  slave because there is another node with the master role?
 
  Thank you!
 
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
 
 
 
 



 --
  -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[nk 11]

oh, so the configuration must be manualy changed?
Can't something be passed at (re)start time?


   2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com wrote:
  Ok so the VIP will point to the new master. but what makes a slave
 promoted
  to a master? Only the fact that it will receive add/update requests?
  And I suppose that this hot promotion is possible only if the slave is
  convigured as master also...
 right.. By default you can setup all slaves to be master also. It does
 not cost anything if it is not serving any requests.

 so , if you have such a setting you will have to disable that slave to
 be a slave and restart it and you will have to make the VIP point to
 this new slave as master.

 so hot promotion is still not possible.
  
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
 
  ideally , we don't do that.
  you can just keep the master host behind a VIP so if you wish to
  change the master make the VIP point to the new host
 
  On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com
 wrote:
   This is more interesting.Such a procedure would involve taking down
 and
   reconfiguring the slave?
  
   On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
   btal...@aeriagames.comwrote:
  
   Or ...
  
   1. Promote existing slave to new master
   2. Add new slave to cluster
  
  
  
  
   -Bryan
  
  
  
  
  
   On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
  
- Migrate configuration files from old master (or backup) to new
   master.
   - Replicate from a slave to the new master.
   - Resume indexing to new master.
  
   -Jay
  
   On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com
 wrote:
  
Nice.
   What if the master fails permanently (like a disk crash...) and
 the
   new
   master is a clean machine?
   2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
  
On Wed, May 13, 2009 at 12:10 PM, nk 11 nick.cass...@gmail.com
   wrote:
  
   Hello
  
   I'm kind of new to Solr and I've read about replication, and the
   fact
  
   that a
  
   node can act as both master and slave.
   I a replica fails and then comes back on line I suppose that it
   will
  
   resyncs
  
   with the master.
  
   right
  
  
   But what happnes if the master fails? A slave that is configured
 as
  
   master
  
   will kick in? What if that slave is not yes fully sync'ed with
 the
  
   failed
  
   master and has old data?
  
   if the master fails you can't index the data. but the slaves will
   continue serving the requests with the last index. You an bring
 back
   the master up and resume indexing.
  
  
   What happens when the original master comes back on line? He
 will
  
   remain
  
   a
  
   slave because there is another node with the master role?
  
   Thank you!
  
  
  
  
   --
   -
   Noble Paul | Principal Engineer| AOL | http://aol.com
  
  
  
  
  
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
 



 --
  -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[Noble Paul നോബിള്‍ नोब्ळ्]

yeah there is a hack
https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12708316

On Thu, May 14, 2009 at 6:07 PM, nk 11 nick.cass...@gmail.com wrote:
 sorry for the mail. I wanted to hit reply :(

 On Thu, May 14, 2009 at 3:37 PM, nk 11 nick.cass...@gmail.com wrote:

 oh, so the configuration must be manualy changed?
 Can't something be passed at (re)start time?

 2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com wrote:
  Ok so the VIP will point to the new master. but what makes a slave
  promoted
  to a master? Only the fact that it will receive add/update requests?
  And I suppose that this hot promotion is possible only if the slave
  is
  convigured as master also...
 right.. By default you can setup all slaves to be master also. It does
 not cost anything if it is not serving any requests.

 so , if you have such a setting you will have to disable that slave to
 be a slave and restart it and you will have to make the VIP point to
 this new slave as master.

 so hot promotion is still not possible.
 
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
 
  ideally , we don't do that.
  you can just keep the master host behind a VIP so if you wish to
  change the master make the VIP point to the new host
 
  On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com
  wrote:
   This is more interesting.Such a procedure would involve taking down
   and
   reconfiguring the slave?
  
   On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
   btal...@aeriagames.comwrote:
  
   Or ...
  
   1. Promote existing slave to new master
   2. Add new slave to cluster
  
  
  
  
   -Bryan
  
  
  
  
  
   On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
  
    - Migrate configuration files from old master (or backup) to new
   master.
   - Replicate from a slave to the new master.
   - Resume indexing to new master.
  
   -Jay
  
   On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com
   wrote:
  
    Nice.
   What if the master fails permanently (like a disk crash...) and
   the
   new
   master is a clean machine?
   2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
  
    On Wed, May 13, 2009 at 12:10 PM, nk 11 nick.cass...@gmail.com
   wrote:
  
   Hello
  
   I'm kind of new to Solr and I've read about replication, and
   the
   fact
  
   that a
  
   node can act as both master and slave.
   I a replica fails and then comes back on line I suppose that it
   will
  
   resyncs
  
   with the master.
  
   right
  
  
   But what happnes if the master fails? A slave that is
   configured as
  
   master
  
   will kick in? What if that slave is not yes fully sync'ed with
   the
  
   failed
  
   master and has old data?
  
   if the master fails you can't index the data. but the slaves
   will
   continue serving the requests with the last index. You an bring
   back
   the master up and resume indexing.
  
  
   What happens when the original master comes back on line? He
   will
  
   remain
  
   a
  
   slave because there is another node with the master role?
  
   Thank you!
  
  
  
  
   --
   -
   Noble Paul | Principal Engineer| AOL | http://aol.com
  
  
  
  
  
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
 



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com






-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[nk 11]

wow! that was just a couple of days old!
thanks as lot!
2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 yeah there is a hack

 https://issues.apache.org/jira/browse/SOLR-1154?focusedCommentId=12708316page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12708316

 On Thu, May 14, 2009 at 6:07 PM, nk 11 nick.cass...@gmail.com wrote:
  sorry for the mail. I wanted to hit reply :(
 
  On Thu, May 14, 2009 at 3:37 PM, nk 11 nick.cass...@gmail.com wrote:
 
  oh, so the configuration must be manualy changed?
  Can't something be passed at (re)start time?
 
  2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
 
  On Thu, May 14, 2009 at 4:07 PM, nk 11 nick.cass...@gmail.com wrote:
   Ok so the VIP will point to the new master. but what makes a slave
   promoted
   to a master? Only the fact that it will receive add/update requests?
   And I suppose that this hot promotion is possible only if the slave
   is
   convigured as master also...
  right.. By default you can setup all slaves to be master also. It does
  not cost anything if it is not serving any requests.
 
  so , if you have such a setting you will have to disable that slave to
  be a slave and restart it and you will have to make the VIP point to
  this new slave as master.
 
  so hot promotion is still not possible.
  
   2009/5/14 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
  
   ideally , we don't do that.
   you can just keep the master host behind a VIP so if you wish to
   change the master make the VIP point to the new host
  
   On Wed, May 13, 2009 at 10:52 PM, nk 11 nick.cass...@gmail.com
   wrote:
This is more interesting.Such a procedure would involve taking
 down
and
reconfiguring the slave?
   
On Wed, May 13, 2009 at 7:55 PM, Bryan Talbot
btal...@aeriagames.comwrote:
   
Or ...
   
1. Promote existing slave to new master
2. Add new slave to cluster
   
   
   
   
-Bryan
   
   
   
   
   
On May 13, 2009, at May 13, 9:48 AM, Jay Hill wrote:
   
 - Migrate configuration files from old master (or backup) to new
master.
- Replicate from a slave to the new master.
- Resume indexing to new master.
   
-Jay
   
On Wed, May 13, 2009 at 4:26 AM, nk 11 nick.cass...@gmail.com
wrote:
   
 Nice.
What if the master fails permanently (like a disk crash...) and
the
new
master is a clean machine?
2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com
   
 On Wed, May 13, 2009 at 12:10 PM, nk 11 
 nick.cass...@gmail.com
wrote:
   
Hello
   
I'm kind of new to Solr and I've read about replication, and
the
fact
   
that a
   
node can act as both master and slave.
I a replica fails and then comes back on line I suppose that
 it
will
   
resyncs
   
with the master.
   
right
   
   
But what happnes if the master fails? A slave that is
configured as
   
master
   
will kick in? What if that slave is not yes fully sync'ed
 with
the
   
failed
   
master and has old data?
   
if the master fails you can't index the data. but the slaves
will
continue serving the requests with the last index. You an
 bring
back
the master up and resume indexing.
   
   
What happens when the original master comes back on line? He
will
   
remain
   
a
   
slave because there is another node with the master role?
   
Thank you!
   
   
   
   
--
-
Noble Paul | Principal Engineer| AOL | http://aol.com
   
   
   
   
   
  
  
  
   --
   -
   Noble Paul | Principal Engineer| AOL | http://aol.com
  
  
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
 
 



 --
  -
 Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[Noble Paul നോബിള്‍ नोब्ळ्]

On Wed, May 13, 2009 at 12:10 PM, nk 11 nick.cass...@gmail.com wrote:
 Hello

 I'm kind of new to Solr and I've read about replication, and the fact that a
 node can act as both master and slave.
 I a replica fails and then comes back on line I suppose that it will resyncs
 with the master.
right

 But what happnes if the master fails? A slave that is configured as master
 will kick in? What if that slave is not yes fully sync'ed with the failed
 master and has old data?
if the master fails you can't index the data. but the slaves will
continue serving the requests with the last index. You an bring back
the master up and resume indexing.


 What happens when the original master comes back on line? He will remain a
 slave because there is another node with the master role?

 Thank you!




-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: master/slave failure scenario

[nk 11]

Nice.
What if the master fails permanently (like a disk crash...) and the new
master is a clean machine?
2009/5/13 Noble Paul നോബിള്‍ नोब्ळ् noble.p...@corp.aol.com

 On Wed, May 13, 2009 at 12:10 PM, nk 11 nick.cass...@gmail.com wrote:
  Hello
 
  I'm kind of new to Solr and I've read about replication, and the fact
 that a
  node can act as both master and slave.
  I a replica fails and then comes back on line I suppose that it will
 resyncs
  with the master.
 right
 
  But what happnes if the master fails? A slave that is configured as
 master
  will kick in? What if that slave is not yes fully sync'ed with the failed
  master and has old data?
 if the master fails you can't index the data. but the slaves will
 continue serving the requests with the last index. You an bring back
 the master up and resume indexing.

 
  What happens when the original master comes back on line? He will remain
 a
  slave because there is another node with the master role?
 
  Thank you!
 



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com