RE: Solr Replication

[Vadim Ivanov]

Using cdcr with new replica types be aware of 
https://issues.apache.org/jira/browse/SOLR-12057?focusedComm

Parallel indexing to both cluster might be an option as well
-- 
Vadim


> -Original Message-
> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de]
> Sent: Monday, January 07, 2019 11:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication
> 
> In SolrCloud there are Data Centers.
> Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2.
> You can then use CDCR (Cross Data Center Replication).
> http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication-
> cdcr.html
> 
> Nevertheless I would spend your Cluster 2 another 2 zookeeper instances.
> 
> Regards, Bernd
> 
> Am 07.01.19 um 06:39 schrieb Mannar mannan:
> > Hi All,
> >
> > I would like to configure master slave between two solr cloud clusters (for
> > failover). Below is the scenario
> >
> > Solr version : 7.0
> >
> > Cluster 1:
> > 3 zookeeper instances :   zk1, zk2, zk3
> > 2 solr instances : solr1, solr2
> >
> > Cluster 2:
> > 1 zookeeper instance : bkpzk1,
> > 1 solr instances : bkpsolr1, bkpsolr2
> >
> > Master / Slave :  solr1 / bkpsolr1
> >solr2 / bkpsolr2
> >
> > Is it possible to have master / slave replication configured for solr
> > instances running in cluster1 & cluster2 (for failover). Kindly let me know
> > the possibility.
> >

Re: Solr Replication

[Bernd Fehling]

In SolrCloud there are Data Centers.
Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2.
You can then use CDCR (Cross Data Center Replication).
http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication-cdcr.html

Nevertheless I would spend your Cluster 2 another 2 zookeeper instances.

Regards, Bernd

Am 07.01.19 um 06:39 schrieb Mannar mannan:

Hi All,

I would like to configure master slave between two solr cloud clusters (for
failover). Below is the scenario

Solr version : 7.0

Cluster 1:
3 zookeeper instances :   zk1, zk2, zk3
2 solr instances : solr1, solr2

Cluster 2:
1 zookeeper instance : bkpzk1,
1 solr instances : bkpsolr1, bkpsolr2

Master / Slave :  solr1 / bkpsolr1
   solr2 / bkpsolr2

Is it possible to have master / slave replication configured for solr
instances running in cluster1 & cluster2 (for failover). Kindly let me know
the possibility.

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

bq: That's evidence enough for me to beat on our systems guys

Beat them with a stick. A large stick. If they insist that they can't
up these limits, I always push back hard with "why?". Usually
reluctance to up them is a misplaced efficiency argument

And I should have said you should see exceptions in your Solr logs if
this is really something that happens.

FWIW,
Erick

On Fri, Jan 19, 2018 at 11:07 AM, Pouliot, Scott
<scott.poul...@peoplefluent.com> wrote:
> Working on that now to see if it helps us out.  Solr process is NOT dying at 
> all.  Searches are still working as expected, but since we load balance 
> requestsif the master/slave are out of sync the search results vary.
>
> The advice is MUCH appreciated!
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 1:49 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 11:27 AM, Shawn Heisey wrote:
>> On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
>>> I do have a ticket in with our systems team to up the file handlers
>>> since I am seeing the "Too many files open" error on occasion on our
>>> prod servers.  Is this the setting you're referring to?  Found we
>>> were set to to 1024 using the "Ulimit" command.
>>
>> No, but that often needs increasing too.  I think you need to increase
>> the process limit even if that's not the cause of this particular problem.
>
> Had another thought.  Either of these limits can cause completely 
> unpredictable problems with Solr.  The open file limit could be the reason 
> for these issues, even if you're not actually hitting the process limit.  As 
> I mentioned before, I would expect a process limit to cause Solr to kill 
> itself, and your other messages don't mention problems like that.
>
> The scale of your Solr installation indicates that you should greatly 
> increase both limits on all of your Solr servers.
>
> Thanks,
> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

Working on that now to see if it helps us out.  Solr process is NOT dying at 
all.  Searches are still working as expected, but since we load balance 
requestsif the master/slave are out of sync the search results vary.  

The advice is MUCH appreciated!

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Friday, January 19, 2018 1:49 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

On 1/19/2018 11:27 AM, Shawn Heisey wrote:
> On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
>> I do have a ticket in with our systems team to up the file handlers 
>> since I am seeing the "Too many files open" error on occasion on our 
>> prod servers.  Is this the setting you're referring to?  Found we 
>> were set to to 1024 using the "Ulimit" command.
> 
> No, but that often needs increasing too.  I think you need to increase 
> the process limit even if that's not the cause of this particular problem.

Had another thought.  Either of these limits can cause completely unpredictable 
problems with Solr.  The open file limit could be the reason for these issues, 
even if you're not actually hitting the process limit.  As I mentioned before, 
I would expect a process limit to cause Solr to kill itself, and your other 
messages don't mention problems like that.

The scale of your Solr installation indicates that you should greatly increase 
both limits on all of your Solr servers.

Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 11:27 AM, Shawn Heisey wrote:

On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
I do have a ticket in with our systems team to up the file handlers 
since I am seeing the "Too many files open" error on occasion on our 
prod servers.  Is this the setting you're referring to?  Found we were 
set to to 1024 using the "Ulimit" command.


No, but that often needs increasing too.  I think you need to increase 
the process limit even if that's not the cause of this particular problem.


Had another thought.  Either of these limits can cause completely 
unpredictable problems with Solr.  The open file limit could be the 
reason for these issues, even if you're not actually hitting the process 
limit.  As I mentioned before, I would expect a process limit to cause 
Solr to kill itself, and your other messages don't mention problems like 
that.


The scale of your Solr installation indicates that you should greatly 
increase both limits on all of your Solr servers.


Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 8:54 AM, Pouliot, Scott wrote:

I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too 
many files open" error on occasion on our prod servers.  Is this the setting you're referring 
to?  Found we were set to to 1024 using the "Ulimit" command.


No, but that often needs increasing too.  I think you need to increase 
the process limit even if that's not the cause of this particular problem.


Sounds like you're running on Linux, though ulimit is probably available 
on other platforms too.


If it's Linux, generally you must increase both the number of processes 
and the open file limit in /etc/security/limits.conf.  Trying to use the 
ulimit command generally doesn't work because the kernel has hard limits 
configured that ulimit can't budge.  If it's not Linux, then you'll need 
to consult with an expert in the OS you're running.


Again, assuming Linux, in the output of "ulimit -a" the value I'm 
talking about is the "-u" value -- "max user processes".  The following 
is the additions that I typically make to /etc/security/limits.conf, to 
increase both the open file limit and the process limit for the solr user:


solr    hard    nproc   61440
solr    soft    nproc   40960

solr    hard    nofile  65535
solr    soft    nofile  49151

Are you running into problems where Solr just disappears?  I would 
expect a process limit to generate OutOfMemoryError exceptions. When 
Solr is started with the included shell script, unless it's running with 
the foreground option, OOME will kill the Solr process.  We have issues 
to bring the OOME death option to running in the foreground, as well as 
when running on Windows.


Thanks,
Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

That's evidence enough for me to beat on our systems guys to get these file 
handles upped and cross my fingers then!

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, January 19, 2018 1:18 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Solr Replication being flaky (6.2.0)

"Could be", certainly. "Definitely is" is iffier ;)...

But the statement "If we restart the Solr service or optimize the core it seems 
to kick back in again.", especially the "optimize" bit (which, by the way you 
should do only if you have the capability of doing it periodically [1]) is some 
evidence that this may be in the vicinity. One of the effects of an optimize is 
to merge your segments files from N to 1. So say you have 10 segments. Each one 
of those may consist of 10-15 individual files, all of which are held open. So 
you'd go from 150 open file handles to 15..

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2F2017%2F10%2F13%2Fsegment-merging-deleted-documents-optimize-may-bad%2F=02%7C01%7CScott.Pouliot%40peoplefluent.com%7Cc2912861f58248e3a92808d55f690eb8%7C8b16fb62c78448b6aba889567990e7fe%7C1%7C0%7C636519827178716698=DxnChrfyTbRDjB7HzqpOE%2BvOJRIxdnrXVCIyfoMjJPU%3D=0

Best,
Erick

On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott 
<scott.poul...@peoplefluent.com> wrote:
> Erick,
>
> Thanks!  Could these settings be toying with replication?  Solr itself seems 
> to be working like a champ, except when things get out of sync.
>
> Scott
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Friday, January 19, 2018 12:27 PM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> Scott:
>
> We usually recommend setting files and processes very, very high. Like 65K 
> high. Or unlimited if you can.
>
> Plus max user processes should also be bumped very high as well, like 65K as 
> well.
>
> Plus max memory and virtual memory should be unlimited.
>
> We've included warnings at startup for open files and processes, see 
> SOLR-11703
>
> Best,
> Erick
>
> On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
> <scott.poul...@peoplefluent.com> wrote:
>> I do have a ticket in with our systems team to up the file handlers since I 
>> am seeing the "Too many files open" error on occasion on our prod servers.  
>> Is this the setting you're referring to?  Found we were set to to 1024 using 
>> the "Ulimit" command.
>>
>> -Original Message-
>> From: Shawn Heisey [mailto:apa...@elyograg.org]
>> Sent: Friday, January 19, 2018 10:48 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Replication being flaky (6.2.0)
>>
>> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>>> seems that the replication stalls or stops functioning every now and again. 
>>>  If we restart the Solr service or optimize the core it seems to kick back 
>>> in again.
>>>
>>> Anyone have any idea what might be causing this?  We do have a good amount 
>>> of cores on each server (@150 or so), but I have heard reports of a LOT 
>>> more than that in use.
>>
>> Have you increased the number of processes that the user running Solr is 
>> allowed to start?  Most operating systems limit the number of 
>> threads/processes a user can start to a low value like 1024.  With 150 
>> cores, particularly with background tasks like replication configured, 
>> chances are that Solr is going to need to start a lot of threads.  This is 
>> an OS setting that a lot of Solr admins end up needing to increase.
>>
>> I ran into the process limit on my servers and I don't have anywhere near 
>> 150 cores.
>>
>> The fact that restarting Solr gets it working again (at least
>> temporarily) would fit with a process limit being the problem.  I'm not 
>> guaranteeing that this is the problem, only saying that it fits.
>>
>> Thanks,
>> Shawn

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

"Could be", certainly. "Definitely is" is iffier ;)...

But the statement "If we restart the Solr service or optimize the core
it seems to kick back in again.", especially the "optimize" bit
(which, by the way you should do only if you have the capability of
doing it periodically [1]) is some evidence that this may be in the
vicinity. One of the effects of an optimize is to merge your segments
files from N to 1. So say you have 10 segments. Each one of those may
consist of 10-15 individual files, all of which are held open. So
you'd go from 150 open file handles to 15..

https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

Best,
Erick

On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott
<scott.poul...@peoplefluent.com> wrote:
> Erick,
>
> Thanks!  Could these settings be toying with replication?  Solr itself seems 
> to be working like a champ, except when things get out of sync.
>
> Scott
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Friday, January 19, 2018 12:27 PM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> Scott:
>
> We usually recommend setting files and processes very, very high. Like 65K 
> high. Or unlimited if you can.
>
> Plus max user processes should also be bumped very high as well, like 65K as 
> well.
>
> Plus max memory and virtual memory should be unlimited.
>
> We've included warnings at startup for open files and processes, see 
> SOLR-11703
>
> Best,
> Erick
>
> On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
> <scott.poul...@peoplefluent.com> wrote:
>> I do have a ticket in with our systems team to up the file handlers since I 
>> am seeing the "Too many files open" error on occasion on our prod servers.  
>> Is this the setting you're referring to?  Found we were set to to 1024 using 
>> the "Ulimit" command.
>>
>> -Original Message-
>> From: Shawn Heisey [mailto:apa...@elyograg.org]
>> Sent: Friday, January 19, 2018 10:48 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Replication being flaky (6.2.0)
>>
>> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>>> seems that the replication stalls or stops functioning every now and again. 
>>>  If we restart the Solr service or optimize the core it seems to kick back 
>>> in again.
>>>
>>> Anyone have any idea what might be causing this?  We do have a good amount 
>>> of cores on each server (@150 or so), but I have heard reports of a LOT 
>>> more than that in use.
>>
>> Have you increased the number of processes that the user running Solr is 
>> allowed to start?  Most operating systems limit the number of 
>> threads/processes a user can start to a low value like 1024.  With 150 
>> cores, particularly with background tasks like replication configured, 
>> chances are that Solr is going to need to start a lot of threads.  This is 
>> an OS setting that a lot of Solr admins end up needing to increase.
>>
>> I ran into the process limit on my servers and I don't have anywhere near 
>> 150 cores.
>>
>> The fact that restarting Solr gets it working again (at least
>> temporarily) would fit with a process limit being the problem.  I'm not 
>> guaranteeing that this is the problem, only saying that it fits.
>>
>> Thanks,
>> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

Erick,

Thanks!  Could these settings be toying with replication?  Solr itself seems to 
be working like a champ, except when things get out of sync.

Scott

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, January 19, 2018 12:27 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Solr Replication being flaky (6.2.0)

Scott:

We usually recommend setting files and processes very, very high. Like 65K 
high. Or unlimited if you can.

Plus max user processes should also be bumped very high as well, like 65K as 
well.

Plus max memory and virtual memory should be unlimited.

We've included warnings at startup for open files and processes, see SOLR-11703

Best,
Erick

On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
<scott.poul...@peoplefluent.com> wrote:
> I do have a ticket in with our systems team to up the file handlers since I 
> am seeing the "Too many files open" error on occasion on our prod servers.  
> Is this the setting you're referring to?  Found we were set to to 1024 using 
> the "Ulimit" command.
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 10:48 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>> seems that the replication stalls or stops functioning every now and again.  
>> If we restart the Solr service or optimize the core it seems to kick back in 
>> again.
>>
>> Anyone have any idea what might be causing this?  We do have a good amount 
>> of cores on each server (@150 or so), but I have heard reports of a LOT more 
>> than that in use.
>
> Have you increased the number of processes that the user running Solr is 
> allowed to start?  Most operating systems limit the number of 
> threads/processes a user can start to a low value like 1024.  With 150 cores, 
> particularly with background tasks like replication configured, chances are 
> that Solr is going to need to start a lot of threads.  This is an OS setting 
> that a lot of Solr admins end up needing to increase.
>
> I ran into the process limit on my servers and I don't have anywhere near 150 
> cores.
>
> The fact that restarting Solr gets it working again (at least
> temporarily) would fit with a process limit being the problem.  I'm not 
> guaranteeing that this is the problem, only saying that it fits.
>
> Thanks,
> Shawn

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

Scott:

We usually recommend setting files and processes very, very high. Like
65K high. Or unlimited if you can.

Plus max user processes should also be bumped very high as well, like
65K as well.

Plus max memory and virtual memory should be unlimited.

We've included warnings at startup for open files and processes, see SOLR-11703

Best,
Erick

On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott
<scott.poul...@peoplefluent.com> wrote:
> I do have a ticket in with our systems team to up the file handlers since I 
> am seeing the "Too many files open" error on occasion on our prod servers.  
> Is this the setting you're referring to?  Found we were set to to 1024 using 
> the "Ulimit" command.
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 10:48 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>> seems that the replication stalls or stops functioning every now and again.  
>> If we restart the Solr service or optimize the core it seems to kick back in 
>> again.
>>
>> Anyone have any idea what might be causing this?  We do have a good amount 
>> of cores on each server (@150 or so), but I have heard reports of a LOT more 
>> than that in use.
>
> Have you increased the number of processes that the user running Solr is 
> allowed to start?  Most operating systems limit the number of 
> threads/processes a user can start to a low value like 1024.  With 150 cores, 
> particularly with background tasks like replication configured, chances are 
> that Solr is going to need to start a lot of threads.  This is an OS setting 
> that a lot of Solr admins end up needing to increase.
>
> I ran into the process limit on my servers and I don't have anywhere near 150 
> cores.
>
> The fact that restarting Solr gets it working again (at least
> temporarily) would fit with a process limit being the problem.  I'm not 
> guaranteeing that this is the problem, only saying that it fits.
>
> Thanks,
> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

I do have a ticket in with our systems team to up the file handlers since I am 
seeing the "Too many files open" error on occasion on our prod servers.  Is 
this the setting you're referring to?  Found we were set to to 1024 using the 
"Ulimit" command.

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Friday, January 19, 2018 10:48 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
> So we're running Solr in a Master/Slave configuration (1 of each) and it 
> seems that the replication stalls or stops functioning every now and again.  
> If we restart the Solr service or optimize the core it seems to kick back in 
> again.
> 
> Anyone have any idea what might be causing this?  We do have a good amount of 
> cores on each server (@150 or so), but I have heard reports of a LOT more 
> than that in use.

Have you increased the number of processes that the user running Solr is 
allowed to start?  Most operating systems limit the number of threads/processes 
a user can start to a low value like 1024.  With 150 cores, particularly with 
background tasks like replication configured, chances are that Solr is going to 
need to start a lot of threads.  This is an OS setting that a lot of Solr 
admins end up needing to increase.

I ran into the process limit on my servers and I don't have anywhere near 150 
cores.

The fact that restarting Solr gets it working again (at least
temporarily) would fit with a process limit being the problem.  I'm not 
guaranteeing that this is the problem, only saying that it fits.

Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 7:50 AM, Pouliot, Scott wrote:

So we're running Solr in a Master/Slave configuration (1 of each) and it seems 
that the replication stalls or stops functioning every now and again.  If we 
restart the Solr service or optimize the core it seems to kick back in again.

Anyone have any idea what might be causing this?  We do have a good amount of 
cores on each server (@150 or so), but I have heard reports of a LOT more than 
that in use.


Have you increased the number of processes that the user running Solr is 
allowed to start?  Most operating systems limit the number of 
threads/processes a user can start to a low value like 1024.  With 150 
cores, particularly with background tasks like replication configured, 
chances are that Solr is going to need to start a lot of threads.  This 
is an OS setting that a lot of Solr admins end up needing to increase.


I ran into the process limit on my servers and I don't have anywhere 
near 150 cores.


The fact that restarting Solr gets it working again (at least 
temporarily) would fit with a process limit being the problem.  I'm not 
guaranteeing that this is the problem, only saying that it fits.


Thanks,
Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

I'm at the point now where I may end up writing a script to compare 
master/slave nightly...and trigger an optimize or solr restart if there are any 
differences.  Of course I have to check 150+ cores...but it could be done.  I'm 
just hoping I don't need to go that route

-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com] 
Sent: Friday, January 19, 2018 10:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

This happens to me quite often as well.  Generally on the replication admin 
screen it will say its downloading a file, but be at 0 or a VERY small kb/sec.  
Then after a restart of the slave its back to downloading at 30 to
100 mg/sec.  Would be curious if there actually is a solution to this aside 
from checking every day if the core replicated.  Im on Solr 5.x by the way -Dave

On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott < 
scott.poul...@peoplefluent.com> wrote:

> So we're running Solr in a Master/Slave configuration (1 of each) and 
> it seems that the replication stalls or stops functioning every now 
> and again.  If we restart the Solr service or optimize the core it 
> seems to kick back in again.
>
> Anyone have any idea what might be causing this?  We do have a good 
> amount of cores on each server (@150 or so), but I have heard reports 
> of a LOT more than that in use.
>
> Here is our master config:
> 
> 
>   
>   startup
>   commit
>
>   
>   00:00:10
> 
> 
> 
> 1
> 
>   
>
> And our slave config:
> 
> 
>
>   
>name="masterUrl">http://server1:8080/solr/${https://na01.safelinks.pro
> tection.outlook.com/?url=solr.core.name=02%7C01%7CScott.Pouliot%4
> 0peoplefluent.com%7C8d43918dd95540a3a11708d55f523302%7C8b16fb62c78448b
> 6aba889567990e7fe%7C1%7C1%7C636519729029923349=Fes6G36gIMRyfahTI
> fftg0eUEVEiVK77B8KpuTr%2FJrA%3D=0}
> 
>
>   
>   00:00:45
> 
>   
>
>   
> 
>   solr-data-config.xml
> 
>   
>

Re: Solr Replication being flaky (6.2.0)

[David Hastings]

This happens to me quite often as well.  Generally on the replication admin
screen it will say its downloading a file, but be at 0 or a VERY small
kb/sec.  Then after a restart of the slave its back to downloading at 30 to
100 mg/sec.  Would be curious if there actually is a solution to this aside
from checking every day if the core replicated.  Im on Solr 5.x by the way
-Dave

On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:

> So we're running Solr in a Master/Slave configuration (1 of each) and it
> seems that the replication stalls or stops functioning every now and
> again.  If we restart the Solr service or optimize the core it seems to
> kick back in again.
>
> Anyone have any idea what might be causing this?  We do have a good amount
> of cores on each server (@150 or so), but I have heard reports of a LOT
> more than that in use.
>
> Here is our master config:
> 
> 
>   
>   startup
>   commit
>
>   
>   00:00:10
> 
> 
> 
> 1
> 
>   
>
> And our slave config:
> 
> 
>
>   
>   http://server1:8080/solr/${solr.core.name}
> 
>
>   
>   00:00:45
> 
>   
>
>   
> 
>   solr-data-config.xml
> 
>   
>

Re: Solr replication

[Satyaprashant Bezwada]

Thanks Eric, fixed the issue. The IT team corrected the solrconfig.xml but 
forgot to execute the zkcli.sh script on solr node. After I executed the script 
its working now.


On 9/20/17, 10:20 AM, "Erick Erickson"  wrote:

WARNING - External email; exercise caution.


Your solrconfig.xml file is mal-formed. The smoking gun is:

 Exception during parsing file: solrconfig.xml

Best,
Erick

On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
 wrote:
> Need some inputs or help in resolving replication across solr nodes. We 
have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes 
configured. Enabled Solr replication in my Solrj client but the replication 
fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 
zookeeper and 3 Solr nodes configured. Here is the exception I see on one of 
the nodes of Solr when I try to create a collection in the environment where it 
fails. I have compared the Solrconfig.xml on both the environments and didn’t 
see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
 status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
/var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
 status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
/var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
Loading solr.xml from SolrHome (not found in ZooKeeper)
> 

Re: Solr replication

[Erick Erickson]

Your solrconfig.xml file is mal-formed. The smoking gun is:

 Exception during parsing file: solrconfig.xml

Best,
Erick

On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
 wrote:
> Need some inputs or help in resolving replication across solr nodes. We have 
> installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes 
> configured. Enabled Solr replication in my Solrj client but the replication 
> fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 zookeeper 
> and 3 Solr nodes configured. Here is the exception I see on one of the nodes 
> of Solr when I try to create a collection in the environment where it fails. 
> I have compared the Solrconfig.xml on both the environments and didn’t see 
> any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR 
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
> action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
> [admin] webapp=null path=/admin/cores 
> params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
>  status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
>  [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
> /overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
> /var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR 
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
> action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
> [admin] webapp=null path=/admin/cores 
> params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
>  status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
>  [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
> /overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
> /var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Loading 
> solr.xml from SolrHome (not found in ZooKeeper)
> 2017-09-19 23:12:22.233 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
> container configuration from /var/solr/data/Node122/solr.xml
> 2017-09-19 23:12:22.644 INFO  (main) [   ] o.a.s.u.UpdateShardHandler 
> Creating UpdateShardHandler HTTP client with params: 
> socketTimeout=60=6=true
> 2017-09-19 23:12:22.650 INFO  (main) [   ] o.a.s.c.ZkContainer 

Re: Solr Replication sometimes coming in log files

[Upayavira]

I bet you have the admin UI open on your second slave. The _=144... is
the give-away. Those requests are the admin UI asking the replication
handler for the status of replication.

Upayavira

On Wed, Sep 9, 2015, at 06:32 AM, Kamal Kishore Aggarwal wrote:
> Hi Team,
> 
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
> 
> 
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1
> is in another server in different zone (say zone B). There is latency of
> 40
> ms between two zones.
> 
> Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> to delay in data replication from Master server. These days we are
> finding
> these below mentioned replication information in log files, but such
> lines
> are not in previous files on the Slave 1 server. Also, such information
> is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
> 
> 
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708786003} status=0 QTime=173
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708787976} status=0 QTime=1807
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708791563} status=0 QTime=7140
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708800450} status=0 QTime=1679
> 
> 
> 
> Please confirm if we our thought that increased replication time (which
> can
> be due to servers connectivity issues) is the reason for high load on
> solr.
> 
> Regards
> Kamal Kishore

Re: Solr Replication sometimes coming in log files

[Mikhail Khludnev]

Hello,

I'd say opposite: high load causes long time to respond. 'command=details'
is rather cheap and fast _I believe_.

On Mon, Sep 14, 2015 at 10:20 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Can anybody suggest me something..
>
> On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
> > Hi Team,
> >
> > I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> > configuration has master & slave ( 2 Slaves) architecture.
> >
> >
> > Master & Slave 2 are in same server location (say zone A) , whereas Slave
> > 1 is in another server in different zone (say zone B). There is latency
> of
> > 40 ms between two zones.
> >
> > Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> > to delay in data replication from Master server. These days we are
> finding
> > these below mentioned replication information in log files, but such
> lines
> > are not in previous files on the Slave 1 server. Also, such information
> is
> > not there in any Slave 2 log files (might be due to same zone of master &
> > slave 2).
> >
> >
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708786003} status=0 QTime=173
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708787976} status=0 QTime=1807
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708791563} status=0 QTime=7140
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708800450} status=0 QTime=1679
> >
> >
> >
> > Please confirm if we our thought that increased replication time (which
> > can be due to servers connectivity issues) is the reason for high load on
> > solr.
> >
> > Regards
> > Kamal Kishore
> >
> >
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

Re: Solr Replication sometimes coming in log files

[Kamal Kishore Aggarwal]

Can anybody suggest me something..

On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi Team,
>
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
>
>
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1 is in another server in different zone (say zone B). There is latency of
> 40 ms between two zones.
>
> Now, a days we are facing high load on Slave 1 & we suspect that it is due
> to delay in data replication from Master server. These days we are finding
> these below mentioned replication information in log files, but such lines
> are not in previous files on the Slave 1 server. Also, such information is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
>
>
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708786003} status=0 QTime=173
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708787976} status=0 QTime=1807
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708791563} status=0 QTime=7140
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708800450} status=0 QTime=1679
>
>
>
> Please confirm if we our thought that increased replication time (which
> can be due to servers connectivity issues) is the reason for high load on
> solr.
>
> Regards
> Kamal Kishore
>
>

Re: solr replication vs. rsync

[Shawn Heisey]

On 1/24/2015 10:56 PM, Dan Davis wrote:
 When I polled the various projects already using Solr at my organization, I
 was greatly surprised that none of them were using Solr replication,
 because they had talked about replicating the data.
 
 But we are not Pinterest, and do not expect to be taking in changes one
 post at a time (at least the engineers don't - just wait until its used for
 a Crud app that wants full-text search on a description field!).Still,
 rsync can be very, very fast with the right options (-W for gigabit
 ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s over
 GigE previously.
 
 Does anyone have any numbers for how fast Solr replication goes, and what
 to do to tune it?
 
 I'm not enthusiastic to give-up recently tested cluster stability for a
 home grown mess, but I am interested in numbers that are out there.

Numbers are included on the Solr replication wiki page, both in graph
and numeric form.  Gathering these numbers must have been pretty easy --
before the HTTP replication made it into Solr, Solr used to contain an
rsync-based implementation.

http://wiki.apache.org/solr/SolrReplication#Performance_numbers

Other data on that wiki page discusses the replication config.  There's
not a lot to tune.

I run a redundant non-SolrCloud index myself through a different method
-- my indexing program indexes each index copy completely independently.
 There is no replication.  This separation allows me to upgrade any
component, or change any part of solrconfig or schema, on either copy of
the index without affecting the other copy at all.  With replication, if
something is changed on the master or the slave, you might find that the
slave no longer works, because it will be handling an index created by
different software or a different config.

Thanks,
Shawn

Re: solr replication vs. rsync

[Erick Erickson]

bq:  I thought SolrCloud replicas were replication, and you imply parallel
indexing

Absolutely! You couldn't get near-real-time indexing if you relied on
replication a-la
3x. And you also couldn't guarantee consistency.

Say you have 1 shard, a leader and a follower (i.e. 2 replicas). Now you
throw a doc
to be indexed. The sequence is:
leader gets the doc
leader forwards the doc to the follower
leader and follower both add the doc to their local index (and tlog).
follower acks back to leader
leader acks back to client.

So yes, the raw document is forwarded to all replicas before the leader
responds
to the client, the docs all get written to the tlogs, etc. That's the only
way to guarantee
that if the leader goes down, the follower can take over without losing
documents.

Best,
Erick

On Sun, Jan 25, 2015 at 6:15 PM, Dan Davis dansm...@gmail.com wrote:

 @Erick,

 Problem space is not constant indexing.   I thought SolrCloud replicas were
 replication, and you imply parallel indexing.  Good to know.

 On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com
 wrote:

  @Shawn: Cool table, thanks!
 
  @Dan:
  Just to throw a different spin on it, if you migrate to SolrCloud, then
  this question becomes moot as the raw documents are sent to each of the
  replicas so you very rarely have to copy the full index. Kind of a
 tradeoff
  between constant load because you're sending the raw documents around
  whenever you index and peak usage when the index replicates.
 
  There are a bunch of other reasons to go to SolrCloud, but you know your
  problem space best.
 
  FWIW,
  Erick
 
  On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org
  javascript:; wrote:
 
   On 1/24/2015 10:56 PM, Dan Davis wrote:
When I polled the various projects already using Solr at my
   organization, I
was greatly surprised that none of them were using Solr replication,
because they had talked about replicating the data.
   
But we are not Pinterest, and do not expect to be taking in changes
 one
post at a time (at least the engineers don't - just wait until its
 used
   for
a Crud app that wants full-text search on a description field!).
   Still,
rsync can be very, very fast with the right options (-W for gigabit
ethernet, and maybe -S for sparse files).   I've clocked it at 48
 MB/s
   over
GigE previously.
   
Does anyone have any numbers for how fast Solr replication goes, and
  what
to do to tune it?
   
I'm not enthusiastic to give-up recently tested cluster stability
 for a
home grown mess, but I am interested in numbers that are out there.
  
   Numbers are included on the Solr replication wiki page, both in graph
   and numeric form.  Gathering these numbers must have been pretty easy
 --
   before the HTTP replication made it into Solr, Solr used to contain an
   rsync-based implementation.
  
   http://wiki.apache.org/solr/SolrReplication#Performance_numbers
  
   Other data on that wiki page discusses the replication config.  There's
   not a lot to tune.
  
   I run a redundant non-SolrCloud index myself through a different method
   -- my indexing program indexes each index copy completely
 independently.
There is no replication.  This separation allows me to upgrade any
   component, or change any part of solrconfig or schema, on either copy
 of
   the index without affecting the other copy at all.  With replication,
 if
   something is changed on the master or the slave, you might find that
 the
   slave no longer works, because it will be handling an index created by
   different software or a different config.
  
   Thanks,
   Shawn
  
  
 

Re: solr replication vs. rsync

[Erick Erickson]

@Shawn: Cool table, thanks!

@Dan:
Just to throw a different spin on it, if you migrate to SolrCloud, then
this question becomes moot as the raw documents are sent to each of the
replicas so you very rarely have to copy the full index. Kind of a tradeoff
between constant load because you're sending the raw documents around
whenever you index and peak usage when the index replicates.

There are a bunch of other reasons to go to SolrCloud, but you know your
problem space best.

FWIW,
Erick

On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 1/24/2015 10:56 PM, Dan Davis wrote:
  When I polled the various projects already using Solr at my
 organization, I
  was greatly surprised that none of them were using Solr replication,
  because they had talked about replicating the data.
 
  But we are not Pinterest, and do not expect to be taking in changes one
  post at a time (at least the engineers don't - just wait until its used
 for
  a Crud app that wants full-text search on a description field!).
 Still,
  rsync can be very, very fast with the right options (-W for gigabit
  ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
 over
  GigE previously.
 
  Does anyone have any numbers for how fast Solr replication goes, and what
  to do to tune it?
 
  I'm not enthusiastic to give-up recently tested cluster stability for a
  home grown mess, but I am interested in numbers that are out there.

 Numbers are included on the Solr replication wiki page, both in graph
 and numeric form.  Gathering these numbers must have been pretty easy --
 before the HTTP replication made it into Solr, Solr used to contain an
 rsync-based implementation.

 http://wiki.apache.org/solr/SolrReplication#Performance_numbers

 Other data on that wiki page discusses the replication config.  There's
 not a lot to tune.

 I run a redundant non-SolrCloud index myself through a different method
 -- my indexing program indexes each index copy completely independently.
  There is no replication.  This separation allows me to upgrade any
 component, or change any part of solrconfig or schema, on either copy of
 the index without affecting the other copy at all.  With replication, if
 something is changed on the master or the slave, you might find that the
 slave no longer works, because it will be handling an index created by
 different software or a different config.

 Thanks,
 Shawn

Re: solr replication vs. rsync

[Dan Davis]

@Erick,

Problem space is not constant indexing.   I thought SolrCloud replicas were
replication, and you imply parallel indexing.  Good to know.

On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com wrote:

 @Shawn: Cool table, thanks!

 @Dan:
 Just to throw a different spin on it, if you migrate to SolrCloud, then
 this question becomes moot as the raw documents are sent to each of the
 replicas so you very rarely have to copy the full index. Kind of a tradeoff
 between constant load because you're sending the raw documents around
 whenever you index and peak usage when the index replicates.

 There are a bunch of other reasons to go to SolrCloud, but you know your
 problem space best.

 FWIW,
 Erick

 On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org
 javascript:; wrote:

  On 1/24/2015 10:56 PM, Dan Davis wrote:
   When I polled the various projects already using Solr at my
  organization, I
   was greatly surprised that none of them were using Solr replication,
   because they had talked about replicating the data.
  
   But we are not Pinterest, and do not expect to be taking in changes one
   post at a time (at least the engineers don't - just wait until its used
  for
   a Crud app that wants full-text search on a description field!).
  Still,
   rsync can be very, very fast with the right options (-W for gigabit
   ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
  over
   GigE previously.
  
   Does anyone have any numbers for how fast Solr replication goes, and
 what
   to do to tune it?
  
   I'm not enthusiastic to give-up recently tested cluster stability for a
   home grown mess, but I am interested in numbers that are out there.
 
  Numbers are included on the Solr replication wiki page, both in graph
  and numeric form.  Gathering these numbers must have been pretty easy --
  before the HTTP replication made it into Solr, Solr used to contain an
  rsync-based implementation.
 
  http://wiki.apache.org/solr/SolrReplication#Performance_numbers
 
  Other data on that wiki page discusses the replication config.  There's
  not a lot to tune.
 
  I run a redundant non-SolrCloud index myself through a different method
  -- my indexing program indexes each index copy completely independently.
   There is no replication.  This separation allows me to upgrade any
  component, or change any part of solrconfig or schema, on either copy of
  the index without affecting the other copy at all.  With replication, if
  something is changed on the master or the slave, you might find that the
  slave no longer works, because it will be handling an index created by
  different software or a different config.
 
  Thanks,
  Shawn
 
 

Re: solr replication vs. rsync

[Dan Davis]

Thanks!

On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com wrote:

 @Shawn: Cool table, thanks!

 @Dan:
 Just to throw a different spin on it, if you migrate to SolrCloud, then
 this question becomes moot as the raw documents are sent to each of the
 replicas so you very rarely have to copy the full index. Kind of a tradeoff
 between constant load because you're sending the raw documents around
 whenever you index and peak usage when the index replicates.

 There are a bunch of other reasons to go to SolrCloud, but you know your
 problem space best.

 FWIW,
 Erick

 On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org
 javascript:; wrote:

  On 1/24/2015 10:56 PM, Dan Davis wrote:
   When I polled the various projects already using Solr at my
  organization, I
   was greatly surprised that none of them were using Solr replication,
   because they had talked about replicating the data.
  
   But we are not Pinterest, and do not expect to be taking in changes one
   post at a time (at least the engineers don't - just wait until its used
  for
   a Crud app that wants full-text search on a description field!).
  Still,
   rsync can be very, very fast with the right options (-W for gigabit
   ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
  over
   GigE previously.
  
   Does anyone have any numbers for how fast Solr replication goes, and
 what
   to do to tune it?
  
   I'm not enthusiastic to give-up recently tested cluster stability for a
   home grown mess, but I am interested in numbers that are out there.
 
  Numbers are included on the Solr replication wiki page, both in graph
  and numeric form.  Gathering these numbers must have been pretty easy --
  before the HTTP replication made it into Solr, Solr used to contain an
  rsync-based implementation.
 
  http://wiki.apache.org/solr/SolrReplication#Performance_numbers
 
  Other data on that wiki page discusses the replication config.  There's
  not a lot to tune.
 
  I run a redundant non-SolrCloud index myself through a different method
  -- my indexing program indexes each index copy completely independently.
   There is no replication.  This separation allows me to upgrade any
  component, or change any part of solrconfig or schema, on either copy of
  the index without affecting the other copy at all.  With replication, if
  something is changed on the master or the slave, you might find that the
  slave no longer works, because it will be handling an index created by
  different software or a different config.
 
  Thanks,
  Shawn
 
 

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

I haven't seen any activity regarding this in Jira, just curious if it
would be looked into anytime soon...

On Thu, Oct 2, 2014 at 10:11 AM, Phil Black-Knight 
pblackkni...@globalgiving.org wrote:

 see the ticket here:
 https://issues.apache.org/jira/browse/SOLR-6579

 including a patch to fix it.

 On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
  I was helping to look into this with Nick  I think we may have figured
 out
  the core of the problem...
 
  The problem is easily reproducible by starting replication on the slave
 and
  then sending a shutdown command to tomcat (e.g. catalina.sh stop).
 
  With a debugger attached, it looks like the fsyncService thread is
 blocking
  VM shutdown because it is created as a non-daemon thread.

 snip

  I can submit patches if needed... and cross post to the dev mailing
 list...

 File a detailed issue in Jira and attach your patch there.  This is our
 bugtracker.  You need an account on the Apache jira instance to do this:

 https://issues.apache.org/jira/browse/SOLR

 Thanks,
 Shawn

RE: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

I was helping to look into this with Nick  I think we may have figured out
the core of the problem...

The problem is easily reproducible by starting replication on the slave and
then sending a shutdown command to tomcat (e.g. catalina.sh stop).

With a debugger attached, it looks like the fsyncService thread is blocking
VM shutdown because it is created as a non-daemon thread.

Essentially what seems to be happening is that the fsyncService thread is
running when 'catalina.sh stop' is executed. This goes in and calls
SnapPuller.destroy() which aborts the current sync. Around line 517 of the
SnapPuller, there is code that is supposed to cleanup the fsyncService
thread, but I don't think it is getting executed because the thread that
called SnapPuller.fetchLatestIndex() is configured as a daemon Thread, so
the JVM ends up shutting that down before it can cleanup the fysncService...

So... it seems like:

if (fsyncService != null)
ExecutorUtil.shutdownNowAndAwaitTermination(fsyncService);
could be added around line 1706 of SnapPuller.java,  or

  puller.setDaemon(*false*);
could be added around line 230 of ReplicationHandler.java, however this
needs some additional work (and I think it might need to be added
regardless) since the cleanup code in SnapPuller(around 517) that shuts
down the fsync thread never gets execute since
logReplicationTimeAndConfFiles() can throw IO exceptions bypassing the rest
of the finally block...So the call to
logReplicationTimeAndConfFiles() around line 512 would need to get wrapped
with a try/catch block to catch the IO exception...

I can submit patches if needed... and cross post to the dev mailing list...

-Phil

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Shawn Heisey]

On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
 I was helping to look into this with Nick  I think we may have figured out
 the core of the problem...
 
 The problem is easily reproducible by starting replication on the slave and
 then sending a shutdown command to tomcat (e.g. catalina.sh stop).
 
 With a debugger attached, it looks like the fsyncService thread is blocking
 VM shutdown because it is created as a non-daemon thread.

snip

 I can submit patches if needed... and cross post to the dev mailing list...

File a detailed issue in Jira and attach your patch there.  This is our
bugtracker.  You need an account on the Apache jira instance to do this:

https://issues.apache.org/jira/browse/SOLR

Thanks,
Shawn

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

see the ticket here:
https://issues.apache.org/jira/browse/SOLR-6579

including a patch to fix it.

On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
  I was helping to look into this with Nick  I think we may have figured
 out
  the core of the problem...
 
  The problem is easily reproducible by starting replication on the slave
 and
  then sending a shutdown command to tomcat (e.g. catalina.sh stop).
 
  With a debugger attached, it looks like the fsyncService thread is
 blocking
  VM shutdown because it is created as a non-daemon thread.

 snip

  I can submit patches if needed... and cross post to the dev mailing
 list...

 File a detailed issue in Jira and attach your patch there.  This is our
 bugtracker.  You need an account on the Apache jira instance to do this:

 https://issues.apache.org/jira/browse/SOLR

 Thanks,
 Shawn

Re: SolR replication issue

[Shawn Heisey]

On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote:
 I'm using Solr 4.3.1 with a master/slave configuration.
 
 Configuration:
 
 Master:
 *  lst name=master*
 * str name=replicateAftercommit/str*
 * str name=replicateAfterstartup/str*
 * str name=confFilesschema.xml,stopwords.txt/str*
 *   /lst*
 
 
 Slave:
  * lst name=slave*
 * str name=masterUrlhttp://10.xx.xx.xx:9081/solr
 http://10.xx.xx.xx:9081/solr/str*
 * str name=pollInterval00:00:60/str*
 *   /lst*
 
 The replication sometimes fails with the exception
 
 *Error closing old IndexWriter.
 core=collection1:java.lang.IllegalArgumentException: Unknown directory:
 NRTCachingDirectory...*
 *ReplicationHandler SnapPull failed :org.apache.solr.common.SolrException:
 Index fetch failed :*
 
 This is happening with any index size.

We would need the entire stacktrace from that error message to make any
real determination, but without that, I offer this:

If it's happening with a tiny index (kilobytes or only a few megabytes),
then I would suspect a bug in Solr.  4.3.1 is ancient history now --
Solr's development and release schedule is very aggressive.  There have
been ten new versions in the 14 months since 4.3.1 was announced, and
the 4.10 release is imminent.  There have been a number of replication
bugs fixed in those releases.

If any index size means that some of them are in the range of several
gigabytes, then it may simply be a configuration problem.  The
commitReserveDuration parameter may need increasing beyond its default
of ten seconds.  Large updates after optimizing or a major automatic
merge can take many minutes to transfer, and if that parameter is set
too low, the old index can disappear before it can finish replicating.

Thanks,
Shawn

Re: SolR replication issue

[Mauricio Ferreyra]

The entire stacktrace:

ERROR SolrIndexWriter
Coud not unlock directory after seemingly failed IndexWriter#close()
org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully
unlock a NativeFSLock which is held by another indexer component:
/home/miapp/collection1/data/index.20140901140800014/write.lock
 at
org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:295)
at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4162)
 at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:156)
at
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:164)
 at
org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:624)
at
org.apache.solr.handler.SnapPuller.openNewWriterAndSearcher(SnapPuller.java:622)
 at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:446)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317)
 at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16:39:00
ERROR DefaultSolrCoreState
Error closing old IndexWriter.
core=collection1:java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77b024a9;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
{NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6de47a0f;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=0;path=/home/miapp/collection1/data;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800016
lockFactory=org.apache.lucene.store.NativeFSLockFactory@791e6360;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=-3;path=/home/miapp/collection1/data/index.20140901140800016;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901163900022
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a6de862;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=1;path=/home/miapp/collection1/data/index.20140901163900022;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@5863c9e0;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=1;path=/home/miapp/collection1/data/index.20140901140800014;done=false}


ERROR SolrIndexWriter SolrIndexWriter was not closed prior to finalize(),​
indicates a bug -- POSSIBLE RESOURCE LEAK!!!






On Mon, Sep 1, 2014 at 4:28 PM, Shawn Heisey s...@elyograg.org wrote:

 On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote:
  I'm using Solr 4.3.1 with a master/slave configuration.
 
  Configuration:
 
  Master:
  *  lst name=master*
  * str name=replicateAftercommit/str*
  * str name=replicateAfterstartup/str*
  * str name=confFilesschema.xml,stopwords.txt/str*
  *   /lst*
 
 
  Slave:
   * lst name=slave*
  * str name=masterUrlhttp://10.xx.xx.xx:9081/solr
  http://10.xx.xx.xx:9081/solr/str*
  * str name=pollInterval00:00:60/str*
  *   /lst*
 
  The replication sometimes fails with the exception
 
  *Error closing old IndexWriter.
  core=collection1:java.lang.IllegalArgumentException: Unknown directory:
  NRTCachingDirectory...*
  *ReplicationHandler SnapPull failed
 :org.apache.solr.common.SolrException:
  Index fetch failed :*
 
  This is happening with any index size.

 We would need the entire stacktrace from that error message to make any
 real determination, but without that, I offer this:

 If it's happening with a tiny index (kilobytes or only a few megabytes),
 then I would suspect a bug in Solr.  4.3.1 is ancient history now --
 Solr's development and release schedule is very aggressive.  There have
 been ten new versions in the 14 months since 4.3.1 was announced, and
 the 4.10 release is imminent.  There have been a number of replication
 bugs fixed in those releases.

 If any index size means that some of them are in the range of several
 gigabytes, then it may simply be a configuration problem.  The
 commitReserveDuration parameter may need increasing beyond its default
 of ten seconds.  Large updates after optimizing or a major 

Re: Solr Replication Issue : Incorrect Base_URL

[Alan Woodward]

Hi Pramod,

You need to set hostContext in your solr.xml.  See 
https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml

Alan Woodward
www.flax.co.uk


On 13 Jun 2014, at 00:44, pramodEbay wrote:

 Hi,
 I am deploying Solr in a larger web application. The standalone solr
 instance works fine. The path-prefix I use is raptorslrweb. A standalone
 SOLR query to my instance that works is as follows:
 
 http://hostname:8080/raptorslrweb/solr/reviews/select?q=*%3A*wt=jsonindent=true
 
 However, when I configure a solr cloud, I get the following error in
 RecoveryStrategy:
 msg:org.apache.solr.client.solrj.SolrServerException: Server at
 http://hostname:8080/solr/reviews sent back a redirect (302).,
 
 The reason is the base_url does not seem to honor the path-prefix.
 clusterstate.json shows the following for the node:
 {reviews:{
shards:{shard1:{
range:null,
state:active,
parent:null,
replicas:{
  core_node1:{
state:down,
   * base_url:http://hostname:8080/solr,*   
core:reviews,
node_name:10.98.63.98:8080_solr},
 
 Can someone please tell me where do I tell zookeeper or solr cloud that the
 base url should be hostname:8080/raptorslrweb/solr and not
 hostname:8080/solr.
 
 Thanks,
 Pramod
 
 
 
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-Replication-Issue-Incorrect-Base-URL-tp4141537.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: SOLR replication question?

[Shawn Heisey]

 I am currently using SOLR 4.4. but not planning to use solrcloud in very
near
 future.
 I have 3 master / 3 slave setup. Each master is linked to its
 corresponding
 slave.. I have disabled auto polling..
 We do both push (using MQ) and pull indexing using SOLRJ indexing
program.
 I have enabled soft commit in slave (to view the changes immediately pushed
 by queue).
 I am thinking of doing the batch indexing in master (optimize and hard
commit) and push indexing in both master / slave.
 I am trying to do more testing with my configuration but thought of getting
 to know some answers before diving very deep...
 Since the queue pushes the docs in master / slave there is a possibility of
 slave having more record compared to master (when master is busy doing
batch
 indexing).. What would happen if the slave has additional segments compared
 to Master. will that be deleted when the replication happens.
 If a message is pushed from a queue to both master and slave during
replication, will there be a latency in seeing that document even if we
use
 softcommit in slave?
 We want to make sure that we are not missing any documents from queue
(since
 its updated via UI and we don't really store that data anywhere except
in
 index)

If you are doing replication, then all updates must go to the master
server. You cannot update the slave directly. The replication happens, the
slave will be identical to the master... Any documents aent to only the
slave will be lost.

Replication will happen according to the interval you have configured, or
since you say you have disabled polling, according to whatever schedule
you manually trigger a replication.

SolrCloud would probably be a better fit for you. With a properly
configured SolrCloud you just index to any host in the cloud and documents
end up exactly where they need to go, and all replicas get updated.

Thanks,
Shawn

Re: Solr replication is extremely slow(less then 1MB/s)

[William Bell]

I agree. It is even slower when the slave is being pounded.


On Fri, Jun 21, 2013 at 3:35 AM, Ted zhanghailian...@qq.com wrote:

 Solr replication is extremely slow(less then 1MB/s)

 When the replication is runinng,network and disk occupancy rate remained at
 a very low level.

 I've tried downloading a piece of the index file with browser.
 like this:

 http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file=
 
 http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file=
 

 it can reach more then 10MB/s.

 so i'm confused,is there anybody can tell me how to check for this problem?

 Environment：
 1.gigabit network
 2.both master  slave run on windows
 3.jdk 1.6u24 i586
 4.solr 3.5
 5.tomcat7

 Looking forward to your reply,tks!



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-replication-is-extremely-slow-less-then-1MB-s-tp4072096.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Solr Replication

[Erick Erickson]

Why do you think that during a backup any write activity would corrupt your
backup? Solr backs up live indexes all the time, that's what's happening,
in effect, when you replicate from a master to a slave. There's no
requirement that the master stop indexing. The replication backup command
Ahmet told you about is using the same mechanism.

Solr/Lucene index segments are write-once. backup/replication essentially
copies closed segments around which, since they aren't changing, aren't
affected by other indexing activity.

So the process is this:

1 a backup/replication starts. At this point, a list is made of all the
currently-closed segments and these segments will NOT be deleted until the
backup/replication is complete.
2 all the segments in 1 are copied. This is a complete snapshot of your
index.
3 after all the segments are copied, they are released and may then be
deleted in the normal merge process.

Of course another way to do backups would be set up a machine that just
does a replication (think of it as a special slave) periodically. Another
would be to have the master keep backups automatically, seesee:
http://wiki.apache.org/solr/SolrReplication,  and look for numberOfBackups.

Best
Erick


On Fri, Mar 15, 2013 at 1:03 AM, vicky desai vicky.de...@germinait.comwrote:

 Hi,

 I have a multi core setup and there is continuous updation going on in each
 core. Hence I dont prefer a bckup as it would either cause a downtime or if
 during a backup there is a write activity my backup will be corrupted. Can
 you please suggest if there is a cleaner way to handle this



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication

[Ahmet Arslan]

Hi Vicky,

May be   str name=replicateAfterstartup/str ?

For backups http://master_host:port/solr/replication?command=backup would be 
more suitable.

or str name=backupAfterstartup/str


--- On Thu, 3/14/13, vicky desai vicky.de...@germinait.com wrote:

 From: vicky desai vicky.de...@germinait.com
 Subject: Solr Replication
 To: solr-user@lucene.apache.org
 Date: Thursday, March 14, 2013, 9:20 AM
 Hi,
 
 I am using solr 4 setup. For the backup purpose once in a
 day I start one
 additional tomcat server with cores having empty data
 folders and which acts
 as a slave server. However it does not replicate data from
 the master unless
 there is a commit on the master. Is there a possibility to
 pull data from
 master core without firing a commit operation on that core 
 
 
 
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266.html
 Sent from the Solr - User mailing list archive at
 Nabble.com.
 

Re: Solr Replication

[vicky desai]

Hi,

I have a multi core setup and there is continuous updation going on in each
core. Hence I dont prefer a bckup as it would either cause a downtime or if
during a backup there is a write activity my backup will be corrupted. Can
you please suggest if there is a cleaner way to handle this



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Victor Ruiz]

After upgrading to 4.2, the problem is not yet solved, in this image you can
see, how slow is the transfer speed. At least, after the update the master
is not blocked during replication
http://lucene.472066.n3.nabble.com/file/n4046951/solr_replication.jpg 

Any idea?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046951.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Victor Ruiz]

While looking at Solr logs, I found a java.lang.OutOfMemoryError: Java heap
space that was happening 2 times per hour 
So I tried to increase the max memory heap assigned to JVM (-Xmx) and since
then  the servers are not crashing, even though the replication takes still
long time to complete. But for now, the 2 slaves can handle with no problems
all the queries.


Regards,
Victor



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046993.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Mark Miller]

Are you using Solr 4.1?

- Mark

On Mar 11, 2013, at 1:53 PM, Victor Ruiz bik1...@gmail.com wrote:

 Hi guys,
 
 I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
 master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
 every server, there are 2 independent instances of solr running (I tried
 also multicore config, but having independent instances has for me better
 performance), every instance having a differente collection. So, we have 2
 masters in server 1, and 2 slaves in server 2.
 
 Index size is currently (for the biggest collection) around 17 million
 documents, with a total size near 12 GB. The files transferred every
 replication cycle are typically not more than 100, with a total size not
 bigger than 50MB. The other collection is not that big, just around 1
 million docs and not bigger than 2 GB and not a high update ratio. The big
 collection has a load around 200 queries per second (MoreLikeThis,
 RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
 is below 50 queries per second
 
 Replication has been working for long time with any problem, but in the last
 weeks the replication cycles started to take long and long time for the big
 collection, even more than 2 minutes, some times even more. During that
 time, slaves are so overloaded, that many queries are timing out, despite
 the timeout in my clients is 30 seconds. 
 
 The servers are in same LAN, gigabit ethernet, so the broadband should not
 be the bottleneck.
 
 Since the index is receiving frequents updates and deletes (update handler
 receives more than 200 request per second for the big collection, but not
 more than 5 per second for the small one), I tried to use the
 maxCommitsToKeep attribute, to ensure that no file was deleted during
 replication, but it has no effect. 
 
 My solrconfig.xml in the big collection is like that:
 
 ?xml version=1.0 encoding=UTF-8 ?
 
 config
 
   luceneMatchVersionLUCENE_40/luceneMatchVersion
 
   directoryFactory name=DirectoryFactory
 
 class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
 
 
   indexConfig
   mergeFactor3/mergeFactor
 
   deletionPolicy class=solr.SolrDeletionPolicy
   
   str name=maxCommitsToKeep10/str
   str name=maxOptimizedCommitsToKeep1/str
   
   str name=maxCommitAge6HOUR/str
 
   /deletionPolicy
 
   /indexConfig
 
   jmx/
 
   updateHandler class=solr.DirectUpdateHandler2
 
   autoCommit
   maxDocs2000/maxDocs
   maxTime3/maxTime
   /autoCommit
 
   autoSoftCommit
   maxTime500/maxTime
   /autoSoftCommit
 
   updateLog
   str name=dir${solr.data.dir:}/str
   /updateLog
 
   /updateHandler
 
   query
   maxBooleanClauses2048/maxBooleanClauses
 
   filterCache
   class=solr.FastLRUCache
   size=2048
   initialSize=1024
   autowarmCount=1024/
 
   queryResultCache
   class=solr.LRUCache
   size=2048
   initialSize=1024
   autowarmCount=1024/
 
   
   documentCache
   class=solr.LRUCache
   size=2048
   initialSize=1024
   autowarmCount=1024/
 
   enableLazyFieldLoadingtrue/enableLazyFieldLoading
 
   queryResultWindowSize50/queryResultWindowSize
 
   queryResultMaxDocsCached50/queryResultMaxDocsCached
 
   listener event=newSearcher class=solr.QuerySenderListener
   arr name=queries
   lst
   str name=q*:*/str
   str name=fqdate:[NOW/DAY-7DAY TO 
 NOW/DAY+1DAY]/str
   str name=rows1000/str
   /lst
   /arr
   /listener
   listener event=firstSearcher 
 class=solr.QuerySenderListener
   arr name=queries
   lst
   str name=q*:*/str
   str name=fqdate:[NOW/DAY-7DAY TO 
 NOW/DAY+1DAY]/str
   str name=rows1000/str
   /lst
   /arr
   /listener
 
   useColdSearchertrue/useColdSearcher
 
   maxWarmingSearchers4/maxWarmingSearchers
   /query
 

Re: Solr replication takes long time

[Victor Ruiz]

no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
issue with the replication, so I decided not to try it for now


Mark Miller-3 wrote
 Are you using Solr 4.1?
 
 - Mark
 
 On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;

 bik1979@

 gt; wrote:
 
 Hi guys,
 
 I have a problem with Solr replication. I have 2 solr servers (Solr
 4.0.0) 1
 master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
 every server, there are 2 independent instances of solr running (I tried
 also multicore config, but having independent instances has for me better
 performance), every instance having a differente collection. So, we have
 2
 masters in server 1, and 2 slaves in server 2.
 
 Index size is currently (for the biggest collection) around 17 million
 documents, with a total size near 12 GB. The files transferred every
 replication cycle are typically not more than 100, with a total size not
 bigger than 50MB. The other collection is not that big, just around 1
 million docs and not bigger than 2 GB and not a high update ratio. The
 big
 collection has a load around 200 queries per second (MoreLikeThis,
 RealTimeGetHandler , TermVectorComponent mainly), and for the small one
 it
 is below 50 queries per second
 
 Replication has been working for long time with any problem, but in the
 last
 weeks the replication cycles started to take long and long time for the
 big
 collection, even more than 2 minutes, some times even more. During that
 time, slaves are so overloaded, that many queries are timing out, despite
 the timeout in my clients is 30 seconds. 
 
 The servers are in same LAN, gigabit ethernet, so the broadband should
 not
 be the bottleneck.
 
 Since the index is receiving frequents updates and deletes (update
 handler
 receives more than 200 request per second for the big collection, but not
 more than 5 per second for the small one), I tried to use the
 maxCommitsToKeep attribute, to ensure that no file was deleted during
 replication, but it has no effect. 
 
 My solrconfig.xml in the big collection is like that:
 
 ?xml version=1.0 encoding=UTF-8 ?
 
 
 config
 
  
 luceneMatchVersion
 LUCENE_40
 /luceneMatchVersion
 
  
 directoryFactory name=DirectoryFactory

 
 class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
 
 
  
 indexConfig
  
 mergeFactor
 3
 /mergeFactor
 
  
 deletionPolicy class=solr.SolrDeletionPolicy
  
  
 str name=maxCommitsToKeep
 10
 /str
  
 str name=maxOptimizedCommitsToKeep
 1
 /str
  
  
 str name=maxCommitAge
 6HOUR
 /str
 
  
 /deletionPolicy
 
  
 /indexConfig
 
  
 jmx/
 
  
 updateHandler class=solr.DirectUpdateHandler2
 
  
 autoCommit
  
 maxDocs
 2000
 /maxDocs
  
 maxTime
 3
 /maxTime
  
 /autoCommit
 
  
 autoSoftCommit
  
 maxTime
 500
 /maxTime
  
 /autoSoftCommit
 
  
 updateLog
  
 str name=dir
 ${solr.data.dir:}
 /str
  
 /updateLog
 
  
 /updateHandler
 
  
 query
  
 maxBooleanClauses
 2048
 /maxBooleanClauses
 
  
 filterCache

   class=solr.FastLRUCache
  size=2048
  initialSize=1024
  autowarmCount=1024/
 
  
 queryResultCache

   class=solr.LRUCache
  size=2048
  initialSize=1024
  autowarmCount=1024/
 
  
  
 documentCache

   class=solr.LRUCache
  size=2048
  initialSize=1024
  autowarmCount=1024/
 
  
 enableLazyFieldLoading
 true
 /enableLazyFieldLoading
 
  
 queryResultWindowSize
 50
 /queryResultWindowSize
 
  
 queryResultMaxDocsCached
 50
 /queryResultMaxDocsCached
 
  
 listener event=newSearcher class=solr.QuerySenderListener
  
 arr name=queries
  
 lst
  
 str name=q
 *:*
 /str
  
 str name=fq
 date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
 /str
  
 str name=rows
 1000
 /str
  
 /lst
  
 /arr
  
 /listener
  
 listener event=firstSearcher class=solr.QuerySenderListener
  
 arr name=queries
  
 lst
  
 str name=q
 *:*
 /str
  
 str name=fq
 date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
 

Re: Solr replication takes long time

[Mark Miller]

Okay - yes, 4.0 is a better choice for replication than 4.1.

It almost sounds like you may be replicating the full index rather than just 
changes or something. 4.0 had a couple issues as well - a couple things that 
were discovered while writing stronger tests for 4.2.

4.2 is spreading onto mirrors now.

- Mark

On Mar 11, 2013, at 2:00 PM, Victor Ruiz bik1...@gmail.com wrote:

 no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
 issue with the replication, so I decided not to try it for now
 
 
 Mark Miller-3 wrote
 Are you using Solr 4.1?
 
 - Mark
 
 On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;
 
 bik1979@
 
 gt; wrote:
 
 Hi guys,
 
 I have a problem with Solr replication. I have 2 solr servers (Solr
 4.0.0) 1
 master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
 every server, there are 2 independent instances of solr running (I tried
 also multicore config, but having independent instances has for me better
 performance), every instance having a differente collection. So, we have
 2
 masters in server 1, and 2 slaves in server 2.
 
 Index size is currently (for the biggest collection) around 17 million
 documents, with a total size near 12 GB. The files transferred every
 replication cycle are typically not more than 100, with a total size not
 bigger than 50MB. The other collection is not that big, just around 1
 million docs and not bigger than 2 GB and not a high update ratio. The
 big
 collection has a load around 200 queries per second (MoreLikeThis,
 RealTimeGetHandler , TermVectorComponent mainly), and for the small one
 it
 is below 50 queries per second
 
 Replication has been working for long time with any problem, but in the
 last
 weeks the replication cycles started to take long and long time for the
 big
 collection, even more than 2 minutes, some times even more. During that
 time, slaves are so overloaded, that many queries are timing out, despite
 the timeout in my clients is 30 seconds. 
 
 The servers are in same LAN, gigabit ethernet, so the broadband should
 not
 be the bottleneck.
 
 Since the index is receiving frequents updates and deletes (update
 handler
 receives more than 200 request per second for the big collection, but not
 more than 5 per second for the small one), I tried to use the
 maxCommitsToKeep attribute, to ensure that no file was deleted during
 replication, but it has no effect. 
 
 My solrconfig.xml in the big collection is like that:
 
 ?xml version=1.0 encoding=UTF-8 ?
 
 
 config
 
 
 luceneMatchVersion
 LUCENE_40
 /luceneMatchVersion
 
 
 directoryFactory name=DirectoryFactory
 

 class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
 
 
 
 indexConfig
 
 mergeFactor
 3
 /mergeFactor
 
 
 deletionPolicy class=solr.SolrDeletionPolicy
 
 
 str name=maxCommitsToKeep
 10
 /str
 
 str name=maxOptimizedCommitsToKeep
 1
 /str
 
 
 str name=maxCommitAge
 6HOUR
 /str
 
 
 /deletionPolicy
 
 
 /indexConfig
 
 
 jmx/
 
 
 updateHandler class=solr.DirectUpdateHandler2
 
 
 autoCommit
 
 maxDocs
 2000
 /maxDocs
 
 maxTime
 3
 /maxTime
 
 /autoCommit
 
 
 autoSoftCommit
 
 maxTime
 500
 /maxTime
 
 /autoSoftCommit
 
 
 updateLog
 
 str name=dir
 ${solr.data.dir:}
 /str
 
 /updateLog
 
 
 /updateHandler
 
 
 query
 
 maxBooleanClauses
 2048
 /maxBooleanClauses
 
 
 filterCache
 
  class=solr.FastLRUCache
 size=2048
 initialSize=1024
 autowarmCount=1024/
 
 
 queryResultCache
 
  class=solr.LRUCache
 size=2048
 initialSize=1024
 autowarmCount=1024/
 
 
 
 documentCache
 
  class=solr.LRUCache
 size=2048
 initialSize=1024
 autowarmCount=1024/
 
 
 enableLazyFieldLoading
 true
 /enableLazyFieldLoading
 
 
 queryResultWindowSize
 50
 /queryResultWindowSize
 
 
 queryResultMaxDocsCached
 50
 /queryResultMaxDocsCached
 
 
 listener event=newSearcher class=solr.QuerySenderListener
 
 arr name=queries
 
 lst
 
 str name=q
 *:*
 /str
 
 str name=fq
 date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
 /str
 
 str name=rows
 1000
 /str
 
 /lst
 

Re: Solr replication takes long time

[Victor Ruiz]

Thanks for your answer Mark. I think I'll try to update to 4.2. I'll keep you
updated.

Anyway, I'd not say that the full index is replicated, I've been monitoring
the replication process in the Solr admin console and there I see that
usually not more than 50-100 files are transferrend, the total size is
rarely greater than 50MB. Is this info trustable?

Victor

Mark Miller-3 wrote
 Okay - yes, 4.0 is a better choice for replication than 4.1.
 
 It almost sounds like you may be replicating the full index rather than
 just changes or something. 4.0 had a couple issues as well - a couple
 things that were discovered while writing stronger tests for 4.2.
 
 4.2 is spreading onto mirrors now.
 
 - Mark
 
 On Mar 11, 2013, at 2:00 PM, Victor Ruiz lt;

 bik1979@

 gt; wrote:
 
 no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was
 an
 issue with the replication, so I decided not to try it for now
 
 
 Mark Miller-3 wrote
 Are you using Solr 4.1?
 
 - Mark
 
 On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;
 
 bik1979@
 
 gt; wrote:
 
 Hi guys,
 
 I have a problem with Solr replication. I have 2 solr servers (Solr
 4.0.0) 1
 master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
 every server, there are 2 independent instances of solr running (I
 tried
 also multicore config, but having independent instances has for me
 better
 performance), every instance having a differente collection. So, we
 have
 2
 masters in server 1, and 2 slaves in server 2.
 
 Index size is currently (for the biggest collection) around 17 million
 documents, with a total size near 12 GB. The files transferred every
 replication cycle are typically not more than 100, with a total size
 not
 bigger than 50MB. The other collection is not that big, just around 1
 million docs and not bigger than 2 GB and not a high update ratio. The
 big
 collection has a load around 200 queries per second (MoreLikeThis,
 RealTimeGetHandler , TermVectorComponent mainly), and for the small one
 it
 is below 50 queries per second
 
 Replication has been working for long time with any problem, but in the
 last
 weeks the replication cycles started to take long and long time for the
 big
 collection, even more than 2 minutes, some times even more. During that
 time, slaves are so overloaded, that many queries are timing out,
 despite
 the timeout in my clients is 30 seconds. 
 
 The servers are in same LAN, gigabit ethernet, so the broadband should
 not
 be the bottleneck.
 
 Since the index is receiving frequents updates and deletes (update
 handler
 receives more than 200 request per second for the big collection, but
 not
 more than 5 per second for the small one), I tried to use the
 maxCommitsToKeep attribute, to ensure that no file was deleted during
 replication, but it has no effect. 
 
 My solrconfig.xml in the big collection is like that:
 
 ?xml version=1.0 encoding=UTF-8 ?
 
 
 
 config
 

 
 luceneMatchVersion
 LUCENE_40
 
 /luceneMatchVersion
 

 
 directoryFactory name=DirectoryFactory

 
  
 class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
 
 

 
 indexConfig

 
 mergeFactor
 3
 
 /mergeFactor
 

 
 deletionPolicy class=solr.SolrDeletionPolicy


 
 str name=maxCommitsToKeep
 10
 
 /str

 
 str name=maxOptimizedCommitsToKeep
 1
 
 /str


 
 str name=maxCommitAge
 6HOUR
 
 /str
 

 
 /deletionPolicy
 

 
 /indexConfig
 

 
 jmx/
 

 
 updateHandler class=solr.DirectUpdateHandler2
 

 
 autoCommit

 
 maxDocs
 2000
 
 /maxDocs

 
 maxTime
 3
 
 /maxTime

 
 /autoCommit
 

 
 autoSoftCommit

 
 maxTime
 500
 
 /maxTime

 
 /autoSoftCommit
 

 
 updateLog

 
 str name=dir
 ${solr.data.dir:}
 
 /str

 
 /updateLog
 

 
 /updateHandler
 

 
 query

 
 maxBooleanClauses
 2048
 
 /maxBooleanClauses
 

 
 filterCache

 
 class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
 

 
 queryResultCache

 
 class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
 


 
 documentCache

 
 class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
 

 
 enableLazyFieldLoading
 true
 
 /enableLazyFieldLoading
 

 
 queryResultWindowSize
 50
 
 /queryResultWindowSize
 

 
 queryResultMaxDocsCached
 50
 
 

Re: Solr replication

[Erick Erickson]

BTW, when I said I think to say anything intelligent I meant I think for
_me_ to say anything intelligent. Didn't mean to be snarky

Erick


On Mon, Nov 26, 2012 at 8:19 AM, Erick Erickson erickerick...@gmail.comwrote:

 both servers are master and slave in the config file. So that means
 they're polling each other? I think to say anything intelligent you're
 going to have to provide more data. Please review:
 http://wiki.apache.org/solr/UsingMailingLists.

 The bottom line here is that it sounds like what you're trying to do is
 unusual enough that it's not a typical use case. I further suspect that
 what you're trying to do will cause problems you haven't tested for.

 So I wouldn't do this. Simply configuring a proper master/slave setup will
 give you HA out of the box. If a master goes down there's some complexity
 in promoting one of your slaves and indexing any stale data.

 But I'd go with SolrCloud to fire and forget rather than try to force
 Master/Slave to do something it wasn't intended to do.

 Best
 Erick


 On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes 
 jacques.cor...@laposte.net wrote:

 Thanks Antoine.

 In fact, both solr servers are master and slave in config file.
 They have the same config file and replicate from the same master url with
 the vip.
 So, the master is on the server with the vip mounted on.
 And if heartbeat toggle, the role is toggled too.

 The question is : what can happend in this configuration?

 With tests, I can't see nothing bad, but I don't know the intern
 mecanisms.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication

[jacques.cortes]

Thanks Antoine.

In fact, both solr servers are master and slave in config file.
They have the same config file and replicate from the same master url with
the vip.
So, the master is on the server with the vip mounted on.
And if heartbeat toggle, the role is toggled too.

The question is : what can happend in this configuration?

With tests, I can't see nothing bad, but I don't know the intern mecanisms.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication

[Erick Erickson]

both servers are master and slave in the config file. So that means
they're polling each other? I think to say anything intelligent you're
going to have to provide more data. Please review:
http://wiki.apache.org/solr/UsingMailingLists.

The bottom line here is that it sounds like what you're trying to do is
unusual enough that it's not a typical use case. I further suspect that
what you're trying to do will cause problems you haven't tested for.

So I wouldn't do this. Simply configuring a proper master/slave setup will
give you HA out of the box. If a master goes down there's some complexity
in promoting one of your slaves and indexing any stale data.

But I'd go with SolrCloud to fire and forget rather than try to force
Master/Slave to do something it wasn't intended to do.

Best
Erick


On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes
jacques.cor...@laposte.netwrote:

 Thanks Antoine.

 In fact, both solr servers are master and slave in config file.
 They have the same config file and replicate from the same master url with
 the vip.
 So, the master is on the server with the vip mounted on.
 And if heartbeat toggle, the role is toggled too.

 The question is : what can happend in this configuration?

 With tests, I can't see nothing bad, but I don't know the intern mecanisms.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[deniz]

Erik Hatcher-4 wrote
 There's an open issue (with a patch!) that enables this, it seems:
 lt;https://issues.apache.org/jira/browse/SOLR-3911gt;
 
   Erik

well patch seems not doing that... i have tried and still getting some error
lines about the dir types




-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018670.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Erik Hatcher]

There's an open issue (with a patch!) that enables this, it seems: 
https://issues.apache.org/jira/browse/SOLR-3911

Erik

On Nov 5, 2012, at 07:41 , deniz wrote:

 Michael Della Bitta-2 wrote
 No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
 ends up storing the index in RAM and more efficiently so, plus it's
 backed by disk.
 
 Just be sure to not set a big heap because MMapDirectory works outside of
 heap.
 
 for my tests, i dont think index is ended up in ram with mmap... i gave
 4gigs for heap while using mmap and got mapping error while indexing...
 while index should be something around 2 gigs, ram consumption was around
 300mbs... 
 
 Can anyone explain why RAMDirectory cant be used for replication? I cant see
 why the master is set for using RAMDirectory and replica is using MMap or
 some other? As far as I understand SolrCloud is some kinda pushing from
 master to replica/slave... so why it is not possible to push from RAM to
 HDD? If my logic is wrong, someone can please explain me all these? 
 
 
 
 -
 Zeki ama calismiyor... Calissa yapar...
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[deniz]

Erik Hatcher-4 wrote
 There's an open issue (with a patch!) that enables this, it seems:
 lt;https://issues.apache.org/jira/browse/SOLR-3911gt;

 i will check it for sure, thank you Erik :) 


Shawn Heisey-4 wrote
 ... transparently mapping the files on disk to a virtual memory space and
 using excess RAM to cache that data and make it fast.  If you have
 enough extra memory (disk cache) to fit the entire index, the OS will
 never have to read any part of the index from disk more than once

so for disk cache, are there any disks with 1 gigs or more of caches? if am
not wrong there are mostly 16 or 32mb cache disks around (or i am checking
the wrong stuff? ) if so, that amount definitely too small... 





-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Shawn Heisey]

 Shawn Heisey-4 wrote
 ... transparently mapping the files on disk to a virtual memory space
 and
 using excess RAM to cache that data and make it fast.  If you have
 enough extra memory (disk cache) to fit the entire index, the OS will
 never have to read any part of the index from disk more than once

 so for disk cache, are there any disks with 1 gigs or more of caches? if
 am
 not wrong there are mostly 16 or 32mb cache disks around (or i am checking
 the wrong stuff? ) if so, that amount definitely too small...


I am not talking about the cache on the actual disk drive, or even cache
on your hard drive controller. I am talking about the operating system
using RAM, specifically RAM not being used by programs, to cache data on
your hard drive. All modern operating systems do it, even the one made in
Redmond that people love to hate.

If you have 16 GB of RAM and all your programs use up 4.5 GB, you can
count on the OS using at least another half GB, so you have about 11 GB
left. The OS is going to put data that it reads and writes to/from your
disk in this space. If you start up another program that wants 2GB, the OS
will simply throw away 2 GB of data in its cache (it's still on the disk,
after all) and give that RAM to the new program.


Solr counts on this OS capability for good performance.

Thanks,
Shawn

Re: Solr Replication is not Possible on RAMDirectory?

[Michael Della Bitta]

Here's some reading:

http://en.wikipedia.org/wiki/Page_cache

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Mon, Nov 5, 2012 at 8:02 PM, deniz denizdurmu...@gmail.com wrote:
 Erik Hatcher-4 wrote
 There's an open issue (with a patch!) that enables this, it seems:
 lt;https://issues.apache.org/jira/browse/SOLR-3911gt;

  i will check it for sure, thank you Erik :)


 Shawn Heisey-4 wrote
 ... transparently mapping the files on disk to a virtual memory space and
 using excess RAM to cache that data and make it fast.  If you have
 enough extra memory (disk cache) to fit the entire index, the OS will
 never have to read any part of the index from disk more than once

 so for disk cache, are there any disks with 1 gigs or more of caches? if am
 not wrong there are mostly 16 or 32mb cache disks around (or i am checking
 the wrong stuff? ) if so, that amount definitely too small...





 -
 Zeki ama calismiyor... Calissa yapar...
 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[deniz]

Michael Della Bitta-2 wrote
 No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
 ends up storing the index in RAM and more efficiently so, plus it's
 backed by disk.
 
 Just be sure to not set a big heap because MMapDirectory works outside of
 heap.

for my tests, i dont think index is ended up in ram with mmap... i gave
4gigs for heap while using mmap and got mapping error while indexing...
while index should be something around 2 gigs, ram consumption was around
300mbs... 

Can anyone explain why RAMDirectory cant be used for replication? I cant see
why the master is set for using RAMDirectory and replica is using MMap or
some other? As far as I understand SolrCloud is some kinda pushing from
master to replica/slave... so why it is not possible to push from RAM to
HDD? If my logic is wrong, someone can please explain me all these? 



-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Shawn Heisey]

On 11/4/2012 11:41 PM, deniz wrote:

Michael Della Bitta-2 wrote

No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
ends up storing the index in RAM and more efficiently so, plus it's
backed by disk.

Just be sure to not set a big heap because MMapDirectory works outside of
heap.

for my tests, i dont think index is ended up in ram with mmap... i gave
4gigs for heap while using mmap and got mapping error while indexing...
while index should be something around 2 gigs, ram consumption was around
300mbs...


With mmap, the ram is not actually consumed by your application, which 
in this case is Java. The operating system is handling it -- 
transparently mapping the files on disk to a virtual memory space and 
using excess RAM to cache that data and make it fast.  If you have 
enough extra memory (disk cache) to fit the entire index, the OS will 
never have to read any part of the index from disk more than once.  With 
RAMDirectory, the index has to go into the Java heap, which is much less 
efficient at memory management than the native operating system.



Can anyone explain why RAMDirectory cant be used for replication? I cant see
why the master is set for using RAMDirectory and replica is using MMap or
some other? As far as I understand SolrCloud is some kinda pushing from
master to replica/slave... so why it is not possible to push from RAM to
HDD? If my logic is wrong, someone can please explain me all these?


With RAMDirectory, there are no files to copy.  Replication does not 
copy Solr (Lucene) documents, it copies files.


Thanks,
Shawn

Re: Solr Replication is not Possible on RAMDirectory?

[Michael Della Bitta]

 so it is not possible to use RAMdirectory for replication?

No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
ends up storing the index in RAM and more efficiently so, plus it's
backed by disk.

Just be sure to not set a big heap because MMapDirectory works outside of heap.

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Fri, Nov 2, 2012 at 4:44 AM, deniz denizdurmu...@gmail.com wrote:
 so it is not possible to use RAMdirectory for replication?

Re: solr replication against active indexing on master

[Jie Sun]

thanks ...

could you please point me to some more detailed explanation on line or I
will have to read the code to find out? I would like to understand a little
more on how this is achieved. thanks!

Jie



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-against-active-indexing-on-master-tp4017696p4017707.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication against active indexing on master

[Otis Gospodnetic]

Hi,

I don't have sources handy but you could start from SnapPuller.Java.

Otis
--
Performance Monitoring - http://sematext.com/spm
On Nov 1, 2012 6:18 PM, Jie Sun jsun5...@yahoo.com wrote:

 thanks ...

 could you please point me to some more detailed explanation on line or I
 will have to read the code to find out? I would like to understand a little
 more on how this is achieved. thanks!

 Jie



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-against-active-indexing-on-master-tp4017696p4017707.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication against active indexing on master

[Jie Sun]

thanks... I just read the related code ... now I understand it seems the
master keeps replicable snapshots (version), so it should be static. thank
you Otis!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-against-active-indexing-on-master-tp4017696p4017743.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication hangs on multiple slave nodes

[Otis Gospodnetic]

Hi,

I haven't seen this error before.

Some questions/suggestions...
Have you tried with 3.6.1?
Is the disk full?
Have you tried watching the network with
http://code.google.com/p/tcpmon/ or tcpdump?

Otis
--
Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/index.html


On Thu, Oct 4, 2012 at 12:06 PM, Justin Babuscio
jbabus...@linchpinsoftware.com wrote:
 After a large index rebuild (16-masters with ~15GB each), some slaves fail
 to completely replicate.

 We are running Solr v3.5 with 16 masters and 2 slaves each for a total of
 48 servers.

 4 of the 32 slaves sit in a stalled replication state with similar messages:

 Files Downloaded:  254/260
 Downloaded: 12.09 GB / 12.09 GB [ 100% ]
 Downloading File: _t6.fdt, Downloaded: 3.1 MB / 3.1 MB [ 100 % ]
 Time Elapsed: 3215s, EStimated Time REmaining: 0s, Speed: 24.5 MB/s


 As you'll notice, all download sizes appear to be complete but the files
 downloaded are not.  This also prevents the servers from polling for a new
 update from the masters.  When searching, we are occasionally seeing 500
 responses from the slaves that fail to replicate.  The errors are

 ArrayIndexOutOfBounds - this occurs when writing the HTTP Response (our
 container is WebSphere)
 NullPointerExceptions - org.apache.lucnee.queryParser.QueryParser.parse
 (QueryParser.java:203 )

 We have tried to stop the slave, delete the /data directory, and restart.
  This started downloading the index but stalled as expected.

 Thanks,
 Justin

Re: Solr Replication and Autocommit

[Erick Erickson]

I'll echo Otis, nothing comes to mind...

Unless you were indexing stuff to the _slaves_, which you should
never do, now or in the past

Erick

On Thu, Sep 27, 2012 at 12:00 AM, Aleksey Vorona avor...@ea.com wrote:
 Hi,

 I remember having some issues with replication and autocommit previously.
 But now we are using Solr 3.6.1. Are there any known issues or any other
 reasons to avoid autocommit while using replication? I guess not, just want
 confirmation from someone confident and competent.

 -- Aleksey

Re: Solr Replication and Autocommit

[Aleksey Vorona]

Thank both of you for the responses!

-- Aleksey

On 12-09-27 03:51 AM, Erick Erickson wrote:

I'll echo Otis, nothing comes to mind...

Unless you were indexing stuff to the _slaves_, which you should
never do, now or in the past

Erick

On Thu, Sep 27, 2012 at 12:00 AM, Aleksey Vorona avor...@ea.com wrote:

Hi,

I remember having some issues with replication and autocommit previously.
But now we are using Solr 3.6.1. Are there any known issues or any other
reasons to avoid autocommit while using replication? I guess not, just want
confirmation from someone confident and competent.

-- Aleksey

Re: Solr Replication and Autocommit

[Otis Gospodnetic]

No issues that I know of, never encountered such an issue with any if our
clients using Solr.

Otis
--
Performance Monitoring - http://sematext.com/spm
On Sep 27, 2012 12:00 AM, Aleksey Vorona avor...@ea.com wrote:

 Hi,

 I remember having some issues with replication and autocommit previously.
 But now we are using Solr 3.6.1. Are there any known issues or any other
 reasons to avoid autocommit while using replication? I guess not, just want
 confirmation from someone confident and competent.

 -- Aleksey

Re: solr replication lag

[Michael Della Bitta]

Hello, Boris,

If I remember correctly, older versions of Solr report the version of
the as-of-yet uncommitted core in the replication page. So if you did
a commit on the master and then a replication, you'd see that version
on the client.

Michael Della Bitta


Appinions, Inc. -- Where Influence Isn’t a Game.
http://www.appinions.com


On Thu, Jun 7, 2012 at 3:53 AM, Boris Vorotnikov bori...@auto.ru wrote:
 Hello,

 My name is Boris Vorotnikov. I am a developer of project parts.auto.ru. My 
 team uses solr v.3.5 in this project. Several days ago I noticed that 
 replication between master and slave had a time lag. There was no such lags 
 before. I tried to find the source of trouble but it was unsuccessfully. All 
 I found is that master tells more recent version of index than it has. And 
 when I press the button Replicate now on slave it receive information about 
 index that it already has and do nothing.

 This is configuration of master and slave replication:
 requestHandler name=/replication class=solr.ReplicationHandler 
    lst name=master
        str name=enable${enable.master:false}/str
        str name=replicateAfterstartup/str
        str name=replicateAfteroptimize/str
        str 
 name=confFilesschema.xml,stopwords.txt,synonyms.txt,spellings.txt/str
    /lst
    lst name=slave
        str name=enable${enable.slave:false}/str
        str 
 name=masterUrlhttp://__solr.master_url__:__solr.port__/solr/parts/replication/str
        str name=pollInterval01:00:00/str
    /lst
    str name=maxNumberOfBackups3/str
 /requestHandler

 enable.master - parameter of command line. It works.
 Solr replicates only once a day when we do index optimization by cron.
 Is there any parameter responsible for keeping index version or something 
 else?




 Best regards,
 Boris Vorotnikov
 Developer auto.ru

 Tel: +7 (499) 780 3780 # 424
 E-mail: bori...@auto.ru

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[geeky2]

hello,

sorry - i overlooked this message - thanks for checking back and thanks for
the info.

yes - replication seems to be working now:

tailed from logs just now:

2012-04-26 09:21:33,284 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-26 09:21:53,279 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-26 09:22:13,279 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-26 09:22:33,279 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.



 

--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3941447.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[Mark Miller]

On Apr 23, 2012, at 12:10 PM, geeky2 wrote:

 http://someip:someport/somepath/somecore/admin/replication/ is not
 available. Index fetch failed. Exception: Invalid version (expected 2, but
 10) or the data in not in 'javabin' format

This is kind of a bug. When Solr tries to talk in javabin and gets an http 
response instead (like a 404 response - what this likely is) it does this. 
Really it should detect this case and give you the proper error. I almost think 
someone made this change already in trunk based on what I was seeing yesterday, 
but I'm not sure.

- Mark Miller
lucidimagination.com

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[Rahul Warawdekar]

Hi,

Is the replication still failing or working fine with that change ?

On Tue, Apr 24, 2012 at 2:16 PM, geeky2 gee...@hotmail.com wrote:

 that was it!

 thank you.

 i did notice something else in the logs now ...

 what is the meaning or implication of the message, Connection reset.?



 2012-04-24 12:59:19,996 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 12:59:39,998 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 *2012-04-24 12:59:59,997 SEVERE [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Master at:
 http://bogus:bogusport/somepath/somecore/replication/ is not available.
 Index fetch failed. Exception: Connection reset*
 2012-04-24 13:00:19,998 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:00:40,004 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:00:59,992 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:01:19,993 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:01:39,992 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:01:59,989 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:02:19,990 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:02:39,989 INFO  [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Slave in sync with master.
 2012-04-24 13:02:59,991 INFO  [org.a

 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3936107.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Thanks and Regards
Rahul A. Warawdekar

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[geeky2]

hello,

thank you for the reply,

yes - master has been indexed.

ok - makes sense - the polling interval needs to change

i did check the solr war file on both boxes (master and slave).  they are
identical.  actually - if they were not indentical - this would point to a
different issue altogether - since our deployment infrastructure - rolls the
war file to the slaves when you do a deployment on the master.

this has me stumped - not sure what to check next.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3935699.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[Rahul Warawdekar]

Hi,

In Solr wiki, for replication, the master url is defined as follows
str name=masterUrlhttp://master_host:port
/solr/corename/replication/str

This url does not contain admin in its path where as in the master url
provided by you, you have an additional admin in the url.
Not very sure if this might be an issue but you can just check removing
admin and check if replication works.


On Tue, Apr 24, 2012 at 11:49 AM, geeky2 gee...@hotmail.com wrote:

 hello,

 thank you for the reply,

 yes - master has been indexed.

 ok - makes sense - the polling interval needs to change

 i did check the solr war file on both boxes (master and slave).  they are
 identical.  actually - if they were not indentical - this would point to a
 different issue altogether - since our deployment infrastructure - rolls
 the
 war file to the slaves when you do a deployment on the master.

 this has me stumped - not sure what to check next.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3935699.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Thanks and Regards
Rahul A. Warawdekar

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[geeky2]

that was it!

thank you.

i did notice something else in the logs now ...

what is the meaning or implication of the message, Connection reset.?



2012-04-24 12:59:19,996 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 12:59:39,998 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
*2012-04-24 12:59:59,997 SEVERE [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Master at:
http://bogus:bogusport/somepath/somecore/replication/ is not available.
Index fetch failed. Exception: Connection reset*
2012-04-24 13:00:19,998 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:00:40,004 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:00:59,992 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:01:19,993 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:01:39,992 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:01:59,989 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:02:19,990 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:02:39,989 INFO  [org.apache.solr.handler.SnapPuller]
(pool-12-thread-1) Slave in sync with master.
2012-04-24 13:02:59,991 INFO  [org.a

--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3936107.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication failing with error: Master at: is not available. Index fetch failed

[Erick Erickson]

Hmmm, does your master have an index? In other words have you
added anything to it? I actually doubt that's an issue, but

An aside, a polling interval of 20 seconds is rather short, beware of
your autowarming time exceeding your index updates

But my _first_ guess is that somehow you're Solrs aren't the same
version or you have a foo'd index on your master.

Best
Erick

On Mon, Apr 23, 2012 at 12:10 PM, geeky2 gee...@hotmail.com wrote:
 hello all,

 enviornment: centOS and solr 3.5

 i am attempting to set up replication betweeen two solr boxes (master and
 slave).

 i am getting the following in the logs on the slave box.

 2012-04-23 10:54:59,985 SEVERE [org.apache.solr.handler.SnapPuller]
 (pool-12-thread-1) Master at:
 http://someip:someport/somepath/somecore/admin/replication/ is not
 available. Index fetch failed. Exception: Invalid version (expected 2, but
 10) or the data in not in 'javabin' format

 master jvm (jboss host) is being started like this:

 -Denable.master=true

 slave jvm (jboss host) is being started like this:

 -Denable.slave=true

 does anyone have any ideas?

 i have done the following:

 used curl http://someip:someport/somepath/somecore/admin/replication/ from
 slave to successfully see master

 used ping from slave to master

 switched out the dns name for master to hard coded ip address

 made sure i can see
 http://someip:someport/somepath/somecore/admin/replication/ in a browser


 this is my request handler - i am using the same config file on both the
 master and slave - but sending in the appropriate switch on start up (per
 the solr wiki page on replication)

    lst name=master

      str name=enable${enable.master:false}/str
      str name=replicateAfterstartup/str
      str name=replicateAftercommit/str



      str name=confFilesschema.xml,stopwords.txt,elevate.xml/str

      str name=commitReserveDuration00:00:10/str
    /lst

    str name=maxNumberOfBackups1/str
    lst name=slave

      str name=enable${enable.slave:false}/str
      str
 name=masterUrlhttp://someip:someport/somecore/admin/replication//str

      str name=pollInterval00:00:20/str


      str name=compressioninternal/str

      str name=httpConnTimeout5000/str
      str name=httpReadTimeout1/str

    /lst
  /requestHandler


 any suggestions would be great

 thank you,
 mark



 --
 View this message in context: 
 http://lucene.472066.n3.nabble.com/solr-replication-failing-with-error-Master-at-is-not-available-Index-fetch-failed-tp3932921p3932921.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[darul]

Here is the way I see it (and implemented it), while using SolrJ api you have
to fire :

- Indexation commands to your /indexation solr instance/ (master) example :
http://myMaster:80/myCore/
- Query commands to your /search solr instance/ (slave). You may have
several slaves, and also find alternative as broker to make load balancing
betweeen each
http://mySlave1:80/myCore/
http://mySlave2:80/myCore/
...

You do not need any changes in code normally, replication is made
automatically and defined in your solrconfig.xml configuration file.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687168.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[Parvin Gasimzade]

Then as you say , shouldn't i define three SolrServer() using SolrJ?
For indexing call solrMasterServer, and for querying call solrSlaveServer1
or solrSlaveServer2?

On Wed, Jan 25, 2012 at 11:09 AM, darul daru...@gmail.com wrote:

 Here is the way I see it (and implemented it), while using SolrJ api you
 have
 to fire :

 - Indexation commands to your /indexation solr instance/ (master) example :
 http://myMaster:80/myCore/
 - Query commands to your /search solr instance/ (slave). You may have
 several slaves, and also find alternative as broker to make load balancing
 betweeen each
 http://mySlave1:80/myCore/
 http://mySlave2:80/myCore/
 ...

 You do not need any changes in code normally, replication is made
 automatically and defined in your solrconfig.xml configuration file.

 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687168.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[darul]

You may define your specific configuration as a Grid with all your solr
instances and then using SolrJ and CommonsHttpSolrServer choose the right
url depending on indexation or search task.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687208.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[Parvin Gasimzade]

Thank you for your response.
What you mean by Grid? Can you please send me any example or any link?


On Wed, Jan 25, 2012 at 11:30 AM, darul daru...@gmail.com wrote:

 You may define your specific configuration as a Grid with all your solr
 instances and then using SolrJ and CommonsHttpSolrServer choose the right
 url depending on indexation or search task.

 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687208.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[darul]

I mean by grid the list of your instances :

String masterUrl = http://masterUrl/core/...;
String[] slaveUrls = {http://slaveUrl/core/...;,
http://slaveUrl/core/...}

Then use your business logic to use the correct one with Http solrJ facade.

--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687314.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[Parvin Gasimzade]

Ok thank you for your response.

On Wed, Jan 25, 2012 at 12:24 PM, darul daru...@gmail.com wrote:

 I mean by grid the list of your instances :

 String masterUrl = http://masterUrl/core/...;
 String[] slaveUrls = {http://slaveUrl/core/...;,
 http://slaveUrl/core/...}

 Then use your business logic to use the correct one with Http solrJ facade.

 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/solr-replication-tp3687106p3687314.html
 Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr replication

[Anderson vasconcelos]

Hi Parvin

I did something that may help you. I set up apache (with mod_proxy and mode
balance) like a front-end and use this to distruted the request of my
aplication. Request for /update or /optmize, i'm redirect to master (or
masters) server and requests /search i redirect to slaves. Example:

Proxy balancer://solrclusterindex
BalancerMember http://127.0.0.1:8080/apache-solr-1.4.1/ disablereuse=On
route=jvm1
/Proxy

Proxy balancer://solrclustersearch
BalancerMember http://127.0.0.1:8080/apache-solr-1.4.1/ disablereuse=On
route=jvm1
BalancerMember http://10.16.129.61:8080/apache-solr-1.4.1/ disablereuse=On
route=jvm2
/Proxy

ProxyPassMatch /solrcluster(.*)/update(.*)$
balancer://solrclusterindex$1/update$2
ProxyPassMatch /solrcluster(.*)/select(.*)$
balancer://solrclustersearch$1/select$2

I hope it helps you

Re: solr replication

[Parvin Gasimzade]

Hi Anderson,

Thank you for your effort.I will try this.
Hope it will solve my problem.

Regards

On Wed, Jan 25, 2012 at 2:27 PM, Anderson vasconcelos 
anderson.v...@gmail.com wrote:

 Hi Parvin

 I did something that may help you. I set up apache (with mod_proxy and mode
 balance) like a front-end and use this to distruted the request of my
 aplication. Request for /update or /optmize, i'm redirect to master (or
 masters) server and requests /search i redirect to slaves. Example:

 Proxy balancer://solrclusterindex
 BalancerMember http://127.0.0.1:8080/apache-solr-1.4.1/ disablereuse=On
 route=jvm1
 /Proxy

 Proxy balancer://solrclustersearch
 BalancerMember http://127.0.0.1:8080/apache-solr-1.4.1/ disablereuse=On
 route=jvm1
 BalancerMember http://10.16.129.61:8080/apache-solr-1.4.1/ disablereuse=On
 route=jvm2
 /Proxy

 ProxyPassMatch /solrcluster(.*)/update(.*)$
 balancer://solrclusterindex$1/update$2
 ProxyPassMatch /solrcluster(.*)/select(.*)$
 balancer://solrclustersearch$1/select$2

 I hope it helps you

Re: Solr Replication: relative path in confFiles Element?

[Yury Kats]

On 10/25/2011 11:24 AM, Mark Schoy wrote:
 Hi,
 
 is ist possible to define a relative path in confFile?
 
 For example:
 
 str name=confFiles../../x.xml/str
 
 If yes, to which location will the file be copied at the slave?

I don;t think it's possible. Replication copies confFiles from master core's
confDir to slave core's confDir.

Re: Solr replication / repeater

[Rene Lehmann]

Hey Erik, Hey Chris,

thank you for your answers. I think i now understand better the
functioning.

kind regards

Rene

Re: Solr replication / repeater

[Rene Lehmann]

Hi,

no ideas? :(

kind regards,

Rene



From:   Rene Lehmann rlehm...@timocom.com
To: solr-user@lucene.apache.org solr-user@lucene.apache.org
Date:   01.09.2011 15:36
Subject:Solr replication / repeater





Hi there,

i´m really new in Solr and have a question about the Solr replication.
We want to use Solr in two data centers (dedicated fibre channel lane, like
intranet) behind a load balancer. Is the following infrastructure possible?

- one repeater and one slave per data center
- the repeaters used to each other for replication
- the slaves uses the local repeater for replication

Such a construction is possible? Or do I need a pure master server?

kind regards,

Rene

P.S. Sorry for the second mail. But now this is an own thread.

Re: Solr Replication / Repeater

[Erick Erickson]

It looks good except for the repeaters used to each other for replication.
Having the masters use each other for replication implies that you're
indexing data to both of them, in which case you wouldn't need them to
update from each other!

So you really have two choices I think
1 designate one indexing as the master and just have a repeater in
the other data center.
all the slaves in a data center use the local master as a source.
or
2 index your data to both masters and then the masters don't know
about each other.

Of the two, I like 1 best but there are arguments either way.

Best
Erick

On Thu, Sep 1, 2011 at 4:36 AM, Rene Lehmann rlehm...@timocom.com wrote:

 Hi there,

 i´m really new in Solr and have a question about the Solr replication.
 We want to use Solr in two data centers (dedicated fibre channel lane, like
 intranet) behind a load balancer. Is the following infrastructure possible?

 - one repeater and one slave per data center
 - the repeaters used to each other for replication
 - the slaves uses the local repeater for replication

 Such a construction is possible? Or do I need a pure master server?

 kind regards,

 Rene

Re: Solr replication / repeater

[Chris Hostetter]

: i´m really new in Solr and have a question about the Solr replication.
: We want to use Solr in two data centers (dedicated fibre channel lane, like
: intranet) behind a load balancer. Is the following infrastructure possible?
: 
: - one repeater and one slave per data center
: - the repeaters used to each other for replication
: - the slaves uses the local repeater for replication
: 
: Such a construction is possible? Or do I need a pure master server?

you can't have the repeaters talking to eachother -- there has to be one 
actual master that is where you index your data.

you can think of the problem as having a cluster in each data center 
consisting of a local master and local slaves and then configure all 
of the individual local masters to actually be repeaters pulling from 
one global master which (if you want) could also be a local master in 
a particular data center.


-Hoss

Re: Solr replication, HAproxy and data management

[Paolo Castagna]

Paolo Castagna wrote:

Hi,
we are using Solr v1.4.x with multi-cores and a master/slaves 
configuration.

We also use HAProxy [1] to load balance search requests amongst slaves.
Finally, we use MapReduce to create new Solr indexes.

I'd like to share with you what I am doing when I need to:

 1. add a new index
 2. replace an existing index with an new/updated one
 3. add a slave
 4. remove a slave (or a slave died)

I am interested in knowing what are the best practices in these scenarios.


[...]


Does all this seems sensible to you?

Do you have best practices, suggestions to share?



Well, maybe these are two too broad questions...

I have a very specific one, related to all this.

Let's say I have a Solr master with multi-cores and I want to add a new
slave. Can I tell the slave to replicate all the indexes for the master?
How?

Any comment/advice regarding my original message are still more than
welcome.

Thank you,
Paolo

Re: Solr Replication Questions

[Noble Paul നോബിള്‍ नोब्ळ्]

On Wed, Jan 6, 2010 at 2:51 AM, Giovanni Fernandez-Kincade
gfernandez-kinc...@capitaliq.com wrote:
 http://wiki.apache.org/solr/SolrReplication

 I've been looking over this replication wiki and I'm still unclear on a two 
 points about Solr Replication:

 1.     If there have been small changes to the index on the master, does the 
 slave copy the entire contents of the index files that were affected?

only the delta is copied.

 a.     Let's say I add one document to the master. Presumably that causes 
 changes to the position file, amidst a few others. Does the slave download 
 the entire position file? Or just the portion that was changed?

Lucene never modifies a file which was written by previous commits. So
if you add a new document and commit , it is written to new files.
Solr replication will only replicate those new files
 2.     If you have a multi-core slave, is it possible to share one 
 configuration file (i.e. one instance directory) amidst the multiple cores, 
 and yet each core poll a different master?

 a.     Can you set the masterUrl for each core separately in the server.xml?


 Thanks for your help,
 Gio.




-- 
-
Noble Paul | Systems Architect| AOL | http://aol.com

Re: SOLR: Replication

[Yonik Seeley]

On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi f...@efendi.ca wrote:
 I tried... I set APR to improve performance... server is slow while replica;
 but top shows only 1% of I/O wait... it is probably environment specific;

So you're saying that stock tomcat (non-native APR) was also 10 times slower?

 but the same happened in my home-based network, rsync was 10 times faster...
 I don't know details of HTTP-replica, it could be base64 or something like
 that; RAM-buffer, flush to disk, etc.

The HTTP replication is using binary.
If you look here, it was benchmarked to be nearly as fast as rsync:
http://wiki.apache.org/solr/SolrReplication

It does do a fsync to make sure that the files are on disk after
downloading, but that shouldn't make too much difference.

-Yonik
http://www.lucidimagination.com

RE: SOLR: Replication

[Fuad Efendi]

Thank you Yonik, excellent WIKI! I'll try without APR, I believe it's
environmental issue; 100Mbps switched should do 10 times faster (current
replica speed is 1Mbytes/sec)


 -Original Message-
 From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
 Seeley
 Sent: January-03-10 10:03 AM
 To: solr-user@lucene.apache.org
 Subject: Re: SOLR: Replication
 
 On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi f...@efendi.ca wrote:
  I tried... I set APR to improve performance... server is slow while
 replica;
  but top shows only 1% of I/O wait... it is probably environment
 specific;
 
 So you're saying that stock tomcat (non-native APR) was also 10 times
 slower?
 
  but the same happened in my home-based network, rsync was 10 times
 faster...
  I don't know details of HTTP-replica, it could be base64 or something
 like
  that; RAM-buffer, flush to disk, etc.
 
 The HTTP replication is using binary.
 If you look here, it was benchmarked to be nearly as fast as rsync:
 http://wiki.apache.org/solr/SolrReplication
 
 It does do a fsync to make sure that the files are on disk after
 downloading, but that shouldn't make too much difference.
 
 -Yonik
 http://www.lucidimagination.com

Re: SOLR: Replication

[Peter Wolanin]

Related to the difference between rsync and native Solr replication -
we are seeing issues with Solr 1.4 where search queries that come in
during a replication request hang for excessive amount of time (up to
100's of seconds for a result normally that takes ~50 ms).

We are replicating pretty often (every 90 sec for multiple cores to
one slave server), but still did not think that replication would make
the master server unable to handle search requests.  Is there some
configuration option we are missing which would handle this situation
better?

Thanks,

Peter

On Sun, Jan 3, 2010 at 11:27 AM, Fuad Efendi f...@efendi.ca wrote:
 Thank you Yonik, excellent WIKI! I'll try without APR, I believe it's
 environmental issue; 100Mbps switched should do 10 times faster (current
 replica speed is 1Mbytes/sec)


 -Original Message-
 From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
 Seeley
 Sent: January-03-10 10:03 AM
 To: solr-user@lucene.apache.org
 Subject: Re: SOLR: Replication

 On Sat, Jan 2, 2010 at 11:35 PM, Fuad Efendi f...@efendi.ca wrote:
  I tried... I set APR to improve performance... server is slow while
 replica;
  but top shows only 1% of I/O wait... it is probably environment
 specific;

 So you're saying that stock tomcat (non-native APR) was also 10 times
 slower?

  but the same happened in my home-based network, rsync was 10 times
 faster...
  I don't know details of HTTP-replica, it could be base64 or something
 like
  that; RAM-buffer, flush to disk, etc.

 The HTTP replication is using binary.
 If you look here, it was benchmarked to be nearly as fast as rsync:
 http://wiki.apache.org/solr/SolrReplication

 It does do a fsync to make sure that the files are on disk after
 downloading, but that shouldn't make too much difference.

 -Yonik
 http://www.lucidimagination.com






-- 
Peter M. Wolanin, Ph.D.
Momentum Specialist,  Acquia. Inc.
peter.wola...@acquia.com

Re: SOLR: Replication

[Yonik Seeley]

On Sun, Jan 3, 2010 at 2:55 PM, Peter Wolanin peter.wola...@acquia.com wrote:
 Related to the difference between rsync and native Solr replication -
 we are seeing issues with Solr 1.4 where search queries that come in
 during a replication request hang for excessive amount of time (up to
 100's of seconds for a result normally that takes ~50 ms).

 We are replicating pretty often (every 90 sec for multiple cores to
 one slave server), but still did not think that replication would make
 the master server unable to handle search requests.  Is there some
 configuration option we are missing which would handle this situation
 better?

Hmmm, any other clues about what's happening during this time?
If it's not a bug, it could simply be that reading a large index to
serve it to a slave could throw out the important parts of the OS
cache that caused searches to be faster.

If it is a bug, well then we certainly want to get to the bottom of it!

-Yonik
http://www.lucidimagination.com

Re: SOLR: Replication

[Yonik Seeley]

On Sat, Jan 2, 2010 at 5:48 PM, Fuad Efendi f...@efendi.ca wrote:
 I used RSYNC before, and 20Gb replica took less than an hour (20-40
 minutes); now, HTTP, and it takes 5-6 hours...
 Admin screen shows 952Kb/sec average speed; 100Mbps network, full-duplex; I
 am using Tomcat Native for APR. 10x times slow...

Hmmm, did you try w/o native APR?

-Yonik
http://www.lucidimagination.com

RE: SOLR: Replication

[Fuad Efendi]

Hi Yonik,

I tried... I set APR to improve performance... server is slow while replica;
but top shows only 1% of I/O wait... it is probably environment specific;
but the same happened in my home-based network, rsync was 10 times faster...
I don't know details of HTTP-replica, it could be base64 or something like
that; RAM-buffer, flush to disk, etc.
-Fuad


 -Original Message-
 From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik
 Seeley
 Sent: January-02-10 5:52 PM
 To: solr-user@lucene.apache.org
 Subject: Re: SOLR: Replication
 
 On Sat, Jan 2, 2010 at 5:48 PM, Fuad Efendi f...@efendi.ca wrote:
  I used RSYNC before, and 20Gb replica took less than an hour (20-40
  minutes); now, HTTP, and it takes 5-6 hours...
  Admin screen shows 952Kb/sec average speed; 100Mbps network, full-
 duplex; I
  am using Tomcat Native for APR. 10x times slow...
 
 Hmmm, did you try w/o native APR?
 
 -Yonik
 http://www.lucidimagination.com

Re: Solr replication 1.3 issue

[Lance Norskog]

This is a Unix security question. rsyncd' is a system daemon and
should be managed in the OS scripts. The rsyncd-* scripts include a
security setting for the 'solr' account that limits the account to the
solr data directory (and this code does not support multicore).

For security reasons, I personally would not make a sudoer out of a
user with an automated remote login; but every site is different.

On Mon, Dec 21, 2009 at 7:54 PM, Maduranga Kannangara
mkannang...@infomedia.com.au wrote:
 Hi All,

 We're trying to replicate indexes on Solr 1.3 across from 
 Dev-QA-Staging-Prod etc.
 So at each stage other than Dev and Prod, each would live as a master and a 
 slave at a given time.

 We hit a bottle neck (may be?) when we try to start rsyncd-start on the 
 master from the slave machine.

 Commands used:

 ssh -o StrictHostKeyChecking=no ad...@192.168.22.1 
 /solr/SolrHome/bin/rsyncd-enable
 ssh -o StrictHostKeyChecking=no ad...@192.168.22.1  /solr / SolrHome 
 /bin/rsyncd-start -p 18003

 On slave following error is displayed:

 @RSYNCD: 29
 @ERROR: protocol startup error

 On master logs following were found:

 2009/12/21 22:46:05 enabled by admin
 2009/12/21 22:46:05 command: / solr/SolrHome /bin/rsyncd-enable
 2009/12/21 22:46:05 ended (elapsed time: 0 sec)
 2009/12/21 22:46:09 started by admin
 2009/12/21 22:46:09 command: /solr/SolrHome/bin/rsyncd-start -p 18993
 2009/12/21 22:46:09 [16964] forward name lookup for devserver002 failed: 
 ai_family not supported
 2009/12/21 22:46:09 [16964] connect from UNKNOWN (localhost)
 2009/12/21 22:46:29 [16964] rsync: connection unexpectedly closed (0 bytes 
 received so far) [receiver]
 2009/12/21 22:46:29 [16964] rsync error: error in rsync protocol data stream 
 (code 12) at io.c(463) [receiver=2.6.8]
 2009/12/21 22:46:44 rsyncd not accepting connections, exiting
 2009/12/21 22:46:57 enabled by admin
 2009/12/21 22:46:57 command: /solr/SolrHome/bin/rsyncd-enable
 2009/12/21 22:46:57 rsyncd already currently enabled
 2009/12/21 22:46:57 exited (elapsed time: 0 sec)
 2009/12/21 22:47:00 started by admin
 2009/12/21 22:47:00 command: /solr/SolrHome/bin/rsyncd-start -p 18993
 2009/12/21 22:47:00 [17115] forward name lookup for devserver002 failed: 
 ai_family not supported
 2009/12/21 22:47:00 [17115] connect from UNKNOWN (localhost)
 2009/12/21 22:49:18 rsyncd not accepting connections, exiting


 Is it not possible to start the rsync daemon on master from the slave?
 The user that we use is on the sudoers list as well.

 Thanks
 Madu







-- 
Lance Norskog
goks...@gmail.com

RE: Solr replication 1.3 issue

[Maduranga Kannangara]

Yes, that makes sense Lance. Thanks.

For the moment we broke our script to do the master's part (starting rsyncd) on 
the master server itself. Our problem is that we have so many instances running 
in different environment and we'd really love to minimize the number of them. 
:-)

Thanks again.
Madu

-Original Message-
From: Lance Norskog [mailto:goks...@gmail.com] 
Sent: Wednesday, 23 December 2009 9:33 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr replication 1.3 issue

This is a Unix security question. rsyncd' is a system daemon and
should be managed in the OS scripts. The rsyncd-* scripts include a
security setting for the 'solr' account that limits the account to the
solr data directory (and this code does not support multicore).

For security reasons, I personally would not make a sudoer out of a
user with an automated remote login; but every site is different.

On Mon, Dec 21, 2009 at 7:54 PM, Maduranga Kannangara
mkannang...@infomedia.com.au wrote:
 Hi All,

 We're trying to replicate indexes on Solr 1.3 across from 
 Dev-QA-Staging-Prod etc.
 So at each stage other than Dev and Prod, each would live as a master and a 
 slave at a given time.

 We hit a bottle neck (may be?) when we try to start rsyncd-start on the 
 master from the slave machine.

 Commands used:

 ssh -o StrictHostKeyChecking=no ad...@192.168.22.1 
 /solr/SolrHome/bin/rsyncd-enable
 ssh -o StrictHostKeyChecking=no ad...@192.168.22.1  /solr / SolrHome 
 /bin/rsyncd-start -p 18003

 On slave following error is displayed:

 @RSYNCD: 29
 @ERROR: protocol startup error

 On master logs following were found:

 2009/12/21 22:46:05 enabled by admin
 2009/12/21 22:46:05 command: / solr/SolrHome /bin/rsyncd-enable
 2009/12/21 22:46:05 ended (elapsed time: 0 sec)
 2009/12/21 22:46:09 started by admin
 2009/12/21 22:46:09 command: /solr/SolrHome/bin/rsyncd-start -p 18993
 2009/12/21 22:46:09 [16964] forward name lookup for devserver002 failed: 
 ai_family not supported
 2009/12/21 22:46:09 [16964] connect from UNKNOWN (localhost)
 2009/12/21 22:46:29 [16964] rsync: connection unexpectedly closed (0 bytes 
 received so far) [receiver]
 2009/12/21 22:46:29 [16964] rsync error: error in rsync protocol data stream 
 (code 12) at io.c(463) [receiver=2.6.8]
 2009/12/21 22:46:44 rsyncd not accepting connections, exiting
 2009/12/21 22:46:57 enabled by admin
 2009/12/21 22:46:57 command: /solr/SolrHome/bin/rsyncd-enable
 2009/12/21 22:46:57 rsyncd already currently enabled
 2009/12/21 22:46:57 exited (elapsed time: 0 sec)
 2009/12/21 22:47:00 started by admin
 2009/12/21 22:47:00 command: /solr/SolrHome/bin/rsyncd-start -p 18993
 2009/12/21 22:47:00 [17115] forward name lookup for devserver002 failed: 
 ai_family not supported
 2009/12/21 22:47:00 [17115] connect from UNKNOWN (localhost)
 2009/12/21 22:49:18 rsyncd not accepting connections, exiting


 Is it not possible to start the rsync daemon on master from the slave?
 The user that we use is on the sudoers list as well.

 Thanks
 Madu







-- 
Lance Norskog
goks...@gmail.com

Re: Solr Replication: How to restore data from last snapshot

[Chris Hostetter]

: Subject: Solr Replication: How to restore data from last snapshot
: References: 8950e934db69a040a1783438e67293d813da3f6...@delmail.sapient.com
:  26230840.p...@talk.nabble.com
:  f4d979d0911061116l14fa3a49sc5a71d855857c...@mail.gmail.com
: In-Reply-To: f4d979d0911061116l14fa3a49sc5a71d855857c...@mail.gmail.com

http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists

When starting a new discussion on a mailing list, please do not reply to 
an existing message, instead start a fresh email.  Even if you change the 
subject line of your email, other mail headers still track which thread 
you replied to and your question is hidden in that thread and gets less 
attention.   It makes following discussions in the mailing list archives 
particularly difficult.
See Also:  http://en.wikipedia.org/wiki/Thread_hijacking



-Hoss

Re: Solr Replication: How to restore data from last snapshot

[Noble Paul നോബിള്‍ नोब्ळ्]

if it is a single core you will have to restart the master

On Sat, Nov 7, 2009 at 1:55 AM, Osborn Chan oc...@shutterfly.com wrote:
 Thanks. But I have following use cases:

 1) Master index is corrupted, but it didn't replicate to slave servers.
        - In this case, I only need to restore to last snapshot.
 2) Master index is corrupted, and it has replicated to slave servers.
        - In this case, I need to restore to last snapshot, and make sure 
 slave servers replicate the restored index from index server as well.

 Assuming both cases are in production environment, and I cannot shutdown the 
 master and slave servers.
 Is there any rest API call or something else I can do without manually using 
 linux command and restart?

 Thanks,

 Osborn

 -Original Message-
 From: Matthew Runo [mailto:matthew.r...@gmail.com]
 Sent: Friday, November 06, 2009 12:20 PM
 To: solr-user@lucene.apache.org
 Subject: Re: Solr Replication: How to restore data from last snapshot

 If your master index is corrupt and it hasn't been replicated out, you
 should be able to shut down the server and remove the corrupted index
 files. Then copy the replicated index back onto the master and start
 everything back up.

 As far as I know, the indexes on the replicated slaves are exactly
 what you'd have on the master, so this method should work.

 --Matthew Runo

 On Fri, Nov 6, 2009 at 11:41 AM, Osborn Chan oc...@shutterfly.com wrote:
 Hi,

 I have followed Solr set up ReplicationHandler for index replication to 
 slave.
 Do anyone know how to restore corrupted index from snapshot in master, and 
 force replication of the restored index to slave?


 Thanks,

 Osborn





-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: Solr Replication

[Shalin Shekhar Mangar]

On Fri, Aug 28, 2009 at 2:04 AM, J G skinny_joe...@hotmail.com wrote:


 We have multiple solr webapps all running from the same WAR file. Each
 webapp is running under the same Tomcat container and I consider each webapp
 the same thing as a slice (or instance). I've configured the Tomcat
 container to enable JMX and when I connect using JConsole I only see the
 replication handler for one of the webapps in the server. I was under the
 impression each webapp gets its own replication handler. Is this not true?

 It would be nice to be able to have a JMX MBean for each replication
 handler in the container so we can get all the same replication information
 using JMX as in using the replication admin page for each web app.


The problem is that all Solr instances are registering themselves to the
same platform mbean server (since all instances share the same tomcat and
jvm). You should think about having separate mbean server for each of the
solr instance by specifying unique service urls in solrconfig.xml

If you setup had multiple solr cores with the same solr war file, then you'd
have seen different mbeans for each core.

-- 
Regards,
Shalin Shekhar Mangar.

Re: Solr Replication

[Noble Paul നോബിള്‍ नोब्ळ्]

when you say a slice you mean one instance of solr? So your JMX
console is connecting to only one solr?

On Thu, Aug 27, 2009 at 3:19 AM, J Gskinny_joe...@hotmail.com wrote:

 Thanks for the response.

 It's interesting because when I run jconsole all I can see is one 
 ReplicationHandler jmx mbean. It looks like it is defaulting to the first 
 slice it finds on its path. Is there anyway to have multiple replication 
 handlers or at least obtain replication on a per slice/instance via JMX 
 like how you can see attributes for each slice/instance via each 
 replication admin jsp page?

 Thanks again.

 From: noble.p...@corp.aol.com
 Date: Wed, 26 Aug 2009 11:05:34 +0530
 Subject: Re: Solr Replication
 To: solr-user@lucene.apache.org

 The ReplicationHandler is not enforced as a singleton , but for all
 practical purposes it is a singleton for one core.

 If an instance  (a slice as you say) is setup as a repeater, It can
 act as both a master and slave

 in the repeater the configuration should be as follows

 MASTER
   |_SLAVE (I am a slave of MASTER)
   |
 REPEATER (I am a slave of MASTER and master to my slaves )
  |
  |
 REPEATER_SLAVE( of REPEATER)


 the point is that REPEATER will have a slave section has a masterUrl
 which points to master and REPEATER_SLAVE will have a slave section
 which has a masterurl pointing to repeater






 On Wed, Aug 26, 2009 at 12:40 AM, J Gskinny_joe...@hotmail.com wrote:
 
  Hello,
 
  We are running multiple slices in our environment. I have enabled JMX and 
  I am inspecting the replication handler mbean to obtain some information 
  about the master/slave configuration for replication. Is the replication 
  handler mbean a singleton? I only see one mbean for the entire server and 
  it's picking an arbitrary slice to report on. So I'm curious if every 
  slice gets its own replication handler mbean? This is important because I 
  have no way of knowing in this specific server any information about the 
  other slices, in particular, information about the master/slave value for 
  the other slices.
 
  Reading through the Solr 1.4 replication strategy, I saw that a slice can 
  be configured to be a master and a slave, i.e. a repeater. I'm wondering 
  how repeaters work because let's say I have a slice named 'A' and the 
  master is on server 1 and the slave is on server 2 then how are these two 
  servers communicating to replicate? Looking at the jmx information I have 
  in the MBean both the isSlave and isMaster is set to true for my repeater 
  so how does this solr slice know if it's the master or slave? I'm a bit 
  confused.
 
  Thanks.
 
 
 
 
  _
  With Windows Live, you can organize, edit, and share your photos.
  http://www.windowslive.com/Desktop/PhotoGallery



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

 _
 Hotmail® is up to 70% faster. Now good news travels really fast.
 http://windowslive.com/online/hotmail?ocid=PID23391::T:WLMTAGL:ON:WL:en-US:WM_HYGN_faster:082009



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

RE: Solr Replication

[J G]

We have multiple solr webapps all running from the same WAR file. Each webapp 
is running under the same Tomcat container and I consider each webapp the same 
thing as a slice (or instance). I've configured the Tomcat container to 
enable JMX and when I connect using JConsole I only see the replication handler 
for one of the webapps in the server. I was under the impression each webapp 
gets its own replication handler. Is this not true? 

It would be nice to be able to have a JMX MBean for each replication handler in 
the container so we can get all the same replication information using JMX as 
in using the replication admin page for each web app.

Thanks.





 From: noble.p...@corp.aol.com
 Date: Thu, 27 Aug 2009 13:04:38 +0530
 Subject: Re: Solr Replication
 To: solr-user@lucene.apache.org
 
 when you say a slice you mean one instance of solr? So your JMX
 console is connecting to only one solr?
 
 On Thu, Aug 27, 2009 at 3:19 AM, J Gskinny_joe...@hotmail.com wrote:
 
  Thanks for the response.
 
  It's interesting because when I run jconsole all I can see is one 
  ReplicationHandler jmx mbean. It looks like it is defaulting to the first 
  slice it finds on its path. Is there anyway to have multiple replication 
  handlers or at least obtain replication on a per slice/instance via JMX 
  like how you can see attributes for each slice/instance via each 
  replication admin jsp page?
 
  Thanks again.
 
  From: noble.p...@corp.aol.com
  Date: Wed, 26 Aug 2009 11:05:34 +0530
  Subject: Re: Solr Replication
  To: solr-user@lucene.apache.org
 
  The ReplicationHandler is not enforced as a singleton , but for all
  practical purposes it is a singleton for one core.
 
  If an instance  (a slice as you say) is setup as a repeater, It can
  act as both a master and slave
 
  in the repeater the configuration should be as follows
 
  MASTER
|_SLAVE (I am a slave of MASTER)
|
  REPEATER (I am a slave of MASTER and master to my slaves )
   |
   |
  REPEATER_SLAVE( of REPEATER)
 
 
  the point is that REPEATER will have a slave section has a masterUrl
  which points to master and REPEATER_SLAVE will have a slave section
  which has a masterurl pointing to repeater
 
 
 
 
 
 
  On Wed, Aug 26, 2009 at 12:40 AM, J Gskinny_joe...@hotmail.com wrote:
  
   Hello,
  
   We are running multiple slices in our environment. I have enabled JMX 
   and I am inspecting the replication handler mbean to obtain some 
   information about the master/slave configuration for replication. Is the 
   replication handler mbean a singleton? I only see one mbean for the 
   entire server and it's picking an arbitrary slice to report on. So I'm 
   curious if every slice gets its own replication handler mbean? This is 
   important because I have no way of knowing in this specific server any 
   information about the other slices, in particular, information about the 
   master/slave value for the other slices.
  
   Reading through the Solr 1.4 replication strategy, I saw that a slice 
   can be configured to be a master and a slave, i.e. a repeater. I'm 
   wondering how repeaters work because let's say I have a slice named 'A' 
   and the master is on server 1 and the slave is on server 2 then how are 
   these two servers communicating to replicate? Looking at the jmx 
   information I have in the MBean both the isSlave and isMaster is set to 
   true for my repeater so how does this solr slice know if it's the master 
   or slave? I'm a bit confused.
  
   Thanks.
  
  
  
  
   _
   With Windows Live, you can organize, edit, and share your photos.
   http://www.windowslive.com/Desktop/PhotoGallery
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
  _
  Hotmail® is up to 70% faster. Now good news travels really fast.
  http://windowslive.com/online/hotmail?ocid=PID23391::T:WLMTAGL:ON:WL:en-US:WM_HYGN_faster:082009
 
 
 
 -- 
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

_
With Windows Live, you can organize, edit, and share your photos.
http://www.windowslive.com/Desktop/PhotoGallery

Re: Solr Replication

[Noble Paul നോബിള്‍ नोब्ळ्]

Each instance has its own ReplicationHandler instance/MBean. I guess
the problem is with the jmx implementation. both MBeans may be
registered with the same name

On Fri, Aug 28, 2009 at 2:04 AM, J Gskinny_joe...@hotmail.com wrote:

 We have multiple solr webapps all running from the same WAR file. Each webapp 
 is running under the same Tomcat container and I consider each webapp the 
 same thing as a slice (or instance). I've configured the Tomcat container 
 to enable JMX and when I connect using JConsole I only see the replication 
 handler for one of the webapps in the server. I was under the impression each 
 webapp gets its own replication handler. Is this not true?

 It would be nice to be able to have a JMX MBean for each replication handler 
 in the container so we can get all the same replication information using JMX 
 as in using the replication admin page for each web app.

 Thanks.





 From: noble.p...@corp.aol.com
 Date: Thu, 27 Aug 2009 13:04:38 +0530
 Subject: Re: Solr Replication
 To: solr-user@lucene.apache.org

 when you say a slice you mean one instance of solr? So your JMX
 console is connecting to only one solr?

 On Thu, Aug 27, 2009 at 3:19 AM, J Gskinny_joe...@hotmail.com wrote:
 
  Thanks for the response.
 
  It's interesting because when I run jconsole all I can see is one 
  ReplicationHandler jmx mbean. It looks like it is defaulting to the first 
  slice it finds on its path. Is there anyway to have multiple replication 
  handlers or at least obtain replication on a per slice/instance via 
  JMX like how you can see attributes for each slice/instance via each 
  replication admin jsp page?
 
  Thanks again.
 
  From: noble.p...@corp.aol.com
  Date: Wed, 26 Aug 2009 11:05:34 +0530
  Subject: Re: Solr Replication
  To: solr-user@lucene.apache.org
 
  The ReplicationHandler is not enforced as a singleton , but for all
  practical purposes it is a singleton for one core.
 
  If an instance  (a slice as you say) is setup as a repeater, It can
  act as both a master and slave
 
  in the repeater the configuration should be as follows
 
  MASTER
    |_SLAVE (I am a slave of MASTER)
    |
  REPEATER (I am a slave of MASTER and master to my slaves )
   |
   |
  REPEATER_SLAVE( of REPEATER)
 
 
  the point is that REPEATER will have a slave section has a masterUrl
  which points to master and REPEATER_SLAVE will have a slave section
  which has a masterurl pointing to repeater
 
 
 
 
 
 
  On Wed, Aug 26, 2009 at 12:40 AM, J Gskinny_joe...@hotmail.com wrote:
  
   Hello,
  
   We are running multiple slices in our environment. I have enabled JMX 
   and I am inspecting the replication handler mbean to obtain some 
   information about the master/slave configuration for replication. Is 
   the replication handler mbean a singleton? I only see one mbean for the 
   entire server and it's picking an arbitrary slice to report on. So I'm 
   curious if every slice gets its own replication handler mbean? This is 
   important because I have no way of knowing in this specific server any 
   information about the other slices, in particular, information about 
   the master/slave value for the other slices.
  
   Reading through the Solr 1.4 replication strategy, I saw that a slice 
   can be configured to be a master and a slave, i.e. a repeater. I'm 
   wondering how repeaters work because let's say I have a slice named 'A' 
   and the master is on server 1 and the slave is on server 2 then how are 
   these two servers communicating to replicate? Looking at the jmx 
   information I have in the MBean both the isSlave and isMaster is set to 
   true for my repeater so how does this solr slice know if it's the 
   master or slave? I'm a bit confused.
  
   Thanks.
  
  
  
  
   _
   With Windows Live, you can organize, edit, and share your photos.
   http://www.windowslive.com/Desktop/PhotoGallery
 
 
 
  --
  -
  Noble Paul | Principal Engineer| AOL | http://aol.com
 
  _
  Hotmail® is up to 70% faster. Now good news travels really fast.
  http://windowslive.com/online/hotmail?ocid=PID23391::T:WLMTAGL:ON:WL:en-US:WM_HYGN_faster:082009



 --
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

 _
 With Windows Live, you can organize, edit, and share your photos.
 http://www.windowslive.com/Desktop/PhotoGallery



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

RE: Solr Replication

[J G]

Thanks for the response.

It's interesting because when I run jconsole all I can see is one 
ReplicationHandler jmx mbean. It looks like it is defaulting to the first slice 
it finds on its path. Is there anyway to have multiple replication handlers or 
at least obtain replication on a per slice/instance via JMX like how you 
can see attributes for each slice/instance via each replication admin jsp 
page? 

Thanks again.

 From: noble.p...@corp.aol.com
 Date: Wed, 26 Aug 2009 11:05:34 +0530
 Subject: Re: Solr Replication
 To: solr-user@lucene.apache.org
 
 The ReplicationHandler is not enforced as a singleton , but for all
 practical purposes it is a singleton for one core.
 
 If an instance  (a slice as you say) is setup as a repeater, It can
 act as both a master and slave
 
 in the repeater the configuration should be as follows
 
 MASTER
   |_SLAVE (I am a slave of MASTER)
   |
 REPEATER (I am a slave of MASTER and master to my slaves )
  |
  |
 REPEATER_SLAVE( of REPEATER)
 
 
 the point is that REPEATER will have a slave section has a masterUrl
 which points to master and REPEATER_SLAVE will have a slave section
 which has a masterurl pointing to repeater
 
 
 
 
 
 
 On Wed, Aug 26, 2009 at 12:40 AM, J Gskinny_joe...@hotmail.com wrote:
 
  Hello,
 
  We are running multiple slices in our environment. I have enabled JMX and I 
  am inspecting the replication handler mbean to obtain some information 
  about the master/slave configuration for replication. Is the replication 
  handler mbean a singleton? I only see one mbean for the entire server and 
  it's picking an arbitrary slice to report on. So I'm curious if every slice 
  gets its own replication handler mbean? This is important because I have no 
  way of knowing in this specific server any information about the other 
  slices, in particular, information about the master/slave value for the 
  other slices.
 
  Reading through the Solr 1.4 replication strategy, I saw that a slice can 
  be configured to be a master and a slave, i.e. a repeater. I'm wondering 
  how repeaters work because let's say I have a slice named 'A' and the 
  master is on server 1 and the slave is on server 2 then how are these two 
  servers communicating to replicate? Looking at the jmx information I have 
  in the MBean both the isSlave and isMaster is set to true for my repeater 
  so how does this solr slice know if it's the master or slave? I'm a bit 
  confused.
 
  Thanks.
 
 
 
 
  _
  With Windows Live, you can organize, edit, and share your photos.
  http://www.windowslive.com/Desktop/PhotoGallery
 
 
 
 -- 
 -
 Noble Paul | Principal Engineer| AOL | http://aol.com

_
Hotmail® is up to 70% faster. Now good news travels really fast. 
http://windowslive.com/online/hotmail?ocid=PID23391::T:WLMTAGL:ON:WL:en-US:WM_HYGN_faster:082009

Re: Solr Replication

[Noble Paul നോബിള്‍ नोब्ळ्]

The ReplicationHandler is not enforced as a singleton , but for all
practical purposes it is a singleton for one core.

If an instance  (a slice as you say) is setup as a repeater, It can
act as both a master and slave

in the repeater the configuration should be as follows

MASTER
  |_SLAVE (I am a slave of MASTER)
  |
REPEATER (I am a slave of MASTER and master to my slaves )
 |
 |
REPEATER_SLAVE( of REPEATER)


the point is that REPEATER will have a slave section has a masterUrl
which points to master and REPEATER_SLAVE will have a slave section
which has a masterurl pointing to repeater






On Wed, Aug 26, 2009 at 12:40 AM, J Gskinny_joe...@hotmail.com wrote:

 Hello,

 We are running multiple slices in our environment. I have enabled JMX and I 
 am inspecting the replication handler mbean to obtain some information about 
 the master/slave configuration for replication. Is the replication handler 
 mbean a singleton? I only see one mbean for the entire server and it's 
 picking an arbitrary slice to report on. So I'm curious if every slice gets 
 its own replication handler mbean? This is important because I have no way of 
 knowing in this specific server any information about the other slices, in 
 particular, information about the master/slave value for the other slices.

 Reading through the Solr 1.4 replication strategy, I saw that a slice can be 
 configured to be a master and a slave, i.e. a repeater. I'm wondering how 
 repeaters work because let's say I have a slice named 'A' and the master is 
 on server 1 and the slave is on server 2 then how are these two servers 
 communicating to replicate? Looking at the jmx information I have in the 
 MBean both the isSlave and isMaster is set to true for my repeater so how 
 does this solr slice know if it's the master or slave? I'm a bit confused.

 Thanks.




 _
 With Windows Live, you can organize, edit, and share your photos.
 http://www.windowslive.com/Desktop/PhotoGallery



-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com

Re: Solr replication and spellcheck data

[Noble Paul നോബിള്‍ नोब्ळ्]

This is not supported by the Java Replication . but planned for later
https://issues.apache.org/jira/browse/SOLR-866

On Wed, Jul 29, 2009 at 4:04 AM, Ian Sugariansu...@gmail.com wrote:
 Hi

 I would like to make use of the new replication mechanism [1] to set up a
 master-slaves configuration, but from quick reading and searching around, I
 can't seem to find a way to replicate the spelling index in addition to the
 main search index. (We use the spellcheck component)

 Is there a way to do it, or would we have to go the cron/script/rsync way
 [2]?

 Any pointers appreciated. I probably missed something!

 Ian

 [1] http://wiki.apache.org/solr/SolrReplication
 [2] http://wiki.apache.org/solr/CollectionDistribution




-- 
-
Noble Paul | Principal Engineer| AOL | http://aol.com