recommendation for solr replication throttling
Hello, We are needing a recommendation for solr replication throttling. What are your recommendations for maxWriteMBPerSec value? Our indexes contain 18 locales and size for all indexes is 188GB and growing. Also will replication throttling work with solr 4.10.3? Thanks, Pino Alu | HCL Commerce Administrator :: Emerson.com | Enterprise IT Emerson | 8000 West Florissant Ave. | St. Louis | MO | 63136 | USA T +1 314 553 1785 pino@emerson.com<mailto:pino@emerson.com> Delivering Technology Solutions With Purpose
RE: Solr Replication
Using cdcr with new replica types be aware of https://issues.apache.org/jira/browse/SOLR-12057?focusedComm Parallel indexing to both cluster might be an option as well -- Vadim > -Original Message- > From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > Sent: Monday, January 07, 2019 11:10 AM > To: solr-user@lucene.apache.org > Subject: Re: Solr Replication > > In SolrCloud there are Data Centers. > Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2. > You can then use CDCR (Cross Data Center Replication). > http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication- > cdcr.html > > Nevertheless I would spend your Cluster 2 another 2 zookeeper instances. > > Regards, Bernd > > Am 07.01.19 um 06:39 schrieb Mannar mannan: > > Hi All, > > > > I would like to configure master slave between two solr cloud clusters (for > > failover). Below is the scenario > > > > Solr version : 7.0 > > > > Cluster 1: > > 3 zookeeper instances : zk1, zk2, zk3 > > 2 solr instances : solr1, solr2 > > > > Cluster 2: > > 1 zookeeper instance : bkpzk1, > > 1 solr instances : bkpsolr1, bkpsolr2 > > > > Master / Slave : solr1 / bkpsolr1 > >solr2 / bkpsolr2 > > > > Is it possible to have master / slave replication configured for solr > > instances running in cluster1 & cluster2 (for failover). Kindly let me know > > the possibility. > >
Re: Solr Replication
In SolrCloud there are Data Centers. Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2. You can then use CDCR (Cross Data Center Replication). http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication-cdcr.html Nevertheless I would spend your Cluster 2 another 2 zookeeper instances. Regards, Bernd Am 07.01.19 um 06:39 schrieb Mannar mannan: Hi All, I would like to configure master slave between two solr cloud clusters (for failover). Below is the scenario Solr version : 7.0 Cluster 1: 3 zookeeper instances : zk1, zk2, zk3 2 solr instances : solr1, solr2 Cluster 2: 1 zookeeper instance : bkpzk1, 1 solr instances : bkpsolr1, bkpsolr2 Master / Slave : solr1 / bkpsolr1 solr2 / bkpsolr2 Is it possible to have master / slave replication configured for solr instances running in cluster1 & cluster2 (for failover). Kindly let me know the possibility.
Solr Replication
Hi All, I would like to configure master slave between two solr cloud clusters (for failover). Below is the scenario Solr version : 7.0 Cluster 1: 3 zookeeper instances : zk1, zk2, zk3 2 solr instances : solr1, solr2 Cluster 2: 1 zookeeper instance : bkpzk1, 1 solr instances : bkpsolr1, bkpsolr2 Master / Slave : solr1 / bkpsolr1 solr2 / bkpsolr2 Is it possible to have master / slave replication configured for solr instances running in cluster1 & cluster2 (for failover). Kindly let me know the possibility.
Re: Solr Replication being flaky (6.2.0)
bq: That's evidence enough for me to beat on our systems guys Beat them with a stick. A large stick. If they insist that they can't up these limits, I always push back hard with "why?". Usually reluctance to up them is a misplaced efficiency argument And I should have said you should see exceptions in your Solr logs if this is really something that happens. FWIW, Erick On Fri, Jan 19, 2018 at 11:07 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Working on that now to see if it helps us out. Solr process is NOT dying at > all. Searches are still working as expected, but since we load balance > requestsif the master/slave are out of sync the search results vary. > > The advice is MUCH appreciated! > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Friday, January 19, 2018 1:49 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr Replication being flaky (6.2.0) > > On 1/19/2018 11:27 AM, Shawn Heisey wrote: >> On 1/19/2018 8:54 AM, Pouliot, Scott wrote: >>> I do have a ticket in with our systems team to up the file handlers >>> since I am seeing the "Too many files open" error on occasion on our >>> prod servers. Is this the setting you're referring to? Found we >>> were set to to 1024 using the "Ulimit" command. >> >> No, but that often needs increasing too. I think you need to increase >> the process limit even if that's not the cause of this particular problem. > > Had another thought. Either of these limits can cause completely > unpredictable problems with Solr. The open file limit could be the reason > for these issues, even if you're not actually hitting the process limit. As > I mentioned before, I would expect a process limit to cause Solr to kill > itself, and your other messages don't mention problems like that. > > The scale of your Solr installation indicates that you should greatly > increase both limits on all of your Solr servers. > > Thanks, > Shawn
RE: Solr Replication being flaky (6.2.0)
Working on that now to see if it helps us out. Solr process is NOT dying at all. Searches are still working as expected, but since we load balance requestsif the master/slave are out of sync the search results vary. The advice is MUCH appreciated! -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Friday, January 19, 2018 1:49 PM To: solr-user@lucene.apache.org Subject: Re: Solr Replication being flaky (6.2.0) On 1/19/2018 11:27 AM, Shawn Heisey wrote: > On 1/19/2018 8:54 AM, Pouliot, Scott wrote: >> I do have a ticket in with our systems team to up the file handlers >> since I am seeing the "Too many files open" error on occasion on our >> prod servers. Is this the setting you're referring to? Found we >> were set to to 1024 using the "Ulimit" command. > > No, but that often needs increasing too. I think you need to increase > the process limit even if that's not the cause of this particular problem. Had another thought. Either of these limits can cause completely unpredictable problems with Solr. The open file limit could be the reason for these issues, even if you're not actually hitting the process limit. As I mentioned before, I would expect a process limit to cause Solr to kill itself, and your other messages don't mention problems like that. The scale of your Solr installation indicates that you should greatly increase both limits on all of your Solr servers. Thanks, Shawn
Re: Solr Replication being flaky (6.2.0)
On 1/19/2018 11:27 AM, Shawn Heisey wrote: On 1/19/2018 8:54 AM, Pouliot, Scott wrote: I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command. No, but that often needs increasing too. I think you need to increase the process limit even if that's not the cause of this particular problem. Had another thought. Either of these limits can cause completely unpredictable problems with Solr. The open file limit could be the reason for these issues, even if you're not actually hitting the process limit. As I mentioned before, I would expect a process limit to cause Solr to kill itself, and your other messages don't mention problems like that. The scale of your Solr installation indicates that you should greatly increase both limits on all of your Solr servers. Thanks, Shawn
Re: Solr Replication being flaky (6.2.0)
On 1/19/2018 8:54 AM, Pouliot, Scott wrote: I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command. No, but that often needs increasing too. I think you need to increase the process limit even if that's not the cause of this particular problem. Sounds like you're running on Linux, though ulimit is probably available on other platforms too. If it's Linux, generally you must increase both the number of processes and the open file limit in /etc/security/limits.conf. Trying to use the ulimit command generally doesn't work because the kernel has hard limits configured that ulimit can't budge. If it's not Linux, then you'll need to consult with an expert in the OS you're running. Again, assuming Linux, in the output of "ulimit -a" the value I'm talking about is the "-u" value -- "max user processes". The following is the additions that I typically make to /etc/security/limits.conf, to increase both the open file limit and the process limit for the solr user: solr hard nproc 61440 solr soft nproc 40960 solr hard nofile 65535 solr soft nofile 49151 Are you running into problems where Solr just disappears? I would expect a process limit to generate OutOfMemoryError exceptions. When Solr is started with the included shell script, unless it's running with the foreground option, OOME will kill the Solr process. We have issues to bring the OOME death option to running in the foreground, as well as when running on Windows. Thanks, Shawn
RE: Solr Replication being flaky (6.2.0)
That's evidence enough for me to beat on our systems guys to get these file handles upped and cross my fingers then! -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 1:18 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Solr Replication being flaky (6.2.0) "Could be", certainly. "Definitely is" is iffier ;)... But the statement "If we restart the Solr service or optimize the core it seems to kick back in again.", especially the "optimize" bit (which, by the way you should do only if you have the capability of doing it periodically [1]) is some evidence that this may be in the vicinity. One of the effects of an optimize is to merge your segments files from N to 1. So say you have 10 segments. Each one of those may consist of 10-15 individual files, all of which are held open. So you'd go from 150 open file handles to 15.. https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2F2017%2F10%2F13%2Fsegment-merging-deleted-documents-optimize-may-bad%2F=02%7C01%7CScott.Pouliot%40peoplefluent.com%7Cc2912861f58248e3a92808d55f690eb8%7C8b16fb62c78448b6aba889567990e7fe%7C1%7C0%7C636519827178716698=DxnChrfyTbRDjB7HzqpOE%2BvOJRIxdnrXVCIyfoMjJPU%3D=0 Best, Erick On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Erick, > > Thanks! Could these settings be toying with replication? Solr itself seems > to be working like a champ, except when things get out of sync. > > Scott > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, January 19, 2018 12:27 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Solr Replication being flaky (6.2.0) > > Scott: > > We usually recommend setting files and processes very, very high. Like 65K > high. Or unlimited if you can. > > Plus max user processes should also be bumped very high as well, like 65K as > well. > > Plus max memory and virtual memory should be unlimited. > > We've included warnings at startup for open files and processes, see > SOLR-11703 > > Best, > Erick > > On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott > <scott.poul...@peoplefluent.com> wrote: >> I do have a ticket in with our systems team to up the file handlers since I >> am seeing the "Too many files open" error on occasion on our prod servers. >> Is this the setting you're referring to? Found we were set to to 1024 using >> the "Ulimit" command. >> >> -Original Message- >> From: Shawn Heisey [mailto:apa...@elyograg.org] >> Sent: Friday, January 19, 2018 10:48 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr Replication being flaky (6.2.0) >> >> On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >>> So we're running Solr in a Master/Slave configuration (1 of each) and it >>> seems that the replication stalls or stops functioning every now and again. >>> If we restart the Solr service or optimize the core it seems to kick back >>> in again. >>> >>> Anyone have any idea what might be causing this? We do have a good amount >>> of cores on each server (@150 or so), but I have heard reports of a LOT >>> more than that in use. >> >> Have you increased the number of processes that the user running Solr is >> allowed to start? Most operating systems limit the number of >> threads/processes a user can start to a low value like 1024. With 150 >> cores, particularly with background tasks like replication configured, >> chances are that Solr is going to need to start a lot of threads. This is >> an OS setting that a lot of Solr admins end up needing to increase. >> >> I ran into the process limit on my servers and I don't have anywhere near >> 150 cores. >> >> The fact that restarting Solr gets it working again (at least >> temporarily) would fit with a process limit being the problem. I'm not >> guaranteeing that this is the problem, only saying that it fits. >> >> Thanks, >> Shawn
Re: Solr Replication being flaky (6.2.0)
"Could be", certainly. "Definitely is" is iffier ;)... But the statement "If we restart the Solr service or optimize the core it seems to kick back in again.", especially the "optimize" bit (which, by the way you should do only if you have the capability of doing it periodically [1]) is some evidence that this may be in the vicinity. One of the effects of an optimize is to merge your segments files from N to 1. So say you have 10 segments. Each one of those may consist of 10-15 individual files, all of which are held open. So you'd go from 150 open file handles to 15.. https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/ Best, Erick On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Erick, > > Thanks! Could these settings be toying with replication? Solr itself seems > to be working like a champ, except when things get out of sync. > > Scott > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, January 19, 2018 12:27 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Solr Replication being flaky (6.2.0) > > Scott: > > We usually recommend setting files and processes very, very high. Like 65K > high. Or unlimited if you can. > > Plus max user processes should also be bumped very high as well, like 65K as > well. > > Plus max memory and virtual memory should be unlimited. > > We've included warnings at startup for open files and processes, see > SOLR-11703 > > Best, > Erick > > On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott > <scott.poul...@peoplefluent.com> wrote: >> I do have a ticket in with our systems team to up the file handlers since I >> am seeing the "Too many files open" error on occasion on our prod servers. >> Is this the setting you're referring to? Found we were set to to 1024 using >> the "Ulimit" command. >> >> -Original Message- >> From: Shawn Heisey [mailto:apa...@elyograg.org] >> Sent: Friday, January 19, 2018 10:48 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr Replication being flaky (6.2.0) >> >> On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >>> So we're running Solr in a Master/Slave configuration (1 of each) and it >>> seems that the replication stalls or stops functioning every now and again. >>> If we restart the Solr service or optimize the core it seems to kick back >>> in again. >>> >>> Anyone have any idea what might be causing this? We do have a good amount >>> of cores on each server (@150 or so), but I have heard reports of a LOT >>> more than that in use. >> >> Have you increased the number of processes that the user running Solr is >> allowed to start? Most operating systems limit the number of >> threads/processes a user can start to a low value like 1024. With 150 >> cores, particularly with background tasks like replication configured, >> chances are that Solr is going to need to start a lot of threads. This is >> an OS setting that a lot of Solr admins end up needing to increase. >> >> I ran into the process limit on my servers and I don't have anywhere near >> 150 cores. >> >> The fact that restarting Solr gets it working again (at least >> temporarily) would fit with a process limit being the problem. I'm not >> guaranteeing that this is the problem, only saying that it fits. >> >> Thanks, >> Shawn
RE: Solr Replication being flaky (6.2.0)
Erick, Thanks! Could these settings be toying with replication? Solr itself seems to be working like a champ, except when things get out of sync. Scott -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 12:27 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Solr Replication being flaky (6.2.0) Scott: We usually recommend setting files and processes very, very high. Like 65K high. Or unlimited if you can. Plus max user processes should also be bumped very high as well, like 65K as well. Plus max memory and virtual memory should be unlimited. We've included warnings at startup for open files and processes, see SOLR-11703 Best, Erick On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > I do have a ticket in with our systems team to up the file handlers since I > am seeing the "Too many files open" error on occasion on our prod servers. > Is this the setting you're referring to? Found we were set to to 1024 using > the "Ulimit" command. > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Friday, January 19, 2018 10:48 AM > To: solr-user@lucene.apache.org > Subject: Re: Solr Replication being flaky (6.2.0) > > On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >> So we're running Solr in a Master/Slave configuration (1 of each) and it >> seems that the replication stalls or stops functioning every now and again. >> If we restart the Solr service or optimize the core it seems to kick back in >> again. >> >> Anyone have any idea what might be causing this? We do have a good amount >> of cores on each server (@150 or so), but I have heard reports of a LOT more >> than that in use. > > Have you increased the number of processes that the user running Solr is > allowed to start? Most operating systems limit the number of > threads/processes a user can start to a low value like 1024. With 150 cores, > particularly with background tasks like replication configured, chances are > that Solr is going to need to start a lot of threads. This is an OS setting > that a lot of Solr admins end up needing to increase. > > I ran into the process limit on my servers and I don't have anywhere near 150 > cores. > > The fact that restarting Solr gets it working again (at least > temporarily) would fit with a process limit being the problem. I'm not > guaranteeing that this is the problem, only saying that it fits. > > Thanks, > Shawn
Re: Solr Replication being flaky (6.2.0)
Scott: We usually recommend setting files and processes very, very high. Like 65K high. Or unlimited if you can. Plus max user processes should also be bumped very high as well, like 65K as well. Plus max memory and virtual memory should be unlimited. We've included warnings at startup for open files and processes, see SOLR-11703 Best, Erick On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > I do have a ticket in with our systems team to up the file handlers since I > am seeing the "Too many files open" error on occasion on our prod servers. > Is this the setting you're referring to? Found we were set to to 1024 using > the "Ulimit" command. > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Friday, January 19, 2018 10:48 AM > To: solr-user@lucene.apache.org > Subject: Re: Solr Replication being flaky (6.2.0) > > On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >> So we're running Solr in a Master/Slave configuration (1 of each) and it >> seems that the replication stalls or stops functioning every now and again. >> If we restart the Solr service or optimize the core it seems to kick back in >> again. >> >> Anyone have any idea what might be causing this? We do have a good amount >> of cores on each server (@150 or so), but I have heard reports of a LOT more >> than that in use. > > Have you increased the number of processes that the user running Solr is > allowed to start? Most operating systems limit the number of > threads/processes a user can start to a low value like 1024. With 150 cores, > particularly with background tasks like replication configured, chances are > that Solr is going to need to start a lot of threads. This is an OS setting > that a lot of Solr admins end up needing to increase. > > I ran into the process limit on my servers and I don't have anywhere near 150 > cores. > > The fact that restarting Solr gets it working again (at least > temporarily) would fit with a process limit being the problem. I'm not > guaranteeing that this is the problem, only saying that it fits. > > Thanks, > Shawn
RE: Solr Replication being flaky (6.2.0)
I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command. -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Friday, January 19, 2018 10:48 AM To: solr-user@lucene.apache.org Subject: Re: Solr Replication being flaky (6.2.0) On 1/19/2018 7:50 AM, Pouliot, Scott wrote: > So we're running Solr in a Master/Slave configuration (1 of each) and it > seems that the replication stalls or stops functioning every now and again. > If we restart the Solr service or optimize the core it seems to kick back in > again. > > Anyone have any idea what might be causing this? We do have a good amount of > cores on each server (@150 or so), but I have heard reports of a LOT more > than that in use. Have you increased the number of processes that the user running Solr is allowed to start? Most operating systems limit the number of threads/processes a user can start to a low value like 1024. With 150 cores, particularly with background tasks like replication configured, chances are that Solr is going to need to start a lot of threads. This is an OS setting that a lot of Solr admins end up needing to increase. I ran into the process limit on my servers and I don't have anywhere near 150 cores. The fact that restarting Solr gets it working again (at least temporarily) would fit with a process limit being the problem. I'm not guaranteeing that this is the problem, only saying that it fits. Thanks, Shawn
Re: Solr Replication being flaky (6.2.0)
On 1/19/2018 7:50 AM, Pouliot, Scott wrote: So we're running Solr in a Master/Slave configuration (1 of each) and it seems that the replication stalls or stops functioning every now and again. If we restart the Solr service or optimize the core it seems to kick back in again. Anyone have any idea what might be causing this? We do have a good amount of cores on each server (@150 or so), but I have heard reports of a LOT more than that in use. Have you increased the number of processes that the user running Solr is allowed to start? Most operating systems limit the number of threads/processes a user can start to a low value like 1024. With 150 cores, particularly with background tasks like replication configured, chances are that Solr is going to need to start a lot of threads. This is an OS setting that a lot of Solr admins end up needing to increase. I ran into the process limit on my servers and I don't have anywhere near 150 cores. The fact that restarting Solr gets it working again (at least temporarily) would fit with a process limit being the problem. I'm not guaranteeing that this is the problem, only saying that it fits. Thanks, Shawn
RE: Solr Replication being flaky (6.2.0)
I'm at the point now where I may end up writing a script to compare
master/slave nightly...and trigger an optimize or solr restart if there are any
differences. Of course I have to check 150+ cores...but it could be done. I'm
just hoping I don't need to go that route
-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com]
Sent: Friday, January 19, 2018 10:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)
This happens to me quite often as well. Generally on the replication admin
screen it will say its downloading a file, but be at 0 or a VERY small kb/sec.
Then after a restart of the slave its back to downloading at 30 to
100 mg/sec. Would be curious if there actually is a solution to this aside
from checking every day if the core replicated. Im on Solr 5.x by the way -Dave
On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:
> So we're running Solr in a Master/Slave configuration (1 of each) and
> it seems that the replication stalls or stops functioning every now
> and again. If we restart the Solr service or optimize the core it
> seems to kick back in again.
>
> Anyone have any idea what might be causing this? We do have a good
> amount of cores on each server (@150 or so), but I have heard reports
> of a LOT more than that in use.
>
> Here is our master config:
>
>
>
> startup
> commit
>
>
> 00:00:10
>
>
>
> 1
>
>
>
> And our slave config:
>
>
>
>
>name="masterUrl">http://server1:8080/solr/${https://na01.safelinks.pro
> tection.outlook.com/?url=solr.core.name=02%7C01%7CScott.Pouliot%4
> 0peoplefluent.com%7C8d43918dd95540a3a11708d55f523302%7C8b16fb62c78448b
> 6aba889567990e7fe%7C1%7C1%7C636519729029923349=Fes6G36gIMRyfahTI
> fftg0eUEVEiVK77B8KpuTr%2FJrA%3D=0}
>
>
>
> 00:00:45
>
>
>
>
>
> solr-data-config.xml
>
>
>
Re: Solr Replication being flaky (6.2.0)
This happens to me quite often as well. Generally on the replication admin
screen it will say its downloading a file, but be at 0 or a VERY small
kb/sec. Then after a restart of the slave its back to downloading at 30 to
100 mg/sec. Would be curious if there actually is a solution to this aside
from checking every day if the core replicated. Im on Solr 5.x by the way
-Dave
On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:
> So we're running Solr in a Master/Slave configuration (1 of each) and it
> seems that the replication stalls or stops functioning every now and
> again. If we restart the Solr service or optimize the core it seems to
> kick back in again.
>
> Anyone have any idea what might be causing this? We do have a good amount
> of cores on each server (@150 or so), but I have heard reports of a LOT
> more than that in use.
>
> Here is our master config:
>
>
>
> startup
> commit
>
>
> 00:00:10
>
>
>
> 1
>
>
>
> And our slave config:
>
>
>
>
> http://server1:8080/solr/${solr.core.name}
>
>
>
> 00:00:45
>
>
>
>
>
> solr-data-config.xml
>
>
>
Solr Replication being flaky (6.2.0)
So we're running Solr in a Master/Slave configuration (1 of each) and it seems
that the replication stalls or stops functioning every now and again. If we
restart the Solr service or optimize the core it seems to kick back in again.
Anyone have any idea what might be causing this? We do have a good amount of
cores on each server (@150 or so), but I have heard reports of a LOT more than
that in use.
Here is our master config:
startup
commit
00:00:10
1
And our slave config:
http://server1:8080/solr/${solr.core.name}
00:00:45
solr-data-config.xml
Re: Solr replication
Thanks Eric, fixed the issue. The IT team corrected the solrconfig.xml but
forgot to execute the zkcli.sh script on solr node. After I executed the script
its working now.
On 9/20/17, 10:20 AM, "Erick Erickson" <erickerick...@gmail.com> wrote:
WARNING - External email; exercise caution.
Your solrconfig.xml file is mal-formed. The smoking gun is:
Exception during parsing file: solrconfig.xml
Best,
Erick
On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
<satyaprashant.bezw...@nasdaq.com> wrote:
> Need some inputs or help in resolving replication across solr nodes. We
have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes
configured. Enabled Solr replication in my Solrj client but the replication
fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1
zookeeper and 3 Solr nodes configured. Here is the exception I see on one of
the nodes of Solr when I try to create a collection in the environment where it
fails. I have compared the Solrconfig.xml on both the environments and didn’t
see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ]
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ]
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ]
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO
Re: Solr replication
Your solrconfig.xml file is mal-formed. The smoking gun is:
Exception during parsing file: solrconfig.xml
Best,
Erick
On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
<satyaprashant.bezw...@nasdaq.com> wrote:
> Need some inputs or help in resolving replication across solr nodes. We have
> installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes
> configured. Enabled Solr replication in my Solrj client but the replication
> fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 zookeeper
> and 3 Solr nodes configured. Here is the exception I see on one of the nodes
> of Solr when I try to create a collection in the environment where it fails.
> I have compared the Solrconfig.xml on both the environments and didn’t see
> any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
> action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
> status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
> [ ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
> /var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
> action=UNLOAD=true=true
> 2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
> [admin] webapp=null path=/admin/cores
> params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
> status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
> [ ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ]
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
> /var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading
> solr.xml from SolrHome (not found in ZooKeeper)
> 2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading
> container configuration from /var/solr/
Solr replication
Need some inputs or help in resolving replication across solr nodes. We have
installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes
configured. Enabled Solr replication in my Solrj client but the replication
fails and is unable to create a collection.
The same code works in our different environment where in I have 1 zookeeper
and 3 Solr nodes configured. Here is the exception I see on one of the nodes of
Solr when I try to create a collection in the environment where it fails. I
have compared the Solrconfig.xml on both the environments and didn’t see any
difference.
2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD=true=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
status=0 QTime=6
2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer
(id=170740497916499012-sr01:8983_solr-n_29) closing
2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
at java.lang.Thread.run(Thread.java:745)
2017-09-19 22:09:35.471 ERROR
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd :
action=UNLOAD=true=true
2017-09-19 22:09:35.486 INFO (qtp401424608-15) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={deleteInstanceDir=true=CIUZLW_shard1_replica1=/admin/cores=true=UNLOAD=javabin=2}
status=0 QTime=6
2017-09-19 22:09:36.194 INFO
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [ ]
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection
for [CIUZLW]
2017-09-19 22:09:38.008 INFO
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
[ ] o.a.s.c.OverseerTaskQueue Response ZK path:
/overseer/collection-queue-work/qnr-000410 doesn't exist. Requestor may
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO (ShutdownMonitor) [ ] o.a.s.c.CoreContainer
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO (ShutdownMonitor) [ ] o.a.s.c.Overseer Overseer
(id=170740497916499012-sr01:8983_solr-n_29) closing
2017-09-19 22:38:36.645 INFO
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [ ]
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO (ShutdownMonitor) [ ] o.a.s.m.SolrMetricManager
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f
/var/solr/logs/solr.log
2017-09-19 23:12:22.230 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading
solr.xml from SolrHome (not found in ZooKeeper)
2017-09-19 23:12:22.233 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading
container configuration from /var/solr/data/Node122/solr.xml
2017-09-19 23:12:22.644 INFO (main) [ ] o.a.s.u.UpdateShardHandler Creating
UpdateShardHandler HTTP client with params:
socketTimeout=60=6=true
2017-09-19 23:12:22.650 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
client=zk01:2181,zk02:2181,zk03:2181
2017-09-19 23:12:22.731 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated live
nodes from ZooKeeper... (0) -> (1)
2017-09-19 23:12:22.762 INFO (main) [ ] o.a.s.c.Overseer Overseer (id=null)
closing
2017-09-19 23:12:22.780 INFO (main) [ ] o.a.s.c.ZkController Register node
as live in ZooKeeper:/live_nodes/sr01:8983_solr
2017
Solr replication failure then restart.
Hi, I have a solr replication set up from master to slave in legacy ( It's not from the solr cloud). Somehow the first initial replication doesn't finish and when it reaches 99% and got the error as following and then restart from the beginning. I don't know why it is keep retriggering to start the replication over and over. ERROR ReplicationHandler Index fetch failed :org.apache.solr.common.SolrException: Unable to download _55sm.si completely. Downloaded 0!=363 Here is the config in the slave. I wonder if solrconfig has any misconfiguration. startup commit 0 http://url.com 00:05:00 Anyone had a similar experience, please share how to resolve the issue. Thanks.
SOLR replication: different behavior for network cut off vs. machine restart
Hi, We try to set up a SOLR Cloud environment using 1 shard with 2 replicas (1 leader). The replicas are managed by 3 zookeeper instances. The setup seems fine when we do the normal work. The data is being replicated at runtime. Now we try to simulate erroneous behavior in several cases: Turn off one of the replicas in two different scenarios: leader and non-leader Cutting off the network making the non-leader replica down In both cases the data is being written contentiously to the SOLR Cloud. CASE 1: The replication process starts after the failed machine gets boot up again. The complete data set is present in both replicas. Everything works fine. CASE 2: Once reconnected to network the non-leader replica starts the recovery process ,but for some reason the new data from leader is not being replicated onto the previously failed replica. >From what I was able to read from logs comparing both cases I don't understand why SOLR sees RecoveryStrategy ## currentVersions as present and RecoveryStrategy ## startupVersions=[[]] (empty) compared to CASE 1 when RecoveryStrategy ## startupVersions are filled with objects that are in currentVersions in CASE 2 The general question is... why restarting SOLR results in a successful migration process, but reconnecting the network does not? Thanks for any tips / leads! Cheers, Greg
Solr Replication error
I cached this in my logs. Any reason to this happen? My Solr version is 5.3.1. Index fetch failed :org.apache.solr.common.SolrException: Index fetch failed : at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:515) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:380) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:162) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227) Caused by: java.nio.file.NoSuchFileException: /opt/solrcloud-node/collections/collection-2016_shard9_replica2/data/index.20160105005921682/segments_fhj at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) at java.nio.channels.FileChannel.open(FileChannel.java:287) at java.nio.channels.FileChannel.open(FileChannel.java:335) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:236) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109) at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294) at org.apache.solr.handler.IndexFetcher.hasUnusedFiles(IndexFetcher.java:568) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:397) ... 5 more - Best regards -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-error-tp4252929.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication sometimes coming in log files
I bet you have the admin UI open on your second slave. The _=144... is
the give-away. Those requests are the admin UI asking the replication
handler for the status of replication.
Upayavira
On Wed, Sep 9, 2015, at 06:32 AM, Kamal Kishore Aggarwal wrote:
> Hi Team,
>
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
>
>
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1
> is in another server in different zone (say zone B). There is latency of
> 40
> ms between two zones.
>
> Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> to delay in data replication from Master server. These days we are
> finding
> these below mentioned replication information in log files, but such
> lines
> are not in previous files on the Slave 1 server. Also, such information
> is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
>
>
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708786003} status=0 QTime=173
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708787976} status=0 QTime=1807
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708791563} status=0 QTime=7140
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json=details&_=1441708800450} status=0 QTime=1679
>
>
>
> Please confirm if we our thought that increased replication time (which
> can
> be due to servers connectivity issues) is the reason for high load on
> solr.
>
> Regards
> Kamal Kishore
Re: Solr Replication sometimes coming in log files
Hello,
I'd say opposite: high load causes long time to respond. 'command=details'
is rather cheap and fast _I believe_.
On Mon, Sep 14, 2015 at 10:20 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:
> Can anybody suggest me something..
>
> On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
> > Hi Team,
> >
> > I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> > configuration has master & slave ( 2 Slaves) architecture.
> >
> >
> > Master & Slave 2 are in same server location (say zone A) , whereas Slave
> > 1 is in another server in different zone (say zone B). There is latency
> of
> > 40 ms between two zones.
> >
> > Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> > to delay in data replication from Master server. These days we are
> finding
> > these below mentioned replication information in log files, but such
> lines
> > are not in previous files on the Slave 1 server. Also, such information
> is
> > not there in any Slave 2 log files (might be due to same zone of master &
> > slave 2).
> >
> >
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708786003} status=0 QTime=173
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708787976} status=0 QTime=1807
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708791563} status=0 QTime=7140
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json=details&_=1441708800450} status=0 QTime=1679
> >
> >
> >
> > Please confirm if we our thought that increased replication time (which
> > can be due to servers connectivity issues) is the reason for high load on
> > solr.
> >
> > Regards
> > Kamal Kishore
> >
> >
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
Re: Solr Replication sometimes coming in log files
Can anybody suggest me something..
On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:
> Hi Team,
>
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
>
>
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1 is in another server in different zone (say zone B). There is latency of
> 40 ms between two zones.
>
> Now, a days we are facing high load on Slave 1 & we suspect that it is due
> to delay in data replication from Master server. These days we are finding
> these below mentioned replication information in log files, but such lines
> are not in previous files on the Slave 1 server. Also, such information is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
>
>
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708786003} status=0 QTime=173
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708787976} status=0 QTime=1807
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708791563} status=0 QTime=7140
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json=details&_=1441708800450} status=0 QTime=1679
>
>
>
> Please confirm if we our thought that increased replication time (which
> can be due to servers connectivity issues) is the reason for high load on
> solr.
>
> Regards
> Kamal Kishore
>
>
Solr Replication sometimes coming in log files
Hi Team,
I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
configuration has master & slave ( 2 Slaves) architecture.
Master & Slave 2 are in same server location (say zone A) , whereas Slave 1
is in another server in different zone (say zone B). There is latency of 40
ms between two zones.
Now, a days we are facing high load on Slave 1 & we suspect that it is due
to delay in data replication from Master server. These days we are finding
these below mentioned replication information in log files, but such lines
are not in previous files on the Slave 1 server. Also, such information is
not there in any Slave 2 log files (might be due to same zone of master &
slave 2).
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708786003} status=0 QTime=173
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708787976} status=0 QTime=1807
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708791563} status=0 QTime=7140
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json=details&_=1441708800450} status=0 QTime=1679
Please confirm if we our thought that increased replication time (which can
be due to servers connectivity issues) is the reason for high load on solr.
Regards
Kamal Kishore
Re: Weird Solr Replication Slave out of sync
Hi, This sounds quite strange. Do you see any error messages either in the solr admin's replication page or in the master's OR slave's logs? When we had issues with slave replicating from the master, they related to slave running out of disk. I'm sure there could be a bunch of other reasons for failed replication, but those should generally be evident in the logs. On Tue, Feb 17, 2015 at 7:46 AM, Summer Shire shiresum...@gmail.com wrote: Hi All, My master and slave index version and generation is the same yet the index is not in sync because when I execute the same query on both master and slave I see old docs on slave which should not be there. I also tried to fetch a specific indexversion on slave using command=fetchindexindexversion=latestMasterVersion This is very spooky because I do not get any errors on master or slave. Also I see in the logs that the slave is polling the master every 15 mins I was able to find this issue only because I was looking at the specific old document. Now I can manually delete the index folder on slave and restart my slave. But I really want to find out what could be going on. Because these type of issues are going to be hard to find especially when there are on errors. What could be happening. and how can I avoid this from happening ? Thanks, Summer -- Dmitry Kan Luke Toolbox: http://github.com/DmitryKey/luke Blog: http://dmitrykan.blogspot.com Twitter: http://twitter.com/dmitrykan SemanticAnalyzer: www.semanticanalyzer.info
Weird Solr Replication Slave out of sync
Hi All, My master and slave index version and generation is the same yet the index is not in sync because when I execute the same query on both master and slave I see old docs on slave which should not be there. I also tried to fetch a specific indexversion on slave using command=fetchindexindexversion=latestMasterVersion This is very spooky because I do not get any errors on master or slave. Also I see in the logs that the slave is polling the master every 15 mins I was able to find this issue only because I was looking at the specific old document. Now I can manually delete the index folder on slave and restart my slave. But I really want to find out what could be going on. Because these type of issues are going to be hard to find especially when there are on errors. What could be happening. and how can I avoid this from happening ? Thanks, Summer
Re: solr replication vs. rsync
On 1/24/2015 10:56 PM, Dan Davis wrote: When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!).Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there. Numbers are included on the Solr replication wiki page, both in graph and numeric form. Gathering these numbers must have been pretty easy -- before the HTTP replication made it into Solr, Solr used to contain an rsync-based implementation. http://wiki.apache.org/solr/SolrReplication#Performance_numbers Other data on that wiki page discusses the replication config. There's not a lot to tune. I run a redundant non-SolrCloud index myself through a different method -- my indexing program indexes each index copy completely independently. There is no replication. This separation allows me to upgrade any component, or change any part of solrconfig or schema, on either copy of the index without affecting the other copy at all. With replication, if something is changed on the master or the slave, you might find that the slave no longer works, because it will be handling an index created by different software or a different config. Thanks, Shawn
Re: solr replication vs. rsync
bq: I thought SolrCloud replicas were replication, and you imply parallel indexing Absolutely! You couldn't get near-real-time indexing if you relied on replication a-la 3x. And you also couldn't guarantee consistency. Say you have 1 shard, a leader and a follower (i.e. 2 replicas). Now you throw a doc to be indexed. The sequence is: leader gets the doc leader forwards the doc to the follower leader and follower both add the doc to their local index (and tlog). follower acks back to leader leader acks back to client. So yes, the raw document is forwarded to all replicas before the leader responds to the client, the docs all get written to the tlogs, etc. That's the only way to guarantee that if the leader goes down, the follower can take over without losing documents. Best, Erick On Sun, Jan 25, 2015 at 6:15 PM, Dan Davis dansm...@gmail.com wrote: @Erick, Problem space is not constant indexing. I thought SolrCloud replicas were replication, and you imply parallel indexing. Good to know. On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com wrote: @Shawn: Cool table, thanks! @Dan: Just to throw a different spin on it, if you migrate to SolrCloud, then this question becomes moot as the raw documents are sent to each of the replicas so you very rarely have to copy the full index. Kind of a tradeoff between constant load because you're sending the raw documents around whenever you index and peak usage when the index replicates. There are a bunch of other reasons to go to SolrCloud, but you know your problem space best. FWIW, Erick On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org javascript:; wrote: On 1/24/2015 10:56 PM, Dan Davis wrote: When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!). Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there. Numbers are included on the Solr replication wiki page, both in graph and numeric form. Gathering these numbers must have been pretty easy -- before the HTTP replication made it into Solr, Solr used to contain an rsync-based implementation. http://wiki.apache.org/solr/SolrReplication#Performance_numbers Other data on that wiki page discusses the replication config. There's not a lot to tune. I run a redundant non-SolrCloud index myself through a different method -- my indexing program indexes each index copy completely independently. There is no replication. This separation allows me to upgrade any component, or change any part of solrconfig or schema, on either copy of the index without affecting the other copy at all. With replication, if something is changed on the master or the slave, you might find that the slave no longer works, because it will be handling an index created by different software or a different config. Thanks, Shawn
Re: solr replication vs. rsync
@Shawn: Cool table, thanks! @Dan: Just to throw a different spin on it, if you migrate to SolrCloud, then this question becomes moot as the raw documents are sent to each of the replicas so you very rarely have to copy the full index. Kind of a tradeoff between constant load because you're sending the raw documents around whenever you index and peak usage when the index replicates. There are a bunch of other reasons to go to SolrCloud, but you know your problem space best. FWIW, Erick On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org wrote: On 1/24/2015 10:56 PM, Dan Davis wrote: When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!). Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there. Numbers are included on the Solr replication wiki page, both in graph and numeric form. Gathering these numbers must have been pretty easy -- before the HTTP replication made it into Solr, Solr used to contain an rsync-based implementation. http://wiki.apache.org/solr/SolrReplication#Performance_numbers Other data on that wiki page discusses the replication config. There's not a lot to tune. I run a redundant non-SolrCloud index myself through a different method -- my indexing program indexes each index copy completely independently. There is no replication. This separation allows me to upgrade any component, or change any part of solrconfig or schema, on either copy of the index without affecting the other copy at all. With replication, if something is changed on the master or the slave, you might find that the slave no longer works, because it will be handling an index created by different software or a different config. Thanks, Shawn
Re: solr replication vs. rsync
@Erick, Problem space is not constant indexing. I thought SolrCloud replicas were replication, and you imply parallel indexing. Good to know. On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com wrote: @Shawn: Cool table, thanks! @Dan: Just to throw a different spin on it, if you migrate to SolrCloud, then this question becomes moot as the raw documents are sent to each of the replicas so you very rarely have to copy the full index. Kind of a tradeoff between constant load because you're sending the raw documents around whenever you index and peak usage when the index replicates. There are a bunch of other reasons to go to SolrCloud, but you know your problem space best. FWIW, Erick On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org javascript:; wrote: On 1/24/2015 10:56 PM, Dan Davis wrote: When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!). Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there. Numbers are included on the Solr replication wiki page, both in graph and numeric form. Gathering these numbers must have been pretty easy -- before the HTTP replication made it into Solr, Solr used to contain an rsync-based implementation. http://wiki.apache.org/solr/SolrReplication#Performance_numbers Other data on that wiki page discusses the replication config. There's not a lot to tune. I run a redundant non-SolrCloud index myself through a different method -- my indexing program indexes each index copy completely independently. There is no replication. This separation allows me to upgrade any component, or change any part of solrconfig or schema, on either copy of the index without affecting the other copy at all. With replication, if something is changed on the master or the slave, you might find that the slave no longer works, because it will be handling an index created by different software or a different config. Thanks, Shawn
Re: solr replication vs. rsync
Thanks! On Sunday, January 25, 2015, Erick Erickson erickerick...@gmail.com wrote: @Shawn: Cool table, thanks! @Dan: Just to throw a different spin on it, if you migrate to SolrCloud, then this question becomes moot as the raw documents are sent to each of the replicas so you very rarely have to copy the full index. Kind of a tradeoff between constant load because you're sending the raw documents around whenever you index and peak usage when the index replicates. There are a bunch of other reasons to go to SolrCloud, but you know your problem space best. FWIW, Erick On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey apa...@elyograg.org javascript:; wrote: On 1/24/2015 10:56 PM, Dan Davis wrote: When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!). Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there. Numbers are included on the Solr replication wiki page, both in graph and numeric form. Gathering these numbers must have been pretty easy -- before the HTTP replication made it into Solr, Solr used to contain an rsync-based implementation. http://wiki.apache.org/solr/SolrReplication#Performance_numbers Other data on that wiki page discusses the replication config. There's not a lot to tune. I run a redundant non-SolrCloud index myself through a different method -- my indexing program indexes each index copy completely independently. There is no replication. This separation allows me to upgrade any component, or change any part of solrconfig or schema, on either copy of the index without affecting the other copy at all. With replication, if something is changed on the master or the slave, you might find that the slave no longer works, because it will be handling an index created by different software or a different config. Thanks, Shawn
solr replication vs. rsync
When I polled the various projects already using Solr at my organization, I was greatly surprised that none of them were using Solr replication, because they had talked about replicating the data. But we are not Pinterest, and do not expect to be taking in changes one post at a time (at least the engineers don't - just wait until its used for a Crud app that wants full-text search on a description field!).Still, rsync can be very, very fast with the right options (-W for gigabit ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over GigE previously. Does anyone have any numbers for how fast Solr replication goes, and what to do to tune it? I'm not enthusiastic to give-up recently tested cluster stability for a home grown mess, but I am interested in numbers that are out there.
Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail
I haven't seen any activity regarding this in Jira, just curious if it would be looked into anytime soon... On Thu, Oct 2, 2014 at 10:11 AM, Phil Black-Knight pblackkni...@globalgiving.org wrote: see the ticket here: https://issues.apache.org/jira/browse/SOLR-6579 including a patch to fix it. On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey apa...@elyograg.org wrote: On 10/2/2014 7:25 AM, Phil Black-Knight wrote: I was helping to look into this with Nick I think we may have figured out the core of the problem... The problem is easily reproducible by starting replication on the slave and then sending a shutdown command to tomcat (e.g. catalina.sh stop). With a debugger attached, it looks like the fsyncService thread is blocking VM shutdown because it is created as a non-daemon thread. snip I can submit patches if needed... and cross post to the dev mailing list... File a detailed issue in Jira and attach your patch there. This is our bugtracker. You need an account on the Apache jira instance to do this: https://issues.apache.org/jira/browse/SOLR Thanks, Shawn
RE: Solr Replication during Tomcat shutdown causes shutdown to hang/fail
I was helping to look into this with Nick I think we may have figured out the core of the problem... The problem is easily reproducible by starting replication on the slave and then sending a shutdown command to tomcat (e.g. catalina.sh stop). With a debugger attached, it looks like the fsyncService thread is blocking VM shutdown because it is created as a non-daemon thread. Essentially what seems to be happening is that the fsyncService thread is running when 'catalina.sh stop' is executed. This goes in and calls SnapPuller.destroy() which aborts the current sync. Around line 517 of the SnapPuller, there is code that is supposed to cleanup the fsyncService thread, but I don't think it is getting executed because the thread that called SnapPuller.fetchLatestIndex() is configured as a daemon Thread, so the JVM ends up shutting that down before it can cleanup the fysncService... So... it seems like: if (fsyncService != null) ExecutorUtil.shutdownNowAndAwaitTermination(fsyncService); could be added around line 1706 of SnapPuller.java, or puller.setDaemon(*false*); could be added around line 230 of ReplicationHandler.java, however this needs some additional work (and I think it might need to be added regardless) since the cleanup code in SnapPuller(around 517) that shuts down the fsync thread never gets execute since logReplicationTimeAndConfFiles() can throw IO exceptions bypassing the rest of the finally block...So the call to logReplicationTimeAndConfFiles() around line 512 would need to get wrapped with a try/catch block to catch the IO exception... I can submit patches if needed... and cross post to the dev mailing list... -Phil
Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail
On 10/2/2014 7:25 AM, Phil Black-Knight wrote: I was helping to look into this with Nick I think we may have figured out the core of the problem... The problem is easily reproducible by starting replication on the slave and then sending a shutdown command to tomcat (e.g. catalina.sh stop). With a debugger attached, it looks like the fsyncService thread is blocking VM shutdown because it is created as a non-daemon thread. snip I can submit patches if needed... and cross post to the dev mailing list... File a detailed issue in Jira and attach your patch there. This is our bugtracker. You need an account on the Apache jira instance to do this: https://issues.apache.org/jira/browse/SOLR Thanks, Shawn
Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail
see the ticket here: https://issues.apache.org/jira/browse/SOLR-6579 including a patch to fix it. On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey apa...@elyograg.org wrote: On 10/2/2014 7:25 AM, Phil Black-Knight wrote: I was helping to look into this with Nick I think we may have figured out the core of the problem... The problem is easily reproducible by starting replication on the slave and then sending a shutdown command to tomcat (e.g. catalina.sh stop). With a debugger attached, it looks like the fsyncService thread is blocking VM shutdown because it is created as a non-daemon thread. snip I can submit patches if needed... and cross post to the dev mailing list... File a detailed issue in Jira and attach your patch there. This is our bugtracker. You need an account on the Apache jira instance to do this: https://issues.apache.org/jira/browse/SOLR Thanks, Shawn
Solr Replication during Tomcat shutdown causes shutdown to hang/fail
Hello,
I have solr 4.10 running on tomcat 7. I'm doing replication from one master
to about 10 slaves, with standard configuration:
requestHandler name=/replication class=solr.ReplicationHandler
lst name=master
str name=enable${enable.master:false}/str
str name=replicateAftercommit/str
str name=replicateAfterstartup/str
str name=confFilesschema.xml,stopwords.txt/str
!-- str name=commitReserveDuration00:00:10/str --
/lst
lst name=slave
str name=enable${enable.slave:false}/str
str name=masterUrlhttp://master:8080/solr/mycore/str
str name=pollInterval00:00:60/str
/lst
/requestHandler
It appears that if tomcat gets shutdown while solr is replicating, it
prevents tomcat from shutting down fully. Immediately after receiving the
shutdown command, a thread dump is logged into catalina.out (this may have
been turned on by some configuration someone else on my team made). I
removed some threads that didn't look related, mostly about tomcat session
replication, or with names like http-bio-8080-exec-10.
62252 [http-bio-8080-exec-1] INFO org.apache.solr.core.SolrCore –
[mycore] webapp=/solr path=/replication
params={command=details_=1412014928648wt=json} status=0 QTime=6
63310 [http-bio-8080-exec-1] INFO org.apache.solr.core.SolrCore –
[mycore] webapp=/solr path=/replication
params={command=details_=1412014929699wt=json} status=0 QTime=6
2014-09-29 14:22:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed
mode):
fsyncService-12-thread-1 prio=10 tid=0x7f3bd4002000 nid=0x203d
waiting on condition [0x7f3c271f]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007e1ff4458 (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
explicit-fetchindex-cmd daemon prio=10 tid=0x7f3c0413e800
nid=0x203c runnable [0x7f3c272f1000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:174)
at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:80)
at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:114)
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:152)
at
org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1239)
at
org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1187)
at
org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:774)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:424)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
at
org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:227)
process reaper daemon prio=10 tid=0x7f3c0409c000 nid=0x203b
waiting on condition [0x7f3c984e9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007dfbfd890 (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
SolR replication issue
Hi folks, I'm using Solr 4.3.1 with a master/slave configuration. Configuration: Master: * lst name=master* * str name=replicateAftercommit/str* * str name=replicateAfterstartup/str* * str name=confFilesschema.xml,stopwords.txt/str* * /lst* Slave: * lst name=slave* * str name=masterUrlhttp://10.xx.xx.xx:9081/solr http://10.xx.xx.xx:9081/solr/str* * str name=pollInterval00:00:60/str* * /lst* The replication sometimes fails with the exception *Error closing old IndexWriter. core=collection1:java.lang.IllegalArgumentException: Unknown directory: NRTCachingDirectory...* *ReplicationHandler SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :* This is happening with any index size. Any suggestions would be great Thanks, -- *Mauri Ferreyra* Cordoba - Argentina
Re: SolR replication issue
On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote: I'm using Solr 4.3.1 with a master/slave configuration. Configuration: Master: * lst name=master* * str name=replicateAftercommit/str* * str name=replicateAfterstartup/str* * str name=confFilesschema.xml,stopwords.txt/str* * /lst* Slave: * lst name=slave* * str name=masterUrlhttp://10.xx.xx.xx:9081/solr http://10.xx.xx.xx:9081/solr/str* * str name=pollInterval00:00:60/str* * /lst* The replication sometimes fails with the exception *Error closing old IndexWriter. core=collection1:java.lang.IllegalArgumentException: Unknown directory: NRTCachingDirectory...* *ReplicationHandler SnapPull failed :org.apache.solr.common.SolrException: Index fetch failed :* This is happening with any index size. We would need the entire stacktrace from that error message to make any real determination, but without that, I offer this: If it's happening with a tiny index (kilobytes or only a few megabytes), then I would suspect a bug in Solr. 4.3.1 is ancient history now -- Solr's development and release schedule is very aggressive. There have been ten new versions in the 14 months since 4.3.1 was announced, and the 4.10 release is imminent. There have been a number of replication bugs fixed in those releases. If any index size means that some of them are in the range of several gigabytes, then it may simply be a configuration problem. The commitReserveDuration parameter may need increasing beyond its default of ten seconds. Large updates after optimizing or a major automatic merge can take many minutes to transfer, and if that parameter is set too low, the old index can disappear before it can finish replicating. Thanks, Shawn
Re: SolR replication issue
The entire stacktrace:
ERROR SolrIndexWriter
Coud not unlock directory after seemingly failed IndexWriter#close()
org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully
unlock a NativeFSLock which is held by another indexer component:
/home/miapp/collection1/data/index.20140901140800014/write.lock
at
org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:295)
at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4162)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:156)
at
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:164)
at
org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:624)
at
org.apache.solr.handler.SnapPuller.openNewWriterAndSearcher(SnapPuller.java:622)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:446)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317)
at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16:39:00
ERROR DefaultSolrCoreState
Error closing old IndexWriter.
core=collection1:java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77b024a9;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
{NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6de47a0f;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=0;path=/home/miapp/collection1/data;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800016
lockFactory=org.apache.lucene.store.NativeFSLockFactory@791e6360;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=-3;path=/home/miapp/collection1/data/index.20140901140800016;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901163900022
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a6de862;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=1;path=/home/miapp/collection1/data/index.20140901163900022;done=false,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@5863c9e0;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDirrefCount=1;path=/home/miapp/collection1/data/index.20140901140800014;done=false}
ERROR SolrIndexWriter SolrIndexWriter was not closed prior to finalize(),
indicates a bug -- POSSIBLE RESOURCE LEAK!!!
On Mon, Sep 1, 2014 at 4:28 PM, Shawn Heisey s...@elyograg.org wrote:
On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote:
I'm using Solr 4.3.1 with a master/slave configuration.
Configuration:
Master:
* lst name=master*
* str name=replicateAftercommit/str*
* str name=replicateAfterstartup/str*
* str name=confFilesschema.xml,stopwords.txt/str*
* /lst*
Slave:
* lst name=slave*
* str name=masterUrlhttp://10.xx.xx.xx:9081/solr
http://10.xx.xx.xx:9081/solr/str*
* str name=pollInterval00:00:60/str*
* /lst*
The replication sometimes fails with the exception
*Error closing old IndexWriter.
core=collection1:java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory...*
*ReplicationHandler SnapPull failed
:org.apache.solr.common.SolrException:
Index fetch failed :*
This is happening with any index size.
We would need the entire stacktrace from that error message to make any
real determination, but without that, I offer this:
If it's happening with a tiny index (kilobytes or only a few megabytes),
then I would suspect a bug in Solr. 4.3.1 is ancient history now --
Solr's development and release schedule is very aggressive. There have
been ten new versions in the 14 months since 4.3.1 was announced, and
the 4.10 release is imminent. There have been a number of replication
bugs fixed in those releases.
If any index size means that some of them are in the range of several
gigabytes, then it may simply be a configuration problem. The
commitReserveDuration parameter may need increasing beyond its default
of ten seconds. Large updates after optimizing or a major
Re: Solr Replication Issue : Incorrect Base_URL
Hi Pramod,
You need to set hostContext in your solr.xml. See
https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml
Alan Woodward
www.flax.co.uk
On 13 Jun 2014, at 00:44, pramodEbay wrote:
Hi,
I am deploying Solr in a larger web application. The standalone solr
instance works fine. The path-prefix I use is raptorslrweb. A standalone
SOLR query to my instance that works is as follows:
http://hostname:8080/raptorslrweb/solr/reviews/select?q=*%3A*wt=jsonindent=true
However, when I configure a solr cloud, I get the following error in
RecoveryStrategy:
msg:org.apache.solr.client.solrj.SolrServerException: Server at
http://hostname:8080/solr/reviews sent back a redirect (302).,
The reason is the base_url does not seem to honor the path-prefix.
clusterstate.json shows the following for the node:
{reviews:{
shards:{shard1:{
range:null,
state:active,
parent:null,
replicas:{
core_node1:{
state:down,
* base_url:http://hostname:8080/solr,*
core:reviews,
node_name:10.98.63.98:8080_solr},
Can someone please tell me where do I tell zookeeper or solr cloud that the
base url should be hostname:8080/raptorslrweb/solr and not
hostname:8080/solr.
Thanks,
Pramod
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-Replication-Issue-Incorrect-Base-URL-tp4141537.html
Sent from the Solr - User mailing list archive at Nabble.com.
Inconsistent behavior with Solr replication
Hi, I am using solr 4.4 , replication set with one Master and 1 slave. Master is set to replicate after startup and commit. It has an internal autocommit of maxTime:15000. Slave polls the master every 45sec to check for updates. I indexed Master with DIH, First, indexed half million docs in Master and it was replicated to slave. Second, indexed next half million docs and same was in slave. Third, while starting the index, by mistake i enalbed clean=true but immediately gave another import with clean=false. now the Master has 1.5 M docs but the slave has zero docs. How is this possible. If i give, replication?command=indexversion in both, the version and generation numbers are same. but slave has zero. *Master:* long name=indexversion1402644323820/long long name=generation695/long *Slave:* long name=indexversion1402644323820/long long name=generation695/long *Admin UI Master REplication page:* Index Version Gen Size Master (Searching) 1402644188557 694 63.6 MB Master (Replicable) 1402647815765 711 - Settings (Master): - replication enable: - replicateAfter:commit, startup *Admin UI Slave replication page:* Index Version Gen Size Master (Searching) 1402644188557 694 96.4 MB Master (Replicable) 1402647831725 712 - Slave (Searching) 1402647831725 712 67.69 MB Settings: - master url:http:/ip:port/solr/col1 - polling enable: (interval: 00:00:45) Settings (Master): - replication enable: - replicateAfter:commit, startup Thanks, Prasi
Solr Replication Issue : Incorrect Base_URL
Hi,
I am deploying Solr in a larger web application. The standalone solr
instance works fine. The path-prefix I use is raptorslrweb. A standalone
SOLR query to my instance that works is as follows:
http://hostname:8080/raptorslrweb/solr/reviews/select?q=*%3A*wt=jsonindent=true
However, when I configure a solr cloud, I get the following error in
RecoveryStrategy:
msg:org.apache.solr.client.solrj.SolrServerException: Server at
http://hostname:8080/solr/reviews sent back a redirect (302).,
The reason is the base_url does not seem to honor the path-prefix.
clusterstate.json shows the following for the node:
{reviews:{
shards:{shard1:{
range:null,
state:active,
parent:null,
replicas:{
core_node1:{
state:down,
* base_url:http://hostname:8080/solr,*
core:reviews,
node_name:10.98.63.98:8080_solr},
Can someone please tell me where do I tell zookeeper or solr cloud that the
base url should be hostname:8080/raptorslrweb/solr and not
hostname:8080/solr.
Thanks,
Pramod
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-Replication-Issue-Incorrect-Base-URL-tp4141537.html
Sent from the Solr - User mailing list archive at Nabble.com.
Solr Replication Index directory disappears
Hi, We are using solr 4.4 version and am facing some issue with replication, we are trying to replicate very large index about 105G. Once the replication is done, I see the index.timestamp directory but the index directory is not present ? Why is this happening ? Also, I see with every iteration the slave is fetching the entire content (full replication) from the master, even if nothing is changed on the master. What could be reason ? Thanks, Anand
SOLR replication question?
I am currently using SOLR 4.4. but not planning to use solrcloud in very near future. I have 3 master / 3 slave setup. Each master is linked to its corresponding slave.. I have disabled auto polling.. We do both push (using MQ) and pull indexing using SOLRJ indexing program. I have enabled soft commit in slave (to view the changes immediately pushed by queue). I am thinking of doing the batch indexing in master (optimize and hard commit) and push indexing in both master / slave. I am trying to do more testing with my configuration but thought of getting to know some answers before diving very deep... Since the queue pushes the docs in master / slave there is a possibility of slave having more record compared to master (when master is busy doing batch indexing).. What would happen if the slave has additional segments compared to Master. will that be deleted when the replication happens. If a message is pushed from a queue to both master and slave during replication, will there be a latency in seeing that document even if we use softcommit in slave? We want to make sure that we are not missing any documents from queue (since its updated via UI and we don't really store that data anywhere except in index). -- View this message in context: http://lucene.472066.n3.nabble.com/SOLR-replication-question-tp4081161.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR replication question?
I am currently using SOLR 4.4. but not planning to use solrcloud in very near future. I have 3 master / 3 slave setup. Each master is linked to its corresponding slave.. I have disabled auto polling.. We do both push (using MQ) and pull indexing using SOLRJ indexing program. I have enabled soft commit in slave (to view the changes immediately pushed by queue). I am thinking of doing the batch indexing in master (optimize and hard commit) and push indexing in both master / slave. I am trying to do more testing with my configuration but thought of getting to know some answers before diving very deep... Since the queue pushes the docs in master / slave there is a possibility of slave having more record compared to master (when master is busy doing batch indexing).. What would happen if the slave has additional segments compared to Master. will that be deleted when the replication happens. If a message is pushed from a queue to both master and slave during replication, will there be a latency in seeing that document even if we use softcommit in slave? We want to make sure that we are not missing any documents from queue (since its updated via UI and we don't really store that data anywhere except in index) If you are doing replication, then all updates must go to the master server. You cannot update the slave directly. The replication happens, the slave will be identical to the master... Any documents aent to only the slave will be lost. Replication will happen according to the interval you have configured, or since you say you have disabled polling, according to whatever schedule you manually trigger a replication. SolrCloud would probably be a better fit for you. With a properly configured SolrCloud you just index to any host in the cloud and documents end up exactly where they need to go, and all replicas get updated. Thanks, Shawn
Re: problems about solr replication in 4.3
Well, a full import is going to re-import everything in the database, and the presumption is that each every document would be replaced (because presumably you're uniqueKey is the same). So every document will be deleted and re-added. So essentially you'll get a completely new index every time. In 3.6 are you sure you weren't doing a delta query? Best Erick On Thu, Jul 25, 2013 at 9:57 PM, xiaoqi belivexia...@gmail.com wrote: thank u for replying very much . in fact ,we make a process for this problem , we found when master building index, it will clean self index when building index . so slave every minute to sync index, destroy self index folder. by the way : we building index using dataimport0?command=full-importclean=false ,dataimport1?command=full-importclean=false, dataimport2?command=full-importclean=false . when i using in solr3.6 has no problem ,never delete at first . does solr 4 need to special config anything ? thanks a lot . -- View this message in context: http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665p4080480.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: problems about solr replication in 4.3
thank u for replying very much . in fact ,we make a process for this problem , we found when master building index, it will clean self index when building index . so slave every minute to sync index, destroy self index folder. by the way : we building index using dataimport0?command=full-importclean=false ,dataimport1?command=full-importclean=false, dataimport2?command=full-importclean=false . when i using in solr3.6 has no problem ,never delete at first . does solr 4 need to special config anything ? thanks a lot . -- View this message in context: http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665p4080480.html Sent from the Solr - User mailing list archive at Nabble.com.
problems about solr replication in 4.3
hi,all i have two solr ,one is master , one is replication , before i use them under 3.5 version . it works fine . when i upgrade to 4.3version , i found when replication solr copying index from master , it will clean current index and copy new version to self folder . slave can't search during this process ! i am newer to solr 4 , does this normal ? any ideas , thanks ! -- View this message in context: http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: problems about solr replication in 4.3
Are you mixing SolrCloud and old-style master/slave? There was a bug a while ago (search the JIRA) where replication was copying the entire index unnecessarily, but I think that was fixed by 4.3. Best Erick On Tue, Jul 23, 2013 at 6:33 AM, xiaoqi belivexia...@gmail.com wrote: hi,all i have two solr ,one is master , one is replication , before i use them under 3.5 version . it works fine . when i upgrade to 4.3version , i found when replication solr copying index from master , it will clean current index and copy new version to self folder . slave can't search during this process ! i am newer to solr 4 , does this normal ? any ideas , thanks ! -- View this message in context: http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication is extremely slow(less then 1MB/s)
I agree. It is even slower when the slave is being pounded. On Fri, Jun 21, 2013 at 3:35 AM, Ted zhanghailian...@qq.com wrote: Solr replication is extremely slow(less then 1MB/s) When the replication is runinng,network and disk occupancy rate remained at a very low level. I've tried downloading a piece of the index file with browser. like this: http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file= http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file= it can reach more then 10MB/s. so i'm confused,is there anybody can tell me how to check for this problem? Environment： 1.gigabit network 2.both master slave run on windows 3.jdk 1.6u24 i586 4.solr 3.5 5.tomcat7 Looking forward to your reply,tks! -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-is-extremely-slow-less-then-1MB-s-tp4072096.html Sent from the Solr - User mailing list archive at Nabble.com. -- Bill Bell billnb...@gmail.com cell 720-256-8076
Solr replication is extremely slow(less then 1MB/s)
Solr replication is extremely slow(less then 1MB/s) When the replication is runinng,network and disk occupancy rate remained at a very low level. I've tried downloading a piece of the index file with browser. like this: http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file= http://master/solr/product/replication?command=filecontentwt=filestreamindexversion=11file= it can reach more then 10MB/s. so i'm confused,is there anybody can tell me how to check for this problem? Environment： 1.gigabit network 2.both master slave run on windows 3.jdk 1.6u24 i586 4.solr 3.5 5.tomcat7 Looking forward to your reply,tks! -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-is-extremely-slow-less-then-1MB-s-tp4072096.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Troubles with solr replication
: Also when I checked the solr log. : : [org.apache.solr.handler.SnapPuller] Master at: : http://192.168.2.204:8080/solr/replication is not available. Index fetch : failed. Exception: Connection refused : : : BTW, I was able to fetch the replication file with wget directly. Are you certian that the network setup for your master slave machines alows them to talk to eachother? you said you could fetch the files from the master via wget, but i'm guessing you were running this from your local machine -- are you certain that when logged in to 192.168.2.174 you can reach port 8080 of 192.168.2.204? -Hoss
Troubles with solr replication
Hi guys, I have some problems with Solr replication and can see some unexpected behavior. Would be nice to have some answers where am I wrong, or what is the best way to solve the problem. I have a replication master-slave. http://192.168.2.204:8080/solr/ is master and http://192.168.2.174:8080/solr/ is slave. With quite simple config. requestHandler name=/replication class=solr.ReplicationHandler lst name=master str name=enable false /str str name=replicateAfter commit /str str name=replicateAfter startup /str str name=confFiles schema.xml,boosting.txt /str /lst lst name=slave str name=enable true /str str name=masterUrlhttp://192.168.2.204:8080/solr/replication/str str name=pollInterval00:00:60/str str name=httpConnTimeout 5000 /str str name=httpReadTimeout 1 /str /lst /requestHandler The main idea when I started playing around Solr is to replicate some boosting values. I wanted to use confFile option for it. An here is my first problem. I wasn't able to replicate files from master. On slave I was able to see only schema.xml. I wanted to check, do I actually have the files and everything correct in solr config. So I checked on master the file list and it returns the list of all files http://192.168.2.204:8080/solr/replication?command=filelistindexversion=1341328964983 but for slave I can't see anything http://192.168.2.174:8080/solr/replication?command=filelistindexversion=1341328964983 returns str name=statusinvalid indexversion/str Seems like we don't have this index version. After I tried to find what is wrong with that. On slave http://192.168.2.174:8080/solr/replication?command=indexversion returns only 0 long name=indexversion0/long long name=generation0/long on master I could see the version of current index long name=indexversion1341328964983/long long name=generation3/long but slave's http://192.168.2.174:8080/solr/admin/stats.jsp I can the the right version indexVersion : 1341328964983 generation : 3 Also when I checked the solr log. [org.apache.solr.handler.SnapPuller] Master at: http://192.168.2.204:8080/solr/replication is not available. Index fetch failed. Exception: Connection refused BTW, I was able to fetch the replication file with wget directly. So my question is: What is wrong with my replication or Solr? About version, I use some legacy version of Solr: Solr Specification Version: 3.5.0.2011.11.22.14.54.38, because we have some legacy systems here. And another question what is the best way to migrate to the latest version. I mean to keep alive all the boosting infrastructure based on ExternalFileField options. Thank you in advance for your time and help you can provide, Sergii
Troubles with solr replication
Hi guys, I have some problems with Solr replication and can see some unexpected behavior. Would be nice to have some answers where am I wrong, or what is the best way to solve the problem. I have a replication master-slave. http://192.168.2.204:8080/solr/ is master and http://192.168.2.174:8080/solr/ is slave. With quite simple config. requestHandler name=/replication class=solr.ReplicationHandler lst name=master str name=enable false /str str name=replicateAfter commit /str str name=replicateAfter startup /str str name=confFiles schema.xml,boosting.txt /str /lst lst name=slave str name=enable true /str str name=masterUrlhttp://192.168.2.204:8080/solr/replication/str str name=pollInterval00:00:60/str str name=httpConnTimeout 5000 /str str name=httpReadTimeout 1 /str /lst /requestHandler The main idea when I started playing around Solr is to replicate some boosting values. I wanted to use confFile option for it. An here is my first problem. I wasn't able to replicate files from master. On slave I was able to see only schema.xml. I wanted to check, do I actually have the files and everything correct in solr config. So I checked on master the file list and it returns the list of all files http://192.168.2.204:8080/solr/replication?command=filelistindexversion=1341328964983 but for slave I can't see anything http://192.168.2.174:8080/solr/replication?command=filelistindexversion=1341328964983 returns str name=statusinvalid indexversion/str Seems like we don't have this index version. After I tried to find what is wrong with that. On slave http://192.168.2.174:8080/solr/replication?command=indexversion returns only 0 long name=indexversion0/long long name=generation0/long on master I could see the version of current index long name=indexversion1341328964983/long long name=generation3/long but slave's http://192.168.2.174:8080/solr/admin/stats.jsp I can the the right version indexVersion : 1341328964983 generation : 3 Also when I checked the solr log. [org.apache.solr.handler.SnapPuller] Master at: http://192.168.2.204:8080/solr/replication is not available. Index fetch failed. Exception: Connection refused BTW, I was able to fetch the replication file with wget directly. So my question is: What is wrong with my replication or Solr? About version, I use some legacy version of Solr: Solr Specification Version: 3.5.0.2011.11.22.14.54.38, because we have some legacy systems here. And another question what is the best way to migrate to the latest version. I mean to keep alive all the boosting infrastructure based on ExternalFileField options. Thank you in advance for your time and help you can provide, Sergii
Problem with Solr replication in solr 4.2
Hey, Currently we are using solr 4.0 with a master slave setup. The data gets indexed on the master and then we issue a fetchindex command to replicate it on the slave. The slave has a postCommit listener which gets kicked off when replication finishes and we depend on this listener to know whn replication is done. But when I tried to do the same with 4.2, the commit does not seem to be happening. Is this a known issue? is there any other way to know that replication is done? Also, initially when i tried solr 4.2, i noticed with this setup, i noticed that with the fetchIndex command, the fields were downloaded to the temp folder, but it was never pulled into the index directory on the slave. The only file which made it was the lock file. This problem does not happen anymore? Thanks, Rohit
Re: Problem with Solr replication in solr 4.2
On Mar 21, 2013, at 12:08 PM, Rohit Harchandani rhar...@gmail.com wrote: Hey, Currently we are using solr 4.0 with a master slave setup. The data gets indexed on the master and then we issue a fetchindex command to replicate it on the slave. The slave has a postCommit listener which gets kicked off when replication finishes and we depend on this listener to know whn replication is done. But when I tried to do the same with 4.2, the commit does not seem to be happening. Is this a known issue? is there any other way to know that replication is done? Yeah, we stopped doing this commit because it changes some index meta data and shouldn't be necessary. I'd open a JIRA issue about being able to listen for replication finishing. Also, initially when i tried solr 4.2, i noticed with this setup, i noticed that with the fetchIndex command, the fields were downloaded to the temp folder, but it was never pulled into the index directory on the slave. The only file which made it was the lock file. This problem does not happen anymore? I don't know, does it? Can you file a JIRA with instructions to replicate? 4.2.1 is about to go out, if we are quick, perhaps we can address something here if there is a problem. - Mark Thanks, Rohit
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
Thanks for the info. I understand that the latest replicateable version of index may differ from the actual version of index on master/slave. But why when I use /solr/replication?command=indexversion On master: response lst name=responseHeader int name=status0/int int name=QTime0/int /lst long name=indexversion1363263304585/long long name=generation4/long /response And on slave: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst long name=indexversion1363263600323/long long name=generation5/long /response Why do I get higher version on slave than on master? The page https://wiki.apache.org/solr/SolrReplication states: Get the latest replicateable index on master: http://master_host:port/solr/replication?command=indexversion; - ok, returns latest replicatable index version Get version number of the index: http://host:port/solr/replication?command=indexversion; - for me it means that on slave I should get actual index version and not an index version appropriate for replication (it is a slave, it should not be the source of replication) Yes, it is confusing overall :) My question is: how to get actual index version on master and slave for monitoring purpose? I can use: /solr/replication?command=details On master: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response On slave: response lst name=responseHeader int name=status0/int int name=QTime25/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263600323/long long name=generation5/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str strsegments_5/str str_2.tvf/str /arr /lst /arr str name=isMasterfalse/str str name=isSlavetrue/str long name=indexVersion1363263304585/long long name=generation4/long lst name=slave lst name=masterDetails str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=masterUrlhttp://172.18.19.204:8080/solr/str str name=pollInterval00:00:60/str str name=nextExecutionAtWed Mar 20 08:48:00 CET 2013/str str name=indexReplicatedAtThu Mar 14 13:20:00 CET 2013/str arr name=indexReplicatedAtList strThu Mar 14 13:20:00 CET 2013/str strThu Mar 14 13:19:00 CET 2013/str strThu Mar 14 12:18:00 CET 2013/str strThu Mar 14 12:17:00 CET 2013/str strFri Mar 08 14:55:00 CET 2013/str strFri Mar 08 14:50:52 CET 2013/str strFri Mar 08 14:32:00 CET 2013/str /arr str name=timesIndexReplicated7/str str name=lastCycleBytesDownloaded23212/str str name=previousCycleTimeInSeconds0/str str name=currentDateWed Mar 20 08:47:03 CET 2013/str str name=isPollingDisabledfalse/str str name=isReplicatingfalse/str /lst /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response Which line with indexVersion should I use? Best regards, Rafal Radecki. 2013/3/19 Mark Miller markrmil...@gmail.com: On Mar 15, 2013, at 6:43 AM, Rafał Radecki radecki.ra...@gmail.com wrote: I use http and get /solr/replication?command=indexversion urls to get index versions on master and slave. The replication
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
Hmm, I'd have to look, but first to make, this subject says 4.1? In 4.1 the slave will be ahead because it commits after installing the index. In 4.2 it shouldn't. Your on? - Mark On Mar 20, 2013, at 4:03 AM, Rafał Radecki radecki.ra...@gmail.com wrote: Thanks for the info. I understand that the latest replicateable version of index may differ from the actual version of index on master/slave. But why when I use /solr/replication?command=indexversion On master: response lst name=responseHeader int name=status0/int int name=QTime0/int /lst long name=indexversion1363263304585/long long name=generation4/long /response And on slave: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst long name=indexversion1363263600323/long long name=generation5/long /response Why do I get higher version on slave than on master? The page https://wiki.apache.org/solr/SolrReplication states: Get the latest replicateable index on master: http://master_host:port/solr/replication?command=indexversion; - ok, returns latest replicatable index version Get version number of the index: http://host:port/solr/replication?command=indexversion; - for me it means that on slave I should get actual index version and not an index version appropriate for replication (it is a slave, it should not be the source of replication) Yes, it is confusing overall :) My question is: how to get actual index version on master and slave for monitoring purpose? I can use: /solr/replication?command=details On master: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response On slave: response lst name=responseHeader int name=status0/int int name=QTime25/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263600323/long long name=generation5/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str strsegments_5/str str_2.tvf/str /arr /lst /arr str name=isMasterfalse/str str name=isSlavetrue/str long name=indexVersion1363263304585/long long name=generation4/long lst name=slave lst name=masterDetails str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=masterUrlhttp://172.18.19.204:8080/solr/str str name=pollInterval00:00:60/str str name=nextExecutionAtWed Mar 20 08:48:00 CET 2013/str str name=indexReplicatedAtThu Mar 14 13:20:00 CET 2013/str arr name=indexReplicatedAtList strThu Mar 14 13:20:00 CET 2013/str strThu Mar 14 13:19:00 CET 2013/str strThu Mar 14 12:18:00 CET 2013/str strThu Mar 14 12:17:00 CET 2013/str strFri Mar 08 14:55:00 CET 2013/str strFri Mar 08 14:50:52 CET 2013/str strFri Mar 08 14:32:00 CET 2013/str /arr str name=timesIndexReplicated7/str str name=lastCycleBytesDownloaded23212/str str name=previousCycleTimeInSeconds0/str str name=currentDateWed Mar 20 08:47:03 CET 2013/str str name=isPollingDisabledfalse/str str name
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
Yes, this is solr 4.1. 2013/3/20 Mark Miller markrmil...@gmail.com: Hmm, I'd have to look, but first to make, this subject says 4.1? In 4.1 the slave will be ahead because it commits after installing the index. In 4.2 it shouldn't. Your on? - Mark On Mar 20, 2013, at 4:03 AM, Rafał Radecki radecki.ra...@gmail.com wrote: Thanks for the info. I understand that the latest replicateable version of index may differ from the actual version of index on master/slave. But why when I use /solr/replication?command=indexversion On master: response lst name=responseHeader int name=status0/int int name=QTime0/int /lst long name=indexversion1363263304585/long long name=generation4/long /response And on slave: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst long name=indexversion1363263600323/long long name=generation5/long /response Why do I get higher version on slave than on master? The page https://wiki.apache.org/solr/SolrReplication states: Get the latest replicateable index on master: http://master_host:port/solr/replication?command=indexversion; - ok, returns latest replicatable index version Get version number of the index: http://host:port/solr/replication?command=indexversion; - for me it means that on slave I should get actual index version and not an index version appropriate for replication (it is a slave, it should not be the source of replication) Yes, it is confusing overall :) My question is: how to get actual index version on master and slave for monitoring purpose? I can use: /solr/replication?command=details On master: response lst name=responseHeader int name=status0/int int name=QTime1/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response On slave: response lst name=responseHeader int name=status0/int int name=QTime25/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263600323/long long name=generation5/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str strsegments_5/str str_2.tvf/str /arr /lst /arr str name=isMasterfalse/str str name=isSlavetrue/str long name=indexVersion1363263304585/long long name=generation4/long lst name=slave lst name=masterDetails str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledtrue/str long name=replicatableGeneration4/long /lst /lst str name=masterUrlhttp://172.18.19.204:8080/solr/str str name=pollInterval00:00:60/str str name=nextExecutionAtWed Mar 20 08:48:00 CET 2013/str str name=indexReplicatedAtThu Mar 14 13:20:00 CET 2013/str arr name=indexReplicatedAtList strThu Mar 14 13:20:00 CET 2013/str strThu Mar 14 13:19:00 CET 2013/str strThu Mar 14 12:18:00 CET 2013/str strThu Mar 14 12:17:00 CET 2013/str strFri Mar 08 14:55:00 CET 2013/str strFri Mar 08 14:50:52 CET 2013/str strFri Mar 08 14:32:00 CET 2013/str /arr str name=timesIndexReplicated7/str str name=lastCycleBytesDownloaded23212/str str name=previousCycleTimeInSeconds0/str str name=currentDateWed Mar 20 08:47:03 CET
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
Deja-vu? http://mail-archives.apache.org/mod_mbox/lucene-general/201303.mbox/%3CCAHd9_iR-HtNDu-3a9A5ekTFdb+5mo1eWVcu4Shp8AD=qtpq...@mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/lucene-general/201303.mbox/%3Calpine.DEB.2.02.1303081644040.5502@frisbee%3E https://issues.apache.org/jira/browse/SOLR-3681 : Date: Fri, 15 Mar 2013 11:43:05 +0100 : From: Rafał Radecki radecki.ra...@gmail.com : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: Re: Solr 4.1 monitoring with /solr/replication?command=details - : indexVersion? : : I use http and get /solr/replication?command=indexversion urls to get : index versions on master and slave. The replication works fine but : index versions from /solr/replication?command=indexversion differ. : : Best regards, : Rafal. : : 2013/3/14 Mark Miller markrmil...@gmail.com: : What calls are you using to get the versions? Or is it the admin UI? : : Also can you add any details about your setup - if this is a problem, we need to duplicate it in one of our unit tests. : : Also, is it affecting proper replication in any way that you can tell. : : - Mark : : On Mar 14, 2013, at 11:12 AM, richardg richa...@dvdempire.com wrote: : : I believe this is the same issue as described, I'm running 4.2 and as you can : see my slave is a couple versions ahead of the master (all three slaves show : the same behavior). This was never the case until I upgraded from 4.0 to : 4.2. : : Master: : 1363272681951 : 93 : 1,022.31 MB : Slave: : 1363273274085 : 95 : 1,022.31 MB : : : : -- : View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html : Sent from the Solr - User mailing list archive at Nabble.com. : : -Hoss
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
On Mar 15, 2013, at 6:43 AM, Rafał Radecki radecki.ra...@gmail.com wrote: I use http and get /solr/replication?command=indexversion urls to get index versions on master and slave. The replication works fine but index versions from /solr/replication?command=indexversion differ. I think thats normal - it's a little misleading, but the replication index version is that of the most recent 'replicatable' commit point. That is determined by things like replicate on startup, replicate on commit, etc. These are very likely to be different from mater to slave. - Mark
Re: Solr Replication
Why do you think that during a backup any write activity would corrupt your backup? Solr backs up live indexes all the time, that's what's happening, in effect, when you replicate from a master to a slave. There's no requirement that the master stop indexing. The replication backup command Ahmet told you about is using the same mechanism. Solr/Lucene index segments are write-once. backup/replication essentially copies closed segments around which, since they aren't changing, aren't affected by other indexing activity. So the process is this: 1 a backup/replication starts. At this point, a list is made of all the currently-closed segments and these segments will NOT be deleted until the backup/replication is complete. 2 all the segments in 1 are copied. This is a complete snapshot of your index. 3 after all the segments are copied, they are released and may then be deleted in the normal merge process. Of course another way to do backups would be set up a machine that just does a replication (think of it as a special slave) periodically. Another would be to have the master keep backups automatically, seesee: http://wiki.apache.org/solr/SolrReplication, and look for numberOfBackups. Best Erick On Fri, Mar 15, 2013 at 1:03 AM, vicky desai vicky.de...@germinait.comwrote: Hi, I have a multi core setup and there is continuous updation going on in each core. Hence I dont prefer a bckup as it would either cause a downtime or if during a backup there is a write activity my backup will be corrupted. Can you please suggest if there is a cleaner way to handle this -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
I use http and get /solr/replication?command=indexversion urls to get index versions on master and slave. The replication works fine but index versions from /solr/replication?command=indexversion differ. Best regards, Rafal. 2013/3/14 Mark Miller markrmil...@gmail.com: What calls are you using to get the versions? Or is it the admin UI? Also can you add any details about your setup - if this is a problem, we need to duplicate it in one of our unit tests. Also, is it affecting proper replication in any way that you can tell. - Mark On Mar 14, 2013, at 11:12 AM, richardg richa...@dvdempire.com wrote: I believe this is the same issue as described, I'm running 4.2 and as you can see my slave is a couple versions ahead of the master (all three slaves show the same behavior). This was never the case until I upgraded from 4.0 to 4.2. Master: 1363272681951 93 1,022.31 MB Slave: 1363273274085 95 1,022.31 MB -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr Replication
Hi, I am using solr 4 setup. For the backup purpose once in a day I start one additional tomcat server with cores having empty data folders and which acts as a slave server. However it does not replicate data from the master unless there is a commit on the master. Is there a possibility to pull data from master core without firing a commit operation on that core -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication
Hi Vicky, May be str name=replicateAfterstartup/str ? For backups http://master_host:port/solr/replication?command=backup would be more suitable. or str name=backupAfterstartup/str --- On Thu, 3/14/13, vicky desai vicky.de...@germinait.com wrote: From: vicky desai vicky.de...@germinait.com Subject: Solr Replication To: solr-user@lucene.apache.org Date: Thursday, March 14, 2013, 9:20 AM Hi, I am using solr 4 setup. For the backup purpose once in a day I start one additional tomcat server with cores having empty data folders and which acts as a slave server. However it does not replicate data from the master unless there is a commit on the master. Is there a possibility to pull data from master core without firing a commit operation on that core -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
Hi All. I am monitoring two solr 4.1 solr instances in master-slave setup. On both nodes I check url /solr/replication?command=details and parse it to get: - on master: if replication is enabled - field replicationEnabled - on slave: if replication is enabled - field replicationEnabled - on slave: if polling is disabled - field isPollingDisabled For solr 3.6 I've als used url: solr/replication?command=indexversion but for 4.1 it gives me different results on master and slave, on slave the version is higher despite the fact that replication is enabled, polling is enabled and in admin gui /solr/#/collection1/replication I have: Index Version Gen Size Master: 1363259808632 3 22.59 KB Slave: 1363259808632 3 22.59 KB So as I see it master and slave have the same version of index despite the fact that /solr/replication?command=indexversion gives: - on master: long name=indexversion1363259808632/long - on slave: long name=indexversion1363259880360/long - higher value Is this a bug? Best regards, Rafal Radecki.
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
In the output of: /solr/replication?command=details there is indexVersion mentioned many times: response lst name=responseHeader int name=status0/int int name=QTime3/int /lst lst name=details str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363259880360/long long name=generation4/long arr name=filelist str_1.tvx/str str_1_nrm.cfs/str str_1_Lucene41_0.doc/str str_1_Lucene41_0.tim/str str_1_Lucene41_0.tip/str str_1.fnm/str str_1_nrm.cfe/str str_1.fdx/str str_1_Lucene41_0.pos/str str_1.tvf/str str_1.fdt/str str_1_Lucene41_0.pay/str str_1.si/str str_1.tvd/str strsegments_4/str /arr /lst /arr str name=isMasterfalse/str str name=isSlavetrue/str long name=indexVersion1363259808632/long long name=generation3/long lst name=slave lst name=masterDetails str name=indexSize22.59 KB/str str name=indexPath/usr/share/solr/data/index//str arr name=commits lst long name=indexVersion1363263304585/long long name=generation4/long arr name=filelist str_2_Lucene41_0.pos/str str_2.si/str str_2_Lucene41_0.tim/str str_2.fdt/str str_2_Lucene41_0.doc/str str_2_Lucene41_0.tip/str str_2.fdx/str str_2.tvx/str str_2.fnm/str str_2_nrm.cfe/str str_2.tvd/str str_2_Lucene41_0.pay/str str_2_nrm.cfs/str str_2.tvf/str strsegments_4/str /arr /lst /arr str name=isMastertrue/str str name=isSlavefalse/str long name=indexVersion1363263304585/long long name=generation4/long lst name=master str name=confFilesschema.xml,stopwords.txt/str arr name=replicateAfter strcommit/str strstartup/str /arr str name=replicationEnabledfalse/str long name=replicatableGeneration4/long /lst /lst str name=masterUrlhttp://172.18.19.204:8080/solr/str str name=pollInterval00:00:60/str str name=nextExecutionAtPolling disabled/str str name=indexReplicatedAtThu Mar 14 12:18:00 CET 2013/str arr name=indexReplicatedAtList strThu Mar 14 12:18:00 CET 2013/str strThu Mar 14 12:17:00 CET 2013/str strFri Mar 08 14:55:00 CET 2013/str strFri Mar 08 14:50:52 CET 2013/str strFri Mar 08 14:32:00 CET 2013/str /arr str name=timesIndexReplicated5/str str name=lastCycleBytesDownloaded23214/str str name=previousCycleTimeInSeconds0/str str name=currentDateThu Mar 14 13:15:53 CET 2013/str str name=isPollingDisabledtrue/str str name=isReplicatingfalse/str /lst /lst str name=WARNING This response format is experimental. It is likely to change in the future. /str /response Which one should be used? Is there any other way to monitor idex version on master and slave? Best regards, Rafał Radecki. 2013/3/14 Rafał Radecki radecki.ra...@gmail.com: Hi All. I am monitoring two solr 4.1 solr instances in master-slave setup. On both nodes I check url /solr/replication?command=details and parse it to get: - on master: if replication is enabled - field replicationEnabled - on slave: if replication is enabled - field replicationEnabled - on slave: if polling is disabled - field isPollingDisabled For solr 3.6 I've als used url: solr/replication?command=indexversion but for 4.1 it gives me different results on master and slave, on slave the version is higher despite the fact that replication is enabled, polling is enabled and in admin gui /solr/#/collection1/replication I have: Index Version Gen Size Master: 1363259808632 3 22.59 KB Slave: 1363259808632 3 22.59 KB So as I see it master and slave have the same version of index despite the fact that /solr/replication?command=indexversion gives: - on master: long name=indexversion1363259808632/long - on slave: long name=indexversion1363259880360/long - higher value Is this a bug? Best regards, Rafal Radecki.
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
On Mar 14, 2013, at 8:10 AM, Rafał Radecki radecki.ra...@gmail.com wrote: Is this a bug? Yes, 4.1 had some replication issues just as you seem to describe here. It all should be fixed in 4.2 which is available now and is a simple upgrade. - Mark
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
I believe this is the same issue as described, I'm running 4.2 and as you can see my slave is a couple versions ahead of the master (all three slaves show the same behavior). This was never the case until I upgraded from 4.0 to 4.2. Master: 1363272681951 93 1,022.31 MB Slave: 1363273274085 95 1,022.31 MB -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?
What calls are you using to get the versions? Or is it the admin UI? Also can you add any details about your setup - if this is a problem, we need to duplicate it in one of our unit tests. Also, is it affecting proper replication in any way that you can tell. - Mark On Mar 14, 2013, at 11:12 AM, richardg richa...@dvdempire.com wrote: I believe this is the same issue as described, I'm running 4.2 and as you can see my slave is a couple versions ahead of the master (all three slaves show the same behavior). This was never the case until I upgraded from 4.0 to 4.2. Master: 1363272681951 93 1,022.31 MB Slave: 1363273274085 95 1,022.31 MB -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication
Hi, I have a multi core setup and there is continuous updation going on in each core. Hence I dont prefer a bckup as it would either cause a downtime or if during a backup there is a write activity my backup will be corrupted. Can you please suggest if there is a cleaner way to handle this -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
After upgrading to 4.2, the problem is not yet solved, in this image you can see, how slow is the transfer speed. At least, after the update the master is not blocked during replication http://lucene.472066.n3.nabble.com/file/n4046951/solr_replication.jpg Any idea? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046951.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication takes long time
While looking at Solr logs, I found a java.lang.OutOfMemoryError: Java heap space that was happening 2 times per hour So I tried to increase the max memory heap assigned to JVM (-Xmx) and since then the servers are not crashing, even though the replication takes still long time to complete. But for now, the 2 slaves can handle with no problems all the queries. Regards, Victor -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046993.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr replication takes long time
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have 2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
is below 50 queries per second
Replication has been working for long time with any problem, but in the last
weeks the replication cycles started to take long and long time for the big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
indexConfig
mergeFactor3/mergeFactor
deletionPolicy class=solr.SolrDeletionPolicy
str name=maxCommitsToKeep10/str
str name=maxOptimizedCommitsToKeep1/str
str name=maxCommitAge6HOUR/str
/deletionPolicy
/indexConfig
jmx/
updateHandler class=solr.DirectUpdateHandler2
autoCommit
maxDocs2000/maxDocs
maxTime3/maxTime
/autoCommit
autoSoftCommit
maxTime500/maxTime
/autoSoftCommit
updateLog
str name=dir${solr.data.dir:}/str
/updateLog
/updateHandler
query
maxBooleanClauses2048/maxBooleanClauses
filterCache
class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
queryResultCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
enableLazyFieldLoadingtrue/enableLazyFieldLoading
queryResultWindowSize50/queryResultWindowSize
queryResultMaxDocsCached50/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=q*:*/str
str name=fqdate:[NOW/DAY-7DAY TO
NOW/DAY+1DAY]/str
str name=rows1000/str
/lst
/arr
/listener
listener event=firstSearcher
class=solr.QuerySenderListener
arr name=queries
lst
str name=q*:*/str
str name=fqdate:[NOW/DAY-7DAY TO
NOW/DAY+1DAY]/str
str name=rows1000/str
/lst
/arr
/listener
useColdSearchertrue/useColdSearcher
maxWarmingSearchers4/maxWarmingSearchers
/query
requestHandler name=/replication class=solr.ReplicationHandler
lst
Re: Solr replication takes long time
Are you using Solr 4.1?
- Mark
On Mar 11, 2013, at 1:53 PM, Victor Ruiz bik1...@gmail.com wrote:
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have 2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
is below 50 queries per second
Replication has been working for long time with any problem, but in the last
weeks the replication cycles started to take long and long time for the big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersionLUCENE_40/luceneMatchVersion
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
indexConfig
mergeFactor3/mergeFactor
deletionPolicy class=solr.SolrDeletionPolicy
str name=maxCommitsToKeep10/str
str name=maxOptimizedCommitsToKeep1/str
str name=maxCommitAge6HOUR/str
/deletionPolicy
/indexConfig
jmx/
updateHandler class=solr.DirectUpdateHandler2
autoCommit
maxDocs2000/maxDocs
maxTime3/maxTime
/autoCommit
autoSoftCommit
maxTime500/maxTime
/autoSoftCommit
updateLog
str name=dir${solr.data.dir:}/str
/updateLog
/updateHandler
query
maxBooleanClauses2048/maxBooleanClauses
filterCache
class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
queryResultCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
enableLazyFieldLoadingtrue/enableLazyFieldLoading
queryResultWindowSize50/queryResultWindowSize
queryResultMaxDocsCached50/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=q*:*/str
str name=fqdate:[NOW/DAY-7DAY TO
NOW/DAY+1DAY]/str
str name=rows1000/str
/lst
/arr
/listener
listener event=firstSearcher
class=solr.QuerySenderListener
arr name=queries
lst
str name=q*:*/str
str name=fqdate:[NOW/DAY-7DAY TO
NOW/DAY+1DAY]/str
str name=rows1000/str
/lst
/arr
/listener
useColdSearchertrue/useColdSearcher
maxWarmingSearchers4/maxWarmingSearchers
/query
Re: Solr replication takes long time
no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
issue with the replication, so I decided not to try it for now
Mark Miller-3 wrote
Are you using Solr 4.1?
- Mark
On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;
bik1979@
gt; wrote:
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr
4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have
2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The
big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one
it
is below 50 queries per second
Replication has been working for long time with any problem, but in the
last
weeks the replication cycles started to take long and long time for the
big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should
not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update
handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersion
LUCENE_40
/luceneMatchVersion
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
indexConfig
mergeFactor
3
/mergeFactor
deletionPolicy class=solr.SolrDeletionPolicy
str name=maxCommitsToKeep
10
/str
str name=maxOptimizedCommitsToKeep
1
/str
str name=maxCommitAge
6HOUR
/str
/deletionPolicy
/indexConfig
jmx/
updateHandler class=solr.DirectUpdateHandler2
autoCommit
maxDocs
2000
/maxDocs
maxTime
3
/maxTime
/autoCommit
autoSoftCommit
maxTime
500
/maxTime
/autoSoftCommit
updateLog
str name=dir
${solr.data.dir:}
/str
/updateLog
/updateHandler
query
maxBooleanClauses
2048
/maxBooleanClauses
filterCache
class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
queryResultCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
enableLazyFieldLoading
true
/enableLazyFieldLoading
queryResultWindowSize
50
/queryResultWindowSize
queryResultMaxDocsCached
50
/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=q
*:*
/str
str name=fq
date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
/str
str name=rows
1000
/str
/lst
/arr
/listener
listener event=firstSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=q
*:*
/str
str name=fq
date:[NOW/DAY-7DAY TO NOW/DAY+1DAY
Re: Solr replication takes long time
Okay - yes, 4.0 is a better choice for replication than 4.1.
It almost sounds like you may be replicating the full index rather than just
changes or something. 4.0 had a couple issues as well - a couple things that
were discovered while writing stronger tests for 4.2.
4.2 is spreading onto mirrors now.
- Mark
On Mar 11, 2013, at 2:00 PM, Victor Ruiz bik1...@gmail.com wrote:
no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
issue with the replication, so I decided not to try it for now
Mark Miller-3 wrote
Are you using Solr 4.1?
- Mark
On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;
bik1979@
gt; wrote:
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr
4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have
2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The
big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one
it
is below 50 queries per second
Replication has been working for long time with any problem, but in the
last
weeks the replication cycles started to take long and long time for the
big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should
not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update
handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersion
LUCENE_40
/luceneMatchVersion
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
indexConfig
mergeFactor
3
/mergeFactor
deletionPolicy class=solr.SolrDeletionPolicy
str name=maxCommitsToKeep
10
/str
str name=maxOptimizedCommitsToKeep
1
/str
str name=maxCommitAge
6HOUR
/str
/deletionPolicy
/indexConfig
jmx/
updateHandler class=solr.DirectUpdateHandler2
autoCommit
maxDocs
2000
/maxDocs
maxTime
3
/maxTime
/autoCommit
autoSoftCommit
maxTime
500
/maxTime
/autoSoftCommit
updateLog
str name=dir
${solr.data.dir:}
/str
/updateLog
/updateHandler
query
maxBooleanClauses
2048
/maxBooleanClauses
filterCache
class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
queryResultCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
enableLazyFieldLoading
true
/enableLazyFieldLoading
queryResultWindowSize
50
/queryResultWindowSize
queryResultMaxDocsCached
50
/queryResultMaxDocsCached
listener event=newSearcher class=solr.QuerySenderListener
arr name=queries
lst
str name=q
*:*
/str
str name=fq
date:[NOW/DAY-7DAY TO NOW/DAY+1DAY]
/str
str name=rows
1000
/str
/lst
Re: Solr replication takes long time
Thanks for your answer Mark. I think I'll try to update to 4.2. I'll keep you
updated.
Anyway, I'd not say that the full index is replicated, I've been monitoring
the replication process in the Solr admin console and there I see that
usually not more than 50-100 files are transferrend, the total size is
rarely greater than 50MB. Is this info trustable?
Victor
Mark Miller-3 wrote
Okay - yes, 4.0 is a better choice for replication than 4.1.
It almost sounds like you may be replicating the full index rather than
just changes or something. 4.0 had a couple issues as well - a couple
things that were discovered while writing stronger tests for 4.2.
4.2 is spreading onto mirrors now.
- Mark
On Mar 11, 2013, at 2:00 PM, Victor Ruiz lt;
bik1979@
gt; wrote:
no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was
an
issue with the replication, so I decided not to try it for now
Mark Miller-3 wrote
Are you using Solr 4.1?
- Mark
On Mar 11, 2013, at 1:53 PM, Victor Ruiz lt;
bik1979@
gt; wrote:
Hi guys,
I have a problem with Solr replication. I have 2 solr servers (Solr
4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11, ext3, each). In
every server, there are 2 independent instances of solr running (I
tried
also multicore config, but having independent instances has for me
better
performance), every instance having a differente collection. So, we
have
2
masters in server 1, and 2 slaves in server 2.
Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size
not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The
big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one
it
is below 50 queries per second
Replication has been working for long time with any problem, but in the
last
weeks the replication cycles started to take long and long time for the
big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out,
despite
the timeout in my clients is 30 seconds.
The servers are in same LAN, gigabit ethernet, so the broadband should
not
be the bottleneck.
Since the index is receiving frequents updates and deletes (update
handler
receives more than 200 request per second for the big collection, but
not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect.
My solrconfig.xml in the big collection is like that:
?xml version=1.0 encoding=UTF-8 ?
config
luceneMatchVersion
LUCENE_40
/luceneMatchVersion
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.NRTCachingDirectoryFactory}/
indexConfig
mergeFactor
3
/mergeFactor
deletionPolicy class=solr.SolrDeletionPolicy
str name=maxCommitsToKeep
10
/str
str name=maxOptimizedCommitsToKeep
1
/str
str name=maxCommitAge
6HOUR
/str
/deletionPolicy
/indexConfig
jmx/
updateHandler class=solr.DirectUpdateHandler2
autoCommit
maxDocs
2000
/maxDocs
maxTime
3
/maxTime
/autoCommit
autoSoftCommit
maxTime
500
/maxTime
/autoSoftCommit
updateLog
str name=dir
${solr.data.dir:}
/str
/updateLog
/updateHandler
query
maxBooleanClauses
2048
/maxBooleanClauses
filterCache
class=solr.FastLRUCache
size=2048
initialSize=1024
autowarmCount=1024/
queryResultCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
documentCache
class=solr.LRUCache
size=2048
initialSize=1024
autowarmCount=1024/
enableLazyFieldLoading
true
/enableLazyFieldLoading
queryResultWindowSize
50
/queryResultWindowSize
queryResultMaxDocsCached
50
Re: Downloading files from the solr replication Handler
They are the '\0' character. what is a marker? Gettting the following with a wget HTTP request sent, awaiting response... 200 OK Length: unspecified [application/xml] On Fri, Nov 30, 2012 at 4:58 PM, Alexandre Rafalovitch arafa...@gmail.comwrote: What mime type you get for binary files? Maybe server is misconfigured for that extension and sends them as text. Then they could be the markers. Do they look like markers? Regards, Alex On 30 Nov 2012 04:06, Eva Lacy e...@lacy.ie wrote: Doesn't make much sense if they are in binary files as well. On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog goks...@gmail.com wrote: Maybe these are text encoding markers? - Original Message - | From: Eva Lacy e...@lacy.ie | To: solr-user@lucene.apache.org | Sent: Thursday, November 29, 2012 3:53:07 AM | Subject: Re: Downloading files from the solr replication Handler | | I tried downloading them with my browser and also with a c# | WebRequest. | If I skip the first and last 4 bytes it seems work fine. | | | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson | erickerick...@gmail.comwrote: | | How are you downloading them? I suspect the issue is | with the download process rather than Solr, but I'm just guessing. | | Best | Erick | | | On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: | | Just to add to that, I'm using solr 3.6.1 | | | On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: | |I downloaded some configuration and data files directly from |solr in an |attempt to develop a backup solution. |I noticed there is some characters at the start and end of the |file | that |aren't in configuration files, I notice the same characters at |the | start |and end of the data files. |Anyone with any idea how I can download these files without the |extra |characters or predict how many there are going to be so I can |skip | them? | | | |
Re: Downloading files from the solr replication Handler
Doesn't make much sense if they are in binary files as well. On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog goks...@gmail.com wrote: Maybe these are text encoding markers? - Original Message - | From: Eva Lacy e...@lacy.ie | To: solr-user@lucene.apache.org | Sent: Thursday, November 29, 2012 3:53:07 AM | Subject: Re: Downloading files from the solr replication Handler | | I tried downloading them with my browser and also with a c# | WebRequest. | If I skip the first and last 4 bytes it seems work fine. | | | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson | erickerick...@gmail.comwrote: | | How are you downloading them? I suspect the issue is | with the download process rather than Solr, but I'm just guessing. | | Best | Erick | | | On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: | | Just to add to that, I'm using solr 3.6.1 | | | On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: | |I downloaded some configuration and data files directly from |solr in an |attempt to develop a backup solution. |I noticed there is some characters at the start and end of the |file | that |aren't in configuration files, I notice the same characters at |the | start |and end of the data files. |Anyone with any idea how I can download these files without the |extra |characters or predict how many there are going to be so I can |skip | them? | | | |
Re: Downloading files from the solr replication Handler
tag me baffled. But these are copied around all the time so I'm guessing an interaction between your servlet container and your request, which is like saying it must be magic. You can tell I'm into places where I'm clueless Sorry I can't be more help Erick On Fri, Nov 30, 2012 at 4:06 AM, Eva Lacy e...@lacy.ie wrote: Doesn't make much sense if they are in binary files as well. On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog goks...@gmail.com wrote: Maybe these are text encoding markers? - Original Message - | From: Eva Lacy e...@lacy.ie | To: solr-user@lucene.apache.org | Sent: Thursday, November 29, 2012 3:53:07 AM | Subject: Re: Downloading files from the solr replication Handler | | I tried downloading them with my browser and also with a c# | WebRequest. | If I skip the first and last 4 bytes it seems work fine. | | | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson | erickerick...@gmail.comwrote: | | How are you downloading them? I suspect the issue is | with the download process rather than Solr, but I'm just guessing. | | Best | Erick | | | On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: | | Just to add to that, I'm using solr 3.6.1 | | | On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: | |I downloaded some configuration and data files directly from |solr in an |attempt to develop a backup solution. |I noticed there is some characters at the start and end of the |file | that |aren't in configuration files, I notice the same characters at |the | start |and end of the data files. |Anyone with any idea how I can download these files without the |extra |characters or predict how many there are going to be so I can |skip | them? | | | |
Re: Downloading files from the solr replication Handler
What mime type you get for binary files? Maybe server is misconfigured for that extension and sends them as text. Then they could be the markers. Do they look like markers? Regards, Alex On 30 Nov 2012 04:06, Eva Lacy e...@lacy.ie wrote: Doesn't make much sense if they are in binary files as well. On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog goks...@gmail.com wrote: Maybe these are text encoding markers? - Original Message - | From: Eva Lacy e...@lacy.ie | To: solr-user@lucene.apache.org | Sent: Thursday, November 29, 2012 3:53:07 AM | Subject: Re: Downloading files from the solr replication Handler | | I tried downloading them with my browser and also with a c# | WebRequest. | If I skip the first and last 4 bytes it seems work fine. | | | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson | erickerick...@gmail.comwrote: | | How are you downloading them? I suspect the issue is | with the download process rather than Solr, but I'm just guessing. | | Best | Erick | | | On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: | | Just to add to that, I'm using solr 3.6.1 | | | On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: | |I downloaded some configuration and data files directly from |solr in an |attempt to develop a backup solution. |I noticed there is some characters at the start and end of the |file | that |aren't in configuration files, I notice the same characters at |the | start |and end of the data files. |Anyone with any idea how I can download these files without the |extra |characters or predict how many there are going to be so I can |skip | them? | | | |
Re: Downloading files from the solr replication Handler
I tried downloading them with my browser and also with a c# WebRequest. If I skip the first and last 4 bytes it seems work fine. On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson erickerick...@gmail.comwrote: How are you downloading them? I suspect the issue is with the download process rather than Solr, but I'm just guessing. Best Erick On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: Just to add to that, I'm using solr 3.6.1 On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: I downloaded some configuration and data files directly from solr in an attempt to develop a backup solution. I noticed there is some characters at the start and end of the file that aren't in configuration files, I notice the same characters at the start and end of the data files. Anyone with any idea how I can download these files without the extra characters or predict how many there are going to be so I can skip them?
Re: Downloading files from the solr replication Handler
Maybe these are text encoding markers? - Original Message - | From: Eva Lacy e...@lacy.ie | To: solr-user@lucene.apache.org | Sent: Thursday, November 29, 2012 3:53:07 AM | Subject: Re: Downloading files from the solr replication Handler | | I tried downloading them with my browser and also with a c# | WebRequest. | If I skip the first and last 4 bytes it seems work fine. | | | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson | erickerick...@gmail.comwrote: | | How are you downloading them? I suspect the issue is | with the download process rather than Solr, but I'm just guessing. | | Best | Erick | | | On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: | | Just to add to that, I'm using solr 3.6.1 | | | On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: | |I downloaded some configuration and data files directly from |solr in an |attempt to develop a backup solution. |I noticed there is some characters at the start and end of the |file | that |aren't in configuration files, I notice the same characters at |the | start |and end of the data files. |Anyone with any idea how I can download these files without the |extra |characters or predict how many there are going to be so I can |skip | them? | | | |
Downloading files from the solr replication Handler
I downloaded some configuration and data files directly from solr in an attempt to develop a backup solution. I noticed there is some characters at the start and end of the file that aren't in configuration files, I notice the same characters at the start and end of the data files. Anyone with any idea how I can download these files without the extra characters or predict how many there are going to be so I can skip them?
Re: Downloading files from the solr replication Handler
Just to add to that, I'm using solr 3.6.1 On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: I downloaded some configuration and data files directly from solr in an attempt to develop a backup solution. I noticed there is some characters at the start and end of the file that aren't in configuration files, I notice the same characters at the start and end of the data files. Anyone with any idea how I can download these files without the extra characters or predict how many there are going to be so I can skip them?
Re: Downloading files from the solr replication Handler
How are you downloading them? I suspect the issue is with the download process rather than Solr, but I'm just guessing. Best Erick On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy e...@lacy.ie wrote: Just to add to that, I'm using solr 3.6.1 On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy e...@lacy.ie wrote: I downloaded some configuration and data files directly from solr in an attempt to develop a backup solution. I noticed there is some characters at the start and end of the file that aren't in configuration files, I notice the same characters at the start and end of the data files. Anyone with any idea how I can download these files without the extra characters or predict how many there are going to be so I can skip them?
Re: Solr replication
BTW, when I said I think to say anything intelligent I meant I think for _me_ to say anything intelligent. Didn't mean to be snarky Erick On Mon, Nov 26, 2012 at 8:19 AM, Erick Erickson erickerick...@gmail.comwrote: both servers are master and slave in the config file. So that means they're polling each other? I think to say anything intelligent you're going to have to provide more data. Please review: http://wiki.apache.org/solr/UsingMailingLists. The bottom line here is that it sounds like what you're trying to do is unusual enough that it's not a typical use case. I further suspect that what you're trying to do will cause problems you haven't tested for. So I wouldn't do this. Simply configuring a proper master/slave setup will give you HA out of the box. If a master goes down there's some complexity in promoting one of your slaves and indexing any stale data. But I'd go with SolrCloud to fire and forget rather than try to force Master/Slave to do something it wasn't intended to do. Best Erick On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes jacques.cor...@laposte.net wrote: Thanks Antoine. In fact, both solr servers are master and slave in config file. They have the same config file and replicate from the same master url with the vip. So, the master is on the server with the vip mounted on. And if heartbeat toggle, the role is toggled too. The question is : what can happend in this configuration? With tests, I can't see nothing bad, but I don't know the intern mecanisms. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication
Thanks Antoine. In fact, both solr servers are master and slave in config file. They have the same config file and replicate from the same master url with the vip. So, the master is on the server with the vip mounted on. And if heartbeat toggle, the role is toggled too. The question is : what can happend in this configuration? With tests, I can't see nothing bad, but I don't know the intern mecanisms. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr replication
both servers are master and slave in the config file. So that means they're polling each other? I think to say anything intelligent you're going to have to provide more data. Please review: http://wiki.apache.org/solr/UsingMailingLists. The bottom line here is that it sounds like what you're trying to do is unusual enough that it's not a typical use case. I further suspect that what you're trying to do will cause problems you haven't tested for. So I wouldn't do this. Simply configuring a proper master/slave setup will give you HA out of the box. If a master goes down there's some complexity in promoting one of your slaves and indexing any stale data. But I'd go with SolrCloud to fire and forget rather than try to force Master/Slave to do something it wasn't intended to do. Best Erick On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes jacques.cor...@laposte.netwrote: Thanks Antoine. In fact, both solr servers are master and slave in config file. They have the same config file and replicate from the same master url with the vip. So, the master is on the server with the vip mounted on. And if heartbeat toggle, the role is toggled too. The question is : what can happend in this configuration? With tests, I can't see nothing bad, but I don't know the intern mecanisms. -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr replication
I have 2 Solr servers : - 1 master - 1 slave The master server has a mounted VIP by heartbeat and can toggle on the other server. What do you think of the idea to put the same config file on the 2 servers with master's replication handler url pointing on the vip? -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Erik Hatcher-4 wrote There's an open issue (with a patch!) that enables this, it seems: lt;https://issues.apache.org/jira/browse/SOLR-3911gt; Erik well patch seems not doing that... i have tried and still getting some error lines about the dir types - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018670.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
There's an open issue (with a patch!) that enables this, it seems: https://issues.apache.org/jira/browse/SOLR-3911 Erik On Nov 5, 2012, at 07:41 , deniz wrote: Michael Della Bitta-2 wrote No, RAMDirectory doesn't work for replication. Use MMapDirectory... it ends up storing the index in RAM and more efficiently so, plus it's backed by disk. Just be sure to not set a big heap because MMapDirectory works outside of heap. for my tests, i dont think index is ended up in ram with mmap... i gave 4gigs for heap while using mmap and got mapping error while indexing... while index should be something around 2 gigs, ram consumption was around 300mbs... Can anyone explain why RAMDirectory cant be used for replication? I cant see why the master is set for using RAMDirectory and replica is using MMap or some other? As far as I understand SolrCloud is some kinda pushing from master to replica/slave... so why it is not possible to push from RAM to HDD? If my logic is wrong, someone can please explain me all these? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Erik Hatcher-4 wrote There's an open issue (with a patch!) that enables this, it seems: lt;https://issues.apache.org/jira/browse/SOLR-3911gt; i will check it for sure, thank you Erik :) Shawn Heisey-4 wrote ... transparently mapping the files on disk to a virtual memory space and using excess RAM to cache that data and make it fast. If you have enough extra memory (disk cache) to fit the entire index, the OS will never have to read any part of the index from disk more than once so for disk cache, are there any disks with 1 gigs or more of caches? if am not wrong there are mostly 16 or 32mb cache disks around (or i am checking the wrong stuff? ) if so, that amount definitely too small... - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Replication is not Possible on RAMDirectory?
Shawn Heisey-4 wrote ... transparently mapping the files on disk to a virtual memory space and using excess RAM to cache that data and make it fast. If you have enough extra memory (disk cache) to fit the entire index, the OS will never have to read any part of the index from disk more than once so for disk cache, are there any disks with 1 gigs or more of caches? if am not wrong there are mostly 16 or 32mb cache disks around (or i am checking the wrong stuff? ) if so, that amount definitely too small... I am not talking about the cache on the actual disk drive, or even cache on your hard drive controller. I am talking about the operating system using RAM, specifically RAM not being used by programs, to cache data on your hard drive. All modern operating systems do it, even the one made in Redmond that people love to hate. If you have 16 GB of RAM and all your programs use up 4.5 GB, you can count on the OS using at least another half GB, so you have about 11 GB left. The OS is going to put data that it reads and writes to/from your disk in this space. If you start up another program that wants 2GB, the OS will simply throw away 2 GB of data in its cache (it's still on the disk, after all) and give that RAM to the new program. Solr counts on this OS capability for good performance. Thanks, Shawn
Re: Solr Replication is not Possible on RAMDirectory?
Here's some reading: http://en.wikipedia.org/wiki/Page_cache Michael Della Bitta Appinions 18 East 41st Street, 2nd Floor New York, NY 10017-6271 www.appinions.com Where Influence Isn’t a Game On Mon, Nov 5, 2012 at 8:02 PM, deniz denizdurmu...@gmail.com wrote: Erik Hatcher-4 wrote There's an open issue (with a patch!) that enables this, it seems: lt;https://issues.apache.org/jira/browse/SOLR-3911gt; i will check it for sure, thank you Erik :) Shawn Heisey-4 wrote ... transparently mapping the files on disk to a virtual memory space and using excess RAM to cache that data and make it fast. If you have enough extra memory (disk cache) to fit the entire index, the OS will never have to read any part of the index from disk more than once so for disk cache, are there any disks with 1 gigs or more of caches? if am not wrong there are mostly 16 or 32mb cache disks around (or i am checking the wrong stuff? ) if so, that amount definitely too small... - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html Sent from the Solr - User mailing list archive at Nabble.com.
