RE: Solr Replication being flaky (6.2.0)
Working on that now to see if it helps us out. Solr process is NOT dying at all. Searches are still working as expected, but since we load balance requestsif the master/slave are out of sync the search results vary. The advice is MUCH appreciated! -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Friday, January 19, 2018 1:49 PM To: solr-user@lucene.apache.org Subject: Re: Solr Replication being flaky (6.2.0) On 1/19/2018 11:27 AM, Shawn Heisey wrote: > On 1/19/2018 8:54 AM, Pouliot, Scott wrote: >> I do have a ticket in with our systems team to up the file handlers >> since I am seeing the "Too many files open" error on occasion on our >> prod servers. Is this the setting you're referring to? Found we >> were set to to 1024 using the "Ulimit" command. > > No, but that often needs increasing too. I think you need to increase > the process limit even if that's not the cause of this particular problem. Had another thought. Either of these limits can cause completely unpredictable problems with Solr. The open file limit could be the reason for these issues, even if you're not actually hitting the process limit. As I mentioned before, I would expect a process limit to cause Solr to kill itself, and your other messages don't mention problems like that. The scale of your Solr installation indicates that you should greatly increase both limits on all of your Solr servers. Thanks, Shawn
RE: Solr Replication being flaky (6.2.0)
That's evidence enough for me to beat on our systems guys to get these file handles upped and cross my fingers then! -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 1:18 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Solr Replication being flaky (6.2.0) "Could be", certainly. "Definitely is" is iffier ;)... But the statement "If we restart the Solr service or optimize the core it seems to kick back in again.", especially the "optimize" bit (which, by the way you should do only if you have the capability of doing it periodically [1]) is some evidence that this may be in the vicinity. One of the effects of an optimize is to merge your segments files from N to 1. So say you have 10 segments. Each one of those may consist of 10-15 individual files, all of which are held open. So you'd go from 150 open file handles to 15.. https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2F2017%2F10%2F13%2Fsegment-merging-deleted-documents-optimize-may-bad%2F=02%7C01%7CScott.Pouliot%40peoplefluent.com%7Cc2912861f58248e3a92808d55f690eb8%7C8b16fb62c78448b6aba889567990e7fe%7C1%7C0%7C636519827178716698=DxnChrfyTbRDjB7HzqpOE%2BvOJRIxdnrXVCIyfoMjJPU%3D=0 Best, Erick On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Erick, > > Thanks! Could these settings be toying with replication? Solr itself seems > to be working like a champ, except when things get out of sync. > > Scott > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Friday, January 19, 2018 12:27 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Solr Replication being flaky (6.2.0) > > Scott: > > We usually recommend setting files and processes very, very high. Like 65K > high. Or unlimited if you can. > > Plus max user processes should also be bumped very high as well, like 65K as > well. > > Plus max memory and virtual memory should be unlimited. > > We've included warnings at startup for open files and processes, see > SOLR-11703 > > Best, > Erick > > On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott > <scott.poul...@peoplefluent.com> wrote: >> I do have a ticket in with our systems team to up the file handlers since I >> am seeing the "Too many files open" error on occasion on our prod servers. >> Is this the setting you're referring to? Found we were set to to 1024 using >> the "Ulimit" command. >> >> -Original Message----- >> From: Shawn Heisey [mailto:apa...@elyograg.org] >> Sent: Friday, January 19, 2018 10:48 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Solr Replication being flaky (6.2.0) >> >> On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >>> So we're running Solr in a Master/Slave configuration (1 of each) and it >>> seems that the replication stalls or stops functioning every now and again. >>> If we restart the Solr service or optimize the core it seems to kick back >>> in again. >>> >>> Anyone have any idea what might be causing this? We do have a good amount >>> of cores on each server (@150 or so), but I have heard reports of a LOT >>> more than that in use. >> >> Have you increased the number of processes that the user running Solr is >> allowed to start? Most operating systems limit the number of >> threads/processes a user can start to a low value like 1024. With 150 >> cores, particularly with background tasks like replication configured, >> chances are that Solr is going to need to start a lot of threads. This is >> an OS setting that a lot of Solr admins end up needing to increase. >> >> I ran into the process limit on my servers and I don't have anywhere near >> 150 cores. >> >> The fact that restarting Solr gets it working again (at least >> temporarily) would fit with a process limit being the problem. I'm not >> guaranteeing that this is the problem, only saying that it fits. >> >> Thanks, >> Shawn
RE: Solr Replication being flaky (6.2.0)
Erick, Thanks! Could these settings be toying with replication? Solr itself seems to be working like a champ, except when things get out of sync. Scott -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Friday, January 19, 2018 12:27 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Solr Replication being flaky (6.2.0) Scott: We usually recommend setting files and processes very, very high. Like 65K high. Or unlimited if you can. Plus max user processes should also be bumped very high as well, like 65K as well. Plus max memory and virtual memory should be unlimited. We've included warnings at startup for open files and processes, see SOLR-11703 Best, Erick On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > I do have a ticket in with our systems team to up the file handlers since I > am seeing the "Too many files open" error on occasion on our prod servers. > Is this the setting you're referring to? Found we were set to to 1024 using > the "Ulimit" command. > > -Original Message- > From: Shawn Heisey [mailto:apa...@elyograg.org] > Sent: Friday, January 19, 2018 10:48 AM > To: solr-user@lucene.apache.org > Subject: Re: Solr Replication being flaky (6.2.0) > > On 1/19/2018 7:50 AM, Pouliot, Scott wrote: >> So we're running Solr in a Master/Slave configuration (1 of each) and it >> seems that the replication stalls or stops functioning every now and again. >> If we restart the Solr service or optimize the core it seems to kick back in >> again. >> >> Anyone have any idea what might be causing this? We do have a good amount >> of cores on each server (@150 or so), but I have heard reports of a LOT more >> than that in use. > > Have you increased the number of processes that the user running Solr is > allowed to start? Most operating systems limit the number of > threads/processes a user can start to a low value like 1024. With 150 cores, > particularly with background tasks like replication configured, chances are > that Solr is going to need to start a lot of threads. This is an OS setting > that a lot of Solr admins end up needing to increase. > > I ran into the process limit on my servers and I don't have anywhere near 150 > cores. > > The fact that restarting Solr gets it working again (at least > temporarily) would fit with a process limit being the problem. I'm not > guaranteeing that this is the problem, only saying that it fits. > > Thanks, > Shawn
RE: Solr Replication being flaky (6.2.0)
I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too many files open" error on occasion on our prod servers. Is this the setting you're referring to? Found we were set to to 1024 using the "Ulimit" command. -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Friday, January 19, 2018 10:48 AM To: solr-user@lucene.apache.org Subject: Re: Solr Replication being flaky (6.2.0) On 1/19/2018 7:50 AM, Pouliot, Scott wrote: > So we're running Solr in a Master/Slave configuration (1 of each) and it > seems that the replication stalls or stops functioning every now and again. > If we restart the Solr service or optimize the core it seems to kick back in > again. > > Anyone have any idea what might be causing this? We do have a good amount of > cores on each server (@150 or so), but I have heard reports of a LOT more > than that in use. Have you increased the number of processes that the user running Solr is allowed to start? Most operating systems limit the number of threads/processes a user can start to a low value like 1024. With 150 cores, particularly with background tasks like replication configured, chances are that Solr is going to need to start a lot of threads. This is an OS setting that a lot of Solr admins end up needing to increase. I ran into the process limit on my servers and I don't have anywhere near 150 cores. The fact that restarting Solr gets it working again (at least temporarily) would fit with a process limit being the problem. I'm not guaranteeing that this is the problem, only saying that it fits. Thanks, Shawn
RE: Solr Replication being flaky (6.2.0)
I'm at the point now where I may end up writing a script to compare
master/slave nightly...and trigger an optimize or solr restart if there are any
differences. Of course I have to check 150+ cores...but it could be done. I'm
just hoping I don't need to go that route
-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com]
Sent: Friday, January 19, 2018 10:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)
This happens to me quite often as well. Generally on the replication admin
screen it will say its downloading a file, but be at 0 or a VERY small kb/sec.
Then after a restart of the slave its back to downloading at 30 to
100 mg/sec. Would be curious if there actually is a solution to this aside
from checking every day if the core replicated. Im on Solr 5.x by the way -Dave
On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:
> So we're running Solr in a Master/Slave configuration (1 of each) and
> it seems that the replication stalls or stops functioning every now
> and again. If we restart the Solr service or optimize the core it
> seems to kick back in again.
>
> Anyone have any idea what might be causing this? We do have a good
> amount of cores on each server (@150 or so), but I have heard reports
> of a LOT more than that in use.
>
> Here is our master config:
>
>
>
> startup
> commit
>
>
> 00:00:10
>
>
>
> 1
>
>
>
> And our slave config:
>
>
>
>
>name="masterUrl">http://server1:8080/solr/${https://na01.safelinks.pro
> tection.outlook.com/?url=solr.core.name=02%7C01%7CScott.Pouliot%4
> 0peoplefluent.com%7C8d43918dd95540a3a11708d55f523302%7C8b16fb62c78448b
> 6aba889567990e7fe%7C1%7C1%7C636519729029923349=Fes6G36gIMRyfahTI
> fftg0eUEVEiVK77B8KpuTr%2FJrA%3D=0}
>
>
>
> 00:00:45
>
>
>
>
>
> solr-data-config.xml
>
>
>
Solr Replication being flaky (6.2.0)
So we're running Solr in a Master/Slave configuration (1 of each) and it seems
that the replication stalls or stops functioning every now and again. If we
restart the Solr service or optimize the core it seems to kick back in again.
Anyone have any idea what might be causing this? We do have a good amount of
cores on each server (@150 or so), but I have heard reports of a LOT more than
that in use.
Here is our master config:
startup
commit
00:00:10
1
And our slave config:
http://server1:8080/solr/${solr.core.name}
00:00:45
solr-data-config.xml
Re: Limit to the number of cores supported?
It doesn't seem to report anything at all, which is part of the problem. No error for me to track down as of yet Get Outlook for iOS<https://aka.ms/o0ukef> From: Erick Erickson <erickerick...@gmail.com> Sent: Monday, July 17, 2017 3:23:24 PM To: solr-user Subject: Re: Limit to the number of cores supported? I know of thousands of cores on a single Solr instance. Operationally there's not problem there, although there may be some practical issues (i.e. startup time and the like). What does your Solr log show? Two popular issues: OutOfMemory issues Not enough file handles (fix with ulimit) But without more specific info about what Solr reports in the log it's impossible to say much. Best, Erick On Mon, Jul 17, 2017 at 10:41 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Hey guys. > > We're running SOLR 6.2.0 in a master/slave configuration and I was wondering > if there is a limit to the number of cores this setup can support? We're > having random issue where a core or 2 will stop responding to POSTS (GETS > work fine) until we restart SOLR. > > We've currently got 140+ cores on this setup and wondering if that could be > part of the problem? > > Anyone ever run into this before? > > Scott
Limit to the number of cores supported?
Hey guys. We're running SOLR 6.2.0 in a master/slave configuration and I was wondering if there is a limit to the number of cores this setup can support? We're having random issue where a core or 2 will stop responding to POSTS (GETS work fine) until we restart SOLR. We've currently got 140+ cores on this setup and wondering if that could be part of the problem? Anyone ever run into this before? Scott
RE: Master/Slave out of sync
I figured the attachments would get stripped, but it was worth a shot! It was
just a screenshot showing the version numbers off from each other.
Here are the Master/Slave commit settings:
18
false
6
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Tuesday, June 27, 2017 11:17 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Master/Slave out of sync
First, attachments are almost always stripped by the mail program, so we can't
see anything.
Hmmm, does look odd. What happens if you issue a commit against the slave via a
url? I.e.
http://server:port/solr/core/update?commit=true?
And what are the autocommit settings on the slave?
Best,
Erick
On Tue, Jun 27, 2017 at 7:22 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:
> Hey guys…
>
>
>
> Does anyone else have a problem with the master/slave setup getting
> out of sync and staying that way until I either optimize the core or
> restart SOLR? It seems to be happening more and more frequently these
> days and I’m looking for a solution here. Running SOLR 6.2 on these
> instances using jetty.
>
>
>
> I do see some log entries like the following at the moment, but it has
> happened WITHOUT these errors in the past as well. This error just
> looks like the core is being loaded, so it can’t replicate (as far as I can
> tell):
>
>
>
> 2017-06-23 00:44:08.624 ERROR (indexFetcher-677-thread-1) [ x:Client1]
> o.a.s.h.IndexFetcher Master at: http://master:8080/solr/Client1 is not
> available. Index fetch failed. Exception: Error from server at
> http://master:8080/solr/Client1: Expected mime type
> application/octet-stream but got text/html.
>
>
>
>
>
> Error 503 {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},m
> sg=SolrCore
> is loading,code=503}
>
>
>
> HTTP ERROR 503
>
> Problem accessing /solr/Client1/replication. Reason:
>
> {metadata={error-class=org.apache.solr.common.
> SolrException,root-error-class=org.apache.solr.common.SolrException},m
> sg=SolrCore
> is loading,code=503}
>
>
>
>
>
>
>
> Our setup looks something like this:
>
>
>
> Master
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Slave
>
> Client Core 1
>
> Client Core 2
>
> Client Core 3
>
>
>
> Master Config
>
> class="solr.ReplicationHandler"
> >
>
>
>
>
>
> startup
>
> commit
>
>
>
>
>
> 00:00:10
>
>
>
>
>
>
>
> 1
>
>
>
>
>
>
>
> Slave Config
>
> class="solr.ReplicationHandler"
> >
>
>
>
>
>
>
>
> http://master:8080/solr/${solr.core.name}
>
>
>
>
>
>
> 00:00:45
>
>
>
>
>
>
>
> Master screenshot
>
>
>
>
>
> Slave Screenshot
>
>
Master/Slave out of sync
Hey guys...
Does anyone else have a problem with the master/slave setup getting out of sync
and staying that way until I either optimize the core or restart SOLR? It
seems to be happening more and more frequently these days and I'm looking for a
solution here. Running SOLR 6.2 on these instances using jetty.
I do see some log entries like the following at the moment, but it has happened
WITHOUT these errors in the past as well. This error just looks like the core
is being loaded, so it can't replicate (as far as I can tell):
2017-06-23 00:44:08.624 ERROR (indexFetcher-677-thread-1) [ x:Client1]
o.a.s.h.IndexFetcher Master at: http://master:8080/solr/Client1 is not
available. Index fetch failed. Exception: Error from server at
http://master:8080/solr/Client1: Expected mime type application/octet-stream
but got text/html.
Error 503
{metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore
is loading,code=503}
HTTP ERROR 503
Problem accessing /solr/Client1/replication. Reason:
{metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore
is loading,code=503}
Our setup looks something like this:
Master
Client Core 1
Client Core 2
Client Core 3
Slave
Client Core 1
Client Core 2
Client Core 3
Master Config
startup
commit
00:00:10
1
Slave Config
http://master:8080/solr/${solr.core.name}
00:00:45
Master screenshot
[cid:image001.png@01D2EF2F.53DF4C10]
Slave Screenshot
[cid:image002.png@01D2EF2F.53DF4C10]
RE: Solr 6.2 - Creating cores via replication from master?
Yeah...I need to get SOLR Cloud up and running. For some reason, I have yet to succeed with it using an external Zookeeper for some reason. Ugghhh Thanks for the confirmation! -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Wednesday, April 12, 2017 4:19 PM To: solr-user@lucene.apache.org Subject: Re: Solr 6.2 - Creating cores via replication from master? On 4/12/2017 2:05 PM, Pouliot, Scott wrote: > Is it possible to create a core on a master SOLR server and have it > automatically replicated to a new slave core? We're running SOLR 6.2 at the > moment, and manually creating the core on the master, and then the slave. > Once we feed the master we're good to go. My manager approached me with a > change to our setup, and in order to facilitate itI need to somehow get > the core replicated automatically from master to slave at creation > timewithout manually calling create core on the slave. > > Is this even possible? I know that the master knows absolutely nothing about > it's slaves out of the box...and I have yet to find any documentation that > tells me otherwise, but figured I'd hit up you experts out here before I > called this a wash. No, that is not possible. This is one of the big advantages of SolrCloud over the old master-slave replication. If you create a new collection and tell it that you want a replicationFactor of 3, then 3 copies of that collection will exist on different machines in the cloud. There are no masters and no slaves -- one of those replicas will be elected as the leader. Thanks, Shawn
Solr 6.2 - Creating cores via replication from master?
Is it possible to create a core on a master SOLR server and have it automatically replicated to a new slave core? We're running SOLR 6.2 at the moment, and manually creating the core on the master, and then the slave. Once we feed the master we're good to go. My manager approached me with a change to our setup, and in order to facilitate itI need to somehow get the core replicated automatically from master to slave at creation timewithout manually calling create core on the slave. Is this even possible? I know that the master knows absolutely nothing about it's slaves out of the box...and I have yet to find any documentation that tells me otherwise, but figured I'd hit up you experts out here before I called this a wash.
RE: SOLR 4.8.0 Master/Slave Replication Issues
Looks like changing the autoCommit maxTime setting is what did it for the
replication issues. Thanks Andrea/Erick for the reminders and pointers!
Scott
-Original Message-
From: Pouliot, Scott [mailto:scott.poul...@peoplefluent.com]
Sent: Friday, March 10, 2017 11:09 AM
To: solr-user@lucene.apache.org
Subject: RE: SOLR 4.8.0 Master/Slave Replication Issues
Valid point on the soft commits. I'll try disabling them to see if there if
any change in performance as well. Worth a shot ;-)
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Friday, March 10, 2017 11:01 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
Glad to hear that. Do note that soft commits are irrelevant to replication, all
that matters are hard commits.
I'd go farther and disable soft commits, they don't serve much point in
master/slave, at least usually. The supposition in M/S setups is that the
master indexes but does not serve queries and the slaves serve queries but do
not index documents. Since soft commits are about being able to search
documents before a hard commit
(openSearcher=true) happens and assuming you're not searching on the master,
they do no good. They won't cause updates to the slaves to happen for instance.
Or, put another way, soft commits are there to support NRT searches.
Master/slave setups are, by definition, not NRT so there's no reason to do soft
commits.
Soft commits are cheaper than hard commits, but they're not free. So disabling
them will decrease the work the master has to do. Whether you'll even notice or
not is an open question, I suspect you'd have to be running your master at
breakneck speed to be able to measure indexing throughput differences with soft
commits enabled/disabled, especially with a 60 second interval, more a style
thing I suppose.
Best,
Erick
On Fri, Mar 10, 2017 at 7:23 AM, Pouliot, Scott
<scott.poul...@peoplefluent.com> wrote:
> My apologies. We're running 2 different sets of SOLR instances, the version
> with the replication problems is 6.2.0 and NOT 4.8.0.
>
> After reading your questions, I double checked the config and we're set to
> autoCommit after a LONG time.
>
>3600
>true
>
>
> Also using softCommit as well:
>
>
>6
>
>
> After looking at our other servers, I changed the autoCommit maxTime down to
> 18 to match. Hoping this might be our issue ;-) I'll report back in a
> bit though!
>
> Thanks for the reminder about commits. We may have forgotten to change these
> values when these new servers got setup.
>
> Scott
>
> -Original Message-
> From: Andrea Gazzarini [mailto:gxs...@gmail.com]
> Sent: Friday, March 10, 2017 3:12 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
>
> Hi Scott,
> that could depend on a lot of things. Some questions:
>
> * What is your commit policy? Explicit / auto / soft / hard ...
> * "Other days things are off by a small amount between master and
> slave"...what do you mean exactly? What is the behaviour you see in
> terms of index versions between nodes?
> * Any error / message in the log file?
>
> Honestly I never used Solr 4.8.0 (this specific version) but I have a lot of
> customers with other 4.x (mostly 4.10.4) master / slave configuration and
> they don't have such problems.
>
> Best,
> Andrea
>
> On 09/03/17 22:26, Pouliot, Scott wrote:
>> So we've been having oddball replication issues between master and slave
>> (Not SOLRCloud) for a few weeks now. Some days...everything lines up just
>> fine. Other days things are off by a small amount between master and slave.
>>
>> Here is the config on the MASTER server replication config:
>>
>>
>>
>>
>>
>>startup
>>commit
>>
>>
>>00:00:10
>>
>>
>>
>> 1
>>
>>
>> Here is the SLAVE server replication config:
>>
>>
>>
>>
>>
>>> name="masterUrl">http://server01:8080/solr/${solr.core.name}
>>
>>
>>00:01:00
>>
>>
>>
>> It seems very sporadic at best. Sometimes I can manually click "Replicate
>> Now" from the replication page in the SOLR UI and it will work, sometimes it
>> does nothing. Sometimes I can go to the master server, and if the
>> "Optimize" button is available and I click itit will replicate as soon
>> as the Optimization is done.
>>
>> Is this a bug in SOLR? Could we possibly have some settings out of whack
>> here? I've been digging around online for a bit and not finding much info
>> here.
>>
>> This is on an older version of SOLR though, so wondering if maybe it used to
>> be a bug?
>>
>> Thanks!
>>
>> Scott
>
RE: SOLR 4.8.0 Master/Slave Replication Issues
Valid point on the soft commits. I'll try disabling them to see if there if
any change in performance as well. Worth a shot ;-)
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Friday, March 10, 2017 11:01 AM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
Glad to hear that. Do note that soft commits are irrelevant to replication, all
that matters are hard commits.
I'd go farther and disable soft commits, they don't serve much point in
master/slave, at least usually. The supposition in M/S setups is that the
master indexes but does not serve queries and the slaves serve queries but do
not index documents. Since soft commits are about being able to search
documents before a hard commit
(openSearcher=true) happens and assuming you're not searching on the master,
they do no good. They won't cause updates to the slaves to happen for instance.
Or, put another way, soft commits are there to support NRT searches.
Master/slave setups are, by definition, not NRT so there's no reason to do soft
commits.
Soft commits are cheaper than hard commits, but they're not free. So disabling
them will decrease the work the master has to do. Whether you'll even notice or
not is an open question, I suspect you'd have to be running your master at
breakneck speed to be able to measure indexing throughput differences with soft
commits enabled/disabled, especially with a 60 second interval, more a style
thing I suppose.
Best,
Erick
On Fri, Mar 10, 2017 at 7:23 AM, Pouliot, Scott
<scott.poul...@peoplefluent.com> wrote:
> My apologies. We're running 2 different sets of SOLR instances, the version
> with the replication problems is 6.2.0 and NOT 4.8.0.
>
> After reading your questions, I double checked the config and we're set to
> autoCommit after a LONG time.
>
>3600
>true
>
>
> Also using softCommit as well:
>
>
>6
>
>
> After looking at our other servers, I changed the autoCommit maxTime down to
> 18 to match. Hoping this might be our issue ;-) I'll report back in a
> bit though!
>
> Thanks for the reminder about commits. We may have forgotten to change these
> values when these new servers got setup.
>
> Scott
>
> -Original Message-
> From: Andrea Gazzarini [mailto:gxs...@gmail.com]
> Sent: Friday, March 10, 2017 3:12 AM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
>
> Hi Scott,
> that could depend on a lot of things. Some questions:
>
> * What is your commit policy? Explicit / auto / soft / hard ...
> * "Other days things are off by a small amount between master and
> slave"...what do you mean exactly? What is the behaviour you see in
> terms of index versions between nodes?
> * Any error / message in the log file?
>
> Honestly I never used Solr 4.8.0 (this specific version) but I have a lot of
> customers with other 4.x (mostly 4.10.4) master / slave configuration and
> they don't have such problems.
>
> Best,
> Andrea
>
> On 09/03/17 22:26, Pouliot, Scott wrote:
>> So we've been having oddball replication issues between master and slave
>> (Not SOLRCloud) for a few weeks now. Some days...everything lines up just
>> fine. Other days things are off by a small amount between master and slave.
>>
>> Here is the config on the MASTER server replication config:
>>
>>
>>
>>
>>
>>startup
>>commit
>>
>>
>>00:00:10
>>
>>
>>
>> 1
>>
>>
>> Here is the SLAVE server replication config:
>>
>>
>>
>>
>>
>>> name="masterUrl">http://server01:8080/solr/${solr.core.name}
>>
>>
>>00:01:00
>>
>>
>>
>> It seems very sporadic at best. Sometimes I can manually click "Replicate
>> Now" from the replication page in the SOLR UI and it will work, sometimes it
>> does nothing. Sometimes I can go to the master server, and if the
>> "Optimize" button is available and I click itit will replicate as soon
>> as the Optimization is done.
>>
>> Is this a bug in SOLR? Could we possibly have some settings out of whack
>> here? I've been digging around online for a bit and not finding much info
>> here.
>>
>> This is on an older version of SOLR though, so wondering if maybe it used to
>> be a bug?
>>
>> Thanks!
>>
>> Scott
>
RE: SOLR 4.8.0 Master/Slave Replication Issues
My apologies. We're running 2 different sets of SOLR instances, the version
with the replication problems is 6.2.0 and NOT 4.8.0.
After reading your questions, I double checked the config and we're set to
autoCommit after a LONG time.
3600
true
Also using softCommit as well:
6
After looking at our other servers, I changed the autoCommit maxTime down to
18 to match. Hoping this might be our issue ;-) I'll report back in a
bit though!
Thanks for the reminder about commits. We may have forgotten to change these
values when these new servers got setup.
Scott
-Original Message-
From: Andrea Gazzarini [mailto:gxs...@gmail.com]
Sent: Friday, March 10, 2017 3:12 AM
To: solr-user@lucene.apache.org
Subject: Re: SOLR 4.8.0 Master/Slave Replication Issues
Hi Scott,
that could depend on a lot of things. Some questions:
* What is your commit policy? Explicit / auto / soft / hard ...
* "Other days things are off by a small amount between master and
slave"...what do you mean exactly? What is the behaviour you see in
terms of index versions between nodes?
* Any error / message in the log file?
Honestly I never used Solr 4.8.0 (this specific version) but I have a lot of
customers with other 4.x (mostly 4.10.4) master / slave configuration and they
don't have such problems.
Best,
Andrea
On 09/03/17 22:26, Pouliot, Scott wrote:
> So we've been having oddball replication issues between master and slave (Not
> SOLRCloud) for a few weeks now. Some days...everything lines up just fine.
> Other days things are off by a small amount between master and slave.
>
> Here is the config on the MASTER server replication config:
>
>
>
>
>
>startup
>commit
>
>
>00:00:10
>
>
>
> 1
>
>
> Here is the SLAVE server replication config:
>
>
>
>
>
> name="masterUrl">http://server01:8080/solr/${solr.core.name}
>
>
>00:01:00
>
>
>
> It seems very sporadic at best. Sometimes I can manually click "Replicate
> Now" from the replication page in the SOLR UI and it will work, sometimes it
> does nothing. Sometimes I can go to the master server, and if the "Optimize"
> button is available and I click itit will replicate as soon as the
> Optimization is done.
>
> Is this a bug in SOLR? Could we possibly have some settings out of whack
> here? I've been digging around online for a bit and not finding much info
> here.
>
> This is on an older version of SOLR though, so wondering if maybe it used to
> be a bug?
>
> Thanks!
>
> Scott
SOLR 4.8.0 Master/Slave Replication Issues
So we've been having oddball replication issues between master and slave (Not
SOLRCloud) for a few weeks now. Some days...everything lines up just fine.
Other days things are off by a small amount between master and slave.
Here is the config on the MASTER server replication config:
startup
commit
00:00:10
1
Here is the SLAVE server replication config:
http://server01:8080/solr/${solr.core.name}
00:01:00
It seems very sporadic at best. Sometimes I can manually click "Replicate Now"
from the replication page in the SOLR UI and it will work, sometimes it does
nothing. Sometimes I can go to the master server, and if the "Optimize" button
is available and I click itit will replicate as soon as the Optimization is
done.
Is this a bug in SOLR? Could we possibly have some settings out of whack here?
I've been digging around online for a bit and not finding much info here.
This is on an older version of SOLR though, so wondering if maybe it used to be
a bug?
Thanks!
Scott
RE: Getting an error: was indexed without position data; cannot run PhraseQuery
We are NOT using SOLRCloud yet. I'm still trying to figure out how to get SOLRCloud running. We're using old school master/slave replication still. So sounds like it can be done if I get to that point. I've got a few non SOLR tasks to get done today, so hoping to dig into this later in the week though. -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Tuesday, March 7, 2017 11:05 AM To: solr-user <solr-user@lucene.apache.org> Subject: Re: Getting an error: was indexed without position data; cannot run PhraseQuery First, it's not clear whether you're using SolrCloud or not, so there may be some irrelevant info in here bq: .could I do it on another instance running the same SOLR version (4.8.0) and then copy the database into place instead In a word "yes", if you're careful. Assuming you have more than one shard you have to be sure to copy the shards faithfully. By that I mean look at your admin UI>>cloud>>tree>>(clusterstate.json or >>collection>>state.json). You'll see a bunch of information for each replica but the critical bit is that the hash range should be the same for the source and destination. It'll be something like 0x-0x7fff for one shard (each replica on a shard has the same hash range). etc. The implication of course is that both collections need to have the same number of shards. If you don't have any shards, don't worry about it... Another possibility, depending on your resources is to create another collection with the same number of shards and index to _that_. Then use the Collections API CREATEALIAS command to atomically switch. This assumes you have enough extra capacity that you can do the reindexing without unduly impacting prod. And there are a number of variants on this. > index to a leader-only collection > during a small maintenance window you shut down prod and ADDREPLICA > for all the shards to build out your new collection blow away your old > collection when you're comfortable. But the bottom line is that indexes may be freely copied wherever you want as long as the bookkeeping is respected wrt hash ranges. I used to build Lucene indexes on a Windows box and copy it to a Unix server as long as I used binary copy.... Best, Erick On Tue, Mar 7, 2017 at 7:04 AM, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > Welcome to IT right? We're always in some sort of pickle ;-) I'm going to > play with settings on one of our internal environments and see if I can > replicate the issue and go from there with some test fixes. > > Here's a question though... If I need to re-indexcould I do it on > another instance running the same SOLR version (4.8.0) and then copy the > database into place instead? We're using some crappy custom Groovy script > run through Aspire to do our indexing and it's horribly slow. 50GB would > take at least a day...maybe 2 and I obviously can't have a client down for > that long in Production, but if I did it on a backup SOLR boxcopying 50GB > into place is much much quicker. > > -Original Message- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Monday, March 6, 2017 8:48 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: Re: Getting an error: was indexed without position > data; cannot run PhraseQuery > > You're in a pickle then. If you change the definition you need to re-index. > > But you claim you haven't changed anything in years as far as the schema is > concerned so maybe you're going to get lucky ;). > > The error you reported is because somehow there's a phrase search going on > against this field. You could have changed something in the query parsers or > eDismax definitions or the query generated on the app side to have phrase > query get through. I'm not quite sure if you'll get information back when the > query fails, but try adding =query to the URL and see what the > parsed_query and parsed_query_toString() to see where phrases are getting > generated. > > Best, > Erick > > On Mon, Mar 6, 2017 at 5:26 PM, Pouliot, Scott > <scott.poul...@peoplefluent.com> wrote: >> Hmm. We haven’t changed data or the definition in YEARS now. I'll >> have to do some more digging I guess. Not sure re-indexing is a >> great thing to do though since this is a production setup and the >> database for this user is @ 50GB. It would take quite a long time to >> reindex all that data from scratch. H >> >> Thanks for the quick reply Erick! >> >> -Original Message- >> From: Erick Erickson [mailto:erickerick...@gmail.com] >> Sent: Monday, March 6, 2017 5:33 PM >> To: solr-user <solr-user@lucene.apache.org> >> Subject: Re: Ge
RE: Getting an error: was indexed without position data; cannot run PhraseQuery
Welcome to IT right? We're always in some sort of pickle ;-) I'm going to
play with settings on one of our internal environments and see if I can
replicate the issue and go from there with some test fixes.
Here's a question though... If I need to re-indexcould I do it on another
instance running the same SOLR version (4.8.0) and then copy the database into
place instead? We're using some crappy custom Groovy script run through Aspire
to do our indexing and it's horribly slow. 50GB would take at least a
day...maybe 2 and I obviously can't have a client down for that long in
Production, but if I did it on a backup SOLR boxcopying 50GB into place is
much much quicker.
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Monday, March 6, 2017 8:48 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Getting an error: was indexed without position data;
cannot run PhraseQuery
You're in a pickle then. If you change the definition you need to re-index.
But you claim you haven't changed anything in years as far as the schema is
concerned so maybe you're going to get lucky ;).
The error you reported is because somehow there's a phrase search going on
against this field. You could have changed something in the query parsers or
eDismax definitions or the query generated on the app side to have phrase
query get through. I'm not quite sure if you'll get information back when the
query fails, but try adding =query to the URL and see what the
parsed_query and parsed_query_toString() to see where phrases are getting
generated.
Best,
Erick
On Mon, Mar 6, 2017 at 5:26 PM, Pouliot, Scott <scott.poul...@peoplefluent.com>
wrote:
> Hmm. We haven’t changed data or the definition in YEARS now. I'll
> have to do some more digging I guess. Not sure re-indexing is a great
> thing to do though since this is a production setup and the database
> for this user is @ 50GB. It would take quite a long time to reindex
> all that data from scratch. H
>
> Thanks for the quick reply Erick!
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Monday, March 6, 2017 5:33 PM
> To: solr-user <solr-user@lucene.apache.org>
> Subject: Re: Getting an error: was indexed without position
> data; cannot run PhraseQuery
>
> Usually an _s field is a "string" type, so be sure you didn't change the
> definition without completely re-indexing. In fact I generally either index
> to a new collection or remove the data directory entirely.
>
> right, the field isn't indexed with position information. That combined with
> (probably) the WordDelimiterFilterFactory in text_en_splitting is generating
> multiple tokens for inputs like 3799H.
> See the admin/analysis page for how that gets broken up. Term positions are
> usually enable by default, so I'm not quite sure why they're gone unless you
> disabled them.
>
> But you're on the right track regardless. you have to
> 1> include term positions for anything that generates phrase queries
> or
> 2> make sure you don't generate phrase queries. edismax can do this if
> you have it configured to, and then there's autoGeneratePhrasQueries that you
> may find.
>
> And do reindex completely from scratch if you change the definitions.
>
> Best,
> Erick
>
> On Mon, Mar 6, 2017 at 1:41 PM, Pouliot, Scott
> <scott.poul...@peoplefluent.com> wrote:
>> We keep getting this in our Tomcat/SOLR Logs and I was wondering if a simple
>> schema change will alleviate this issue:
>>
>> INFO - 2017-03-06 07:26:58.751; org.apache.solr.core.SolrCore;
>> [Client_AdvanceAutoParts] webapp=/solr path=/select
>> params={fl=candprofileid,+candid=0=*:*=json=issearchable:1+AND+cpentitymodifiedon:[2017-01-20T00:00:00.000Z+TO+*]+AND+clientreqid:17672+AND+folderid:132+AND+(engagedid_s:(0)+AND+atleast21_s:(1))+AND+(preferredlocations_s:(3799H))=1000}
>> status=500 QTime=1480 ERROR - 2017-03-06 07:26:58.766;
>> org.apache.solr.common.SolrException; null:java.lang.IllegalStateException:
>> field "preferredlocations_s" was indexed without position data; cannot run
>> PhraseQuery (term=3799)
>> at
>> org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277)
>> at
>> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:351)
>> at
>> org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
>> at
>> org.apache.lucene.search.BooleanQuery$BooleanWeight.bulkScorer(BooleanQuery.java:313)
>> at
>> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
>>
RE: Getting an error: was indexed without position data; cannot run PhraseQuery
Hmm. We haven’t changed data or the definition in YEARS now. I'll have to do
some more digging I guess. Not sure re-indexing is a great thing to do though
since this is a production setup and the database for this user is @ 50GB. It
would take quite a long time to reindex all that data from scratch. H
Thanks for the quick reply Erick!
-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Monday, March 6, 2017 5:33 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Getting an error: was indexed without position data;
cannot run PhraseQuery
Usually an _s field is a "string" type, so be sure you didn't change the
definition without completely re-indexing. In fact I generally either index to
a new collection or remove the data directory entirely.
right, the field isn't indexed with position information. That combined with
(probably) the WordDelimiterFilterFactory in text_en_splitting is generating
multiple tokens for inputs like 3799H.
See the admin/analysis page for how that gets broken up. Term positions are
usually enable by default, so I'm not quite sure why they're gone unless you
disabled them.
But you're on the right track regardless. you have to
1> include term positions for anything that generates phrase queries
or
2> make sure you don't generate phrase queries. edismax can do this if
you have it configured to, and then there's autoGeneratePhrasQueries that you
may find.
And do reindex completely from scratch if you change the definitions.
Best,
Erick
On Mon, Mar 6, 2017 at 1:41 PM, Pouliot, Scott <scott.poul...@peoplefluent.com>
wrote:
> We keep getting this in our Tomcat/SOLR Logs and I was wondering if a simple
> schema change will alleviate this issue:
>
> INFO - 2017-03-06 07:26:58.751; org.apache.solr.core.SolrCore;
> [Client_AdvanceAutoParts] webapp=/solr path=/select
> params={fl=candprofileid,+candid=0=*:*=json=issearchable:1+AND+cpentitymodifiedon:[2017-01-20T00:00:00.000Z+TO+*]+AND+clientreqid:17672+AND+folderid:132+AND+(engagedid_s:(0)+AND+atleast21_s:(1))+AND+(preferredlocations_s:(3799H))=1000}
> status=500 QTime=1480 ERROR - 2017-03-06 07:26:58.766;
> org.apache.solr.common.SolrException; null:java.lang.IllegalStateException:
> field "preferredlocations_s" was indexed without position data; cannot run
> PhraseQuery (term=3799)
> at
> org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:351)
> at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
> at
> org.apache.lucene.search.BooleanQuery$BooleanWeight.bulkScorer(BooleanQuery.java:313)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1158)
> at
> org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:846)
> at
> org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1004)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1517)
> at
> org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1397)
> at
> org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:478)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:461)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
&
Getting an error: was indexed without position data; cannot run PhraseQuery
We keep getting this in our Tomcat/SOLR Logs and I was wondering if a simple
schema change will alleviate this issue:
INFO - 2017-03-06 07:26:58.751; org.apache.solr.core.SolrCore;
[Client_AdvanceAutoParts] webapp=/solr path=/select
params={fl=candprofileid,+candid=0=*:*=json=issearchable:1+AND+cpentitymodifiedon:[2017-01-20T00:00:00.000Z+TO+*]+AND+clientreqid:17672+AND+folderid:132+AND+(engagedid_s:(0)+AND+atleast21_s:(1))+AND+(preferredlocations_s:(3799H))=1000}
status=500 QTime=1480
ERROR - 2017-03-06 07:26:58.766; org.apache.solr.common.SolrException;
null:java.lang.IllegalStateException: field "preferredlocations_s" was indexed
without position data; cannot run PhraseQuery (term=3799)
at
org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:277)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:351)
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.bulkScorer(BooleanQuery.java:313)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
at
org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:1158)
at
org.apache.solr.search.SolrIndexSearcher.getPositiveDocSet(SolrIndexSearcher.java:846)
at
org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(SolrIndexSearcher.java:1004)
at
org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1517)
at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1397)
at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:478)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:461)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
The field in question "preferredlocations_s" is not defined in schema.xml
explicitly, but we have a dynamicField schema entry that covers it.
Would adding omitTermFreqAndPositions="false" to this schema line help out
here? Should I explicitly define this "preferredlocations_s" field in the
schema instead and add it there? We do have a handful of dynamic fields that
all get covered by this rule, but it seems the "preferredlocations_s" field is
the only one throwing errors. All it stores is a CSV string with location IDs
in it.
RE: SOLRCloud on 6.4 on Ubuntu
I actually started out using 3.4.9, and a tutorial that was recent recommended using 3.3.x instead since 3.4 "wasn't ready for production". I'm fine with either really! I did read that in production zk should utilize odd numbers of servers for sure. 1,3,5 etc etc etc for redundancy purposes for a chunk of your cloud doesn't go dead with your zk server. 3 servers provides better coverage since if one dies, you still have 66% of your cloud up etc etc etc. I'm doing this setup in Azure more as a proof of concept and to figure out how in the world to get SOLR Cloud up and running reliably so we can talk about migrating over. I've definitely read over the 2 links you shared, and while I understand themthe lightbulb still hasn’t lit up yet in my head for that "ah ha!" moment. ;-) I plan to try and spin up some new VMs this weekend and start the process over again. It's gotta work one of these times! Thanks for the info! -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: Friday, February 24, 2017 11:34 AM To: solr-user@lucene.apache.org Subject: Re: SOLRCloud on 6.4 on Ubuntu On 2/23/2017 2:12 PM, Pouliot, Scott wrote: > I'm trying to find a good beginner level guide to setting up SolrCloud NOT > using the example configs that are provided with SOLR. > > Here are my goals (and the steps I have done so far!): > > 1. Use an external Zookeeper server > a. wget > http://apache.claz.org/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz Solr includes the 3.4.6 version of the Zookeeper client. I would strongly recommend that the servers be running the latest 3.4.x version, currently 3.4.9. Although I cannot say for sure, it's entirely possible that Solr uses ZK client features that are not supported by an earlier server version. I've omitted the rest of the zookeeper steps you mentioned. They look fine, as long as the configuration is OK and the version is new enough. Another bit of info: You do know that Zookeeper requires three separate physical servers for a redundant install, I hope? One or two servers is not enough. > 2. Install SOLR on both nodes > a. wget http://www.us.apache.org/dist/lucene/solr/6.4.1/solr-6.4.1.tgz > b. tar xzf solr-6.4.1.tgz solr-6.4.1/bin/install_solr_service.sh > --strip-components=2 > c. ./install_solr_service.sh solr-6.4.1.tgz > d. Update solr.in.sh to include the ZKHome variable set to my ZK > server's ip on port 2181 > > Now it seems if I start SOLR manually with bin/solr start -c -p 8080 -z IP>:2181 then it will actually load, but if I let it auto start, I get an > HTTP 500 error on the Admin UI for SOLR. Again ... you need three ZK servers for redundancy, so the setting for -z needs to reference all three, and probably should have a chroot. You can set all of those startup parameters by configuring variables in /etc/default/solr.in.shinstead of starting it manually. The copy of solr.in.sh that's in the bin directory is NOT used when running as a service. > I also can't seem to figure out what I need to upload into Zookeeper as far > as configuration files go. I created a test collection on the instance when > I got it up one time...but it has yet to start properly again for me. Use the upconfig command with zkcli or the zk command on the solr script. The directory you are uploading should contain everything in a core config that's normally in the "conf" directory -- solrconfig.xml, the schema, and any files referenced by either of those. > Are there any GOOD tutorials out there? I have read most of the > documentation I can get my hands on thus far from Apache, and blogs > and such, but the light bulb still has not lit up for me yet and I > feel like a n00b ;-) There's a quick start. This URL shows how to start a SolrCloud example where Zookeeper is embedded within one of the Solr nodes, and everything's on one machine. This setup is not suitable for production. http://lucene.apache.org/solr/quickstart.html This is some more detailed info about migrating to production: https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production Information about setting up a redundant external Zookeeper is best obtained from the Zookeeper project. They understand their software best. > My company is currently running SOLR in the old master/slave config and I'm > trying to setup a SOLRCloud so that we can toy with it in a Dev/QA > Environment and see what it's capable of. We're currently running 4 separate > master/slave SOLR server pairs in production to spread out the load a bit, > but I'd rather see us migrate towards a cluster/cloud scenario to gain some > computing power here! What SolrCloud offers is much easier management and a true cluster with no masters and no slaves. Depending on how th
RE: SOLRCloud on 6.4 on Ubuntu
I did see that and was trying to go down that path, but apparently messed something up! Seems if I start SOLR with the following command, it starts, but the Admin UI only throws a 500 error. bin/solr start -c -p 8080 -z 1.1.1.1:2181 It ends up tossing this: org.apache.solr.common.SolrException: Error processing the request. CoreContainer is either not initialized or shutting down. at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:745) I'll have to start over with some fresh Azure VMs and see if I can get this going. I did manage to get everything up using the internal Zookeeper once...and create a collection and all that jazz. Using the external, proved to smack me down though ;-) -Original Message- From: Erick Erickson [mailto:erickerick...@gmail.com] Sent: Thursday, February 23, 2017 5:44 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: SOLRCloud on 6.4 on Ubuntu Getting configs up (and down) from solr is done either with zkCli or bin/solr. Personally I find the latter easier if only because it's in a single place. Try bin/solr zk -help and you'll see a bunch of options. Once you do upload the config, you must reload the collection for it to "take". Best, Erick On Thu, Feb 23, 2017 at 1:51 PM, Alexandre Rafalovitch <arafa...@gmail.com> wrote: > I don't know which of these you read, so it is a bit of a grab bag. > And I haven't reviewed some of them in depth. But hopefully, there is > a nugget of gold somewhere in there for you: > > https://github.com/LucidWorks/solr-scale-tk > https://www.slideshare.net/thelabdude/apache-con-managingsolrcloudinth > ecloud > https://systemsarchitect.net/2013/04/06/painless-guide-to-solr-cloud-c > onfiguration/ https://github.com/bloomreach/solrcloud-haft > http://www.francelabs.com/blog/tutorial-solrcloud-5-amazon-ec2/ > (oldish) https://github.com/freedev/solrcloud-zookeeper-docker > https://sematext.com/blog/2016/12/13/solr-master-slave-solrcloud-migra > tion/ > http://dlightdaily.com/2016/11/30/solr-cloud-installation-zookeeper/ > https://sbdevel.wordpress.com/2016/11/30/70tb-16b-docs-4-machines-1-so > lrcloud/ (just to drool, but it may also be useful) > > Hope it helps, >Alex. > > http://www.solr-start.com/ - Resources for Solr users, new and > experienced > > > On 23 February 2017 at 16:12, Pouliot, Scott > <scott.poul...@peoplefluent.com> wrote: >> I'm trying to find a good beginner level guid
RE: SOLRCloud on 6.4 on Ubuntu
Thanks Alex. I'll thumb through them for sure Most were good when I read them, but seemed like a few steps they skipped out on adding the details needed to make it completely painless ;-) -Original Message- From: Alexandre Rafalovitch [mailto:arafa...@gmail.com] Sent: Thursday, February 23, 2017 4:52 PM To: solr-user <solr-user@lucene.apache.org> Subject: Re: SOLRCloud on 6.4 on Ubuntu I don't know which of these you read, so it is a bit of a grab bag. And I haven't reviewed some of them in depth. But hopefully, there is a nugget of gold somewhere in there for you: https://github.com/LucidWorks/solr-scale-tk https://www.slideshare.net/thelabdude/apache-con-managingsolrcloudinthecloud https://systemsarchitect.net/2013/04/06/painless-guide-to-solr-cloud-configuration/ https://github.com/bloomreach/solrcloud-haft http://www.francelabs.com/blog/tutorial-solrcloud-5-amazon-ec2/ (oldish) https://github.com/freedev/solrcloud-zookeeper-docker https://sematext.com/blog/2016/12/13/solr-master-slave-solrcloud-migration/ http://dlightdaily.com/2016/11/30/solr-cloud-installation-zookeeper/ https://sbdevel.wordpress.com/2016/11/30/70tb-16b-docs-4-machines-1-solrcloud/ (just to drool, but it may also be useful) Hope it helps, Alex. http://www.solr-start.com/ - Resources for Solr users, new and experienced On 23 February 2017 at 16:12, Pouliot, Scott <scott.poul...@peoplefluent.com> wrote: > I'm trying to find a good beginner level guide to setting up SolrCloud NOT > using the example configs that are provided with SOLR. > > Here are my goals (and the steps I have done so far!): > > > 1. Use an external Zookeeper server > > a. wget > http://apache.claz.org/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz > > b. uncompress into /apps folder (Our company uses this type of standard > folder, so I'm following suit here) > > c. Copy zoo_sample.cfg to zoo.cfg > > d. Update data folder to: /apps/zookeeperData > > e. Bin/zkServer.sh start > > 2. Install SOLR on both nodes > > a. wget http://www.us.apache.org/dist/lucene/solr/6.4.1/solr-6.4.1.tgz > > b. tar xzf solr-6.4.1.tgz solr-6.4.1/bin/install_solr_service.sh > --strip-components=2 > > c. ./install_solr_service.sh solr-6.4.1.tgz > > d. Update solr.in.sh to include the ZKHome variable set to my ZK > server's ip on port 2181 > > Now it seems if I start SOLR manually with bin/solr start -c -p 8080 -z IP>:2181 then it will actually load, but if I let it auto start, I get an > HTTP 500 error on the Admin UI for SOLR. > > I also can't seem to figure out what I need to upload into Zookeeper as far > as configuration files go. I created a test collection on the instance when > I got it up one time...but it has yet to start properly again for me. > > Are there any GOOD tutorials out there? I have read most of the > documentation I can get my hands on thus far from Apache, and blogs > and such, but the light bulb still has not lit up for me yet and I > feel like a n00b ;-) > > My company is currently running SOLR in the old master/slave config and I'm > trying to setup a SOLRCloud so that we can toy with it in a Dev/QA > Environment and see what it's capable of. We're currently running 4 separate > master/slave SOLR server pairs in production to spread out the load a bit, > but I'd rather see us migrate towards a cluster/cloud scenario to gain some > computing power here! > > Any help is GREATLY appreciated! > > Scott
SOLRCloud on 6.4 on Ubuntu
I'm trying to find a good beginner level guide to setting up SolrCloud NOT using the example configs that are provided with SOLR. Here are my goals (and the steps I have done so far!): 1. Use an external Zookeeper server a. wget http://apache.claz.org/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz b. uncompress into /apps folder (Our company uses this type of standard folder, so I'm following suit here) c. Copy zoo_sample.cfg to zoo.cfg d. Update data folder to: /apps/zookeeperData e. Bin/zkServer.sh start 2. Install SOLR on both nodes a. wget http://www.us.apache.org/dist/lucene/solr/6.4.1/solr-6.4.1.tgz b. tar xzf solr-6.4.1.tgz solr-6.4.1/bin/install_solr_service.sh --strip-components=2 c. ./install_solr_service.sh solr-6.4.1.tgz d. Update solr.in.sh to include the ZKHome variable set to my ZK server's ip on port 2181 Now it seems if I start SOLR manually with bin/solr start -c -p 8080 -z :2181 then it will actually load, but if I let it auto start, I get an HTTP 500 error on the Admin UI for SOLR. I also can't seem to figure out what I need to upload into Zookeeper as far as configuration files go. I created a test collection on the instance when I got it up one time...but it has yet to start properly again for me. Are there any GOOD tutorials out there? I have read most of the documentation I can get my hands on thus far from Apache, and blogs and such, but the light bulb still has not lit up for me yet and I feel like a n00b ;-) My company is currently running SOLR in the old master/slave config and I'm trying to setup a SOLRCloud so that we can toy with it in a Dev/QA Environment and see what it's capable of. We're currently running 4 separate master/slave SOLR server pairs in production to spread out the load a bit, but I'd rather see us migrate towards a cluster/cloud scenario to gain some computing power here! Any help is GREATLY appreciated! Scott
