ExternalFileField management strategy with SolrCloud

2018-04-26 Thread Tom Peters
Is there a recommended way of managing external files with SolrCloud. At first glance it appears that I would need to manually manage the placement of the external_.txt file in each shard's data directory. Is there a better way of managing this (Solr API, interface, etc?) This message and any

Re: CDCR Bootstrap

2018-04-26 Thread Tom Peters
I'm not sure under what conditions it will be automatically triggered, but if you manually wanted to trigger a CDCR Bootstrap you need to issue the following query to the leader in your target data center. /solr//cdcr?action=BOOTSTRAP= The masterUrl will look something like (change the

Re: Does CDCR Bootstrap sync leaves replica's out of sync

2018-04-16 Thread Tom Peters
There are two ways I've gotten around this issue: 1. Add replicas in the target data center after CDCR bootstrapping has completed. -or- 2. After the bootstrapping has completed, restart the replica nodes one-at-time in the target data center (restart, wait for replica to catch up, then

Re: CDCR performance issues

2018-03-23 Thread Tom Peters
cdcr.update=,_stateVer_=synacor:199},delByQ=null,docsMap=[MapEntry[SolrInputDocument(fields: [solr_id=Mytest, _version_=1595749902698151936]):null]]} > > > > Amrit Sarkar > Search Engineer > Lucidworks, Inc. > 415-589-9269 > www.lucidworks.com > Twitter http://twitter.com/lucidworks

Re: CDCR performance issues

2018-03-12 Thread Tom Peters
I'm also having issue with replicas in the target data center. It will go from recovering to down. And when one of my replicas go to down in the target data center, CDCR will no longer send updates from the source to the target. > On Mar 12, 2018, at 9:24 AM, Tom Peters <tpet...@synac

Re: CDCR performance issues

2018-03-12 Thread Tom Peters
? Or is there something else we can do? Thanks. > On Mar 9, 2018, at 3:59 PM, Tom Peters <tpet...@synacor.com> wrote: > > Thanks. This was helpful. I did some tcpdumps and I'm noticing that the > requests to the target data center are not batched in any way. Each update > comes

Re: CDCR performance issues

2018-03-09 Thread Tom Peters
checkout paper > "Latency performance of SOAP Implementations". Same distribution of skills > - I knew TCP well, but Apache Axis 1.1 not so well. I still improved > response time of Apache Axis 1.1 by 250ms per call with 1-line of code. > > -Original Message

Re: CDCR performance issues

2018-03-08 Thread Tom Peters
of updates (3805 over two hours). Thanks. > On Mar 7, 2018, at 6:19 PM, Tom Peters <tpet...@synacor.com> wrote: > > I'm having issues with the target collection staying up-to-date with indexing > from the source collection using CDCR. > > This is what I'm gett

CDCR performance issues

2018-03-07 Thread Tom Peters
I'm having issues with the target collection staying up-to-date with indexing from the source collection using CDCR. This is what I'm getting back in terms of OPS: curl -s 'solr2-a:8080/solr/mycollection/cdcr?action=OPS' | jq . { "responseHeader": { "status": 0,

Re: Issues with CDCR in Solr 7.1

2018-03-05 Thread Tom Peters
You can ignore this. I think I found the issue (I was missing a block of XML in the source ocnfig). I'm going to monitor it over the next day and see if it was resolved. > On Mar 5, 2018, at 4:29 PM, Tom Peters <tpet...@synacor.com> wrote: > > I'm trying to get Solr CDCR se

Issues with CDCR in Solr 7.1

2018-03-05 Thread Tom Peters
I'm trying to get Solr CDCR setup in Solr 7.1 and I'm having issues post-bootstrap. I have about 5,572,933 documents in the source cluster (index size is 3.77 GB). I'm enabling CDCR in the following manner: 1. Delete the existing cluster in the target data center

Re: /var/solr/data has lots of index* directories

2018-03-05 Thread Tom Peters
ou can look inside the index.properties. The directory name mentioned in > that properties file is the one being used actively. The rest are old > directories that should be cleaned up on Solr restart but you can delete > them yourself without any issues. > > On Mon, Mar 5, 2

/var/solr/data has lots of index* directories

2018-03-05 Thread Tom Peters
While trying to debug an issue with CDCR, I noticed that the /var/solr/data directories on my source cluster have wildly different sizes. % for i in solr2-{a..e}; do echo -n "$i: "; ssh -A $i du -sh /var/solr/data; done solr2-a: 9.5G /var/solr/data solr2-b: 29G/var/solr/data

Re: Indexing timeout issues with SolrCloud 7.1

2018-03-01 Thread Tom Peters
b 24, 2018 at 1:37 AM, Deepak Goel <deic...@gmail.com> wrote: >> From the error list, i can see multiple errors: >> >> 1. Failure to recover replica >> 2. Peer sync error >> 3. Failure to download file >> >> On 24 Feb 2018 03:10, "Tom Peters"

Re: Indexing timeout issues with SolrCloud 7.1

2018-02-23 Thread Tom Peters
node > 'solr-2d' > > On 23 Feb 2018 09:42, "Tom Peters" <tpet...@synacor.com> wrote: > > I'm trying to debug why indexing in SolrCloud 7.1 is having so many issues. > It will hang most of the time, and timeout the rest. > > Here's an example: > >tim

Indexing timeout issues with SolrCloud 7.1

2018-02-22 Thread Tom Peters
I'm trying to debug why indexing in SolrCloud 7.1 is having so many issues. It will hang most of the time, and timeout the rest. Here's an example: time curl -s 'myhost:8080/solr/mycollection/update/json/docs' -d '{"solr_id":"test_001", "data_type":"test"}'|jq . {

Re: Issue with CDCR bootstrapping in Solr 7.1

2017-12-04 Thread Tom Peters
ww.linkedin.com/in/sarkaramrit2 > Medium: https://medium.com/@sarkaramrit2 > > On Fri, Dec 1, 2017 at 4:49 AM, Tom Peters <tpet...@synacor.com> wrote: > >> Hi Amrit, I tried issuing hard commits to the various nodes in the target >> cluster and it does not appear to cause the fol

Re: Issue with CDCR bootstrapping in Solr 7.1

2017-11-30 Thread Tom Peters
m/lucidworks > LinkedIn: https://www.linkedin.com/in/sarkaramrit2 > Medium: https://medium.com/@sarkaramrit2 > > On Thu, Nov 30, 2017 at 10:06 PM, Tom Peters <tpet...@synacor.com> wrote: > >> Hi Amrit, >> >> Starting with more documents doesn't appear to h

Re: Issue with CDCR bootstrapping in Solr 7.1

2017-11-30 Thread Tom Peters
00, unless you have modified in > solrconfig.xml. > > Looking forward to your analysis. > > Amrit Sarkar > Search Engineer > Lucidworks, Inc. > 415-589-9269 > www.lucidworks.com > Twitter http://twitter.com/lucidworks > LinkedIn: https://www.linkedin.com/in

Issue with CDCR bootstrapping in Solr 7.1

2017-11-30 Thread Tom Peters
I'm running into an issue with the initial CDCR bootstrapping of an existing index. In short, after turning on CDCR only the leader replica in the target data center will have the documents replicated and it will not exist in any of the follower replicas in the target data center. All