Re: Solr - zkRun problem

2019-09-05 Thread Tamás Barta
I could solve the problem, the "-Dbootstrap_conf=true" launch parameter caused this problem. After removed it everything is fine. On Thu, Sep 5, 2019 at 3:29 PM Bruno Osiek wrote: > In order to persist docker container files into its host filesystem you > need to link both by -v parameter when

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Doss
Thanks Eric for the explanation. Sum of all our index size is about 138 GB, only 2 indexes are > 19 GB, time to scale up :-). Adding new hardware will require at least couple of days, till that time is there any option to control the replication method? Thanks, Doss. On Thu, Sep 5, 2019 at 6:12

Re: string field max size

2019-09-05 Thread Erick Erickson
Use a text field with stored=true and indexed=false? That'll allow you to return it... On Thu, Sep 5, 2019, 07:04 Vincenzo D'Amore wrote: > Hi all, > > sorry for the silly question, I need to store in Solr a string field larger > than 32k (index="false"). > > Given that storing field larger

Re: string field max size

2019-09-05 Thread Jitendra soni
I agree, stored=true and indexed =false should resolve this size issue. On Thu, 5 Sep 2019 at 21:54, Erick Erickson wrote: > Use a text field with stored=true and indexed=false? That'll allow you to > return it... > > On Thu, Sep 5, 2019, 07:04 Vincenzo D'Amore wrote: > > > Hi all, > > > >

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Erick Erickson
If I'm reading this correctly, you have a huge amount of index in not much memory. You only have 14g allocated across 130 replicas, at least one of which has a 20g index. You don't need as much memory as your aggregate index size, but this system feels severely under provisioned. I suspect that's

Re: Atomic indexing as default indexing mode in Solr

2019-09-05 Thread Erick Erickson
Shankar: Follow the instructions here: http://lucene.apache.org/solr/community.html#mailing-lists-irc. You must use the _exact_ same e-mail as you used to subscribe. If the initial try doesn't work and following the suggestions at the "problems" link doesn't work for you, let us know. But

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Erick Erickson
You say you have three nodes, 130 replicas and a replication factor of 3, so you have 130 cores/node. At least one of those cores has a 20G index, right? What is the sum of all the indexes on a single physical machine? I think your system is under-provisioned and that you’ve been riding at the

Re: string field max size

2019-09-05 Thread Vincenzo D'Amore
Thanks Erick for the prompt answer. What I do not understand is what happens to the Analyzers, Tokenizers, and Filters in the indexing chain. Are they executed or not? Well, answering to my own question I think no, but so what's the difference between string and text when they are not indexed?

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Jörn Franke
1 Node zookeeper ensemble does not sound very healthy > Am 05.09.2019 um 13:07 schrieb Doss : > > Hi, > > We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble. > Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT) > with index size ranging from 700MB to

Need help | NoNodeException | Could not read DIH properties

2019-09-05 Thread Pal Sumit
Hi, I am getting the below log very frequently and I can't find more details about it. ZKPropertiesWriter Could not read DIH properties from /configs//dataimport.properties :class org.apache.zookeeper.KeeperException$NoNodeException Details: We have a Solr cluster containing 2 Solr

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Doss
@Jorn We are adding few more zookeeper nodes soon. Thanks. @ Erick, sorry I couldn't understand it clearly, we have 90GB RAM per node, out of which 14 GB assigned for HEAP, you mean to say we have to allocate more HEAP? or we need add more Physical RAM? This system ran for 8 to 9 months without

Re: Solr - zkRun problem

2019-09-05 Thread Bruno Osiek
In order to persist docker container files into its host filesystem you need to link both by -v parameter when starting the container. As you said the problem emerges only when restarting this would be my guess. Regards, Bruno On Thu, Sep 5, 2019 at 1:19 AM Tamás Barta wrote: > Hi, > > I use

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

2019-09-05 Thread Russell Bahr
Hi Shawn, Sorry for the other link. I figured out after I sent the first one how to share the entire folder. Please try this link and let me know if that works. https://www.dropbox.com/sh/hslknixd3azj7mi/AABnCXex_HInCvRz3kuKLwNna?dl=0 I will pull the GC logs and save them up to the same folder

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Jack Schlederer
I'd defer to the committers if they have any further advice, but you might have to suspend the autoAddReplicas trigger through the autoscaling API ( https://solr.stage.ecommerce.sandbox.directsupply-sandbox.cloud:8985/solr/ ) if you set up your collections with autoAddReplicas enabled. Then, the

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Jack Schlederer
My mistake on the link, which should be this: https://lucene.apache.org/solr/guide/7_1/solrcloud-autoscaling-auto-add-replicas.html#implementation-using-autoaddreplicas-trigger --Jack On Thu, Sep 5, 2019 at 11:02 AM Jack Schlederer wrote: > I'd defer to the committers if they have any further

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

2019-09-05 Thread Shawn Heisey
On 9/4/2019 12:48 PM, Russell Bahr wrote: Thank you for the feedback and advise. I have loaded the 2 screenshots up to drop box. Here is the link. https://www.dropbox.com/s/c5b41a61za0ojw7/solr4_Screen%20Shot%202019-09-03%20at%203.37.08%20PM.png?dl=0 Just one screenshot there. Looking at

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

2019-09-05 Thread Russell Bahr
Hi Toke and Shawn, Any thoughts on what I sent? Thanks in advance, Russ *Manzama*a MODERN GOVERNANCE company Russell Bahr Lead Infrastructure Engineer USA & CAN Office: +1 (541) 306 3271 USA & CAN Support: +1 (541) 706 9393 UK Office & Support: +44 (0)203 282 1633 AUS Office & Support: +61 (0)

Re: SolrClient from inside processAdd function

2019-09-05 Thread Arnold Bronley
Hi Markus, Is there any way to get the information about the current Solr endpoint from within the custom URP? On Wed, Sep 4, 2019 at 3:10 PM Markus Jelsma wrote: > Hello Arnold, > > Yes, we do this too for several cases. > > You can create the SolrClient in the Factory's inform() method, and

host and port for SolrTestCaseJ4 and EmbeddedSolrServer

2019-09-05 Thread Arnold Bronley
Hi, In SolrTestCaseJ4 there is initCore function. After using this function how to know on which host and port the solr is running. Same goes for EmbeddedSolrServer? How to know on which port and host it is running?

Re: string field max size

2019-09-05 Thread Erick Erickson
bq. What I do not understand is what happens to the Analyzers, Tokenizers, and Filters in the indexing chain They are irrelevant. The analysis chain is only executed when indexed=true. Best, Erick > On Sep 5, 2019, at 9:03 AM, Vincenzo D'Amore wrote: > > What I do not understand is what

RE: SolrClient from inside processAdd function

2019-09-05 Thread Markus Jelsma
Hello Arnold, In the Factory's inform() method you receive a SolrCore reference. Using this you can get the CloudDescriptor and the ZkController references. These provide access to what you need to open a connection for SolrClient. Our plugins usually work in cloud and non-cloud environments,

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

2019-09-05 Thread Russell Bahr
Hi Shawn and Toke, I have uploaded the solr_gc.log for solr8 and the catalina.out log or solr4 to the link i shared to my dropbox folder. Did you get a chance to look at the configs I uploaded? If you want I can clear out the comments to make it smaller to read? Thank you, Russ *Manzama*a

Restore from HDFS slow

2019-09-05 Thread Aroop Ganguly
Hey Solr Experts Anyone has idea how to ensure restore collections from HDFS can be made faster? Are there any tuning parameters like number of threads to use, the memory to use etc, that can be configured someplace to enhance/manage the restore process? I am on Solr 7.7.2 btw and the api we

Get host/port information for current Solr

2019-09-05 Thread Arnold Bronley
Hi, is there a way to get host/port information for current Solr from inside custom Solr URP plugin? One way it to do use 'localhost:8983' but I feel little uncomfortable with such hardcoding of the port.

Re: upgrading from solr4 to solr8 searches taking 4 to 10 times as long to return

2019-09-05 Thread David Smiley
I suggest first working with a single machine to see if it responds substantially slower with the new version. Just find one of yours and issue it a query that will resolve locally (distrib=false param). Your current collection level queries are internally issuing such queries, and so with a

Re: Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Doss
Dear Jack, Thanks for your input. Non of our cores were created with autoAddReplicas. The problem we are facing is, upon rebooting leader tries to sync the data with other nodes which are part of the cluster. Thanks, Doss. On Thu, Sep 5, 2019 at 9:46 PM Jack Schlederer wrote: > My mistake on

Re: host and port for SolrTestCaseJ4 and EmbeddedSolrServer

2019-09-05 Thread Mikhail Khludnev
Hello, Arnold. Noone of them bootstrap jetty. Look at Solrj or Cloud tests. On Fri, Sep 6, 2019 at 2:06 AM Arnold Bronley wrote: > Hi, > > In SolrTestCaseJ4 there is initCore function. After using this function how > to know on which host and port the solr is running. Same goes for >

Suggestion Needed: Exclude documents that are already served / viewed by a customer

2019-09-05 Thread Doss
Dear Experts, For a matchmaking portal, we have one requirement where in, if a customer viewed complete details of a bride or groom then we have to exclude that profile id from further search results. Currently, along with other details we are storing the viewed profile ids in a field

Solr - zkRun problem

2019-09-05 Thread Tamás Barta
Hi, I use Solr Cloud with 3 ZK nodes in production and it is good. Now I try to create a "simple" Solr docker container in a test environment. I don't want to create ZK nodes too so I run solr with the -DzkRun parameter to start embedded Zookeeper. After it start I uploaded my configuration to

Re: Atomic indexing as default indexing mode in Solr

2019-09-05 Thread Shankar Ramalingam
Please remove my email id from this list. On Thu, 5 Sep, 2019, 5:55 AM Erick Erickson, wrote: > Because atomic updates require special preparation, specifically all > original fields must be stored which is not a requirement and is, in fact, > an anti-pattern in large installations. > > Best, >

string field max size

2019-09-05 Thread Vincenzo D'Amore
Hi all, sorry for the silly question, I need to store in Solr a string field larger than 32k (index="false"). Given that storing field larger than 32k rises an exception: "DocValuesField "filterQuery" is too large, must be <= 32766", I thought to use predefined type text_ws. Any suggestions?

Production Issue: SOLR node goes to non responsive , restart not helping at peak hours

2019-09-05 Thread Doss
Hi, We are using 3 node SOLR (7.0.1) cloud setup 1 node zookeeper ensemble. Each system has 16CPUs, 90GB RAM (14GB HEAP), 130 cores (3 replicas NRT) with index size ranging from 700MB to 20GB. autoCommit - 10 minutes once softCommit - 30 Sec Once At peak time if a shard goes to recovery mode