Creating 100000 dynamic fields in solr
Hi Is it good idea to create 10 dynamic fields of time pint in solr? I have that many fields to search on actually which come upon based on users. Thanks in advance! And I'm using Solr Cloud in real-time. Regards, Sai Vignan M
Re: Response Time Diff between Collection with low deletes
Why so many shards? > On May 10, 2020, at 9:09 PM, Ganesh Sethuraman > wrote: > > We are using dedicated host, Cent OS in EC2 r5.12xlarge (48 CPU, ~360GB > RAM), 2 nodes. Swapiness set to 1. With General purpose 2T EBS SSD volume. > JVM size of 18gb, with G1 GC enabled. About 92 collection with average of 8 > shards and 2 replica each. Most of updates over daily batch updates. > > While we have Solr disk utilization of about ~800gb. Most of the collection > space are for real time GET, /get call. The issue we are having is for few > collection where we having query use case /need. This has 32 replica (16 > shards 2 replica each). During performance test, issue is few calls where > we have high response time, it is noticeable when test duration is small, > the response time improve when the test is for longer duration. > > Hope this information helps. > > Regards > Ganesh > > Regards > Ganesh > > >> On Sun, May 10, 2020, 8:14 PM Shawn Heisey wrote: >> >>> On 5/10/2020 4:48 PM, Ganesh Sethuraman wrote: >>> The additional info is that when we execute the test for longer (20mins) >> we >>> are seeing better response time, however for a short test (5mins) and >> rerun >>> the test after an hour or so we are seeing slow response times again. >> Note >>> that we don't update the collection during the test or in between the >> test. >>> Does this help to identify the issue? >> >> Assuming Solr is the only software that is running, most operating >> systems would not remove Solr data from the disk cache, so unless you >> have other software running on the machine, it's a little weird that >> performance drops back down after waiting an hour. Windows is an >> example of an OS that *does* proactively change data in the disk cache, >> and on that OS, I would not be surprised by such behavior. You haven't >> mentioned which OS you're running on. >> >>> 3. We have designed our test to mimick reality where filter cache is not >>> hit at all. From solr, we are seeing that there is ZERO Filter cache hit. >>> There is about 4% query and document cache hit in prod and we are seeing >> no >>> filter cache hit in both QA and PROD >> >> If you're getting zero cache hits, you should disable the cache that is >> getting zero hits. There is no reason to waste the memory that the >> cache uses, because there is no benefit. >> >>> Give that, could this be some warming up related issue to keep the Solr / >>> Lucene memory-mapped file in RAM? Is there any way to measure which >>> collection is using memory? we do have 350GB RAM, but we see it full with >>> buffer cache, not really sure what is really using this memory. >> >> You would have to ask the OS which files are contained by the OS disk >> cache, and it's possible that even if the information is available, that >> it is very difficult to get. There is no way Solr can report this. >> >> Thanks, >> Shawn >>
Re: Response Time Diff between Collection with low deletes
We are using dedicated host, Cent OS in EC2 r5.12xlarge (48 CPU, ~360GB RAM), 2 nodes. Swapiness set to 1. With General purpose 2T EBS SSD volume. JVM size of 18gb, with G1 GC enabled. About 92 collection with average of 8 shards and 2 replica each. Most of updates over daily batch updates. While we have Solr disk utilization of about ~800gb. Most of the collection space are for real time GET, /get call. The issue we are having is for few collection where we having query use case /need. This has 32 replica (16 shards 2 replica each). During performance test, issue is few calls where we have high response time, it is noticeable when test duration is small, the response time improve when the test is for longer duration. Hope this information helps. Regards Ganesh Regards Ganesh On Sun, May 10, 2020, 8:14 PM Shawn Heisey wrote: > On 5/10/2020 4:48 PM, Ganesh Sethuraman wrote: > > The additional info is that when we execute the test for longer (20mins) > we > > are seeing better response time, however for a short test (5mins) and > rerun > > the test after an hour or so we are seeing slow response times again. > Note > > that we don't update the collection during the test or in between the > test. > > Does this help to identify the issue? > > Assuming Solr is the only software that is running, most operating > systems would not remove Solr data from the disk cache, so unless you > have other software running on the machine, it's a little weird that > performance drops back down after waiting an hour. Windows is an > example of an OS that *does* proactively change data in the disk cache, > and on that OS, I would not be surprised by such behavior. You haven't > mentioned which OS you're running on. > > > 3. We have designed our test to mimick reality where filter cache is not > > hit at all. From solr, we are seeing that there is ZERO Filter cache hit. > > There is about 4% query and document cache hit in prod and we are seeing > no > > filter cache hit in both QA and PROD > > If you're getting zero cache hits, you should disable the cache that is > getting zero hits. There is no reason to waste the memory that the > cache uses, because there is no benefit. > > > Give that, could this be some warming up related issue to keep the Solr / > > Lucene memory-mapped file in RAM? Is there any way to measure which > > collection is using memory? we do have 350GB RAM, but we see it full with > > buffer cache, not really sure what is really using this memory. > > You would have to ask the OS which files are contained by the OS disk > cache, and it's possible that even if the information is available, that > it is very difficult to get. There is no way Solr can report this. > > Thanks, > Shawn >
Re: Secure communication between Solr and Zookeeper
Hi Jan, Could you advise more detail in step by step how we setup between Solr & Zookeeper? Do we need to put jetty-all.jar of Zookeeper 3.5.5 to the classpath of Solr? or it has been taken care? What's the configuration we need for SSL-enabled Zookeeper to communicate with Solr? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Response Time Diff between Collection with low deletes
On 5/10/2020 4:48 PM, Ganesh Sethuraman wrote: The additional info is that when we execute the test for longer (20mins) we are seeing better response time, however for a short test (5mins) and rerun the test after an hour or so we are seeing slow response times again. Note that we don't update the collection during the test or in between the test. Does this help to identify the issue? Assuming Solr is the only software that is running, most operating systems would not remove Solr data from the disk cache, so unless you have other software running on the machine, it's a little weird that performance drops back down after waiting an hour. Windows is an example of an OS that *does* proactively change data in the disk cache, and on that OS, I would not be surprised by such behavior. You haven't mentioned which OS you're running on. 3. We have designed our test to mimick reality where filter cache is not hit at all. From solr, we are seeing that there is ZERO Filter cache hit. There is about 4% query and document cache hit in prod and we are seeing no filter cache hit in both QA and PROD If you're getting zero cache hits, you should disable the cache that is getting zero hits. There is no reason to waste the memory that the cache uses, because there is no benefit. Give that, could this be some warming up related issue to keep the Solr / Lucene memory-mapped file in RAM? Is there any way to measure which collection is using memory? we do have 350GB RAM, but we see it full with buffer cache, not really sure what is really using this memory. You would have to ask the OS which files are contained by the OS disk cache, and it's possible that even if the information is available, that it is very difficult to get. There is no way Solr can report this. Thanks, Shawn
Re: Response Time Diff between Collection with low deletes
Here is a quick update based on your question, and few additional information that will help The additional info is that when we execute the test for longer (20mins) we are seeing better response time, however for a short test (5mins) and rerun the test after an hour or so we are seeing slow response times again. Note that we don't update the collection during the test or in between the test. Does this help to identify the issue? 1. The schema is exactly the same between prod and QA 2. The solr Config are exactly the same between prod and QA 3. We have designed our test to mimick reality where filter cache is not hit at all. From solr, we are seeing that there is ZERO Filter cache hit. There is about 4% query and document cache hit in prod and we are seeing no filter cache hit in both QA and PROD 4. The GC CPU usage is about 0.2% in prod and about 0.02% in QA. Note sure if that matters 5. We measure it using newrelic, and it has statistical information about the solr transaction times. Give that, could this be some warming up related issue to keep the Solr / Lucene memory-mapped file in RAM? Is there any way to measure which collection is using memory? we do have 350GB RAM, but we see it full with buffer cache, not really sure what is really using this memory. On Sun, May 10, 2020 at 10:37 AM Erick Erickson wrote: > Do not, repeat NOT expungeDelete after each deleteByQuery, it is > a very expensive operation. Perhaps after the nightly batch, but > I doubt that’ll help much anyway. > > 30% deleted docs is quite normal, and should definitely not > change the response time by a factor of 100! So there’s > some other issue in your environment. > > So the things I’d check: > 1> the schema is exactly the same. It’s vaguely possible that > the schema is just a tiny bit different. If that’s the case, you > need to delete your entire collection’s data and re-index from > scratch. You can index to a new collection and use > collection aliasing to do this seamlessly > > 2> Your solrconfig is exactly the same, especially the filterCache > cache settings. I call out filterCache because you specifically > mention filter queries, but check your other caches too. > > 3> Check your filterCache usage statistics. If you see drastically > different hit ratios in the two environments, you need to pursue that. > > 4> Once and always, check your GC performance on the two > environments. It’s a low-probability item, but you may be > just enough different in prod that GC is an issue. > > 5> Take a look at the QTimes recorded in your solr logs to insure > that the difference isn’t outside of Solr. > > While I can’t say what the exact problem is, I’m 99% sure that the number > of deleted docs isn’t the culprit. > > Best, > Erick > > > On May 9, 2020, at 6:22 PM, Ganesh Sethuraman > wrote: > > > > Hi Solr Users, > > > > We use SolrCloud 7.2.1 with 2 Solr nodes in AWS. The shard size for these > > collections does not exceed more than 5G. They have approximately 16 > shards > > with 2 replicas. We do deletes (ByQuery) as well large updates in some > of > > these Solr collections. We are seeing slower filter queries (95% > > 10secs) > > on these collections in production, same collections, and same queries in > > our lower environment with similar setup and configuration we seeing much > > better performance (<100ms). These are NRT indexes, with daily batch > > updates only. > > > > We see a difference however in the lower environment; that we don't see > > updates or deletes, we see in Segment Info for each of the Solr code > there > > are ZERO delete percentages. Could this be the reason for the faster > query > > response time in our lower environment? in our production environment, we > > are seeing about 30-32% of deletes in each core shard/replica pair. > > > > Does this segment delete % has any correlation with query response time? > We > > do delete by Query in a loop. Also updates. > > If it is so, do you suggest to try to do Optimize or expungeDelete at the > > end every day? > > Do we need to expunge delete after each delete ByQuery or do it once at > the > > end? > > > > Regards, > > Ganesh > >
Re: Secure communication between Solr and Zookeeper
Not documented yet, see https://issues.apache.org/jira/browse/SOLR-7893 Jan Høydahl > 10. mai 2020 kl. 12:48 skrev ChienHuaWang : > > Hi, > Does anyone figure how to handle the TLS between Solr and Zookeeper? > From Solr doc, I only find the Solr internal communication, no specific > detail about how zookeeper should be handled. > Appreciate any suggestion, thanks. > > > > -- > Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Which solrconfig.xml and schema file should I start with?
Choose whichever example is closest to what you want to do. Then strip it down removing everything you don’t use. Note that _default configset has schema guessing enabled which you don’t want in production. Jan Høydahl > 9. mai 2020 kl. 22:34 skrev Steven White : > > Hi everyone, > > There are multiple copies with each a bit different of the > files solrconfig.xml and the various schema files. Should I be using > what's under \solr-8.5.1\server\solr\configsets\_default\conf as my > foundation to build on? > > Thanks > > Steve
Re: solr suggest is not replicated
I wrote some Python for updating a collection config. An optional part of that is to go to each replica and start a suggester build. If your collection is sharded and you load from a dictionary, you’ll also need to add distrib=false to the queries, otherwise you’ll get suggest results from every shard. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On May 9, 2020, at 11:55 PM, morph3u...@web.de wrote: > > Hello, > > I want to use solr suggest > (https://lucene.apache.org/solr/guide/8_2/suggester.html) in a solr cloud > with replication. > > We are using solr cloud with 2 nodes. > > To build up the suggest I have to call "suggest.buildAll=true". > I do it this way: > > - Ask Zookeeper for a Solr Node, it will return for example node1 > - Execute suggest.buildAll=true at node1 > > It works exactly for node1. Node2 is not build. > I have to call suggest.buildAll=true specific also on node2 > > If I ask for suggest results, the result will be empty, if the suggest ist > not build at all nodes. > I don`t want to call all nodes manual. It should work to call only one node, > and the replication should sync the data. > > Is this a bug or missing feature? > > I found: > > https://lucene.472066.n3.nabble.com/Suggester-component-replication-td3990967.html > > https://issues.apache.org/jira/browse/SOLR-866 > > This is about spell checking, not exactly my point, but similar. > > > Best > > David
Re: Response Time Diff between Collection with low deletes
Do not, repeat NOT expungeDelete after each deleteByQuery, it is a very expensive operation. Perhaps after the nightly batch, but I doubt that’ll help much anyway. 30% deleted docs is quite normal, and should definitely not change the response time by a factor of 100! So there’s some other issue in your environment. So the things I’d check: 1> the schema is exactly the same. It’s vaguely possible that the schema is just a tiny bit different. If that’s the case, you need to delete your entire collection’s data and re-index from scratch. You can index to a new collection and use collection aliasing to do this seamlessly 2> Your solrconfig is exactly the same, especially the filterCache cache settings. I call out filterCache because you specifically mention filter queries, but check your other caches too. 3> Check your filterCache usage statistics. If you see drastically different hit ratios in the two environments, you need to pursue that. 4> Once and always, check your GC performance on the two environments. It’s a low-probability item, but you may be just enough different in prod that GC is an issue. 5> Take a look at the QTimes recorded in your solr logs to insure that the difference isn’t outside of Solr. While I can’t say what the exact problem is, I’m 99% sure that the number of deleted docs isn’t the culprit. Best, Erick > On May 9, 2020, at 6:22 PM, Ganesh Sethuraman wrote: > > Hi Solr Users, > > We use SolrCloud 7.2.1 with 2 Solr nodes in AWS. The shard size for these > collections does not exceed more than 5G. They have approximately 16 shards > with 2 replicas. We do deletes (ByQuery) as well large updates in some of > these Solr collections. We are seeing slower filter queries (95% > 10secs) > on these collections in production, same collections, and same queries in > our lower environment with similar setup and configuration we seeing much > better performance (<100ms). These are NRT indexes, with daily batch > updates only. > > We see a difference however in the lower environment; that we don't see > updates or deletes, we see in Segment Info for each of the Solr code there > are ZERO delete percentages. Could this be the reason for the faster query > response time in our lower environment? in our production environment, we > are seeing about 30-32% of deletes in each core shard/replica pair. > > Does this segment delete % has any correlation with query response time? We > do delete by Query in a loop. Also updates. > If it is so, do you suggest to try to do Optimize or expungeDelete at the > end every day? > Do we need to expunge delete after each delete ByQuery or do it once at the > end? > > Regards, > Ganesh
Re: Secure communication between Solr and Zookeeper
Hi, Does anyone figure how to handle the TLS between Solr and Zookeeper? >From Solr doc, I only find the Solr internal communication, no specific detail about how zookeeper should be handled. Appreciate any suggestion, thanks. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
solr suggest is not replicated
Hello, I want to use solr suggest (https://lucene.apache.org/solr/guide/8_2/suggester.html) in a solr cloud with replication. We are using solr cloud with 2 nodes. To build up the suggest I have to call "suggest.buildAll=true". I do it this way: - Ask Zookeeper for a Solr Node, it will return for example node1 - Execute suggest.buildAll=true at node1 It works exactly for node1. Node2 is not build. I have to call suggest.buildAll=true specific also on node2 If I ask for suggest results, the result will be empty, if the suggest ist not build at all nodes. I don`t want to call all nodes manual. It should work to call only one node, and the replication should sync the data. Is this a bug or missing feature? I found: https://lucene.472066.n3.nabble.com/Suggester-component-replication-td3990967.html https://issues.apache.org/jira/browse/SOLR-866 This is about spell checking, not exactly my point, but similar. Best David
