Re: Facet Query performance
On 7/8/2019 12:00 PM, Midas A wrote: Number of Docs :50+ docs Index Size: 300 GB RAM: 256 GB JVM: 32 GB Half a million documents producing an index size of 300GB suggests *very* large documents. That typically produces an index with fields that have very high cardinality, due to text tokenization. Is Solr the only thing running on this machine, or does it have other memory-hungry software running on it? The screenshot described at the following URL may provide more insight. It will be important to get the sort correct. If the columns have been customized to show information other than the examples, it may need to be adjusted: https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_help_on_a_memory.2Fperformance_issue Assuming that Solr is the only thing on the machine, then it means you have about 224 GB of memory available to cache your index data, which is at least 300GB. Normally I would think being able to cache two thirds of the index should be enough for good performance, but it's always possible that there is something about your setup that means you don't have enough memory. Are you sure that you need a 32GB heap? Half a million documents should NOT require anywhere near that much heap. Cardinality: cat=44 rol=1005 ind=504 cl=2000 These cardinality values are VERY low. If you are certain about those numbers, it is not likely that these fields are significant contributors to query time, either with or without docValues. How did you obtain those numbers? Those are not the only fields referenced in your query. I also see these: hemp cEmp pEmp is_udis id is_resume upt_date country exp ctc contents currdesig predesig lng ttl kw_sql kw_it QTime: 2988 ms Three seconds for a query with so many facets is something I would probably be pretty happy to get. Our 35% queries takes more than 10 sec. I have no idea what this sentence means. Please suggest the ways to improve response time . Attached queries and schema.xml and solrconfig.xml 1. Is there any other ways to rewrite queries that improve our query performance .? With the information available, the only suggestion I have currently is to replace "q=*" with "q=*:*" -- assuming that the intent is to match all documents with the main query. According to what you attached (which I am very surprised to see -- attachments usually don't make it to the list), your df parameter is "ttl" ... a field that is heavily tokenized. That means that the cardinality of the ttl field is probably VERY high, which would make the wildcard query VERY slow. 2. can we see the DocValues cache in plugin/ stats->cache-> section on solr UI panel ? The admin UI only shows Solr caches. If Lucene even has a docValues cache (and I do not know whether it does), it will not be available in Solr's statistics. I am unaware of any cache in Solr for docValues. The entire point of docValues is to avoid the need to generate and cache large amounts of data, so I suspect there is not going to be anything available in this regard. Thanks, Shawn
Re: Facet Query performance
On 7/8/2019 3:08 AM, Midas A wrote: I have enabled docvalues on facet field but query is still taking time. How i can improve the Query time . docValues="true" multiValued="true" termVectors="true" /> *Query: * There's very little information here -- only a single field definition and the query URL. No information about how many documents, what sort of cardinality there is in the fields being used in the query, no information about memory and settings, etc. You haven't even told us how long the query takes. Your main query is a single * wildcard. A wildcard query is typically quite slow. If you are aiming for all documents, change that to q=*:* instead -- this is special syntax that the query parser understands, and is normally executed very quickly. When a field has DocValues defined, it will automatically be used for field-based sorting, field-based facets, and field-based grouping. DocValues should not be relied on for queries, because indexed data is far faster for that usage. Queries *can* be done with docValues, but it would be VERY slow. Solr will avoid that usage if it can. I'm reasonably certain that docValues will NOT be used for facet.query as long as the field is indexed. You do have three-field based facets -- using the facet.field parameter. If docValues was present on cat for ALL of the indexing that has happened, then they will work for that field, but you have not told us whether rol and pref have them defined. You have a lot of faceting in this query. That can cause things to be slow. Thanks, Shawn
Re: Facet Query performance
Hi How i can know whether DocValues are getting used or not ? Please help me here . On Mon, Jul 8, 2019 at 2:38 PM Midas A wrote: > Hi , > > I have enabled docvalues on facet field but query is still taking time. > > How i can improve the Query time . > docValues="true" multiValued="true" termVectors="true" /> > > *Query: * > http://X.X.X.X: > /solr/search/select?df=ttl=0=true=id,upt=1=true=1=OR=NOT+hemp:(%22xgidx29760%22+%22xmwxmonster%22+%22xmwxmonsterindia%22+%22xmwxcom%22+%22xswxmonster+com%22+%22xswxmonster%22+%22xswxmonsterindia+com%22+%22xswxmonsterindia%22)=NOT+cEmp:(% > 22nomster.com%22+OR+%22utyu%22)=NOT+pEmp:(%22nomster.com > %22+OR+%22utyu%22)=ind:(5)=NOT+is_udis:2=NOT+id:(92197+OR+240613+OR+249717+OR+1007148+OR+2500513+OR+2534675+OR+2813498+OR+9401682)=true=0=is_resume:0^-1000=upt_date:[*+TO+NOW/DAY-36MONTHS]^2=upt_date:[NOW/DAY-36MONTHS+TO+NOW/DAY-24MONTHS]^3=upt_date:[NOW/DAY-24MONTHS+TO+NOW/DAY-12MONTHS]^4=upt_date:[NOW/DAY-12MONTHS+TO+NOW/DAY-9MONTHS]^5=upt_date:[NOW/DAY-9MONTHS+TO+NOW/DAY-6MONTHS]^10=upt_date:[NOW/DAY-6MONTHS+TO+NOW/DAY-3MONTHS]^15=upt_date:[NOW/DAY-3MONTHS+TO+*]^20=NOT+country:isoin^-10=exp:[+10+TO+11+]=exp:[+11+TO+13+]=exp:[+13+TO+15+]=exp:[+15+TO+17+]=exp:[+17+TO+20+]=exp:[+20+TO+25+]=exp:[+25+TO+109+]=ctc:[+100+TO+101+]=ctc:[+101+TO+101.5+]=ctc:[+101.5+TO+102+]=ctc:[+102+TO+103+]=ctc:[+103+TO+104+]=ctc:[+104+TO+105+]=ctc:[+105+TO+107.5+]=ctc:[+107.5+TO+110+]=ctc:[+110+TO+115+]=ctc:[+115+TO+10100+]=0=contents^0.05+currdesig^1.5+predesig^1.5+lng^2+ttl+kw_skl+kw_it=1=false=ttl,kw_skl,kw_it,contents=json=1=0=ind=cat=rol=cl=pref=timing=/resumesearch=1=0=40=2=*=10=id==1=id=id=true=false > >
Re: facet query performance
On Mon, 2016-11-14 at 11:36 +0530, Midas A wrote: > How to improve facet query performance 1) Don't shard unless you really need to. Replicas are fine. 2) If the problem is the first facet call, then enable DocValues and re-index. 3) Keep facet.limit <= 100, especially if you shard. and most important 4) Describe in detail what you have, how you facet and what you expect. Give us something to work. - Toke Eskildsen, State and University Library, Denmark