Thanks for the suggestion. At this time I wont be able to change any code in the API ...my options are limited to changing things at the solr level. Any suggestions regarding solr settings in config or schema changes are something in my control.
On Fri, May 27, 2016 at 7:03 AM, Ahmet Arslan <iori...@yahoo.com> wrote: > Hi Jay, > > Please separate the clauses. Feed one of them to the main q parameter with > content score operator =^ since you are sorting on a structured field(e.g. > date) > > q:fieldB:(123 OR 456)^=1.0 > &fq=dt1:[date1 TO *] > &fq=dt2:[* TO NOW/DAY+1] > &fq=fieldA:abc > &sort=dt1 asc,field2 asc, fieldC desc > > Play with the caches. > Also consider disabling caching, and/or supplying execution order for the > filer queries. > Please see : > https://lucidworks.com/blog/2012/02/10/advanced-filter-caching-in-solr/ > > Ahmet > > > > On Friday, May 27, 2016 4:01 PM, Jay Potharaju <jspothar...@gmail.com> > wrote: > I updated almost 1/3 of the data and ran my queries with new columns as > mentioned earlier. The query returns data in almost half the time as > compared to before. > I am thinking that if I update all the columns there would not be much > difference in query response time. > > <field name="dt1" type="date" indexed="true" stored="true"/> <field name=" > dt1_new" type="tdate" indexed="true" stored="true"/> > > <field name="dt2" type="date" indexed="true" stored="true"/> <field name=" > dt2_new" type="tdate" indexed="true" stored="true" docValues="true" > default=""/> > > Are there any suggestions on how handle filtering/querying/sorting on high > cardinality date fields? > > Index size: 30Million > Solr: 4.3.1 > > Thanks > > On Thu, May 26, 2016 at 6:04 AM, Jay Potharaju <jspothar...@gmail.com> > wrote: > > > Hi, > > Thanks for the feedback. The queries I run are very basic filter queries > > with some sorting. > > > > q:*:*&fq=(dt1:[date1 TO *] && dt2:[* TO NOW/DAY+1]) && fieldA:abc && > > fieldB:(123 OR 456)&sort=dt1 asc,field2 asc, fieldC desc > > > > I noticed that the date fields(dt1,dt2) are using date instead of tdate > > fields & there are no docValues set on any of the fields used for > sorting. > > > > In order to fix this I plan to add a new field using tdate & docvalues > > where required to the schema & update the new columns only for documents > > that have fieldA set to abc. Once the fields are updated query on the new > > fields to measure query performance . > > > > > > - Would the new added fields be used effectively by the solr index > > when querying & filtering? What I am not sure is whether only > populating > > small number of documents(fieldA:abc) that are used for the above > query > > provide performance benefits. > > - Would there be a performance penalty because majority of the > > documents(!fieldA:abc) dont have values in the new columns? > > > > Thanks > > > > On Wed, May 25, 2016 at 8:40 PM, Jay Potharaju <jspothar...@gmail.com> > > wrote: > > > >> Any links that illustrate and talk about solr internals and how > >> indexing/querying works would be a great help. > >> Thanks > >> Jay > >> > >> On Wed, May 25, 2016 at 6:30 PM, Jay Potharaju <jspothar...@gmail.com> > >> wrote: > >> > >>> Hi, > >>> Thanks for the feedback. The queries I run are very basic filter > queries > >>> with some sorting. > >>> > >>> q:*:*&fq=(dt1:[date1 TO *] && dt2:[* TO NOW/DAY+1]) && fieldA:abc && > >>> fieldB:(123 OR 456)&sort=dt1 asc,field2 asc, fieldC desc > >>> > >>> I noticed that the date fields(dt1,dt2) are using date instead of tdate > >>> fields & there are no docValues set on any of the fields used for > sorting. > >>> > >>> In order to fix this I plan to add a new field using tdate & docvalues > >>> where required to the schema & update the new columns only for > documents > >>> that have fieldA set to abc. Once the fields are updated query on the > new > >>> fields to measure query performance . > >>> > >>> > >>> - Would the new added fields be used effectively by the solr index > >>> when querying & filtering? What I am not sure is whether only > populating > >>> small number of documents(fieldA:abc) that are used for the above > query > >>> provide performance benefits. > >>> - Would there be a performance penalty because majority of the > >>> documents(!fieldA:abc) dont have values in the new columns? > >>> > >>> > >>> Thanks > >>> Jay > >>> > >>> On Tue, May 24, 2016 at 8:06 PM, Erick Erickson < > erickerick...@gmail.com > >>> > wrote: > >>> > >>>> Try adding debug=timing, that'll give you an idea of what component is > >>>> taking all the time. > >>>> From there, it's "more art than science". > >>>> > >>>> But you haven't given us much to go on. What is the query? Are you > >>>> grouping? > >>>> Faceting on high-cardinality fields? Returning 10,000 rows? > >>>> > >>>> Best, > >>>> Erick > >>>> > >>>> On Tue, May 24, 2016 at 4:52 PM, Ahmet Arslan > <iori...@yahoo.com.invalid> > >>>> wrote: > >>>> > > >>>> > > >>>> > Hi, > >>>> > > >>>> > Is it QueryComponent taking time? > >>>> > Ot other components? > >>>> > > >>>> > Also make sure there is plenty of RAM for OS cache. > >>>> > > >>>> > Ahmet > >>>> > > >>>> > On Wednesday, May 25, 2016 1:47 AM, Jay Potharaju < > >>>> jspothar...@gmail.com> wrote: > >>>> > > >>>> > > >>>> > > >>>> > Hi, > >>>> > I am trying to debug solr performance problems on an old version of > >>>> solr, > >>>> > 4.3.1. > >>>> > The queries are taking really long -in the range of 2-5 seconds!!. > >>>> > Running filter query with only one condition also takes about a > >>>> second. > >>>> > > >>>> > There is memory available on the box for solr to use. I have been > >>>> looking > >>>> > at the following link but was looking for some more reference that > >>>> would > >>>> > tell me why a particular query is slow. > >>>> > > >>>> > https://wiki.apache.org/solr/SolrPerformanceProblems > >>>> > > >>>> > Solr version:4.3.1 > >>>> > Index size:128 GB > >>>> > Heap:65 GB > >>>> > Index size:75 GB > >>>> > Memory usage:70 GB > >>>> > > >>>> > Even though there is available memory is high all is not being used > >>>> ..i > >>>> > would expect the complete index to be in memory but it doesnt look > >>>> like it > >>>> > is. Any recommendations ?? > > >>>> > > >>>> > -- > >>>> > Thanks > >>>> > Jay > >>>> > >>> > >>> > >>> > >>> -- > >>> Thanks > >>> Jay Potharaju > >>> > >>> > >> > >> > >> > >> -- > >> Thanks > >> Jay Potharaju > >> > >> > > > > > > > > -- > > Thanks > > Jay Potharaju > > > > > > > > -- > Thanks > Jay Potharaju > -- Thanks Jay Potharaju
