[jira] [Comment Edited] (SOLR-5725) Efficient facets without counts for enum method
[ https://issues.apache.org/jira/browse/SOLR-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15461656#comment-15461656 ] Mikhail Khludnev edited comment on SOLR-5725 at 9/3/16 8:38 PM: you know what??? http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3523/ {quote} junit.framework.AssertionFailedError: .facet_counts.facet_fields.t_s.null:0!=null at __randomizedtesting.SeedInfo.seed([1CFBD6041AD904D3:94AFE9DEB425692B]:0) at junit.framework.Assert.fail(Assert.java:50) at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:913) at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:932) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:607) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:574) at org.apache.solr.handler.component.DistributedFacetExistsSmallTest.checkRandomParams(DistributedFacetExistsSmallTest.java:136) NOTE: reproduce with: ant test -Dtestcase=DistributedFacetExistsSmallTest -Dtests.method=test -Dtests.seed=1CFBD6041AD904D3 -Dtests.slow=true -Dtests.locale=es-SV -Dtests.timezone=Pacific/Guadalcanal -Dtests.asserts=true -Dtests.file.encoding=US-ASCII {quote} It's perfectly reproduced with the seed. The reason is: with distrib=false {{facet.limit=0}} bypasses {{facet.missing}} processing. It's AS-WAS pre SOLR-5725 behavior. However, distrib=true calculates missing count disregards {{facet.limit=0}}. I'm going to commit test fix. And think what to do then. was (Author: mkhludnev): you know what??? http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3523/ {quote} junit.framework.AssertionFailedError: .facet_counts.facet_fields.t_s.null:0!=null at __randomizedtesting.SeedInfo.seed([1CFBD6041AD904D3:94AFE9DEB425692B]:0) at junit.framework.Assert.fail(Assert.java:50) at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:913) at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:932) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:607) at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:574) at org.apache.solr.handler.component.DistributedFacetExistsSmallTest.checkRandomParams(DistributedFacetExistsSmallTest.java:136) NOTE: reproduce with: ant test -Dtestcase=DistributedFacetExistsSmallTest -Dtests.method=test -Dtests.seed=1CFBD6041AD904D3 -Dtests.slow=true -Dtests.locale=es-SV -Dtests.timezone=Pacific/Guadalcanal -Dtests.asserts=true -Dtests.file.encoding=US-ASCII {quote} It's perfectly reproduced with the seed. The reason is: with distrib=false {{facet.limit}} bypasses even {{facet.missing}} processing. It's AS-WAS pre SOLR-5725 behavior. However, distrib=true calculates missing count disregards limit. I'm going to commit test fix. And think what to do then. > Efficient facets without counts for enum method > --- > > Key: SOLR-5725 > URL: https://issues.apache.org/jira/browse/SOLR-5725 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Alexey Kozhemiakin >Assignee: Mikhail Khludnev > Fix For: master (7.0), 6.3 > > Attachments: SOLR-5725-5x.patch, SOLR-5725-master.patch, > SOLR-5725.patch, SOLR-5725.patch, SOLR-5725.patch, SOLR-5725.patch, > SOLR-5725.patch, SOLR-5725.patch, SOLR-5725.patch > > > Shot version: > This improves performance for facet.method=enum when it's enough to know that > facet count>0, for example when you it's when you dynamically populate > filters on search form. New method checks if two bitsets intersect instead of > counting intersection size. > Long version: > We have a dataset containing hundreds of millions of records, we facet by > dozens of fields with many of facet-excludes and have relatively small number > of unique values in fields, around thousands. > Before executing search, users work with "advanced search" form, our goal is > to populate dozens of filters with values which are applicable with other > selected values, so basically this is a use case for facets with mincount=1, > but without need in actual counts. > Our performance tests showed that facet.method=enum works much better than > fc\fcs, probably due to a specific ratio of "docset"\"unique terms count". > For example average execution of query time with method fc=1500ms, fcs=2600ms > and with enum=280ms. Profiling indicated the majority time for enum was spent > on intersecting docsets. > Hers's a
[jira] [Comment Edited] (SOLR-5725) Efficient facets without counts for enum method
[ https://issues.apache.org/jira/browse/SOLR-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15442160#comment-15442160 ] Mikhail Khludnev edited comment on SOLR-5725 at 8/29/16 7:30 AM: - squashed commits, tweaked tests - especially {{TestRandomFaceting}} for {{facet.sort=counts}} with this method it's pain. I will work on it, mostly checking new tests and case w/o filterCache ({{facet.enum.cache.minDf>0}}). *Colleagues*, please confirm that you are happy to have a new {{facet.method=enumprobing}}, otherwise chime in! was (Author: mkhludnev): squashed commits, tweaked tests - especially {{TestRandomFaceting}} for {{facet.sort=counts}} with this method it's pain. I will work on it, mostly checking new tests and case w/o filterCache. *Colleagues*, please confirm that you are happy to have a new {{facet.method=enumprobing}}, otherwise chime in! > Efficient facets without counts for enum method > --- > > Key: SOLR-5725 > URL: https://issues.apache.org/jira/browse/SOLR-5725 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Alexey Kozhemiakin >Assignee: Mikhail Khludnev > Fix For: master (7.0), 6.3 > > Attachments: SOLR-5725-5x.patch, SOLR-5725-master.patch, > SOLR-5725.patch, SOLR-5725.patch > > > Shot version: > This improves performance for facet.method=enum when it's enough to know that > facet count>0, for example when you it's when you dynamically populate > filters on search form. New method checks if two bitsets intersect instead of > counting intersection size. > Long version: > We have a dataset containing hundreds of millions of records, we facet by > dozens of fields with many of facet-excludes and have relatively small number > of unique values in fields, around thousands. > Before executing search, users work with "advanced search" form, our goal is > to populate dozens of filters with values which are applicable with other > selected values, so basically this is a use case for facets with mincount=1, > but without need in actual counts. > Our performance tests showed that facet.method=enum works much better than > fc\fcs, probably due to a specific ratio of "docset"\"unique terms count". > For example average execution of query time with method fc=1500ms, fcs=2600ms > and with enum=280ms. Profiling indicated the majority time for enum was spent > on intersecting docsets. > Hers's a patch that introduces an extension to facet calculation for > method=enum. Basically it uses docSetA.intersects(docSetB) instead of > docSetA. intersectionSize (docSetB). > As a result we were able to reduce our average query time from 280ms to 60ms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-5725) Efficient facets without counts for enum method
[ https://issues.apache.org/jira/browse/SOLR-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409262#comment-15409262 ] Mikhail Khludnev edited comment on SOLR-5725 at 8/8/16 3:34 PM: Hello, I skim through the patch. Generally it's ok, here are some notes: * as far the proposed method is an optimization of {{enum}}, it's reasonable to name it starting with it, eg {{enumprobing}}; * it's preferable to avoid code diverging, instead of introducing {{SimpleFacets.getFacetTermIntersection()}} it's better to introduce conditional code branches (yep, this is sad) into {{getFacetTermEnumCounts()}}. * implementation should be consistent, thus it's worth to implement the similar idea in {{else}} branch of {{if (df >= minDfFilterCache) }} * it should be resilient for other parameter, eg it should throw an exception if combined with {{mincount=2}} or more * when merge facet from shards we need to cap them by 1, otherwise it will flip from 1 to number of shards * the test case is beefy enough, but it's also necessary to include new method into existing regression tests with amending asserts, yep. it's pain. was (Author: mkhludnev): Hello, I skim through the patch. Generally it's ok, here are some notes: * as far the proposed method is an optimization of {{enum}}, it's reasonable to name it starting with it, eg {{enumprobing}}; * it's preferable to avoid code diverging, instead of introducing {{SimpleFacets.getFacetTermIntersection()}} it's better to introduce conditional code branches (yep, this is sad) into {{getFacetTermEnumCounts()}}. * implementation should be consistent, thus it's worth to implement the similar idea in {{else}} branch of {{if (df >= minDfFilterCache) }} * it should be resilient for other parameter, eg it should throw an exception if combined with {{mincount=2}} or more * the test case is beefy enough, but it's also necessary to include new method into existing regression tests with amending asserts, yep. it's pain. > Efficient facets without counts for enum method > --- > > Key: SOLR-5725 > URL: https://issues.apache.org/jira/browse/SOLR-5725 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Alexey Kozhemiakin >Assignee: Shalin Shekhar Mangar > Fix For: 6.0 > > Attachments: SOLR-5725-5x.patch, SOLR-5725.patch > > > Shot version: > This improves performance for facet.method=enum when it's enough to know that > facet count>0, for example when you it's when you dynamically populate > filters on search form. New method checks if two bitsets intersect instead of > counting intersection size. > Long version: > We have a dataset containing hundreds of millions of records, we facet by > dozens of fields with many of facet-excludes and have relatively small number > of unique values in fields, around thousands. > Before executing search, users work with "advanced search" form, our goal is > to populate dozens of filters with values which are applicable with other > selected values, so basically this is a use case for facets with mincount=1, > but without need in actual counts. > Our performance tests showed that facet.method=enum works much better than > fc\fcs, probably due to a specific ratio of "docset"\"unique terms count". > For example average execution of query time with method fc=1500ms, fcs=2600ms > and with enum=280ms. Profiling indicated the majority time for enum was spent > on intersecting docsets. > Hers's a patch that introduces an extension to facet calculation for > method=enum. Basically it uses docSetA.intersects(docSetB) instead of > docSetA. intersectionSize (docSetB). > As a result we were able to reduce our average query time from 280ms to 60ms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org