[ https://issues.apache.org/jira/browse/SOLR-12954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672570#comment-16672570 ]
Hoss Man commented on SOLR-12954: --------------------------------- The code that generates shard requests has an optimization for dealing with the per-shard mincount... {code:java} // per-shard mincount & overrequest if ( FacetParams.FACET_SORT_INDEX.equals(sort) && 1 < requestedMinCount && 0 < requestedLimit) { // We can divide the mincount by num shards rounded up, because unless // a single shard has at least that many it can't compete... shardMinCount = (int) Math.ceil((double) requestedMinCount / rb.slices.length); {code} ...but this optimization depends on *all* candidate terms being refined (when using sort=index) – there is evidently some situation where this is not happening properly. ---- With {{cf445ba54998710466a7c6cb489d3162d20d127a}} checked out, the following seed demonstrates the problem... {noformat} [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCloudPivotFacet -Dtests.method=test -Dtests.seed=8A3362235855F8A8 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sr-Latn -Dtests.timezone=America/Guatemala -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] FAILURE 38.9s | TestCloudPivotFacet.test <<< [junit4] > Throwable #1: java.lang.AssertionError: {main(facet=true&facet.pivot={!stats%3Dst3}pivot_dt,pivot_b1&facet.pivot={!stats%3Dst2}pivot_z_s1,pivot_i1,pivot_d1&facet.limit=12&facet.pivot.mincount=4&facet.sort=index&facet.overrequest.count=2&facet.overrequest.ratio=1.1145182),extra(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index)} ==> pivot_dt,pivot_b1: {params(rows=0),defaults({main({main(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index),extra(fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z)}),extra(fq={!term+f%3Dpivot_b1}false)})} expected:<5> but was:<6> [junit4] > at __randomizedtesting.SeedInfo.seed([8A3362235855F8A8:2675DF9F6A99550]:0) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:292) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.test(TestCloudPivotFacet.java:239) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985) [junit4] > at java.lang.Thread.run(Thread.java:748) [junit4] > Caused by: java.lang.AssertionError: pivot_dt,pivot_b1: {params(rows=0),defaults({main({main(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index),extra(fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z)}),extra(fq={!term+f%3Dpivot_b1}false)})} expected:<5> but was:<6> [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertNumFound(TestCloudPivotFacet.java:690) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertPivotData(TestCloudPivotFacet.java:346) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:313) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:324) [junit4] > at org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:282) [junit4] > ... 42 more {noformat} The pivot query being tests... {noformat} [junit4] 2> 35268 INFO (TEST-TestCloudPivotFacet.test-seed#[8A3362235855F8A8]) [ ] o.a.s.c.TestCloudPivotFacet Doing full run: {main(facet=true&facet.pivot={!stats%3Dst3}pivot_dt,pivot_b1&facet.pivot={!stats%3Dst2}pivot_z_s1,pivot_i1,pivot_d1&facet.limit=12&facet.pivot.mincount=4&facet.sort=index&facet.overrequest.count=2&facet.overrequest.ratio=1.1145182),extra(rows=0&q=id:[*+TO+1077]&fq=id:[*+TO+204]&stats=true&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1&_test_min=4&_test_sort=index)} {noformat} The bucket that fails is from the {{pivot_dt,pivot_b1}} pivot for the bucket {{2010-10-31T10:31:15Z , false}} The test verification queries show... {noformat} ### verification queries of 2010-10-31T10:31:15Z (shard1=2 shard2=7)... [junit4] 2> 36749 INFO (qtp1638186575-84) [n:127.0.0.1:51294__ c:collection1 s:shard2 r:core_node42 x:collection1_shard2_replica_n41] o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp=/_ path=/select params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&shard.url=http://127.0.0.1:51294/_/collection1_shard2_replica_n41/|http://127.0.0.1:59609/_/collection1_shard2_replica_n45/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572502&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=7 status=0 QTime=1 [junit4] 2> 36749 INFO (qtp1007717333-122) [n:127.0.0.1:55141__ c:collection1 s:shard1 r:core_node44 x:collection1_shard1_replica_n43] o.a.s.c.S.Request [collection1_shard1_replica_n43] webapp=/_ path=/select params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&shard.url=http://127.0.0.1:55141/_/collection1_shard1_replica_n43/|http://127.0.0.1:38554/_/collection1_shard1_replica_n47/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572502&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=2 status=0 QTime=1 [junit4] 2> 36750 INFO (qtp73311073-158) [n:127.0.0.1:59609__ c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.S.Request [collection1_shard2_replica_n45] webapp=/_ path=/select params={q=id:[*+TO+1077]&stats=true&_test_sort=index&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&_test_min=4&rows=0&wt=javabin&version=2&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=9 status=0 QTime=3 ### verification queries of 2010-10-31T10:31:15Z + false (shard1=1, shard2=5)... [junit4] 2> 36754 INFO (qtp1638186575-84) [n:127.0.0.1:51294__ c:collection1 s:shard2 r:core_node42 x:collection1_shard2_replica_n41] o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp=/_ path=/select params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&shard.url=http://127.0.0.1:51294/_/collection1_shard2_replica_n41/|http://127.0.0.1:59609/_/collection1_shard2_replica_n45/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572507&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=5 status=0 QTime=1 [junit4] 2> 36756 INFO (qtp1007717333-124) [n:127.0.0.1:55141__ c:collection1 s:shard1 r:core_node44 x:collection1_shard1_replica_n43] o.a.s.c.S.Request [collection1_shard1_replica_n43] webapp=/_ path=/select params={df=text&distrib=false&_test_sort=index&fl=id&fl=score&shards.purpose=516&start=0&fsv=true&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&shard.url=http://127.0.0.1:55141/_/collection1_shard1_replica_n43/|http://127.0.0.1:38554/_/collection1_shard1_replica_n47/&rows=0&version=2&q=id:[*+TO+1077]&stats=true&NOW=1541118572507&isShard=true&wt=javabin&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=1 status=0 QTime=1 [junit4] 2> 36758 INFO (qtp973539008-202) [n:127.0.0.1:38554__ c:collection1 s:shard1 r:core_node48 x:collection1_shard1_replica_n47] o.a.s.c.S.Request [collection1_shard1_replica_n47] webapp=/_ path=/select params={q=id:[*+TO+1077]&stats=true&_test_sort=index&fq=id:[*+TO+204]&fq={!term+f%3Dpivot_dt}2010-10-31T10:31:15Z&fq={!term+f%3Dpivot_b1}false&_test_min=4&rows=0&wt=javabin&version=2&stats.field={!key%3Dsk1+tag%3Dst1,st2}pivot_tdt&stats.field={!key%3Dsk2+tag%3Dst2,st3}dense_pivot_ti1&stats.field={!key%3Dsk3+tag%3Dst3,st4}pivot_tdt1} hits=6 status=0 QTime=6 {noformat} Based on the computed shard min count used, and the per-shard numFounds for the verification queries, this bucket would not have been returned by shard1 during the initial query, but would have been returned by shard2 – yet there is nothing in the logs showing any attempt at refining it on shard2, and the resulting assertion failure is consistent with the one doc in this bucket on shard1 not being counted. ---- Forcing the above mentioned code to use {{shardMinCount=1}} causes this seed to pass, and that may be the best course of action if we can't figure out why the buckets aren't getting refined properly. > facet.pivot refinement bug when facet.sort=index and mincount > 2*numShards > --------------------------------------------------------------------------- > > Key: SOLR-12954 > URL: https://issues.apache.org/jira/browse/SOLR-12954 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Hoss Man > Priority: Major > > While testing out SOLR-7804 i discovered a failure in TestCloudPivotFacet > that indicates a problem with the refinement of (nested?) pivot facets when > {{facet.sort=index}} and {{facet.pivot.mincount > 2*numShards}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org