Hello,
I have a brand new installation of Solr 5.4.1 and I am running into a
strange problem with one of my collections. Collection *products* has 5
shards and replication factor of two. Both replicas are up and show green
status on the Cloud page in the UI.
When I run a default search on the query page (q=*:*) I always get a
different numFound although there is no active indexing and everything is
committed. I checked the logs and it looks like every time it runs a
search, it is sent to different shards. Below, search1 went to shard 5, 2
and 4, search2 went to shard 5, 3, 1 and search 3 went to shard 3, 4, 1, 5.
To confirm this, I ran a &distrib=false query on shard 5 and got 8,928,379
items, 8,917,318 for shard 2, and 9,005,295 for shard 4. The results from
shard 2 distrib=false query did not match the results that were in the
distributed query (from the logs). The query returned 8917318. Here is the
log entry for the query.
214467874 INFO (qtp1013423070-21019) [c:products s:shard2 r:core_node7
x:products_shard2_replica2] o.a.s.c.S.Request [products_shard2_replica2]
webapp=/solr path=/select
params={q=*:*&distrib=false&indent=true&wt=json&_=1458056340020}
hits=8917318 status=0 QTime=0
Here are the logs from other queries.
Search 1 - numFound 18309764
213941984 INFO (qtp1013423070-21046) [c:products s:shard5 r:core_node4
x:products_shard5_replica2] o.a.s.c.S.Request [products_shard5_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.211:9000/solr/products_shard5_replica2/|http://192.168.1.212:9000/solr/products_shard5_replica1/&rows=10&version=2&q=*:*&NOW=1458055805759&isShard=true&wt=javabin&_=1458055814096}
hits=8928379 status=0 QTime=3
213941985 INFO (qtp1013423070-21028) [c:products s:shard4 r:core_node6
x:products_shard4_replica2] o.a.s.c.S.Request [products_shard4_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard4_replica1/|http://192.168.1.211:9000/solr/products_shard4_replica2/&rows=10&version=2&q=*:*&NOW=1458055805759&isShard=true&wt=javabin&_=1458055814096}
hits=9005295 status=0 QTime=3
213942045 INFO (qtp1013423070-21042) [c:products s:shard2 r:core_node7
x:products_shard2_replica2] o.a.s.c.S.Request [products_shard2_replica2]
webapp=/solr path=/select
params={q=*:*&indent=true&wt=json&_=1458055814096} hits=18309764 status=0
QTime=81
Search 2 - numFound 27072144
213995779 INFO (qtp1013423070-21046) [c:products s:shard5 r:core_node4
x:products_shard5_replica2] o.a.s.c.S.Request [products_shard5_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.211:9000/solr/products_shard5_replica2/|http://192.168.1.212:9000/solr/products_shard5_replica1/&rows=10&version=2&q=*:*&NOW=1458055859563&isShard=true&wt=javabin&_=1458055867894}
hits=8928379 status=0 QTime=1
213995781 INFO (qtp1013423070-20985) [c:products s:shard3 r:core_node10
x:products_shard3_replica2] o.a.s.c.S.Request [products_shard3_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard3_replica1/|http://192.168.1.211:9000/solr/products_shard3_replica2/&rows=10&version=2&q=*:*&NOW=1458055859563&isShard=true&wt=javabin&_=1458055867894}
hits=8980542 status=0 QTime=3
213995785 INFO (qtp1013423070-21042) [c:products s:shard1 r:core_node9
x:products_shard1_replica2] o.a.s.c.S.Request [products_shard1_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard1_replica1/|http://192.168.1.211:9000/solr/products_shard1_replica2/&rows=10&version=2&q=*:*&NOW=1458055859563&isShard=true&wt=javabin&_=1458055867894}
hits=8914801 status=0 QTime=3
213995798 INFO (qtp1013423070-21028) [c:products s:shard2 r:core_node7
x:products_shard2_replica2] o.a.s.c.S.Request [products_shard2_replica2]
webapp=/solr path=/select
params={q=*:*&indent=true&wt=json&_=1458055867894} hits=27072144 status=0
QTime=30
Search 3 - numFound 35953734
214022457 INFO (qtp1013423070-21019) [c:products s:shard3 r:core_node10
x:products_shard3_replica2] o.a.s.c.S.Request [products_shard3_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard3_replica1/|http://192.168.1.211:9000/solr/products_shard3_replica2/&rows=10&version=2&q=*:*&NOW=1458055886247&isShard=true&wt=javabin&_=1458055894580}
hits=8980542 status=0 QTime=0
214022458 INFO (qtp1013423070-21036) [c:products s:shard4 r:core_node6
x:products_shard4_replica2] o.a.s.c.S.Request [products_shard4_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard4_replica1/|http://192.168.1.211:9000/solr/products_shard4_replica2/&rows=10&version=2&q=*:*&NOW=1458055886247&isShard=true&wt=javabin&_=1458055894580}
hits=9005295 status=0 QTime=1
214022459 INFO (qtp1013423070-21046) [c:products s:shard1 r:core_node9
x:products_shard1_replica2] o.a.s.c.S.Request [products_shard1_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.212:9000/solr/products_shard1_replica1/|http://192.168.1.211:9000/solr/products_shard1_replica2/&rows=10&version=2&q=*:*&NOW=1458055886247&isShard=true&wt=javabin&_=1458055894580}
hits=8914801 status=0 QTime=0
214022460 INFO (qtp1013423070-20985) [c:products s:shard5 r:core_node4
x:products_shard5_replica2] o.a.s.c.S.Request [products_shard5_replica2]
webapp=/solr path=/select
params={df=text&distrib=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
http://192.168.1.211:9000/solr/products_shard5_replica2/|http://192.168.1.212:9000/solr/products_shard5_replica1/&rows=10&version=2&q=*:*&NOW=1458055886247&isShard=true&wt=javabin&_=1458055894580}
hits=8928379 status=0 QTime=1
214022471 INFO (qtp1013423070-21043) [c:products s:shard2 r:core_node7
x:products_shard2_replica2] o.a.s.c.S.Request [products_shard2_replica2]
webapp=/solr path=/select
params={q=*:*&indent=true&wt=json&_=1458055894580} hits=35953734 status=0
QTime=20
I would really like to avoid re-indexing if possible. Can someone provide a
bit of info on what is happening?
