[
https://issues.apache.org/jira/browse/SOLR-5046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189986#comment-14189986
]
Timo Schmidt edited comment on SOLR-5046 at 10/30/14 12:18 PM:
---------------------------------------------------------------
Hello together,
yesterday i could do dome debugging and just want to share the outcome, ask
some questions and hopefully get a step further.
For me it is not clear if this is a problem insight of lucene or solr
The exceptions is thrown in org.apache.lucene.search.TopDocs on line 143
(branch_4x):
if (fd.fields == null) {
throw new IllegalArgumentException("shard " + shardIDX + " did
not set sort field values (FieldDoc.fields is null); you must pass
fillFields=true to IndexSearcher.search on each shard");
}
From reading the code in "public static TopDocs merge(Sort sort, int
start, int size, TopDocs[] shardHits)“ there are two options:
a) sort is null => in this case the ScoreMergeSortQueue is used
b) sort is defined => the mergedSortQueue will be used (Even
when sort.fields has no Sorting fields).
By default we use Sort.RELEVANCE
What i don’t understand here is, why not just pass „null“ to merge by
score? Or is there a difference between Score and Relevance?
When i change the merge method the following:
if (sort == null) {
queue = new ScoreMergeSortQueue(shardHits);
} else {
queue = new MergeSortQueue(sort, shardHits);
}
to
if (sort == null || sort.equals(Sort.RELEVANCE)) {
queue = new ScoreMergeSortQueue(shardHits);
} else {
queue = new MergeSortQueue(sort, shardHits);
}
the case without any sorting works fine from the user level (but, there
is one broken testcase in lucene)
By reading the exception message „you must pass fillFields=true to“ i checked
the code and found the following in
„org.apache.solr.search.grouping.distributed.shardresultserializer.TopGroupsResultTransformer“:
In the „transformToNative“ method „document.get("sortValues“)“ is used to
fill the sortValues for the document.
For some reason this seems to be empty.
From where do these sortValues come? And when could they be empty? And
what should happen when they are empty?
This serializer (TopGroupsResultTransformer.transformToNative) is used in
„TopGroupsShardResponseProcessor.process“.
Finally „ org.apache.lucene.search.TopDocs.merge“ is used to merge the
topDocs
TopDocs mergedTopDocs = TopDocs.merge(sortWithinGroup, topN,
topDocs.toArray(new TopDocs[topDocs.size()]));
As mentioned in „1“ the default Sorting is sorting byScore but the
topDocs seem to have no „sortFields"
Is it an options to pass here „null“ as sorting when we want to sort by
relevance?
Conclusion:
I think there could be two potential reasons:
We should use/change the lucene api for merging with no sorting / sorting by
relevance?
We should find out why „sortFields“ is empty or if the is a call where search
is used without „fillFields“?
Any ideas?
If i could help somewhere just ask and it would be nice to get some insights
from somebody who is more inside the lucene and solr core.
Cheers
timo
was (Author: timo.schmidt):
Hello together,
yesterday i could do dome debugging and just want to share the outcome, ask
some questions and hopefully get a step further.
For me it is not clear if this is a problem insight of lucene or solr
The exceptions is thrown in org.apache.lucene.search.TopDocs on line 143
(branch_4x):
if (fd.fields == null) {
throw new IllegalArgumentException("shard " + shardIDX + " did
not set sort field values (FieldDoc.fields is null); you must pass
fillFields=true to IndexSearcher.search on each shard");
}
From reading the code in "public static TopDocs merge(Sort sort, int
start, int size, TopDocs[] shardHits)“ there are two options:
a) sort is null => in this case the ScoreMergeSortQueue is used
b) sort is defined => the mergedSortQueue will be used (Even
when sort.fields has no Sorting fields).
By default we use Sort.RELEVANCE
What i don’t understand here is, why not just pass „null“ to merge by
score? Or is there a difference between Score and Relevance?
When i change the merge method the following:
if (sort == null) {
queue = new ScoreMergeSortQueue(shardHits);
} else {
queue = new MergeSortQueue(sort, shardHits);
}
to
if (sort == null || sort.equals(Sort.RELEVANCE)) {
queue = new ScoreMergeSortQueue(shardHits);
} else {
queue = new MergeSortQueue(sort, shardHits);
}
the case without any sorting works fine from the user level (but, there
is one broken testcase in lucene)
By reading the exception message „you must pass fillFields=true to“ i checked
the code and found the following in
„org.apache.solr.search.grouping.distributed.shardresultserializer.TopGroupsResultTransformer“:
In the „transformToNative“ method „document.get("sortValues“)“ is used to
fill the sortValues for the document.
For some reason this seems to be empty.
From where do these sortValues come? And when could they be empty? And
what should happen when they are empty?
This serializer (TopGroupsResultTransformer.transformToNative) is used in
„TopGroupsShardResponseProcessor.process“.
Finally „ org.apache.lucene.search.TopDocs.merge“ is used to merge the
topDocs
TopDocs mergedTopDocs = TopDocs.merge(sortWithinGroup, topN,
topDocs.toArray(new TopDocs[topDocs.size()]));
As mentioned in „1“ the default Sorting is sorting byScore but the
topDocs seem to have no „sortFields"
Is it an options to pass here „null“ as sorting when we want to sort by
relevance?
Conclusion:
I think there could be two potential reasons:
We should use/change the lucene api for merging with no sorting / sorting by
relevance?
We should find out why „sortFields“ is empty or if the is a call where search
is used without „fillFields“?
Any ideas?
If i could help somewhere just ask and it would be nice to get some insights
from somebody who is more inside the lucene and solr core.
Cheers
timo
> IllegalArgumentException using distributed group.query when one shard does
> not match any docs
> ---------------------------------------------------------------------------------------------
>
> Key: SOLR-5046
> URL: https://issues.apache.org/jira/browse/SOLR-5046
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.3
> Reporter: Hoss Man
>
> [Evgeny Salnikov noted this problem on the mailing
> list|http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201307.mbox/%3CCADz7Cx6PbMxExhb8gsCu9%3DP6nphJd2fYayov_%3D%3D%2Bo1sEXswWLw%40mail.gmail.com%3E],
> although the initial report was somewhat convoluted by suspicious
> description of adding shards after the fact.
> Steps to reproduce using 4.3.1 example...
> * startup a 2 node SolrCloud cluster following the "Example A" description on
> the SolrCloud wiki...
> ** cp example example2
> ** cd example && java -Dbootstrap_confdir=./solr/collection1/conf
> -Dcollection.configName=myconf -DzkRun -DnumShards=2 -jar start.jar
> ** cd example2 && java -Djetty.port=7574 -DzkHost=localhost:9983 -jar
> start.jar
> * index exactly one doc (to ensure that subsequent distributed queries get
> results from only one node)
> ** java -jar post.jar utf8-example.xml
> * execute a request using group.query
> ** http://localhost:7574/solr/select?q=*:*&group=true&group.query=cat:software
> stack trace...
> {noformat}
> 166500 [qtp2092063645-19] ERROR org.apache.solr.servlet.SolrDispatchFilter –
> null:java.lang.IllegalArgumentException: shard 1 did not set sort field
> values (FieldDoc.fields is null); you must pass fillFields=true to
> IndexSearcher.search on each shard
> at
> org.apache.lucene.search.TopDocs$MergeSortQueue.<init>(TopDocs.java:143)
> at org.apache.lucene.search.TopDocs.merge(TopDocs.java:214)
> at
> org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:114)
> at
> org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:619)
> at
> org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:602)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
