[jira] [Commented] (SOLR-12457) field(x,min|max) sorting doesn't work on trie or str fields in multi-shard collections
[ https://issues.apache.org/jira/browse/SOLR-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510391#comment-16510391 ] Hoss Man commented on SOLR-12457: - Ok ... here's some spit balling... ultimatley the root of the "ClassCastException: java.lang.String cannot be cast to org.apache.lucene.util.BytesRef" comes down to the age old problem of JavaBinCodec defaulting to {{writeVal(val.getClass().getName() + ':' + val.toString());}} when it doesn't know how to write ab Object. In particular: {{QueryComponent.doFieldSortValues}} tries to be agnostic to the SortFields/Comparators used, and puts the "raw" sort value directly in the response (unless there is a SchemaField that can be used to marshall it) .. in the case of functions like these, that's a BytesRef -- on the other side of things, the resulting "String" (ie: {{org.apache.lucene.util.BytesRef:[DEADBEEF]}} ) gets read in and cases the class cast when it's passed to TermOrdValComparator. In an ideal world, the intra node communication would just happily "work" even with {{BytesRef}} objects -- in which case we wouldn't even need the existing marshall/unmarshall logic that StrField has -- and we could potentially configure an ObjectResolver on the JavaBinCodec used for intra node communication to do so ... but that seems like a pretty big change for this and still wouldn't fix things for anyone still using xml for intra node communication (do these people exist?) a "hack" with a more limited surface area of change would be to make {{QueryComponent.doFieldSortValues}} and {{QueryComponent.unmarshalSortValues}} "smarter" such that they take care of automatically base64 encoding/decoding sort values if/when the SortField uses a TermOrdValComparator ...that's really the best idea i've got. > field(x,min|max) sorting doesn't work on trie or str fields in multi-shard > collections > -- > > Key: SOLR-12457 > URL: https://issues.apache.org/jira/browse/SOLR-12457 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.1 >Reporter: Varun Thacker >Priority: Major > Attachments: SOLR-12457.patch > > > When I go to sort on a multi-valued field in a 2 shard collection, which has > trie fields the query fails. > To reproduce we need 2+ shards, a multi-valued trie field and "desc" sort > criteria. > Here's my schema > {code:java} > multiValued="true" docValues="true"/> > positionIncrementGap="0" precisionStep="0"/> > multiValued="true"/> > > {code} > Now If I add a few docs > {code:java} > [ > {"id" : "1", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "2", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "3", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]} > ]{code} > Works: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_i,max)%20desc] > > Doesn't Work: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_is,max)%20desc] > > To be more clear when I say it doesn't work , the query throws and error and > here's the stack trace for it: > {code:java} > ERROR - 2018-06-06 22:55:06.599; [c:gettingstarted s:shard2 r:core_node8 > x:gettingstarted_shard2_replica_n5] org.apache.solr.common.SolrException; > null:java.lang.ClassCastException: java.lang.String cannot be cast to > org.apache.lucene.util.BytesRef > at > org.apache.lucene.search.FieldComparator$TermOrdValComparator.compareValues(FieldComparator.java:561) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:161) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:153) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:91) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:33) > at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263) > at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140) > at > org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:156) > at > org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:924) > at > org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585) > at > org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.jav
[jira] [Commented] (SOLR-12457) field(x,min|max) sorting doesn't work on trie or str fields in multi-shard collections
[ https://issues.apache.org/jira/browse/SOLR-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510314#comment-16510314 ] Hoss Man commented on SOLR-12457: - attached a patch that converts {{SortByFunctionTest}} to a MiniSolrCloud based test (NOTE: {{TestSortByMinMaxFunction extendsSortByFunctionTest}} ) as a starting point for testing & fixing this > field(x,min|max) sorting doesn't work on trie or str fields in multi-shard > collections > -- > > Key: SOLR-12457 > URL: https://issues.apache.org/jira/browse/SOLR-12457 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.1 >Reporter: Varun Thacker >Priority: Major > Attachments: SOLR-12457.patch > > > When I go to sort on a multi-valued field in a 2 shard collection, which has > trie fields the query fails. > To reproduce we need 2+ shards, a multi-valued trie field and "desc" sort > criteria. > Here's my schema > {code:java} > multiValued="true" docValues="true"/> > positionIncrementGap="0" precisionStep="0"/> > multiValued="true"/> > > {code} > Now If I add a few docs > {code:java} > [ > {"id" : "1", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "2", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "3", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]} > ]{code} > Works: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_i,max)%20desc] > > Doesn't Work: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_is,max)%20desc] > > To be more clear when I say it doesn't work , the query throws and error and > here's the stack trace for it: > {code:java} > ERROR - 2018-06-06 22:55:06.599; [c:gettingstarted s:shard2 r:core_node8 > x:gettingstarted_shard2_replica_n5] org.apache.solr.common.SolrException; > null:java.lang.ClassCastException: java.lang.String cannot be cast to > org.apache.lucene.util.BytesRef > at > org.apache.lucene.search.FieldComparator$TermOrdValComparator.compareValues(FieldComparator.java:561) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:161) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:153) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:91) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:33) > at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263) > at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140) > at > org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:156) > at > org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:924) > at > org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585) > at > org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:423) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720) > at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526){code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12457) field(x,min|max) sorting doesn't work on trie or str fields in multi-shard collections
[ https://issues.apache.org/jira/browse/SOLR-12457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16510308#comment-16510308 ] Hoss Man commented on SOLR-12457: - bq. ... so StrField's marshalSortValue & unmarshalSortValue methods (which handle the BytesRef/String conversion) take care of things for you. FWIW: i started second guessing myself as to *why* I was seeing the implicit syntax work for str fields, because in both cases the {{SortField}} should be identical {{SortedSetSortField}} instances with the field name set on them -- but then i realized {{QueryComponent.unmarshalSortValues}} uses {{SortSpec.getSchemaFields()}} to decide which {{SortFields}} should be unmarshalled -- and _that_ knows the difference between a {{SortField}} whose {{String getField()}} might happen to match a schema field name, vs an actual usage of sorting on a {{SchemaField}} (which is important for field aliasing, etc...) > field(x,min|max) sorting doesn't work on trie or str fields in multi-shard > collections > -- > > Key: SOLR-12457 > URL: https://issues.apache.org/jira/browse/SOLR-12457 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 7.1 >Reporter: Varun Thacker >Priority: Major > > When I go to sort on a multi-valued field in a 2 shard collection, which has > trie fields the query fails. > To reproduce we need 2+ shards, a multi-valued trie field and "desc" sort > criteria. > Here's my schema > {code:java} > multiValued="true" docValues="true"/> > positionIncrementGap="0" precisionStep="0"/> > multiValued="true"/> > > {code} > Now If I add a few docs > {code:java} > [ > {"id" : "1", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "2", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]}, > {"id" : "3", "test_is" : ["1", "2", "3", "4", "5"], "test_i" : ["1", "2", > "3", "4", "5"]} > ]{code} > Works: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_i,max)%20desc] > > Doesn't Work: > [http://localhost:8983/solr/gettingstarted/select?q=*:*&sort=field(test_is,max)%20desc] > > To be more clear when I say it doesn't work , the query throws and error and > here's the stack trace for it: > {code:java} > ERROR - 2018-06-06 22:55:06.599; [c:gettingstarted s:shard2 r:core_node8 > x:gettingstarted_shard2_replica_n5] org.apache.solr.common.SolrException; > null:java.lang.ClassCastException: java.lang.String cannot be cast to > org.apache.lucene.util.BytesRef > at > org.apache.lucene.search.FieldComparator$TermOrdValComparator.compareValues(FieldComparator.java:561) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:161) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue$1.compare(ShardFieldSortedHitQueue.java:153) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:91) > at > org.apache.solr.handler.component.ShardFieldSortedHitQueue.lessThan(ShardFieldSortedHitQueue.java:33) > at org.apache.lucene.util.PriorityQueue.upHeap(PriorityQueue.java:263) > at org.apache.lucene.util.PriorityQueue.add(PriorityQueue.java:140) > at > org.apache.lucene.util.PriorityQueue.insertWithOverflow(PriorityQueue.java:156) > at > org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:924) > at > org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:585) > at > org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:564) > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:423) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2484) > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:720) > at > org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:526){code} > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org