Documents with SOLR function sort are NOT sorted by score
I am having difficulty with my sort function. With the following sort,
documents are not sorted by score if you can see. Why sort function is not
able to sort it properly?
I appreciate your prompt answer
This is my sort function.
sort=map(and(termfreq(CustomersFavourite,852708),exists($exactqq)),1,1,1,0)
desc,map(and(termfreq(CustomersPurchased,852708),exists($exactqq)),1,1,1,0)
desc,map(exists($exactqq),1,1,NumberOfClicks,0)
desc,map(exists($exactqq),1,1,Amount,0)
desc,map(and(termfreq(InPromotion_925,true),exists($exactqq)),1,1,1,0)
desc,map(exists($exactqq),1,1,OrderCount,0) desc
exactqq={!edismax}(ProductModelNameExact:xyz OR ProductModelName_TR:xyz)
But as you can results are not sorted by score
response: {
numFound: 139,
start: 0,
maxScore: 0.6251737,
docs: [
{
score: 0.28109676
},
{
score: 0.25829598
},
{
score: 0.36092186
},
{
score: 0.6251737
},
{
score: 0.1379621
},
{
score: 0.14090014
},
{
score: 0.1379621
},
{
score: 0.14090014
},
{
score: 0.50190175
},
{
score: 0.1379621
},
{
score: 0.12398934
},
{
score: 0.1379621
},
{
score: 0.12398934
},
{
score: 0.4989637
},
{
score: 0.12585841
},
{
score: 0.12585841
},
--
View this message in context:
http://lucene.472066.n3.nabble.com/Documents-with-SOLR-function-sort-are-NOT-sorted-by-score-tp4173928.html
Sent from the Solr - User mailing list archive at Nabble.com.
Dynamic boosting with functions
I need to boost some of document with with a function. I can give boosting for field myField with bq = myField^10 But I like to give dynamic boosting based on whether or not other field someotherfield exists in a document or not. I like to give boosting if (exists(termfreq(someotherfield,green)),100,1) to myField field Is this possible? If so, how? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Dynamic-boosting-with-functions-tp4172387.html Sent from the Solr - User mailing list archive at Nabble.com.
Boosting mixed fiedl types
I have several field types and like to assign correct boosting so that I will get results in correct order. Here is a summary of what I have: 1- Product Title - text field , Boost = 160 2- Product Description - text field , Boost = 80 3-Number of clicks - Integer field, having value [1 TO 1000] , Boost = 40 4- Product Features - text field , Boost = 20 5- AmountPurchased - Float field , Boost = 10 5- Product Properties - text field , Boost = 5 User will make a search q= foo bar and we expect solr will return results based on Boost values assigned above. qf and pf can help me to assign boosting for text fields easily. But I am having difficulty to mix text fields with numeric ones. For example, I want product with Number of clicks = 20 should be listed higher than one with 10 clicks after 1) and 2). I guess solr, based on search results, will re order based boost values in text fields but I want product with number of clicks 10 will be higher than with clicks 5. As result, any products having clicks will have higher ranks that products that has features that includes search keywords. I hope I have explained correctly, Can you please guide me on how to solve this issue? Regards Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Boosting-mixed-fiedl-types-tp4169469.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Boosting mixed fiedl types
Ahmet, Thanks boost parameter is determining boost value for whole query. But I am assigning boost for other individual fields. I worry whether or not boost parameter and invidiual boosts with bf and pdf will rank results properly. Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Boosting-mixed-fiedl-types-tp4169469p4169489.html Sent from the Solr - User mailing list archive at Nabble.com.
Every minute update on solrclound
I am using solr 4.3.1 attached solrconfig.xml file. There is no softCommit enabled in config file but master node is receiving continuously an update document evry minute. I could not figure it out where this update is coming. Solr cloud master: [04/May/2014:01:50:54 +] GET /solr/trcollection2/update/json?commit=true?softCommit=true HTTP/1.1 200 160 [04/May/2014:01:51:53 +] POST /solr/trcollection2/update/json?commit=true HTTP/1.1 200 55 [04/May/2014:01:51:54 +] GET /solr/trcollection2/update/json?commit=true?softCommit=true HTTP/1.1 200 160 [04/May/2014:01:52:53 +] POST /solr/trcollection2/update/json?commit=true HTTP/1.1 200 55 [04/May/2014:01:52:54 +] GET /solr/trcollection2/update/json?commit=true?softCommit=true HTTP/1.1 200 160 [04/May/2014:01:53:53 +] POST /solr/trcollection2/update/json?commit=true HTTP/1.1 200 55 [04/May/2014:01:53:54 +] GET /solr/trcollection2/update/json?commit=true?softCommit=true HTTP/1.1 200 160 [04/May/2014:01:54:53 +] POST /solr/trcollection2/update/json?commit=true HTTP/1.1 200 55 - [04/May/2014:01:54:54 +] GET /solr/trcollection2/update/json?commit=true?softCommit=true HTTP/1.1 200 160 Solr clound slave solrconfig.xml http://lucene.472066.n3.nabble.com/file/n4134489/solrconfig.xml - [04/May/2014:01:54:39 +] POST /solr/trcollection2_shard9_replica1/update HTTP/1.1 200 41 - - [04/May/2014:01:54:40 +] POST /solr/trcollection2_shard9_replica1/update HTTP/1.1 200 41 - - [04/May/2014:01:55:39 +] POST /solr/trcollection2_shard9_replica1/update HTTP/1.1 200 41 - - [04/May/2014:01:55:40 +] POST /solr/trcollection2_shard9_replica1/update HTTP/1.1 200 41 -- View this message in context: http://lucene.472066.n3.nabble.com/Every-minute-update-on-solrclound-tp4134489.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unit of dimension for solr field
Erick, I haven't written any SOLR plugin before so it takes time to understand concepts. This is more simpler to implement and I think this way does not need to write any plugin SOLR, isn't it? Outside process analyses values with dimensions and prepare 2 fields as you described Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Unit-of-dimension-for-solr-field-tp4100209p4100449.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unit of dimension for solr field
Thanks Upayavira It seems it needs too much work. I will have several more fields that will have unit values. Do we have more quicker way of implementing it? We have Currency filed coming as default with SOLR. Can we use it? Creating conversion rate table for each field? What I am expecting from units is similar to currency field Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Unit-of-dimension-for-solr-field-tp4100209p4100295.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unit of dimension for solr field
Ryan and Upayavira, Do we have an example skeleton to do this for schema.xml and solrconfig.xml? Example java class that would help to build UnitResolvingFilterFactory class? Thanks Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Unit-of-dimension-for-solr-field-tp4100209p4100303.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unit of dimension for solr field
Can DelimitedPayloadTokenFilterFactory be used to store unit dimension information? This factory class can store extra information for field. -- View this message in context: http://lucene.472066.n3.nabble.com/Unit-of-dimension-for-solr-field-tp4100209p4100345.html Sent from the Solr - User mailing list archive at Nabble.com.
Unit of dimension for solr field
I would like to have a SOLR field that has multiple unit of dimension. Suppose we store the memory value of a computer in solr field. That can have value 256 MB, 512 MB, or 1 GB where we use MB and GB units. Same case is for hard drive sizes : 256 MB,50GB or 3TB where we use MB,GB and TB units. How can I store these unit of dimensions with values itself? I would like to have range queries on such fields: say bring me desktops that has 256M-1G memory values. I appreciate any guidance Thanks Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Unit-of-dimension-for-solr-field-tp4100209.html Sent from the Solr - User mailing list archive at Nabble.com.
Invalid version (expected 2, but 60) or the data in not in 'javabin' format
I am using Solr 4.3.1 on solrcloud with 10 nodes. I added 3 million documents from a csv file with this command curl 'http://localhost:8080/solr/trcollection2/update/csv?stream.file=/home/hduser/csvFile.csvskipLines=1fieldnames=,cache,segment,digest,tstamp,lang,url,,content,id,title,booststream.contentType=text/p lain;charset=utf-8' Then I query the data, fetching first 100K documents with this. But I am getting Invalid version (expected 2, but 60) or the data in not in 'javabin' format error. I have appended what I got in output file alltrcollection_309mil.csv I appreciate if you can help me on this hduser@host1:~$ curl -o alltrcollection_309mil.csv 'http://localhost:8080/solr/trcollection2/select?q=*%3A*rows=10wt=xmlindent=true' % Total% Received % Xferd Average Speed TimeTime Time Current Dload Upload Total SpentLeft Speed 100 34390 34390 0103 0 --:--:-- 0:00:33 --:--:-- 1049 hduser@host1:~$ more alltrcollection_309mil.csv ?xml version=1.0 encoding=UTF-8? response lst name=responseHeader int name=status500/int int name=QTime33307/int lst name=params str name=indenttrue/str str name=q*:*/str str name=wtxml/str str name=rows10/str /lst /lst lst name=error str name=msgjava.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format/str str name=traceorg.apache.solr.common.SolrException: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:656) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:359) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:931) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.RuntimeException: Invalid version (expected 2, but 60) or the data in not in 'javabin' format at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:109) at org.apache.solr.client.solrj.impl.BinaryResponseParser.processResponse(BinaryResponseParser.java:41) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) ... 3 more /str int name=code500/int /lst /response -- View this message in context: http://lucene.472066.n3.nabble.com/Invalid-version-expected-2-but-60-or-the-data-in-not-in-javabin-format-tp4075739.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Invalid version (expected 2, but 60) or the data in not in 'javabin' format
Thanks for your answer, I can fetch 10K documents without any issue. I don't think we are having out of memory exception because each tomcat server in cluster has 8GB memory allocated. -- View this message in context: http://lucene.472066.n3.nabble.com/Invalid-version-expected-2-but-60-or-the-data-in-not-in-javabin-format-tp4075739p4075749.html Sent from the Solr - User mailing list archive at Nabble.com.
java.io.CharConversionException] Invalid UTF-8 character 0xffff at char #478803, byte #606190)
Hello, I am crawling with apache nutche some sites and index it with solr. It has been working fine until a few days ago. The crawled data can have 200K or more documents inside. When I send it to SOLR to index with bin/nutch solrindex http://.com:8080/solr crawl/crawldb -linkdb crawl/linkdb crawl/segments/* nutch is getting SORL server internal error. SOLR 4.1 logs are getting this error below: It is very tough to find which document are causing this issue. What I need is either to configure SOLR so that it will ignore documents that has bad data inside and continue to index next documents coming from nutch. Or even though I am new to SOLR, maybe, I can write update pre/post processor plugin to SORL update job to ignore XML errors. Do we have solution for this problem? I appreciate your help class java.io.CharConversionException] Invalid UTF-8 character 0x at char #478803, byte #606190).%trace?..java.lang.RuntimeException: [was class java.io.CharConversionException] Invalid UTF-8 character 0x at char #478803, byte #606190) .at com.ctc.wstx.util.ExceptionUtil.throwRuntimeException(ExceptionUtil.java:18) .at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:731) .at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3657) .at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:809) .at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:393) .at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:245) .at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173) .at or g.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92) .at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) .at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) .at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816) .at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448) .at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269) .at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) .at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) .at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) .at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) .at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) .at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorRepo rtValve.java:99) .at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:931) .at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) .at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) .at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) .at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) .at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312) .at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) .at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) .at java.lang.Thread.run(Thread.java:722) Caused by: java.io.CharConversionException: Invalid UTF-8 character 0x at char #478803, byte #606190) .at com.ctc.wstx.io.UTF8Reader.reportInvalid(UTF8Reader.java:335) .at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:249) .at com.ctc.wstx.io.MergedReader.read(Merge dReader.java:101) .at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:84) .at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) .at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:992) .at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4628) .at com.ctc.wstx.sr.BasicStreamReader.readCoalescedText(BasicStreamReader.java:4126) .at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3701) .at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3649) 25 more -- View this message in context: http://lucene.472066.n3.nabble.com/java-io-CharConversionException-Invalid-UTF-8-character-0x-at-char-478803-byte-606190-tp4055323.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Faceted search on SOLR
Thanks for help, Alexandre. It worked as you described. I have other question. Suppose I have product catalogue that has many sub categories, each one has different group of fields. When a user search the catalogue, we should show corresponding facet fields on left based on result set. That means we should be dynamically pick appropriate facets to show based on result products from search. Can we do it with SOLR? Or SOLR can show only static facets configured? It is better to show only related facets I guess. Also, can we filter out facets that has ,say less than 10 results? I appreciate your answer Erol Akarsu -- View this message in context: http://lucene.472066.n3.nabble.com/Faceted-search-on-SOLR-tp4045297p4045563.html Sent from the Solr - User mailing list archive at Nabble.com.
