Re: Too many fields to Sort in Solr
Hi Joel, Might have an answer for this. Initially my servers were on 3.5 and then i moved to Solr 4.0. at this time i use the solrconfig.xml that was in the example and updated is with parameters i changed in 3.5 for the environment. there was no codecFactory class=solr.SchemaCodecFactory/ in the 4.0 example solrconfig.xml file. We continued to us the same file and updated war to 4.1 then 4.2 just by changing the luceneMatchVersion in the existing solrconfig.xml file. I was looking at the 4.2 and comparing it with the one we have and i see that the *codefactory* is in the example solrconfig.xml file. codecFactory class=solr.SchemaCodecFactory/ -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052374.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
OK, that makes sense. How are DocValues working for you? On Fri, Mar 29, 2013 at 9:02 AM, adityab aditya_ba...@yahoo.com wrote: Hi Joel, Might have an answer for this. Initially my servers were on 3.5 and then i moved to Solr 4.0. at this time i use the solrconfig.xml that was in the example and updated is with parameters i changed in 3.5 for the environment. there was no codecFactory class=solr.SchemaCodecFactory/ in the 4.0 example solrconfig.xml file. We continued to us the same file and updated war to 4.1 then 4.2 just by changing the luceneMatchVersion in the existing solrconfig.xml file. I was looking at the 4.2 and comparing it with the one we have and i see that the *codefactory* is in the example solrconfig.xml file. codecFactory class=solr.SchemaCodecFactory/ -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052374.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
Joel, thanks for your excellent idea using docValues. its working exactly as you described. So far my unit test case has no issues and i see low memory foot print. Will be sending the build for performance that should give comparable numbers. Now i see another replication issue in 4.2. there is a thread on that. thanks Aditya -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052486.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
Hi, I tested this config on Solr 4.2 this morning and it worked: fieldType name=long class=solr.TrieLongField precisionStep=0 docValuesFormat=Disk positionIncrementGap=0/ field name=MMDDhh type=long indexed=true stored=true required=true docValues=true multiValued=false / I also loaded data and ran a sort and looked at the heap with jvisualvm and the longs were not loaded into the jvm's heap. The sort was also very fast, although only on 600,000 records. Possibly you are not on Solr 4.2? Can you post both your filedType definition and your field definition? Joel On Thu, Mar 28, 2013 at 12:57 AM, adityab aditya_ba...@yahoo.com wrote: Hi Joel, you are correct, boost function populates the field cache. Well i am not aware of docValue, so while trying the example you provided i see the error when i define the field type Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854) at org.apache.solr.core.SolrCore.init(SolrCore.java:719) ... 13 more My field defination: fieldType name=dvLong class=solr.TrieLongField precisionStep=0 positionIncrementGap=0 docValuesFormat=Disk/ what am i missing here? thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4051960.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
Here is the field type definition. same as what you posted yesterday just a different name. fieldType name=dvLong class=solr.TrieLongField precisionStep=0 docValuesFormat=Disk positionIncrementGap=0/ And Field Definition field name=lcontNumOfDownloads type=dvLong indexed=true stored=true default=0 docValues=true/ as soon as i restart the server i see the exception in log. removing the *docValuesFormat=Disk* from the field type i don't see this exception. 01:49:37,177 ERROR [org.apache.solr.core.CoreContainer] (coreLoadExecutor-3-thread-1) Unable to create core: collection1: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.init(SolrCore.java:806) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.SolrCore.init(SolrCore.java:619) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1021) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09] at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_09] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_09] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09] Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] at org.apache.solr.core.SolrCore.init(SolrCore.java:719) [solr-core-4.2.0.jar:4.2.0 1453694 - rmuir - 2013-03-06 22:32:13] ... 13 more 01:49:37,202 ERROR [org.apache.solr.core.CoreContainer] (coreLoadExecutor-3-thread-1) null:org.apache.solr.common.SolrException: Unable to create core: collection1 at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1672) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1057) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:634) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.init(SolrCore.java:806) at org.apache.solr.core.SolrCore.init(SolrCore.java:619) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1021) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1051) ... 10 more Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854) at org.apache.solr.core.SolrCore.init(SolrCore.java:719) ... 13 more -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052036.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052036.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
still no luck Performed. 1. Stop the Application Server (JBoss) 2. Deleted everything under data 3. Star the server 4. Observe exception in log (i have uploaded the file) on a side note. do i need to have any additional jar files in the solr home lib folder. currently its empty. docValueException.log http://lucene.472066.n3.nabble.com/file/n4052070/docValueException.log -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052070.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
Update --- I was able to fix the exception by adding following line in solrconfig.xml codecFactory name=CodecFactory class=solr.SchemaCodecFactory / Not sure if its mentioned in any document to have this declared in config file. I am now re-indexing and data on the master and will perform test to see if it works as expected. thanks for your support. Aditya -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052091.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
I didn't have to do anything with the codecs to make it work. Checked my solrconfig.xml and the codecFactory element is not present. I'm running the out of the box jetty setup. On Thu, Mar 28, 2013 at 11:58 AM, adityab aditya_ba...@yahoo.com wrote: Update --- I was able to fix the exception by adding following line in solrconfig.xml codecFactory name=CodecFactory class=solr.SchemaCodecFactory / Not sure if its mentioned in any document to have this declared in config file. I am now re-indexing and data on the master and will perform test to see if it works as expected. thanks for your support. Aditya -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052091.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
Wo. that's strange. I tried toggling with the code factory line in solrconfig.xml (attached in this post) commenting gives me error where as un-commenting works. can you please take a look into config and let me know if anything wrong there? thanks Aditya solrconfig.xml http://lucene.472066.n3.nabble.com/file/n4052131/solrconfig.xml -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052131.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
Not, sure that making changes to the solrconfig.xml is going down the right path here. There might something else with your setup that's causing this issue. I'm not sure what it would be though. On Thu, Mar 28, 2013 at 1:38 PM, adityab aditya_ba...@yahoo.com wrote: Wo. that's strange. I tried toggling with the code factory line in solrconfig.xml (attached in this post) commenting gives me error where as un-commenting works. can you please take a look into config and let me know if anything wrong there? thanks Aditya solrconfig.xml http://lucene.472066.n3.nabble.com/file/n4052131/solrconfig.xml -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4052131.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
Hi Joel, you are correct, boost function populates the field cache. Well i am not aware of docValue, so while trying the example you provided i see the error when i define the field type Caused by: org.apache.solr.common.SolrException: FieldType 'dvLong' is configured with a docValues format, but the codec does not support it: class org.apache.solr.core.SolrCore$3 at org.apache.solr.core.SolrCore.initCodec(SolrCore.java:854) at org.apache.solr.core.SolrCore.init(SolrCore.java:719) ... 13 more My field defination: fieldType name=dvLong class=solr.TrieLongField precisionStep=0 positionIncrementGap=0 docValuesFormat=Disk/ what am i missing here? thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4051960.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
I pretty sure that the boost function is going load up the fieldCache for that field, which will have the same memory footprint as sorting. You may want to try out disk based doc values (new 4.2). Documentation is sparse on this but you can specify a field as being docValues=true and set it to a field type where docValuesFormat=Disk example: field name=rec_time type=date_disk indexed=true required=true multiValued=false docValues=true default=NOW / fieldType name=date_disk class=solr.TrieDateField precisionStep=0 docValuesFormat=Disk positionIncrementGap=0/ On Mon, Mar 25, 2013 at 11:14 AM, Erick Erickson erickerick...@gmail.comwrote: Certainly that will be true for the bare q=*:*, I meant with the boosting clause added. Best Erick On Sun, Mar 24, 2013 at 7:01 PM, adityab aditya_ba...@yahoo.com wrote: thanks Eric. in this query q=*:* the Lucene score is always 1 -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4050944.html Sent from the Solr - User mailing list archive at Nabble.com. -- Joel Bernstein Professional Services LucidWorks
Re: Too many fields to Sort in Solr
Certainly that will be true for the bare q=*:*, I meant with the boosting clause added. Best Erick On Sun, Mar 24, 2013 at 7:01 PM, adityab aditya_ba...@yahoo.com wrote: thanks Eric. in this query q=*:* the Lucene score is always 1 -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4050944.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
Seems like a reasonable thing to do. Examine the debug output to insure
that there's no short-circuiting being done as far as ConstantScoreQuery...
Best
Erick
On Tue, Mar 19, 2013 at 7:05 PM, adityab aditya_ba...@yahoo.com wrote:
Hi All,
I want to validate my approach by the experts, just to make sure i am on
doing anything wrong.
#Docs in Solr : 25M
Solr Versin : 4.2
Our requirement is to list top download document based on user country.
So we have a dynamic field *numdownload.** which is evaluate as
*numdownloads.countryId*
Now as sorting is an expensive and also uses large amount of java heap, I
planned to use this field in boosting result.
Old Query
q=*:*fq=countryId:1sort=numdownloads.1 desc
which i changed to
q={!boost b=numdownloads.1}*:*fq=countryId:1
Is my approach correct. Any better alternate ?
thanks
Aditya
--
View this message in context:
http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Too many fields to Sort in Solr
thanks Eric. in this query q=*:* the Lucene score is always 1 -- View this message in context: http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139p4050944.html Sent from the Solr - User mailing list archive at Nabble.com.
Too many fields to Sort in Solr
Hi All,
I want to validate my approach by the experts, just to make sure i am on
doing anything wrong.
#Docs in Solr : 25M
Solr Versin : 4.2
Our requirement is to list top download document based on user country.
So we have a dynamic field *numdownload.** which is evaluate as
*numdownloads.countryId*
Now as sorting is an expensive and also uses large amount of java heap, I
planned to use this field in boosting result.
Old Query
q=*:*fq=countryId:1sort=numdownloads.1 desc
which i changed to
q={!boost b=numdownloads.1}*:*fq=countryId:1
Is my approach correct. Any better alternate ?
thanks
Aditya
--
View this message in context:
http://lucene.472066.n3.nabble.com/Too-many-fields-to-Sort-in-Solr-tp4049139.html
Sent from the Solr - User mailing list archive at Nabble.com.
