RE: Problem with pdf, upgrading Cell
Hello, Please let me know if anybody figured out a way out of this issue. Thanks, Sandhya -Original Message- From: Praveen Agrawal [mailto:pkal...@gmail.com] Sent: Friday, April 30, 2010 11:14 PM To: solr-user@lucene.apache.org Subject: Re: Problem with pdf, upgrading Cell Grant, You can try any of the sample pdfs that come in /docs folder of Solr 1.4 dist'n. I had tried 'Installing Solr in Tomcat.pdf', 'index.pdf' etc. Only metadata i.e. stream_size, content_type apart from my own literals are indexed, and content is missing.. On Fri, Apr 30, 2010 at 8:52 PM, Grant Ingersoll gsing...@apache.orgwrote: Praveen and Marc, Can you share the PDF (feel free to email my private email) that fails in Solr? Thanks, Grant On Apr 30, 2010, at 7:55 AM, Marc Ghorayeb wrote: Hi Nope i didn't get it to work... Just like you, command line version of tika extracts correctly the content, but once included in Solr, no content is extracted. What i tried until now is:- Updating the tika libraries inside Solr 1.4 public version, no luck there.- Downloading the latest SVN version, compiled it, and started from a simple schema, still no luck.- Getting other versions compiled on hudson (nightly builds), and testing them also, still no extraction. I sent a mail on the developpers mailing list but they told me i should just mail here, hope some developper reads this because it's quite an important feature of Solr and somehow it got broke between the 1.4 release, and the last version on the svn. Marc _ Consultez gratuitement vos emails Orange, Gmail, Free, ... directement dans HOTMAIL ! http://www.windowslive.fr/hotmail/agregation/ -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
Re: phrase search - problem
I wanted to do phrase search. What are the analyzers that best suited for phrase search. I tried with textgen, but it did not yield the expected results. I wanted to index: my dear friend If I search for dear friend, I should get the result and if I search for friend dear I should not get any records. Default PhraseQuery is unordered. dear friend returns documents containing friend dear. It is not about Analyzer but QueryParser. So you want ordered phrase queries. With SOLR-1604 you can accomplish what you want. It constructs ordered SpanNearQuery instead of PhraseQuery. For example features:stick memory returns this snippet: SmartMedia, Memory emStick/em, emMemory/em Stick Pro, SD Card http://localhost:8983/solr/select/?q=features:%22stick%20memory%22version=2.2start=0rows=10indent=ondefType=complexphrasedebugQuery=onhl=truehl.fl=features https://issues.apache.org/jira/browse/SOLR-1604
Re: synonym filter problem for string or phrase
Hi Ranveer, I don't see any stemming analyzer in your configuration of the field 'text_sync', also you have filter class=solr.TrimFilterFactory / at query time and not at index time, maybe that is your problem. Regards, Marco Martínez Bautista http://www.paradigmatecnologico.com Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón Tel.: 91 352 59 42 2010/4/30 Jonty Rhods jonty.rh...@gmail.com On 4/29/10 8:50 PM, Marco Martinez wrote: Hi Ranveer, If you don't specify a field type in the q parameter, the search will be done searching in your default search field defined in the solrconfig.xml, its your default field a text_sync field? Regards, Marco Martínez Bautista http://www.paradigmatecnologico.com Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón Tel.: 91 352 59 42 2010/4/29 Ranveerranveer.s...@gmail.com ranveer.s...@gmail.com Hi, I am trying to configure synonym filter. my requirement is: when user searching by phrase like what is solr user? then it should be replace with solr user. something like : what is solr user? = solr user My schema for particular field is: fieldType name=text_sync class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.KeywordTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ /analyzer analyzer type=query tokenizer class=solr.KeywordTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.TrimFilterFactory / filter class=solr.SynonymFilterFactory synonyms=synonyms.txt ignoreCase=true expand=true tokenizerFactory=KeywordTokenizerFactory/ /analyzer /fieldType it seems working fine while trying by analysis.jsp but not by url http://localhost:8080/solr/core0/select?q=what is solr user? or http://localhost:8080/solr/core0/select?q=what is solr user? Please guide me for achieve desire result. Hi Marco, thanks. yes my default search field is text_sync. I am getting result now but not as I expect. following is my synonym.txt what is bone cancer=bone cancer what is bone cancer?=bone cancer what is of bone cancer=bone cancer what is symptom of bone cancer=bone cancer what is symptoms of bone cancer=bone cancer in above I am getting result of all synonym but not the last one what is symptoms of bone cancer=bone cancer. I think due to stemming I am not getting expected result. However when I am checking result from the analysis.jsp, its giving expected result. I am confused.. Also I want to know best approach to configure synonym for my requirement. thanks with regards Hi, I am also facing same type of problem.. I am Newbie please help. thanks Jonty
Re: Skipping duplicates in DataImportHandler based on uniqueKey
You can use deduplication to do that. Create the signature based on the unique field or any field you want. -- View this message in context: http://lucene.472066.n3.nabble.com/Skipping-duplicates-in-DataImportHandler-based-on-uniqueKey-tp771559p772768.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Problem with pdf, upgrading Cell
Hi, Grant, i confirm what Praveen has said, any PDF i try does not work with the new Tika and SVN versions. :( Marc From: sagar...@opentext.com To: solr-user@lucene.apache.org Date: Mon, 3 May 2010 13:05:24 +0530 Subject: RE: Problem with pdf, upgrading Cell Hello, Please let me know if anybody figured out a way out of this issue. Thanks, Sandhya -Original Message- From: Praveen Agrawal [mailto:pkal...@gmail.com] Sent: Friday, April 30, 2010 11:14 PM To: solr-user@lucene.apache.org Subject: Re: Problem with pdf, upgrading Cell Grant, You can try any of the sample pdfs that come in /docs folder of Solr 1.4 dist'n. I had tried 'Installing Solr in Tomcat.pdf', 'index.pdf' etc. Only metadata i.e. stream_size, content_type apart from my own literals are indexed, and content is missing.. On Fri, Apr 30, 2010 at 8:52 PM, Grant Ingersoll gsing...@apache.orgwrote: Praveen and Marc, Can you share the PDF (feel free to email my private email) that fails in Solr? Thanks, Grant On Apr 30, 2010, at 7:55 AM, Marc Ghorayeb wrote: Hi Nope i didn't get it to work... Just like you, command line version of tika extracts correctly the content, but once included in Solr, no content is extracted. What i tried until now is:- Updating the tika libraries inside Solr 1.4 public version, no luck there.- Downloading the latest SVN version, compiled it, and started from a simple schema, still no luck.- Getting other versions compiled on hudson (nightly builds), and testing them also, still no extraction. I sent a mail on the developpers mailing list but they told me i should just mail here, hope some developper reads this because it's quite an important feature of Solr and somehow it got broke between the 1.4 release, and the last version on the svn. Marc _ Consultez gratuitement vos emails Orange, Gmail, Free, ... directement dans HOTMAIL ! http://www.windowslive.fr/hotmail/agregation/ -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search _ Hotmail et MSN dans la poche? HOTMAIL et MSN sont dispo gratuitement sur votre téléphone! http://www.messengersurvotremobile.com/?d=Hotmail
RE: run on reboot on windows
Hi Ahmed, I need to achieve that also. Do you manage to install it as service and start solr with Jetty? After installing and start jetty as service how do you start solr? Thanks, Frederico -Original Message- From: S Ahmed [mailto:sahmed1...@gmail.com] Sent: segunda-feira, 3 de Maio de 2010 01:05 To: solr-user@lucene.apache.org Subject: Re: run on reboot on windows Thanks, for some reason I was looking for a solution outside of jetty/tomcat, when that was the obvious way to get things restarted :) On Sun, May 2, 2010 at 7:53 PM, Dave Searle dave.sea...@magicalia.comwrote: Tomcat is installed as a service on windows. Just go into service control panel and set startup type to automatic Sent from my iPhone On 3 May 2010, at 00:43, S Ahmed sahmed1...@gmail.com wrote: its not tomcat/jetty that's the issue, its how to get things to re- start on a windows server (tomcat and jetty don't run as native windows services) so I am a little confused..thanks. On Sun, May 2, 2010 at 7:37 PM, caman aboxfortheotherst...@gmail.comwrote: Ahmed, Best is if you take a look at the documentation of jetty or tomcat. SOLR can run on any web container, it's up to you how you configure your web container to run Thanks Aboxy From: S Ahmed [via Lucene] [mailto:ml-node+772174-2097041460-124...@n3.nabble.comml-node%2B772174- 2097041460-124...@n3.nabble.com ml-node %2b772174-2097041460-124...@n3.nabble.com ] Sent: Sunday, May 02, 2010 4:33 PM To: caman Subject: Re: run on reboot on windows By default it uses Jetty, so your saying Tomcat on windows server 2008/ IIS7 runs as a native windows service? On Sun, May 2, 2010 at 12:46 AM, Dave Searle [hidden email]wrote: Set tomcat6 service to auto start on boot (if running tomat) Sent from my iPhone On 2 May 2010, at 02:31, S Ahmed [hidden email] wrote: Hi, I'm trying to get Solr to run on windows, such that if it reboots the Solr service will be running. How can I do this? _ View message @ http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 174 . html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node%2B472068-464289649 -124...@n3.nabble.com ml-node %2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 178.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: synonym filter problem for string or phrase
Just for clear terminology: You mean field, not fieldType. FieldType is the definition of tokenizers, filters etc.. You apply a fieldType on a field. And you query against a field, not against a whole fieldType. :-) Kind regards - Mitch Marco Martinez-2 wrote: Hi Ranveer, If you don't specify a field type in the q parameter, the search will be done searching in your default search field defined in the solrconfig.xml, its your default field a text_sync field? Regards, Marco Martínez Bautista http://www.paradigmatecnologico.com Avenida de Europa, 26. Ática 5. 3ª Planta 28224 Pozuelo de Alarcón Tel.: 91 352 59 42 -- View this message in context: http://lucene.472066.n3.nabble.com/synonym-filter-problem-for-string-or-phrase-tp765242p773083.html Sent from the Solr - User mailing list archive at Nabble.com.
Retrieving indexed field data
Hi folks, i'm wondering if there is a way to retrieve the indexed data. The reason is that i'm working on a solrj-based tool that copies one index data into other (allowing you to perform changes in docs ). I know i can't perform any change in an indexed field, just want to copy the chunk of bytes .. Am i missing something? Indexing generated data can't be retrieved anyway? Thanks in advance . -- Lici ~Java Developer~
Commit takes 1 to 2 minutes, CPU usage affects other apps
Hi, we recently began having trouble with our Solr 1.4 instance. We've about 850k documents in the index which is about 1.2GB in size; the JVM which runs tomcat/solr (no other apps are deployed) has been given 2GB. We've a forum and run a process every minute which indexes the new messages. The number of messages updated are from 0 to 20 messages average. The commit takes about 1 or two minutes, but usually when it finished a few seconds later the next batch of documents is processed and the story starts again. So actually it's like Solr is running commits all day long and CPU usage ranges from 80% to 120%. This continuous CPU usage caused ill effects on other services running on the same machine. Our environment is being providing by a company purely using VMWare infrastructure, the Solr index itself is on an NSF for which we get some 33MB/s throughput. So, an easy solution would be to just put more resources into it, e.g. a separate machine. But before I make the decision I'd like to find out whether the app behaves properly under this circumstances or if its possible to shorten the commit time down to a few seconds so the CPU is not drained that long. thanks for any pointers, - Markus
Re: OutOfMemoryError when using query with sort
How many unique terms are in your sort field? On Sun, May 2, 2010 at 11:48 PM, Hamid Vahedi hvb...@yahoo.com wrote: I install 64 bit windows and my problem solved. also i using shard mode (100 M doc per machine with one solr instance) is there better solution? because i insert at least 5M doc per day From: Koji Sekiguchi k...@r.email.ne.jp To: solr-user@lucene.apache.org Sent: Sun, May 2, 2010 9:08:42 PM Subject: Re: OutOfMemoryError when using query with sort Hamid Vahedi wrote: Hi, i using solr that running on windows server 2008 32-bit. I add about 100 million article into solr without set store attribute. (only store document id) (index file size about 164 GB) when try to get query without sort , it's return doc ids in some ms, but when add sort command, i get below error: TTP Status 500 - Java heap space java.lang.OutOfMemoryError: Java heap space at Since sort uses FieldCache and it consumes memory, you got OOM. I think 100M docs/164GB index is considerable large for 32 bit machine. Why don't you use distributed search? Koji -- http://www.rondhuit.com/en/
SpellChecking
Hi there, I want to enable spellchecking, but i got many fields. I tried around with copyfield to copy all with * in one field, but that didnt work. Next try was to copy some fields specified each by name in one field named spell, but that worked only for 2 or 3 fields, but not for 10 or more... My question is, what the best practice is to enable spellchecking on many fields. thanks. greetz, Jan
Re: Skipping duplicates in DataImportHandler based on uniqueKey
Marc Sturlese wrote: You can use deduplication to do that. Create the signature based on the unique field or any field you want. Cool, thanks, I hadn't thought of that. -- View this message in context: http://lucene.472066.n3.nabble.com/Skipping-duplicates-in-DataImportHandler-based-on-uniqueKey-tp771559p773268.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: run on reboot on windows
I don't think jetty can be installed as a service. You'd need to create a bat file and put that in the win startup registry. Sent from my iPhone On 3 May 2010, at 11:26, Frederico Azeiteiro frederico.azeite...@cision.com wrote: Hi Ahmed, I need to achieve that also. Do you manage to install it as service and start solr with Jetty? After installing and start jetty as service how do you start solr? Thanks, Frederico -Original Message- From: S Ahmed [mailto:sahmed1...@gmail.com] Sent: segunda-feira, 3 de Maio de 2010 01:05 To: solr-user@lucene.apache.org Subject: Re: run on reboot on windows Thanks, for some reason I was looking for a solution outside of jetty/tomcat, when that was the obvious way to get things restarted :) On Sun, May 2, 2010 at 7:53 PM, Dave Searle dave.sea...@magicalia.comwrote: Tomcat is installed as a service on windows. Just go into service control panel and set startup type to automatic Sent from my iPhone On 3 May 2010, at 00:43, S Ahmed sahmed1...@gmail.com wrote: its not tomcat/jetty that's the issue, its how to get things to re- start on a windows server (tomcat and jetty don't run as native windows services) so I am a little confused..thanks. On Sun, May 2, 2010 at 7:37 PM, caman aboxfortheotherst...@gmail.comwrote: Ahmed, Best is if you take a look at the documentation of jetty or tomcat. SOLR can run on any web container, it's up to you how you configure your web container to run Thanks Aboxy From: S Ahmed [via Lucene] [mailto:ml-node+772174-2097041460-124...@n3.nabble.comml-node %2B772174- 2097041460-124...@n3.nabble.com ml-node %2b772174-2097041460-124...@n3.nabble.com ] Sent: Sunday, May 02, 2010 4:33 PM To: caman Subject: Re: run on reboot on windows By default it uses Jetty, so your saying Tomcat on windows server 2008/ IIS7 runs as a native windows service? On Sun, May 2, 2010 at 12:46 AM, Dave Searle [hidden email]wrote: Set tomcat6 service to auto start on boot (if running tomat) Sent from my iPhone On 2 May 2010, at 02:31, S Ahmed [hidden email] wrote: Hi, I'm trying to get Solr to run on windows, such that if it reboots the Solr service will be running. How can I do this? _ View message @ http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 174 . html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node %2B472068-464289649 -124...@n3.nabble.com ml-node %2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 178.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SpellChecking
It would help a lot to see your actual config file, and if you provided a bit more detail about what failure looks like Best Erick On Mon, May 3, 2010 at 9:43 AM, Jan Kammer jan.kam...@mni.fh-giessen.dewrote: Hi there, I want to enable spellchecking, but i got many fields. I tried around with copyfield to copy all with * in one field, but that didnt work. Next try was to copy some fields specified each by name in one field named spell, but that worked only for 2 or 3 fields, but not for 10 or more... My question is, what the best practice is to enable spellchecking on many fields. thanks. greetz, Jan
Re: SpellChecking
Hi, if I define one of my normal fields from schema.xml in solrconfig.xml for spellchecking all works fine: lst name=spellchecker ... str name=fieldcontent/str ... /lst But i want to use spellcheck for much more fields. So I tried to define in schema.xml a copyfield like this: copyField source=* dest=spell/ That didnt work, because nothing was in spell after that. Next try was to copy each field in a line to spell: copyField source=content dest=spell/ copyField source=name dest=spell/ copyField source=title dest=spell/ ... This does work up to 3 documents, if i define more, the count for failed documents in dataimporthandler gets higher and higher the more i copy into spell. str name=Total Documents Failed16444/str So my question is, if this is the right way to use the spellchecker with many fields, or is there an other better way... thanks. greetz, Jan Am 03.05.2010 16:08, schrieb Erick Erickson: It would help a lot to see your actual config file, and if you provided a bit more detail about what failure looks like Best Erick On Mon, May 3, 2010 at 9:43 AM, Jan Kammerjan.kam...@mni.fh-giessen.dewrote: Hi there, I want to enable spellchecking, but i got many fields. I tried around with copyfield to copy all with * in one field, but that didnt work. Next try was to copy some fields specified each by name in one field named spell, but that worked only for 2 or 3 fields, but not for 10 or more... My question is, what the best practice is to enable spellchecking on many fields. thanks. greetz, Jan
RE: SpellChecking
We are using copy fields for 40+ fields to do spelling, and it works fine. Are you sure that you actually build the spell index before you try to do spelling? You need to either configure SOLr to build spell index on commit, or manually issue a spell index build request. Regards, Gert. -Original Message- From: Jan Kammer [mailto:jan.kam...@mni.fh-giessen.de] Sent: Montag, 3. Mai 2010 16:26 To: solr-user@lucene.apache.org Subject: Re: SpellChecking Hi, if I define one of my normal fields from schema.xml in solrconfig.xml for spellchecking all works fine: lst name=spellchecker ... str name=fieldcontent/str ... /lst But i want to use spellcheck for much more fields. So I tried to define in schema.xml a copyfield like this: copyField source=* dest=spell/ That didnt work, because nothing was in spell after that. Next try was to copy each field in a line to spell: copyField source=content dest=spell/ copyField source=name dest=spell/ copyField source=title dest=spell/ ... This does work up to 3 documents, if i define more, the count for failed documents in dataimporthandler gets higher and higher the more i copy into spell. str name=Total Documents Failed16444/str So my question is, if this is the right way to use the spellchecker with many fields, or is there an other better way... thanks. greetz, Jan Am 03.05.2010 16:08, schrieb Erick Erickson: It would help a lot to see your actual config file, and if you provided a bit more detail about what failure looks like Best Erick On Mon, May 3, 2010 at 9:43 AM, Jan Kammerjan.kam...@mni.fh-giessen.dewrote: Hi there, I want to enable spellchecking, but i got many fields. I tried around with copyfield to copy all with * in one field, but that didnt work. Next try was to copy some fields specified each by name in one field named spell, but that worked only for 2 or 3 fields, but not for 10 or more... My question is, what the best practice is to enable spellchecking on many fields. thanks. greetz, Jan Please help Logica to respect the environment by not printing this email / Pour contribuer comme Logica au respect de l'environnement, merci de ne pas imprimer ce mail / Bitte drucken Sie diese Nachricht nicht aus und helfen Sie so Logica dabei, die Umwelt zu schützen. / Por favor ajude a Logica a respeitar o ambiente nao imprimindo este correio electronico. This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you.
Re: SpellChecking
Am 03.05.2010 16:43, schrieb Jan Kammer: Hi, It worked fine with a normal field. There must something wrong with copyfield, or why does dataimporthandler add/update no more documents? Did you define your destination field as multivalue? -Michael
Auto-commit does not work
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Running Solr 1.4 with updateHandler class=solr.DirectUpdateHandler2 ? !-- A prefix of solr. for class names is an alias that causes solr to search appropriate packages, including org.apache.solr.(search|update|request|core|analysis) - -- ? autocommit maxDocs1000/maxDocs maxTime6/maxTime /autocommit /updateHandler I inserted 10k documents through a Python script (w/ solrpy bindings) without explict commit. However I do not see that the numDocs increased meanwhile...is there any way to hunt this down? Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkve8MUACgkQCJIWIbr9KYwl6gCg6HxoSyjt9ccPFAohIKjEUzdZ ZrEAoM7QmwP3GWVMSpQaCtABK/6K/6dK =EiRk -END PGP SIGNATURE-
Re: Auto-commit does not work
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ahmet Arslan wrote: I inserted 10k documents through a Python script (w/ solrpy bindings) without explict commit. However I do not see that the numDocs increased meanwhile...is there any way to hunt this down? What does solr/admin/stats.jsp#update page says about autocommits and docsPending? commits : 135 autocommits : 0 optimizes : 0 rollbacks : 0 expungeDeletes : 0 docsPending : 8842 adds : 8842 deletesById : 0 deletesByQuery : 0 errors : 0 cumulative_adds : 8842 cumulative_deletesById : 20390 cumulative_deletesByQuery : 0 cumulative_errors : 0 Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkve9pwACgkQCJIWIbr9KYwKhgCdFB39W6SitIeaL//ioXwWoC+n 4PkAn05TlCFYfJw3Vm/B20Dm7lyY/Qhm =lWbr -END PGP SIGNATURE-
cores and SWAP
Hi, I'm trying to figure out whether I need to reload a core (or both cores?) after performing a swap. When I perform a swap in my sandbox (non-production) environment, I am seeing that one of the cores needs to be reloaded following a swap and the other does not, but I haven't been able to find a pattern to which one it will be. http://wiki.apache.org/solr/CoreAdmin doesn't seem to cover this. Or, maybe I'm doing something wrong. Thanks for any help, Tim
Re: Auto-commit does not work
commits : 135 autocommits : 0 optimizes : 0 rollbacks : 0 expungeDeletes : 0 docsPending : 8842 adds : 8842 deletesById : 0 deletesByQuery : 0 errors : 0 cumulative_adds : 8842 cumulative_deletesById : 20390 cumulative_deletesByQuery : 0 cumulative_errors : 0 I just realized that there is a typo in your autoCommit definition. The letter C sould be capital. autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit
Re: Auto-commit does not work
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ahmet Arslan wrote: I just realized that there is a typo in your autoCommit definition. The letter C sould be capital. autoCommit maxDocs1/maxDocs maxTime1000/maxTime /autoCommit right - and Solr should not swallow errors in the configuration :-) Andreas -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvfB2wACgkQCJIWIbr9KYy+6ACZAXSFy8wHPWa8R+wzEUHhftEp qw0An1GV6PAAdd4ezTKn4OF9WwROAUHP =FeBy -END PGP SIGNATURE-
Re: Auto-commit does not work
: right - and Solr should not swallow errors in the configuration :-) If you have an error in a *known* config declaration, solr will complain about it -- but solr can't complain just because you declare extra stuff in your conig files that it doens't know anything about -- some other plugin might care about it (or it might be there because you wanted special syntax for your own documentation purposes) -Hoss
Re: Auto-commit does not work
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Hostetter wrote: : right - and Solr should not swallow errors in the configuration :-) If you have an error in a *known* config declaration, solr will complain about it -- but solr can't complain just because you declare extra stuff in your conig files that it doens't know anything about -- some other plugin might care about it (or it might be there because you wanted special syntax for your own documentation purposes) I don't care about if this config is a core configuration or a configuration of some plugin. Such kind of misconfiguration should be handled properly - this is the minimum one can expect. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvfCQ4ACgkQCJIWIbr9KYz22QCfeVNIwJt0f5+XfnV1qvsZ0HJm He0AoL5lIoEiUyhUINXpA2rcDB8bgsAy =skdf -END PGP SIGNATURE-
Re: Overlapping onDeckSearchers=2
On Mon, May 3, 2010 at 11:24 AM, revas revas...@gmail.com wrote: Hello, We have a server with many solr instances running (around 40-50) . We are committing documents ,sometimes one or sometimes around 200 documents at a time .to only one instance at a time When i run 2 -3 commits parallely to diff instances or same instance I get this error PERFORMANCE WARNING: Overlapping onDeckSearchers=2 What is the Best approach to solve this You should see that warning only when you run multiple commits within a short period of time to the same Solr instance. You will never see this warning when performing commit on different instances. So, do you really need to commit on the same instance in such short time? Can you batch commits? -- Regards, Shalin Shekhar Mangar.
Re: Overlapping onDeckSearchers=2
: When i run 2 -3 commits parallely to diff instances or same instance I get : this error : : PERFORMANCE WARNING: Overlapping onDeckSearchers=2 : : What is the Best approach to solve this http://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F -Hoss
Re: cores and SWAP
I have 2 cores: core1 and core2. Load the same data set into each and commit. Verify that searches return the same for each core. Delete a document (call it docA) from core2 but not from core1. Commit and verify search results (docA disappears from core2's search results. core1 continues to return the docA) Swap cores. Core2 should now return docA, but it doesn't until I reload core2. thanks, Tim On Mon, May 3, 2010 at 1:41 PM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: On Mon, May 3, 2010 at 10:27 PM, Tim Heckman theck...@gmail.com wrote: Hi, I'm trying to figure out whether I need to reload a core (or both cores?) after performing a swap. When I perform a swap in my sandbox (non-production) environment, I am seeing that one of the cores needs to be reloaded following a swap and the other does not, but I haven't been able to find a pattern to which one it will be. No, you should not need to reload any core after a swap. What is the behavior that you see? -- Regards, Shalin Shekhar Mangar.
Re: Score cutoff
Hi, Can someone give clues on how to implement this feature? This is a very important requirement for us, so any help is greatly appreciated. thanks! On Tue, Apr 27, 2010 at 5:54 PM, Satish Kumar satish.kumar.just.d...@gmail.com wrote: Hi, For some of our queries, the top xx (five or so) results are of very high quality and results after xx are very poor. The difference in score for the high quality and poor quality results is high. For example, 3.5 for high quality and 0.8 for poor quality. We want to exclude results with score value that is less than 60% or so of the first result. Is there a filter that does this? If not, can someone please give some hints on how to implement this (we want to do this as part of solr relevance ranking so that the facet counts, etc will be correct). Thanks, Satish
Re: Problem with pdf, upgrading Cell
I'm investigating. On May 3, 2010, at 5:17 AM, Marc Ghorayeb wrote: Hi, Grant, i confirm what Praveen has said, any PDF i try does not work with the new Tika and SVN versions. :( Marc From: sagar...@opentext.com To: solr-user@lucene.apache.org Date: Mon, 3 May 2010 13:05:24 +0530 Subject: RE: Problem with pdf, upgrading Cell Hello, Please let me know if anybody figured out a way out of this issue. Thanks, Sandhya -Original Message- From: Praveen Agrawal [mailto:pkal...@gmail.com] Sent: Friday, April 30, 2010 11:14 PM To: solr-user@lucene.apache.org Subject: Re: Problem with pdf, upgrading Cell Grant, You can try any of the sample pdfs that come in /docs folder of Solr 1.4 dist'n. I had tried 'Installing Solr in Tomcat.pdf', 'index.pdf' etc. Only metadata i.e. stream_size, content_type apart from my own literals are indexed, and content is missing.. On Fri, Apr 30, 2010 at 8:52 PM, Grant Ingersoll gsing...@apache.orgwrote: Praveen and Marc, Can you share the PDF (feel free to email my private email) that fails in Solr? Thanks, Grant On Apr 30, 2010, at 7:55 AM, Marc Ghorayeb wrote: Hi Nope i didn't get it to work... Just like you, command line version of tika extracts correctly the content, but once included in Solr, no content is extracted. What i tried until now is:- Updating the tika libraries inside Solr 1.4 public version, no luck there.- Downloading the latest SVN version, compiled it, and started from a simple schema, still no luck.- Getting other versions compiled on hudson (nightly builds), and testing them also, still no extraction. I sent a mail on the developpers mailing list but they told me i should just mail here, hope some developper reads this because it's quite an important feature of Solr and somehow it got broke between the 1.4 release, and the last version on the svn. Marc _ Consultez gratuitement vos emails Orange, Gmail, Free, ... directement dans HOTMAIL ! http://www.windowslive.fr/hotmail/agregation/ -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search _ Hotmail et MSN dans la poche? HOTMAIL et MSN sont dispo gratuitement sur votre téléphone! http://www.messengersurvotremobile.com/?d=Hotmail -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
Re: Problem with pdf, upgrading Cell
I've opened https://issues.apache.org/jira/browse/SOLR-1902 to track this. It is indeed a bug somewhere (still investigating). It seems that Tika is now picking an EmptyParser implementation when trying to determine which parser to use, despite the fact that it properly identifies the MIME Type. -Grant On May 3, 2010, at 5:36 PM, Grant Ingersoll wrote: I'm investigating. On May 3, 2010, at 5:17 AM, Marc Ghorayeb wrote: Hi, Grant, i confirm what Praveen has said, any PDF i try does not work with the new Tika and SVN versions. :( Marc From: sagar...@opentext.com To: solr-user@lucene.apache.org Date: Mon, 3 May 2010 13:05:24 +0530 Subject: RE: Problem with pdf, upgrading Cell Hello, Please let me know if anybody figured out a way out of this issue. Thanks, Sandhya -Original Message- From: Praveen Agrawal [mailto:pkal...@gmail.com] Sent: Friday, April 30, 2010 11:14 PM To: solr-user@lucene.apache.org Subject: Re: Problem with pdf, upgrading Cell Grant, You can try any of the sample pdfs that come in /docs folder of Solr 1.4 dist'n. I had tried 'Installing Solr in Tomcat.pdf', 'index.pdf' etc. Only metadata i.e. stream_size, content_type apart from my own literals are indexed, and content is missing.. On Fri, Apr 30, 2010 at 8:52 PM, Grant Ingersoll gsing...@apache.orgwrote: Praveen and Marc, Can you share the PDF (feel free to email my private email) that fails in Solr? Thanks, Grant On Apr 30, 2010, at 7:55 AM, Marc Ghorayeb wrote: Hi Nope i didn't get it to work... Just like you, command line version of tika extracts correctly the content, but once included in Solr, no content is extracted. What i tried until now is:- Updating the tika libraries inside Solr 1.4 public version, no luck there.- Downloading the latest SVN version, compiled it, and started from a simple schema, still no luck.- Getting other versions compiled on hudson (nightly builds), and testing them also, still no extraction. I sent a mail on the developpers mailing list but they told me i should just mail here, hope some developper reads this because it's quite an important feature of Solr and somehow it got broke between the 1.4 release, and the last version on the svn. Marc _ Consultez gratuitement vos emails Orange, Gmail, Free, ... directement dans HOTMAIL ! http://www.windowslive.fr/hotmail/agregation/ -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search _ Hotmail et MSN dans la poche? HOTMAIL et MSN sont dispo gratuitement sur votre téléphone! http://www.messengersurvotremobile.com/?d=Hotmail -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
Re: Problem with pdf, upgrading Cell
Little more info... Seems to be a classloading issue. The tests pass, but they aren't loading the Tika libraries via the Solr ResourceLoader, whereas the example is. Marc, one thing to try is to unjar the Solr WAR file and put the Tika libs in there, as I bet it will then work. Note, however, I haven't tried this. On May 3, 2010, at 6:24 PM, Grant Ingersoll wrote: I've opened https://issues.apache.org/jira/browse/SOLR-1902 to track this. It is indeed a bug somewhere (still investigating). It seems that Tika is now picking an EmptyParser implementation when trying to determine which parser to use, despite the fact that it properly identifies the MIME Type. -Grant On May 3, 2010, at 5:36 PM, Grant Ingersoll wrote: I'm investigating. On May 3, 2010, at 5:17 AM, Marc Ghorayeb wrote: Hi, Grant, i confirm what Praveen has said, any PDF i try does not work with the new Tika and SVN versions. :( Marc From: sagar...@opentext.com To: solr-user@lucene.apache.org Date: Mon, 3 May 2010 13:05:24 +0530 Subject: RE: Problem with pdf, upgrading Cell Hello, Please let me know if anybody figured out a way out of this issue. Thanks, Sandhya -Original Message- From: Praveen Agrawal [mailto:pkal...@gmail.com] Sent: Friday, April 30, 2010 11:14 PM To: solr-user@lucene.apache.org Subject: Re: Problem with pdf, upgrading Cell Grant, You can try any of the sample pdfs that come in /docs folder of Solr 1.4 dist'n. I had tried 'Installing Solr in Tomcat.pdf', 'index.pdf' etc. Only metadata i.e. stream_size, content_type apart from my own literals are indexed, and content is missing.. On Fri, Apr 30, 2010 at 8:52 PM, Grant Ingersoll gsing...@apache.orgwrote: Praveen and Marc, Can you share the PDF (feel free to email my private email) that fails in Solr? Thanks, Grant On Apr 30, 2010, at 7:55 AM, Marc Ghorayeb wrote: Hi Nope i didn't get it to work... Just like you, command line version of tika extracts correctly the content, but once included in Solr, no content is extracted. What i tried until now is:- Updating the tika libraries inside Solr 1.4 public version, no luck there.- Downloading the latest SVN version, compiled it, and started from a simple schema, still no luck.- Getting other versions compiled on hudson (nightly builds), and testing them also, still no extraction. I sent a mail on the developpers mailing list but they told me i should just mail here, hope some developper reads this because it's quite an important feature of Solr and somehow it got broke between the 1.4 release, and the last version on the svn. Marc _ Consultez gratuitement vos emails Orange, Gmail, Free, ... directement dans HOTMAIL ! http://www.windowslive.fr/hotmail/agregation/ -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search _ Hotmail et MSN dans la poche? HOTMAIL et MSN sont dispo gratuitement sur votre téléphone! http://www.messengersurvotremobile.com/?d=Hotmail -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search -- Grant Ingersoll http://www.lucidimagination.com/ Search the Lucene ecosystem using Solr/Lucene: http://www.lucidimagination.com/search
Re: Commit takes 1 to 2 minutes, CPU usage affects other apps
On 5/3/10 9:06 AM, Markus Fischer wrote: Hi, we recently began having trouble with our Solr 1.4 instance. We've about 850k documents in the index which is about 1.2GB in size; the JVM which runs tomcat/solr (no other apps are deployed) has been given 2GB. We've a forum and run a process every minute which indexes the new messages. The number of messages updated are from 0 to 20 messages average. The commit takes about 1 or two minutes, but usually when it finished a few seconds later the next batch of documents is processed and the story starts again. So actually it's like Solr is running commits all day long and CPU usage ranges from 80% to 120%. This continuous CPU usage caused ill effects on other services running on the same machine. Our environment is being providing by a company purely using VMWare infrastructure, the Solr index itself is on an NSF for which we get some 33MB/s throughput. So, an easy solution would be to just put more resources into it, e.g. a separate machine. But before I make the decision I'd like to find out whether the app behaves properly under this circumstances or if its possible to shorten the commit time down to a few seconds so the CPU is not drained that long. thanks for any pointers, - Markus That is certainly not a normal commit time for an index of that size. Note that Solr 1.4 can have issues when working on NFS, but I don't know that it would have anything to do with this. Are you using the simple lock factory rather than the default native lock factory? (as you should do when running on NFS) -- - Mark http://www.lucidimagination.com
Re: Auto-commit does not work
I think his point was, _what_ determines if its a misconfiguration? It can't be Solr because, like he said, a plugin may require it. If there is no such plugin, then what shall be the handler of it properly? nothingergo its ignored. On Mon, 2010-05-03 at 19:34 +0200, Andreas Jung wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chris Hostetter wrote: : right - and Solr should not swallow errors in the configuration :-) If you have an error in a *known* config declaration, solr will complain about it -- but solr can't complain just because you declare extra stuff in your conig files that it doens't know anything about -- some other plugin might care about it (or it might be there because you wanted special syntax for your own documentation purposes) I don't care about if this config is a core configuration or a configuration of some plugin. Such kind of misconfiguration should be handled properly - this is the minimum one can expect. - -aj -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvfCQ4ACgkQCJIWIbr9KYz22QCfeVNIwJt0f5+XfnV1qvsZ0HJm He0AoL5lIoEiUyhUINXpA2rcDB8bgsAy =skdf -END PGP SIGNATURE-
Re: Solr commit issue
This could be caused by HTTP caching. Solr's example solrconfig.xml comes with HTTP caching turned on, and this causes lots of beginners to have problems. The code to turn it off is commented in solrconfig.xml. Notice that the default is to have caching on, so to turn it off you have to have the XML that turns it off. On Sat, May 1, 2010 at 8:01 PM, Indika Tantrigoda indik...@gmail.com wrote: Thanks for the reply. Here is another thread I found similar to this http://www.mail-archive.com/solr-user@lucene.apache.org/msg28236.html From what I understand the IndexReaders get reopened after a commit. Regards, Indika On 2 May 2010 00:29, Erick Erickson erickerick...@gmail.com wrote: The underlying IndexReader must be reopened. If you're searching for a document with a searcher that was opened before the document was indexed, it won't show up on the search results. I'm guessing that your statement that when you search for it with some test is coincidence, but that's just a guess. HTH Erick On Sat, May 1, 2010 at 1:07 PM, Indika Tantrigoda indik...@gmail.com wrote: Hi all, I've been working with Solr for a few weeks and have gotten SolrJ to connect to it, index, search documents. However I am having an issue when a document is committed. When a document is committed it does not show in the search results if I do a *:* search, but if I search for it with some text then it is shown in the results. Only when another document is committed, the previous document is found when I do a *:* search Is this because of the SolrJ client or do I have to pass additional parameters to commit() ? Thanks in advance. Regards, Indika -- Lance Norskog goks...@gmail.com
Facets vs TermV's
Hi, I spent a lot of time on the Wiki and am working with facets and tv's, but I'm still confused about something. Basically, what is the difference between issuing a facet field query that returns facets with counts, and a query with term vectors that also returns document frequency counts for terms in a field? They seem almost similar, but I'm missing something I think. i.e. when is best to use one over the other? I know this is Solr 101, but just want to understand it fully. Thanks for any quick tips again. thanks.
Re: run on reboot on windows
There are a few programs that wrap any java app as a service. http://en.wikipedia.org/wiki/Service_wrapper On Mon, May 3, 2010 at 6:58 AM, Dave Searle dave.sea...@magicalia.com wrote: I don't think jetty can be installed as a service. You'd need to create a bat file and put that in the win startup registry. Sent from my iPhone On 3 May 2010, at 11:26, Frederico Azeiteiro frederico.azeite...@cision.com wrote: Hi Ahmed, I need to achieve that also. Do you manage to install it as service and start solr with Jetty? After installing and start jetty as service how do you start solr? Thanks, Frederico -Original Message- From: S Ahmed [mailto:sahmed1...@gmail.com] Sent: segunda-feira, 3 de Maio de 2010 01:05 To: solr-user@lucene.apache.org Subject: Re: run on reboot on windows Thanks, for some reason I was looking for a solution outside of jetty/tomcat, when that was the obvious way to get things restarted :) On Sun, May 2, 2010 at 7:53 PM, Dave Searle dave.sea...@magicalia.comwrote: Tomcat is installed as a service on windows. Just go into service control panel and set startup type to automatic Sent from my iPhone On 3 May 2010, at 00:43, S Ahmed sahmed1...@gmail.com wrote: its not tomcat/jetty that's the issue, its how to get things to re- start on a windows server (tomcat and jetty don't run as native windows services) so I am a little confused..thanks. On Sun, May 2, 2010 at 7:37 PM, caman aboxfortheotherst...@gmail.comwrote: Ahmed, Best is if you take a look at the documentation of jetty or tomcat. SOLR can run on any web container, it's up to you how you configure your web container to run Thanks Aboxy From: S Ahmed [via Lucene] [mailto:ml-node+772174-2097041460-124...@n3.nabble.comml-node %2B772174- 2097041460-124...@n3.nabble.com ml-node %2b772174-2097041460-124...@n3.nabble.com ] Sent: Sunday, May 02, 2010 4:33 PM To: caman Subject: Re: run on reboot on windows By default it uses Jetty, so your saying Tomcat on windows server 2008/ IIS7 runs as a native windows service? On Sun, May 2, 2010 at 12:46 AM, Dave Searle [hidden email]wrote: Set tomcat6 service to auto start on boot (if running tomat) Sent from my iPhone On 2 May 2010, at 02:31, S Ahmed [hidden email] wrote: Hi, I'm trying to get Solr to run on windows, such that if it reboots the Solr service will be running. How can I do this? _ View message @ http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 174 . html To start a new topic under Solr - User, email ml-node+472068-464289649-124...@n3.nabble.comml-node %2B472068-464289649 -124...@n3.nabble.com ml-node %2b472068-464289649-124...@n3.nabble.com To unsubscribe from Solr - User, click (link removed) GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx here. -- View this message in context: http://lucene.472066.n3.nabble.com/run-on-reboot-on-windows-tp770892p772 178.html Sent from the Solr - User mailing list archive at Nabble.com. -- Lance Norskog goks...@gmail.com
Re: Solr Dismax query - prefix matching
: example: If I have a field called 'booktitle' with the actual values as : 'Code Complete', 'Coding standard 101', then I'd like to search for the : query string 'cod' and have the dismax match against both the book : titles since 'cod' is a prefix match for 'code' and 'coding'. it doesn't sound like you really want prefix queries ... it sounds like you want stemming. It's hard to tell because you only gave one example, so consider whether you want the book codependents of agony to match a search for code ... if hte answer is yes then what you are looking for is preix matching, if the answer is no then you should probably read up on stemming (which can work with the dismax parsing, by configuring it in the analyzer for your fields) -Hoss
