Re: sort by function
Where is your query? You don't search for anything. The q-param is empty. You got two options (untested): remove the q-param or search for something special. I think removing is not a good idea. Instead search for *:* would retrive ALL results that match your filter-query. Kind regards - Mitch -- View this message in context: http://lucene.472066.n3.nabble.com/sort-by-function-tp814380p839167.html Sent from the Solr - User mailing list archive at Nabble.com.
Indexing stalls reads
Hey, I'm using solr 1.4 I've a master / slave setup, I use the slave for all my read operations commits are scheduled every 20 mins or every 1 docs. Now I think slave shouldn't build index but fetch ones created on Master, but I see it creating indexes, during which all read stalls. Now I don't think thats a common behavior or is there any other way to stop this ? Also how do i stop slave from removing the old indexes till AutoWarming is done ? is there a way to achieve this ? Thnx n Regards, - Manish _ The amazing world in sharp snaps http://news.in.msn.com/gallery/archive.aspx
sort by field length
Hi folks, is it possible to sort by field length without having to (redundantly) save the length information in a seperate index field? At first, I thought to accomplish this using a function query, but I couldn't find an appropriate one. Thanks in advance, Sascha
Dynamically augment search with data
So my need is this: I have a site in which a user does a query for other users. The user can filter the query by different parameters that will limit the result set. One of the things about the system is that the user's can like different objects (Products, Services, etc.). When the user searches the index by a query and it returns a list of users I want to be able to calculate the shared likes between the user and each user result in the the returned result set. I would like to then append the calculation in each result in the result set and then sort by the greatest number of shared likes, thereby making the results more relevant to the user. I would like to have this calculation run before the paging process kicks in so this function will be applied to the result set right before paging. I am using Solr 1.4 and have read just a little on FunctionQuery. Is this what I am needing to perform this task? *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ Travis Chase ~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
Re: CFP for Lucene Revolution Conference, Boston, MA October 7 8 2010
I should add that talks on Mahout, Tika, Nutch, etc. are also encouraged. -Grant On May 17, 2010, at 8:43 AM, Grant Ingersoll wrote: Lucene Revolution Call For Participation - Boston, Massachusetts October 7 8, 2010 The first US conference dedicated to Apache Lucene and Solr is coming to Boston, October 7 8, 2010. The conference is sponsored by Lucid Imagination with additional support from community and other commercial co‐sponsors. The audience will include those experienced Solr and Lucene application development, along with those experienced in other enterprise search technologies interested becoming more familiar with Solr and Lucene technologies and the opportunities they present. We are soliciting 45‐minute presentations for the conference. Key Dates: May 12, 2010 Call For Participation Open June 23, 2010Call For Participation Closes June 28, 2010Speaker Acceptance/Rejection Notification October 5‐6, 2010 Lucene and Solr Pre‐conference Training Sessions October 7‐8, 2010 Conference Sessions Topics of interest include: Lucene and Solr in the Enterprise (case studies, implementation, return on investment, etc.) “How We Did It” Development Case Studies Spatial/Geo search Lucene and Solr in the Cloud (Deployment cases as well as tutorials) Scalability and Performance Tuning Large Scale Search Real Time Search Data Integration/Data Management Lucene Solr for Mobile Applications All accepted speakers will qualify for discounted conference admission. Financial assistance is available for speakers that qualify. To submit a 45‐minute presentation proposal, please send an email to c...@lucenerevolution.org with Subject containing: your name, Topic your session title containing the following information in plain text. If you have more than one topic proposed, send a separate email. Do not attach Word or other text file documents. Return all fields completed as follows: 1.Your full name, title, and organization 2.Contact information, including your address, email, phone number 3.The name of your proposed session (keep your title simple, interesting, and relevant to the topic) 4.A 75‐200 word overview of your presentation; in addition to the topic, describe whether your presentation is intended as a tutorial, description of an implementation, an theoretical/academic discussion, etc. 5.A 100‐200‐word speaker bio that includes prior conference speaking or related experience To be considered, proposals must be received by 12 Midnight PDT Wednesday, June 23, 2010. Please email any general questions regarding the conference to i...@lucenerevolution.org. To be added to the conference mailing list, please email sig...@lucenerevolution.org. If your organization is interested in sponsorship opportunities, email spon...@lucenerevolution.org. We look forward to seeing you in Boston!
Re: Embedded Server, Caching, Stats page updates
So you're right i did miss removing the app deployment but removing that still didn't really do that great. The avg request response time is still slower. The bell curve is a lot more streched than it was before but it doesn't seem to give an overall better performance. Thanks for your suggestions, Tony On Wed, May 19, 2010 at 4:37 PM, Chris Hostetter hossman_luc...@fucit.orgwrote: : Switched works for the specific setup i'm using - the server would refer : to itself in the CommonHttpSolrServer request sent, i.e. it would run both : the server and client sides. Removing this and simply using : EmbeddedSolrServer just made the setup a little more sane in that aspect. : Does that make more sense now? not really ... what *exactly* did you change about your setup and your client code? please be specific -- how did you run solr before when you were using CommonsHttpSolrServer? whare are *all* of the steps you did when you switched to EmbeddedSolrServer (specificly: what did the changes to your java client code look like, and what did you hcange about how you run solr) Because if you still have the solr.war running in your servlet container, and all you did is edit your java code to use EmbeddedSolrServer (poiting at the same directory on disk) instead of COmmonsHttpSolrServer, thne you are now running *two* instances of Solr in your VM, both reading from the same indexes. -Hoss
Highlighting a field with a certain value
Hello, How am I able to highlight a field that contains a specific value? If I have a field called type, how am I able to highlight the rows whose values contain something like title?
Problem with extended dismax, minus prefix (to mean NOT) and interaction with mm?
I'm running edismax (on both a 1.4 with patch and a branch_3x version) and I'm seeing something I don't expect. We have our mm set such that 2/2 must match and 2/3 must match (mm=2-1 567% A query of dog cat ...gets interpreted as dog AND cat But a query of dog cat -mouse ...gets interpreted as (dog AND cat) OR (dog AND NOT mouse) OR (cat AND NOT mouse) In other words, the -mouse is being interpreted as a single token (NOT mouse) to be counted for mm. I would expect the query to interpret as: (dog AND cat) AND (NOT mouse) Are my expectations out of whack? Or is this unexpected behavior? [I've pasted the debugQuery info for a similar search below, though I freely admit to not knowing how to read it] Any thoughts on what I'm seeing here? -Bill- lst name=debug str name=rawquerystringdog cat -trilogy/str str name=querystringdog cat -trilogy/str str name=parsedqueryallfields:dog allfields:cat -allfields:trilogi/str str name=parsedquery_toStringallfields:dog allfields:cat -allfields:trilogi/str lst name=explain str name=000107098 2.1741915 = (MATCH) sum of: 1.2620605 = (MATCH) weight(allfields:dog in 3187), product of: 0.7618881 = queryWeight(allfields:dog), product of: 8.744003 = idf(docFreq=64, maxDocs=15) 0.08713264 = queryNorm 1.6564907 = (MATCH) fieldWeight(allfields:dog in 3187), product of: 1.7320508 = tf(termFreq(allfields:dog)=3) 8.744003 = idf(docFreq=64, maxDocs=15) 0.109375 = fieldNorm(field=allfields, doc=3187) 0.912131 = (MATCH) weight(allfields:cat in 3187), product of: 0.64770865 = queryWeight(allfields:cat), product of: 7.4335938 = idf(docFreq=240, maxDocs=15) 0.08713264 = queryNorm 1.4082427 = (MATCH) fieldWeight(allfields:cat in 3187), product of: 1.7320508 = tf(termFreq(allfields:cat)=3) 7.4335938 = idf(docFreq=240, maxDocs=15) 0.109375 = fieldNorm(field=allfields, doc=3187) /str str name=36695 2.1518915 = (MATCH) sum of: 1.249116 = (MATCH) weight(allfields:dog in 36426), product of: 0.7618881 = queryWeight(allfields:dog), product of: 8.744003 = idf(docFreq=64, maxDocs=15) 0.08713264 = queryNorm 1.6395006 = (MATCH) fieldWeight(allfields:dog in 36426), product of: 2.0 = tf(termFreq(allfields:dog)=4) 8.744003 = idf(docFreq=64, maxDocs=15) 0.09375 = fieldNorm(field=allfields, doc=36426) 0.9027756 = (MATCH) weight(allfields:cat in 36426), product of: 0.64770865 = queryWeight(allfields:cat), product of: 7.4335938 = idf(docFreq=240, maxDocs=15) 0.08713264 = queryNorm 1.3937988 = (MATCH) fieldWeight(allfields:cat in 36426), product of: 2.0 = tf(termFreq(allfields:cat)=4) 7.4335938 = idf(docFreq=240, maxDocs=15) 0.09375 = fieldNorm(field=allfields, doc=36426) /str str name=38137 1.4345944 = (MATCH) sum of: 0.832744 = (MATCH) weight(allfields:dog in 37852), product of: 0.7618881 = queryWeight(allfields:dog), product of: 8.744003 = idf(docFreq=64, maxDocs=15) 0.08713264 = queryNorm 1.0930004 = (MATCH) fieldWeight(allfields:dog in 37852), product of: 1.0 = tf(termFreq(allfields:dog)=1) 8.744003 = idf(docFreq=64, maxDocs=15) 0.125 = fieldNorm(field=allfields, doc=37852) 0.6018504 = (MATCH) weight(allfields:cat in 37852), product of: 0.64770865 = queryWeight(allfields:cat), product of: 7.4335938 = idf(docFreq=240, maxDocs=15) 0.08713264 = queryNorm 0.9291992 = (MATCH) fieldWeight(allfields:cat in 37852), product of: 1.0 = tf(termFreq(allfields:cat)=1) 7.4335938 = idf(docFreq=240, maxDocs=15) 0.125 = fieldNorm(field=allfields, doc=37852) /str str name=000134898 1.2629167 = (MATCH) sum of: 0.624558 = (MATCH) weight(allfields:dog in 30673), product of: 0.7618881 = queryWeight(allfields:dog), product of: 8.744003 = idf(docFreq=64, maxDocs=15) 0.08713264 = queryNorm 0.8197503 = (MATCH) fieldWeight(allfields:dog in 30673), product of: 1.0 = tf(termFreq(allfields:dog)=1) 8.744003 = idf(docFreq=64, maxDocs=15) 0.09375 = fieldNorm(field=allfields, doc=30673) 0.6383587 = (MATCH) weight(allfields:cat in 30673), product of: 0.64770865 = queryWeight(allfields:cat), product of: 7.4335938 = idf(docFreq=240, maxDocs=15) 0.08713264 = queryNorm 0.9855646 = (MATCH) fieldWeight(allfields:cat in 30673), product of: 1.4142135 = tf(termFreq(allfields:cat)=2) 7.4335938 = idf(docFreq=240, maxDocs=15) 0.09375 = fieldNorm(field=allfields, doc=30673) /str str name=29964 1.25527 = (MATCH) sum of: 0.728651 = (MATCH) weight(allfields:dog in 29736), product of: 0.7618881 = queryWeight(allfields:dog), product of: 8.744003 = idf(docFreq=64, maxDocs=15) 0.08713264 = queryNorm 0.95637536 = (MATCH) fieldWeight(allfields:dog in 29736), product of: 1.0 =
Re: IndexSearcher and Caches
I am not sure, what you mean with multi-user-scenario. I have an application deployed on an application server (Weblogic). This application uses solr to query an index. Users (sessions) will log in to the application, query and then log out. This login and logout has nothing to do with solr but the application manages them separately. I am using EmbeddedSolrServer here. I think I know where my mistake is. From what you say, it looks to me as though that I should not create a new SolrIndexSearcher object because Solr will do this automatically. In my current implementation, I am explicitly creating a new SolrIndexSearcher object for every new user who logs into the application. Let me provide a code snippet to explain further. This is how I initialize the solr handles required for searching. I am using EmbeddedSolrServer. SolrConfig solrConfig = new SolrConfig(configHome+/solrconfig.xml); IndexSchema indexSchema = new IndexSchema(solrConfig, configHome+/schema.xml, null); File corefile = new File(coreHome, solr.xml); CoreContainer coreContainer = new CoreContainer(coreHome, corefile); CoreDescriptor coreDescriptor = new CoreDescriptor(coreContainer, coreName, solrConfig.getResourceLoader().getInstanceDir()); coreDescriptor.setConfigName(solrConfig.getResourceName()); coreDescriptor.setSchemaName(indexSchema.getResourceName()); SolrCore solrCore = new SolrCore(coreName, indexHome, solrConfig, indexSchema, coreDescriptor); coreContainer.register(coreName, solrCore, false); SolrServer solrServer = new EmbeddedSolrServer( coreContainer, coreName ); //Next two lines executed for every user SolrIndexSearcher solrSearcher = solrCore.newSearcher(s1); SolrRequestParsers solrRequestParsers = new SolrRequestParsers(solrConfig); Many thanks for the response(s). Regards Rahul On Mon, May 24, 2010 at 1:55 AM, MitchK mitc...@web.de wrote: In my case, I have an index which will not be modified after creation. Does this mean that in a multi-user scenario, I can have a static IndexSearcher object that can be shared by multiple users ? I am not sure, what you mean with multi-user-scenario. Can you tell me what you got in mind? If your index never changes, your IndexSearcher won't change. If the IndexSearcher object is threadsafe, then only issues related to concurrency are addressed. What about the case where the IndexSearcher is static? User 1 logs in to the system, queries with the static IndexSearcher, logs out; and then User 2 logs in to the system, queries with the same static IndexSearcher, logs out. In this case, the users 1 and 2 are not querying concurrently but one after another. Will the query information (filters or any other data) of User 1 be retained when User 2 uses this ? I am not sure about the benefit of a static IndexSearcher. What do you hope??? If user 1 uses a filter like fq=name:Samuelq=somethingIWantToKnow and user 2 queries for fq=name:Samuelq=whatIReallyWantToKnow than they use the same cached filter-object, retrived from Solr's internal cache (of course you need to have a cache-size that allows cacheing). The solr wiki states that the caches are per IndexSearcher object i.e if I set my filterCache size to 1000 it means that 1000 entries can be assigned for every IndexSearcher object. Yes. If a new searcher is created than the new Cache is built on the old one. Is this true for queryResultsCache, filterCache and documentCache ? For FilterCache it's true. For queryResultsCache (if I understand the wiki right), too. Please note, that the documentCache's behaviour is different from the already mentioned ones. The wiki says: Note: This cache cannot be used as a source for autowarming because document IDs will change when anything in the index changes so they can't be used by a new searcher. The wiki says that the number of the document cache should not be bigger than the number of _results_ * number of _concurrent_ queries. I never worked with the document cache, so maybe someone else can throw some light into the dark. But from what I have understood it means the following: If you show 10 results per request and you think of up to 500 concurrent queries: 10 * 500 = 5000 But I want to emphasize, that this is only a gues. I actually don't exactly know more about this topic. Kind regards - Mitch -- View this message in context: http://lucene.472066.n3.nabble.com/IndexSearcher-and-Caches-tp833567p838367.html Sent from the Solr - User mailing list archive at Nabble.com.
Highlighting is not happening
Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !-- str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !-- str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Re: Highlighting is not happening
Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !-- str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !-- str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Re: CFP for Lucene Revolution Conference, Boston, MA October 7 8 2010
Are there any of these conferences in India? Dennis Gearon Signature Warning EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Mon, 5/24/10, Grant Ingersoll gsing...@apache.org wrote: From: Grant Ingersoll gsing...@apache.org Subject: Re: CFP for Lucene Revolution Conference, Boston, MA October 7 8 2010 To: Cc: java-u...@lucene.apache.org, solr-user@lucene.apache.org, Lucene mailing list gene...@lucene.apache.org, u...@nutch.apache.org, u...@tika.apache.org, u...@mahout.apache.org, lucene-net-...@lucene.apache.org Date: Monday, May 24, 2010, 8:14 AM I should add that talks on Mahout, Tika, Nutch, etc. are also encouraged. -Grant On May 17, 2010, at 8:43 AM, Grant Ingersoll wrote: Lucene Revolution Call For Participation - Boston, Massachusetts October 7 8, 2010 The first US conference dedicated to Apache Lucene and Solr is coming to Boston, October 7 8, 2010. The conference is sponsored by Lucid Imagination with additional support from community and other commercial co‐sponsors. The audience will include those experienced Solr and Lucene application development, along with those experienced in other enterprise search technologies interested becoming more familiar with Solr and Lucene technologies and the opportunities they present. We are soliciting 45‐minute presentations for the conference. Key Dates: May 12, 2010 Call For Participation Open June 23, 2010 Call For Participation Closes June 28, 2010 Speaker Acceptance/Rejection Notification October 5‐6, 2010 Lucene and Solr Pre‐conference Training Sessions October 7‐8, 2010 Conference Sessions Topics of interest include: Lucene and Solr in the Enterprise (case studies, implementation, return on investment, etc.) “How We Did It” Development Case Studies Spatial/Geo search Lucene and Solr in the Cloud (Deployment cases as well as tutorials) Scalability and Performance Tuning Large Scale Search Real Time Search Data Integration/Data Management Lucene Solr for Mobile Applications All accepted speakers will qualify for discounted conference admission. Financial assistance is available for speakers that qualify. To submit a 45‐minute presentation proposal, please send an email to c...@lucenerevolution.org with Subject containing: your name, Topic your session title containing the following information in plain text. If you have more than one topic proposed, send a separate email. Do not attach Word or other text file documents. Return all fields completed as follows: 1. Your full name, title, and organization 2. Contact information, including your address, email, phone number 3. The name of your proposed session (keep your title simple, interesting, and relevant to the topic) 4. A 75‐200 word overview of your presentation; in addition to the topic, describe whether your presentation is intended as a tutorial, description of an implementation, an theoretical/academic discussion, etc. 5. A 100‐200‐word speaker bio that includes prior conference speaking or related experience To be considered, proposals must be received by 12 Midnight PDT Wednesday, June 23, 2010. Please email any general questions regarding the conference to i...@lucenerevolution.org. To be added to the conference mailing list, please email sig...@lucenerevolution.org. If your organization is interested in sponsorship opportunities, email spon...@lucenerevolution.org. We look forward to seeing you in Boston!
RE: Highlighting is not happening
Hey Daren, Yes the fields for which I am searching are stored and indexed, also they are returned from the query, Also it is not coming, if the entire search keyword is part of the field. Thanks Prakash -Original Message- From: dar...@ontrenet.com [mailto:dar...@ontrenet.com] Sent: Monday, May 24, 2010 9:32 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !-- str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !-- str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Dynamic analyzers
Hi, I have a requirement to dynamically choose a fieldType to analyze text in multiple languages. I will know the language (in a separate field) at index and query time. I've tried implementing this with a custom UpdateRequestProcessorFactory and custom DocumentBuilder.toDocument to change the FieldType, but this dosen't work. I realize I can have e.g. text_en, text_de,... and dynamically populate this with a custom UpdateRequestProcessorFactory, but we are worried with all the languages (lets say 50+) that effectively doing an OR with 50 fields will be a performance issue, is this true? Many thanks in advance, Dan
RE: Solr 1.4 Enterprise Search Server book examples
I have Windows 7 x64. The entire issue is the non-existence of solr/logs folder. Even when I run commands in Administrator mode the folder doesn't get created automagically. After I manually created it, ant index (21 minutes) and java -Xms512M -Xmx1024M -Dfile.encoding=UTF8 -Dsolr.solr.home=cores -Djetty.home=solr -Djetty.logs=solr/logs -jar solr/start.jar both run successfully. --Robert -Original Message- From: David Smiley (@MITRE.org) [mailto:dsmi...@mitre.org] Sent: Friday, May 21, 2010 9:29 PM To: solr-user@lucene.apache.org Subject: RE: Solr 1.4 Enterprise Search Server book examples Hello Rob, Thank you for buying the book. I'm the lead author. There is a README.txt file in the root of the zip which includes a rather full invocation of java to kick off Solr that is to be used for the example data. The options as part of the invocation should elucidate what's going on. The layout of where Solr's home is in relation to where Jetty is does not coincide with a standard Solr distribution's example directory. In hind-site, I should have made it the same so as not to confuse people. Sorry. And I have no idea why the download got corrupted on Packt's server. I made a smaller distribution for them (~127MB vs 300-something) and put the data files on MusicBrainz' servers which are downloaded as part of the setup script you should run. ~ David Smiley - Author: https://www.packtpub.com/solr-1-4-enterprise-search-server/book -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-1-4-Enterprise-Search-Server-boo k-examples-tp756119p835927.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting is not happening
Hi Prakash, more importantly, check the field type and its associated analyzer. In case you use a non-tokenized type (e.g., string), highlighting will not appear if only a partial field match exists (only exact matches, i.e. the query coincides with the field value, will be highlighted). If that's not your intent, you should at least define an tokenizer for the field type. Best, Sascha Doddamani, Prakash wrote: Hey Daren, Yes the fields for which I am searching are stored and indexed, also they are returned from the query, Also it is not coming, if the entire search keyword is part of the field. Thanks Prakash -Original Message- From: dar...@ontrenet.com [mailto:dar...@ontrenet.com] Sent: Monday, May 24, 2010 9:32 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !--str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !--str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
RE: Highlighting is not happening
Thanks Sascha, The type for fields for which I am searching are all text , and I am using solr.TextField fieldType name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.WhitespaceTokenizerFactory/ !-- in this example, we will only use synonyms at query time filter class=solr.SynonymFilterFactory synonyms=index_synonyms.txt ignoreCase=true expand=false/ -- !-- Case insensitive stop word removal. enablePositionIncrements=true ensures that a 'gap' is left to allow for accurate phrase queries. -- filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt enablePositionIncrements=true / filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=1 catenateNumbers=1 catenateAll=0 splitOnCaseChange=1/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.EnglishPorterFilterFactory protected=protwords.txt/ filter class=solr.RemoveDuplicatesTokenFilterFactory/ /analyzer analyzer type=query tokenizer class=solr.WhitespaceTokenizerFactory/ filter class=solr.SynonymFilterFactory synonyms=synonyms.txt ignoreCase=true expand=true/ filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt/ filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=0 catenateNumbers=0 catenateAll=0 splitOnCaseChange=1/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.EnglishPorterFilterFactory protected=protwords.txt/ filter class=solr.RemoveDuplicatesTokenFilterFactory/ /analyzer /fieldType Regards Prakash -Original Message- From: Sascha Szott [mailto:sz...@zib.de] Sent: Monday, May 24, 2010 10:29 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Hi Prakash, more importantly, check the field type and its associated analyzer. In case you use a non-tokenized type (e.g., string), highlighting will not appear if only a partial field match exists (only exact matches, i.e. the query coincides with the field value, will be highlighted). If that's not your intent, you should at least define an tokenizer for the field type. Best, Sascha Doddamani, Prakash wrote: Hey Daren, Yes the fields for which I am searching are stored and indexed, also they are returned from the query, Also it is not coming, if the entire search keyword is part of the field. Thanks Prakash -Original Message- From: dar...@ontrenet.com [mailto:dar...@ontrenet.com] Sent: Monday, May 24, 2010 9:32 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !--str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !--str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Re: Highlighting is not happening
Hi Prakash, can you provide 1. the definition of the relevant field 2. your query 3. the definition of the relevant request handler 4. a field value that is stored in your index and should be highlighted -Sascha Doddamani, Prakash wrote: Thanks Sascha, The type for fields for which I am searching are all text , and I am using solr.TextField fieldType name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.WhitespaceTokenizerFactory/ !-- in this example, we will only use synonyms at query time filter class=solr.SynonymFilterFactory synonyms=index_synonyms.txt ignoreCase=true expand=false/ -- !-- Case insensitive stop word removal. enablePositionIncrements=true ensures that a 'gap' is left to allow for accurate phrase queries. -- filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt enablePositionIncrements=true / filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=1 catenateNumbers=1 catenateAll=0 splitOnCaseChange=1/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.EnglishPorterFilterFactory protected=protwords.txt/ filter class=solr.RemoveDuplicatesTokenFilterFactory/ /analyzer analyzer type=query tokenizer class=solr.WhitespaceTokenizerFactory/ filter class=solr.SynonymFilterFactory synonyms=synonyms.txt ignoreCase=true expand=true/ filter class=solr.StopFilterFactory ignoreCase=true words=stopwords.txt/ filter class=solr.WordDelimiterFilterFactory generateWordParts=1 generateNumberParts=1 catenateWords=0 catenateNumbers=0 catenateAll=0 splitOnCaseChange=1/ filter class=solr.LowerCaseFilterFactory/ filter class=solr.EnglishPorterFilterFactory protected=protwords.txt/ filter class=solr.RemoveDuplicatesTokenFilterFactory/ /analyzer /fieldType Regards Prakash -Original Message- From: Sascha Szott [mailto:sz...@zib.de] Sent: Monday, May 24, 2010 10:29 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Hi Prakash, more importantly, check the field type and its associated analyzer. In case you use a non-tokenized type (e.g., string), highlighting will not appear if only a partial field match exists (only exact matches, i.e. the query coincides with the field value, will be highlighted). If that's not your intent, you should at least define an tokenizer for the field type. Best, Sascha Doddamani, Prakash wrote: Hey Daren, Yes the fields for which I am searching are stored and indexed, also they are returned from the query, Also it is not coming, if the entire search keyword is part of the field. Thanks Prakash -Original Message- From: dar...@ontrenet.com [mailto:dar...@ontrenet.com] Sent: Monday, May 24, 2010 9:32 PM To: solr-user@lucene.apache.org Subject: Re: Highlighting is not happening Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !--str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !--str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below --
Re: IndexSearcher and Caches
Ahh, now I understand. No, you need no second IndexSearcher as long as the Server is alive. You can reuse your searcher for every user. The only commands you are executing per user are those to create a search-query. Kind regards, - Mitch -- View this message in context: http://lucene.472066.n3.nabble.com/IndexSearcher-and-Caches-tp833567p840228.html Sent from the Solr - User mailing list archive at Nabble.com.
getting documents sorted after a faceted search
I apply a faceted search and get document ids from the facet_field I have used. Then I search for these documents given ids. id:(id1 id2 ...) But the order is not predictable. (It applies OR) I do not want to sort documents again. Is there any way to get documents in the given id order? -- Furkan Kuru
Re: IndexSearcher and Caches
Thank you I found the API to get the existing SolrIndexSearcher to be present in SolrCore: SolrCore.getSearcher().get() So if now the Index changes (a commit is done) in between, will I automatically get the new SolrIndexSearcher from this call ? Regards Rahul On Mon, May 24, 2010 at 11:25 PM, MitchK mitc...@web.de wrote: Ahh, now I understand. No, you need no second IndexSearcher as long as the Server is alive. You can reuse your searcher for every user. The only commands you are executing per user are those to create a search-query. Kind regards, - Mitch -- View this message in context: http://lucene.472066.n3.nabble.com/IndexSearcher-and-Caches-tp833567p840228.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: IndexSearcher and Caches
Good question. Well, I never worked productively with SolrJ. But two things: The first: As the documentation says, you *should* get your IndexSearcher from your SolrQueryRequest-object. The second: As a developer of the SolrJ I would do as much as I can automatically behind the curtain. That means that if you do a commit, the index searcher should be automatically renewed. But that's a guess. I can't answer you this question, sorry. Maybe this link helps? http://lucene.472066.n3.nabble.com/Solr-commit-issue-td770315.html#a770453 (searched with the following keywords: solrj commit searcher) I am new to Java and the concept of Java Enterprise Edition's Servlets is not yet fully clear to me. Please, let me ask a question. Let me give you an example: If I use inside my application (it's a Servlet) a SolrServer, I should create him when I start the Servlet. Should I cache the instantiated SolrServer-object with the help of the servlet's cache? And should my cache-implementation should provide a getSolrServer()-method? Maybe this is a question more related to the JavaEE-concept. Thank you. Kind regards - Mitch -- View this message in context: http://lucene.472066.n3.nabble.com/IndexSearcher-and-Caches-tp833567p840479.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Highlighting is not happening
It's not true that highlighting means you have to return the field from the query too. You can specify fl independently of hl.fl Erik On May 24, 2010, at 12:02 PM, dar...@ontrenet.com wrote: Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !-- str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !-- str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Re: Highlighting is not happening
A. Good to know. Thanks! On Mon, 2010-05-24 at 16:34 -0400, Erik Hatcher wrote: It's not true that highlighting means you have to return the field from the query too. You can specify fl independently of hl.fl Erik On May 24, 2010, at 12:02 PM, dar...@ontrenet.com wrote: Check that the field you are highlighting on is stored. It won't work otherwise. Now, this also means that the field is returned from the query. For large text fields to be highlighted only, this means the entire text is returned for each result. There is a pending feature to address this, that allows you to tell Solr to NOT return a specific field (to avoid unecessary transfer of large text fields in this scenario). Darren Hi I am using dismax request handler, I wanted to highlight the search field, So added str name=hltrue/str I was expecting like if I search for keyword Akon resultant docs wherever the Akon is available is bold. But I am not seeing them getting bold, could some one tell me the real path where I should tune If I pass explicitly the hl=true does not work I have added the request handler requestHandler name=dismax class=solr.SearchHandler lst name=defaults str name=defTypedismax/str str name=echoParamsexplicit/str float name=tie0.01/float str name=qf name^20.0 coming^5 playing^4 keywords^0.1 /str str name=bf rord(isclassic)^0.5 ord(listeners)^0.3 /str str name=*,score name, coming, playing, keywords, score /str str name=mm 2lt;-1 5lt;-2 6lt;90% /str int name=ps100/int str name=q.alt*:*/str !-- example highlighter config, enable per-query with hl=true -- str name=hltrue/str !-- str name=hl.simple.preb/str str name=hl.simple.post/b/str -- !-- for this field, we want no fragmenting, just highlighting -- str name=f.name.hl.fragsize0/str !-- instructs Solr to return the field itself if no query terms are found -- !-- str name=f.name.hl.alternateFieldname/str -- str name=f.text.hl.fragmenterregex/str !-- defined below -- /lst /requestHandler regards prakash
Faceted search not working?
I'm trying to perform a faceted search without any luck. Result set doesn't return any facet information... http://localhost:8080/solr/select/?q=title:*facet=onfacet.field=title I'm getting the result set, but no face information present? Is there something else that needs to happen to turn faceting on? I'm using latest Solr 1.4 release. Data is indexed from the database using dataimporter. Thanks. Ilya Sterin
Re: Faceted search not working?
Hi Ilya, Ilya Sterin wrote: I'm trying to perform a faceted search without any luck. Result set doesn't return any facet information... http://localhost:8080/solr/select/?q=title:*facet=onfacet.field=title I'm getting the result set, but no face information present? Is there something else that needs to happen to turn faceting on? No. What does http://localhost:8080/solr/select/?q=title:*fl=titlewt=xml return? -Sascha
Re: Faceted search not working?
it's probably because your query isn't matching anything. try q=title: [* TO *], or maybe even simply *:* if you are trying to match all documents. Erik On May 24, 2010, at 5:11 PM, Ilya Sterin wrote: I'm trying to perform a faceted search without any luck. Result set doesn't return any facet information... http://localhost:8080/solr/select/? q=title:*facet=onfacet.field=title I'm getting the result set, but no face information present? Is there something else that needs to happen to turn faceting on? I'm using latest Solr 1.4 release. Data is indexed from the database using dataimporter. Thanks. Ilya Sterin
Re: Commits, facet autowarming and hung searches
: -Should I be warming the new searcher with a search which uninverts all of : my fields? yes. : -If so, will the warming happen before the new searcher is registered? they will if you do them as part of a newSearcher event in your solrconfig.xml -- the example solrocnfig.xml has some examples using QuerySenderListener to do this. : -Will searches coming in to the index before the new searcher is registered : but after its created go to the old searcher? yes. : -Will my memory usage spike during this interval? it probably already is -- but yes, the warming of hte new Searcher will require it to use a lot of ram to pre-populate these caches data structures, in addition to hte caches and data structures being used by the old searcher still serving queries. -Hoss
Re: query parser for boost query text
: Special characters in the text used for boost queries are not removed. For : example, bq=field1:(what is xyz?)^10 gets parsed into query field1:xyz?10 : (what and is are stop words). Question mark didn't get removed -- field1 : uses standard tokenizer and standard filter, so I expect it to get removed. : When I test it using the analysis page agains field1, question mark did get : removed. unless you are using some special QParser that you haven't mentioned, i supsect that what you are seeing is that since ? is a special character for wildcard searchers, you are still seeing it appear in the debugQuery output showing hte toString of your query. in general, if your analyzer removes punctuation, then the punctuation will be removed -- but the exception is when thta punctuation is a special character for your quer parser -- then it has special meaning. -Hoss
Re: Issues with clustering in multicore
: so i edit the solrconfig.xml in tht file to add clustering information (i : did make sure that the lib declaration points to the correct location). are you sure? ... did you see log messages (prior to the stack trace below) inidcating that the clustering jar was being loaded? SolrResourceLoader logs an info message for every jar it loads, so if you don't see the clustering jar in that list, that's why it can't find it. (and even if you do see it, it might be getting loaded by a differnet core then the one you are trying to use it in, so initializing Solr with a single core first to sanity check may be a good idea) -Hoss
Re: IndexSearcher and Caches
: Thank you I found the API to get the existing SolrIndexSearcher to be : present in SolrCore: : SolrCore.getSearcher().get() I think perhaps you need to take 5 big steps back and explain what your goal is. 99.999% of all solr users should never care about that method -- even the 99.9% of the folks writing java code and using EmbeddedSolr should never ever have a need to call those -- so what exactly is it you are doing, and how did you get along hte path you find yourself on? this thread started with some fairly innoculous questions about how caches worked in regardes to new searchers -- which is all fine and dandy, those concepts that solr users should be aware of ... in the abstract. you should almost never be instantiating those IndexSearchers or Caches yourself. Stick with teh SolrServer abstraction provided by SolrJ... http://wiki.apache.org/solr/Solrj#EmbeddedSolrServer http://lucene.apache.org/solr/api/org/apache/solr/client/solrj/SolrServer.html -Hoss
Re: sort by field length
Are you sure you want to recompute the length when sorting? It's the classic time/space tradeoff, but I'd suggest that when your index is big enough to make taking up some more space a problem, it's far too big to spend the cycles calculating each term length for sorting purposes considering you may be sorting all the terms in your index worst-case. But you could consider payloads for storing the length, although that would still be redundant... Best Erick On Mon, May 24, 2010 at 8:30 AM, Sascha Szott sz...@zib.de wrote: Hi folks, is it possible to sort by field length without having to (redundantly) save the length information in a seperate index field? At first, I thought to accomplish this using a function query, but I couldn't find an appropriate one. Thanks in advance, Sascha
