Re: CommonGrams and SOLR - 1604
Anyone?
On Mon, Jan 17, 2011 at 7:48 PM, Salman Akram
salman.ak...@northbaysolutions.net wrote:
Hi,
I am trying to use CommonGrams with SOLR - 1604 patch but doesn't seem to
work.
If I don't add {!complexphrase} it uses CommonGramsQueryFilterFactory and
proper bi-grams are made but of course doesn't use this patch.
If I add {!complexphrase} it simply does it the old way i.e. ignore
CommonGrams.
Does anyone know how to combine both these features?
Also once they are combined (hopefully they will be) would phrase proximity
search work fine?
Thanks
--
Regards,
Salman Akram
--
Regards,
Salman Akram
Re: NRT
Real NRT in Solr is not implementet yet. But you can configure a near NRT-Search. http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-td1461275.html - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other under 100.000 - Solr1 for Search-Requests - commit every Minute - 4GB Xmx - Solr2 for Update-Request - delta every 2 Minutes - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/NRT-tp2276967p2278477.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: resetting the statistics
why do you dont want to restart ? its a downtime 1 Minute ... !? - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other under 100.000 - Solr1 for Search-Requests - commit every Minute - 4GB Xmx - Solr2 for Update-Request - delta every 2 Minutes - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/resetting-the-statistics-tp2273081p2278496.html Sent from the Solr - User mailing list archive at Nabble.com.
zip in solr
Hallo.. i don't know who i can indexing zip Dokuments, richtext, pdf and office documents works pretty fine, but from the zip files i only get the Name of ziped dokumentds, not the Content. maybe i have to do some other thinks bye indexing zip, but i have read that Tika can read zip and jar and and and.. my configuration is: one PC, with Solr and tika is installed. one other PC as crawler send dokuments with curl like: curl http://192.168.105.66:8983/solr/update/extract?literal.id=zipuprefix=attr_commit=true; -F myfile=@file.zip
Does Solr supports indexing search for Hebrew.
Hello, With reference to below links I haven't found Hebrew support in Solr. http://wiki.apache.org/solr/LanguageAnalysis http://lucene.apache.org/java/3_0_3/api/all/index.html If I want to index and search Hebrew files/data then how would I achieve this? Thanks, Prasad
Re: Does Solr supports indexing search for Hebrew.
You may need to use Hebrew analyzer. http://www.findbestopensource.com/search/?query=hebrew Regards Aditya www.findbestopensource.com On Tue, Jan 18, 2011 at 2:34 PM, prasad deshpande prasad.deshpand...@gmail.com wrote: Hello, With reference to below links I haven't found Hebrew support in Solr. http://wiki.apache.org/solr/LanguageAnalysis http://lucene.apache.org/java/3_0_3/api/all/index.html If I want to index and search Hebrew files/data then how would I achieve this? Thanks, Prasad
Re: sort problem
Thanks for answers, So could i do something like that : fieldType name=string class=solr.TextField sortMissingLast=true omitNorms=true analyzer tokenizer class=solr.KeywordTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ filter class=ISOLatin1AccentFilterFactory/ /analyzer /fieldType ? thanks :) Phil 2011/1/17 Erick Erickson erickerick...@gmail.com Note two things: 1 the lowercasefilter is NOT applied to the STORED data. So the display will still have the original case although the sorting should be what you want. 2 you should NOT be sorting on a tokenized field. Use something like KeywordTokenizer followed by the lowercase filter. String types don't go through filters as I remember. Best Erick On Mon, Jan 17, 2011 at 7:57 AM, Brad Dewar bde...@stfx.ca wrote: Haha, Yes, you're not wrong. The field you are sorting on should be a fieldtype that has the lowercase filter applied. You'll probably have to re-index your data, unless you happen to already have such a field (via copyField, perhaps). Brad -Original Message- From: Salman Akram [mailto:salman.ak...@northbaysolutions.net] Sent: January-17-11 5:47 AM To: solr-user@lucene.apache.org Subject: Re: sort problem Yes. On Mon, Jan 17, 2011 at 2:44 PM, Philippe VINCENT-ROYOL vincent.ro...@gmail.com wrote: Le 17/01/11 10:32, Grijesh a écrit : Use Lowercase filter to lowering your data at both index time and search time it will make case insensitive - Thanx: Grijesh Thanks, so tell me if i m wrong... i need to modify my schema.xml to add lowercase filter and reindex my content? -- Regards, Salman Akram Senior Software Engineer - Tech Lead 80-A, Abu Bakar Block, Garden Town, Pakistan Cell: +92-321-4391210 -- Philippe VINCENT-ROYOL
Re: partitioning documents with fields
Both solutions are working fine for me. I guess the fq performance is slower though, or? Thanks for your feedback. On 1/17/11 7:51 PM, Erick Erickson wrote: As Ahmet says, this is what dismax does. You could also append a filter query (fq=crawl:DIGITALDATA) to your query. eDismax supports fielded queries, see: https://issues.apache.org/jira/browse/SOLR-1553 This is already in the trunk and 3.x code lines I'm pretty sure. Best Erick On Mon, Jan 17, 2011 at 12:15 PM, Claudio Martella claudio.marte...@tis.bz.it wrote: Thanks for your answer. Yes, schema browser shows that the field contains the right values as i expect. From debugQuery=on i see there must be some problem though: str name=rawquerystringcrawl:DIGITALDATA/str str name=querystringcrawl:DIGITALDATA/str str name=parsedquery+DisjunctionMaxQuery((contentEN:crawl (digitaldata crawldigitaldata)^0.8 | title:crawl (digitaldata crawldigitaldata)^1.2 | url:crawl digitaldata^1.5 | contentDE:crawl (digitaldata crawldigitaldata)^0.8 | contentIT:crawl (digitald crawldigitald)^0.8 | anchor:crawl:DIGITALDATA^1.5)~0.1) DisjunctionMaxQuery((contentEN:crawl (digitaldata crawldigitaldata)^0.8 | title:crawl (digitaldata crawldigitaldata)^1.2 | url:crawl digitaldata^1.5 | contentDE:crawl (digitaldata crawldigitaldata)^0.8 | contentIT:crawl (digitald crawldigitald)^0.8 | anchor:crawl:DIGITALDATA^1.5)~0.1)/str str name=parsedquery_toString+(contentEN:crawl (digitaldata crawldigitaldata)^0.8 | title:crawl (digitaldata crawldigitaldata)^1.2 | url:crawl digitaldata^1.5 | contentDE:crawl (digitaldata crawldigitaldata)^0.8 | contentIT:crawl (digitald crawldigitald)^0.8 | anchor:crawl:DIGITALDATA^1.5)~0.1 (contentEN:crawl (digitaldata crawldigitaldata)^0.8 | title:crawl (digitaldata crawldigitaldata)^1.2 | url:crawl digitaldata^1.5 | contentDE:crawl (digitaldata crawldigitaldata)^0.8 | contentIT:crawl (digitald crawldigitald)^0.8 | anchor:crawl:DIGITALDATA^1.5)~0.1/str It looks like there's some problem with my dismax query handler. It doesn't recognize the query with the colon format. Here's the handler definition: requestHandler name=/content class=solr.SearchHandler default=true lst name=defaults str name=defTypedismax/str str name=pftitle^1.2 anchor^1.5 url^1.5 contentEN^0.8 contentIT^0.8 contentDE^0.8/str str name=qftitle^1.2 anchor^1.5 url^1.5 contentEN^0.8 contentIT^0.8 contentDE^0.8/str float name=tie0.1/float bool name=hltrue/bool str name=hl.fltitle url content anchor/str int name=hl.fragsize150/int int name=hl.snippets3/int bool name=hl.mergeContiguoustrue/bool /lst /requestHandler On 1/17/11 6:06 PM, Erick Erickson wrote: String fields are unanalyzed, so case matters. Are you sure you're not using a different case (try KeywordTokenizer + lowercaseFilter if you want these normalized to, say, lower case). If that isn't the problem, could we see the results if you add debugQuery=on to your URL? That often helps diagnose the problem. Take a look at your solr/admin page, schema browser to examine the actual contents of the crawl field and see if they're really what you expect. Best Erick On Mon, Jan 17, 2011 at 11:59 AM, Claudio Martella claudio.marte...@tis.bz.it wrote: Hi, I'm crawling different intranets so i developed a nutch plugin to add a static field for each of these crawls. I do have now in SOLR my documents with their specific craw field. If i search withing solr i can see my documents being returned with that field. The field definition in the schema is: field name=crawl type=string stored=true indexed=true/ I'd like to put a checkbox in my websearch app to choose with partition to search in. So i thought i'd implement it by simply using: /select?indent=onversion=2.2q=crawl%3Avalue+AND+query but nothing is returned. I also just tried crawl:value, which i'd expect to return all the documents from that crawl, but no results are sent back. As the field is indexed and stored and i can see the documents owning that field from normal query results, what could i be missing? -- Claudio Martella Digital Technologies Unit Research Development - Analyst TIS innovation park Via Siemens 19 | Siemensstr. 19 39100 Bolzano | 39100 Bozen Tel. +39 0471 068 123 Fax +39 0471 068 129 claudio.marte...@tis.bz.it http://www.tis.bz.it Short information regarding use of personal data. According to Section 13 of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we process your personal data in order to fulfil contractual and fiscal obligations and also to send you information regarding our services and events. Your personal data are processed with and without electronic means and by respecting data subjects' rights, fundamental freedoms and dignity, particularly with regard to confidentiality, personal identity and the right to personal data protection. At any time and without formalities you can write an
Re: just got 'the book' already have a question
On Mon, Jan 17, 2011 at 11:10 PM, Dennis Gearon gear...@sbcglobal.net wrote: First of all, seems like a good book, Solr-14-Enterprise-Search-Server.pdf Question, is it possible to choose locale at search time? So if my customer is querying across cultural/national/linguistic boundaries and I have the data for him different languages in the same index, can I sort based on his language? http://wiki.apache.org/solr/UnicodeCollation#Sorting_text_for_multiple_languages
Local param tag voodoo ?
Hey,
here are my needs :
- a query that has tagged and untagged contents
- facets that ignore the tagged contents
I tryed :
q=({!tag=toExclude} ignored) taken into account
q={tag=toExclude v='ignored'} take into account
Both resulted in a error.
Is this possible or do I have to try another way ?
Fwd: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
-- Forwarded message --
From: kun xiong xiongku...@gmail.com
Date: 2011/1/18
Subject: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
To: solr-user@lucene.apache.org
Hi all,
I got a ParseException when I query solr with Lucene BooleanQuery
expression (toString()).
I use the default parser : LuceneQParserPlugin,which should support whole
lucene syntax,right?
Java Code:
BooleanQuery bq = new BooleanQuery();
Query q1 = new TermQuery(new Term(I_NAME_ENUM, KFC));
Query q2 = new TermQuery(new Term(I_NAME_ENUM, MCD));
bq.add(q1, Occur.SHOULD);
bq.add(q2, Occur.SHOULD);
bq.setMinimumNumberShouldMatch(1);
String solrQuery = bq.toString();
query string is : q=(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1
Exceptions :
*message* *org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... *
*description* *The request sent by the client was syntactically incorrect
(org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... ).*
*
*
Anyone could help?
Thanks
Kun
*
*
Re: Not storing, but highlighting from document sentences
Hi A simple solution to this could be, for all such searches (foo and bar), search them as it is from 1st(primary index) and while sending these queries to secondary index replace and with or. But in this particular scenario u could also have problem with proximity and phrase queries that is much difficult to tackle. Regards Ahsan From: Otis Gospodnetic otis_gospodne...@yahoo.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:25:12 PM Subject: Re: Not storing, but highlighting from document sentences Hi Tarjei, :) Yeah, that is the solution we are going with, actually. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Tarjei Huse tar...@scanmine.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 1:33:44 AM Subject: Re: Not storing, but highlighting from document sentences On 01/12/2011 12:02 PM, Otis Gospodnetic wrote: Hello, I'm indexing some content (articles) whose text I cannot store in its original form for copyright reason. So I can index the content, but cannot store it. However, I need snippets and search term highlighting. Any way to accomplish this elegantly? Or even not so elegantly? Here is one idea: * Create 2 indices: main index for indexing (but not storing) the original content, the secondary index for storing individual sentences from the original article. How about storing the sentences in the same index in a separate field but with random ordering, would that be ok? Tarjei * That is, before indexing an article, split it into sentences. Then index the article in the main index, and index+store each sentence in the secondary index. So for each doc in the main index there will be multiple docs in the secondary index with individual sentences. Each sentence doc includes an ID of the parent document. * Then run queries against the main index, and pull individual sentences from the secondary index for snippet+highlight purposes. The problem I see with this approach (and there may be other ones that I am not seeing yet) is with queries like foo AND bar. In this case foo may be a match from sentence #1, and bar may be a match from sentence #7. Or maybe foo is a match in sentence #1, and bar is a match in multiple sentences: #7 and #10 and #23. Regardless, when a query is run against the main index, you don't know where the match was, so you don't know which sentences to go get from the secondary index. Does anyone have any suggestions for how to handle this? Thanks, Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ -- Regards / Med vennlig hilsen Tarjei Huse Mobil: 920 63 413
HTTP Status 400 - org.apache.lucene.queryParser.ParseException
Hi all,
I got a ParseException when I query solr with Lucene BooleanQuery
expression (toString()).
I use the default parser : LuceneQParserPlugin,which should support whole
lucene syntax,right?
Java Code:
BooleanQuery bq = new BooleanQuery();
Query q1 = new TermQuery(new Term(I_NAME_ENUM, KFC));
Query q2 = new TermQuery(new Term(I_NAME_ENUM, MCD));
bq.add(q1, Occur.SHOULD);
bq.add(q2, Occur.SHOULD);
bq.setMinimumNumberShouldMatch(1);
String solrQuery = bq.toString();
query string is : q=(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1
Exceptions :
*message* *org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... *
*description* *The request sent by the client was syntactically incorrect
(org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... ).*
*
*
Anyone could help?
Thanks
Kun
*
*
Carrot2 clustering component
Hi, Can anyone help me to solve the error: Class org.carrot2.util.pool.SoftUnboundedPool does not implement the requested interface org.carrot2.util.pool.IParameterizedPool at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:77) at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:62) at org.carrot2.core.ControllerFactory.create(ControllerFactory.java:158) at org.carrot2.core.ControllerFactory.createPooling(ControllerFactory.java:71) at org.apache.solr.handler.clustering.carrot2.CarrotClusteringEngine.init(CarrotClusteringEngine.java:61) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.Class.newInstance0(Class.java:355) at java.lang.Class.newInstance(Class.java:308) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:396) at org.apache.solr.handler.clustering.ClusteringComponent.inform(ClusteringComponent.java:121) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:486) at org.apache.solr.core.SolrCore.init(SolrCore.java:588) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83) at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:99) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:594) at org.mortbay.jetty.servlet.Context.startContext(Context.java:139) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1218) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:500) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:448) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:147) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:147) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117) at org.mortbay.jetty.Server.doStart(Server.java:210) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:929) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.mortbay.start.Main.invokeMain(Main.java:183) at org.mortbay.start.Main.start(Main.java:497) at org.mortbay.start.Main.main(Main.java:115) 18 Jan, 2011 6:03:30 PM org.apache.solr.common.SolrException log SEVERE: java.lang.IncompatibleClassChangeError: Class org.carrot2.util.pool.SoftUnboundedPool does not implement the requested interface org.carrot2.util.pool.IParameterizedPool at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:77) at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:62) at org.carrot2.core.ControllerFactory.create(ControllerFactory.java:158) at org.carrot2.core.ControllerFactory.createPooling(ControllerFactory.java:71) at org.apache.solr.handler.clustering.carrot2.CarrotClusteringEngine.init(CarrotClusteringEngine.java:61) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.Class.newInstance0(Class.java:355) at java.lang.Class.newInstance(Class.java:308) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:396) at org.apache.solr.handler.clustering.ClusteringComponent.inform(ClusteringComponent.java:121) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:486) at
Re: Carrot2 clustering component
Hi, I think the exception is caused by the fact that you're trying to use the latest version of Carrot2 with Solr 1.4.x. There are two alternative solutions here: * as described in http://wiki.apache.org/solr/ClusteringComponent, invoke ant get-libraries to get the compatible JAR files. or * use the latest version of Carrot2 with Solr 1.4.x by installing the compatibility package, documentation is here: http://download.carrot2.org/stable/manual/#section.solr Cheers, Staszek On Tue, Jan 18, 2011 at 13:36, Isha Garg isha.g...@orkash.com wrote: Hi, Can anyone help me to solve the error: Class org.carrot2.util.pool.SoftUnboundedPool does not implement the requested interface org.carrot2.util.pool.IParameterizedPool at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:77) at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:62) at org.carrot2.core.ControllerFactory.create(ControllerFactory.java:158) at org.carrot2.core.ControllerFactory.createPooling(ControllerFactory.java:71) at org.apache.solr.handler.clustering.carrot2.CarrotClusteringEngine.init(CarrotClusteringEngine.java:61) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at java.lang.Class.newInstance0(Class.java:355) at java.lang.Class.newInstance(Class.java:308) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:396) at org.apache.solr.handler.clustering.ClusteringComponent.inform(ClusteringComponent.java:121) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:486) at org.apache.solr.core.SolrCore.init(SolrCore.java:588) at org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:137) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:83) at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:99) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:594) at org.mortbay.jetty.servlet.Context.startContext(Context.java:139) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1218) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:500) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:448) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:147) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:147) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117) at org.mortbay.jetty.Server.doStart(Server.java:210) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40) at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:929) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.mortbay.start.Main.invokeMain(Main.java:183) at org.mortbay.start.Main.start(Main.java:497) at org.mortbay.start.Main.main(Main.java:115) 18 Jan, 2011 6:03:30 PM org.apache.solr.common.SolrException log SEVERE: java.lang.IncompatibleClassChangeError: Class org.carrot2.util.pool.SoftUnboundedPool does not implement the requested interface org.carrot2.util.pool.IParameterizedPool at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:77) at org.carrot2.core.PoolingProcessingComponentManager.init(PoolingProcessingComponentManager.java:62) at org.carrot2.core.ControllerFactory.create(ControllerFactory.java:158) at org.carrot2.core.ControllerFactory.createPooling(ControllerFactory.java:71) at org.apache.solr.handler.clustering.carrot2.CarrotClusteringEngine.init(CarrotClusteringEngine.java:61) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at
Re: Does Solr supports indexing search for Hebrew.
take a look at : http://github.com/synhershko/HebMorph with more info at http://www.code972.com/blog/hebmorph/ On Tue, Jan 18, 2011 at 11:04 AM, prasad deshpande prasad.deshpand...@gmail.com wrote: Hello, With reference to below links I haven't found Hebrew support in Solr. http://wiki.apache.org/solr/LanguageAnalysis http://lucene.apache.org/java/3_0_3/api/all/index.html If I want to index and search Hebrew files/data then how would I achieve this? Thanks, Prasad
Re: Any way to query by offset?
Ahhh, I see. I don't know of any way to do what you want. Best Erick On Mon, Jan 17, 2011 at 7:25 PM, 5 Diamond IT i...@smallbusinessconsultingexperts.com wrote: I want to start at row 1000, 2000, and 3000 and retrieve those 3 rows ONLY from the result set of whatever search was used. Yes, I can do 3 queries, start=1000 and limit 1, etc., but, want ONE query to get those 3 rows from the result set. It's the poor mans way of doing price buckets the way I want them to be. So, what I need that they do not provide is the ability to find those 3 rows out of the result set in one query. Was hoping for a function, a parser that supported this perhaps, some hidden field I am not aware of I could simply match on, any trick that would work. On Jan 17, 2011, at 6:13 PM, Erick Erickson wrote: Have you seen the start and rows parameters? If they don't work, perhaps you could explain what you need that they don't provide. Best Erick On Mon, Jan 17, 2011 at 4:58 PM, 5 Diamond IT i...@smallbusinessconsultingexperts.com wrote: Say I do a query that matches 4000 documents. Is there a query syntax or parser that would allow me to say retrieve offsets 1000, 2000, 3000? I would prefer to not do multiple starts and limit 1's. Thanks in advance. Steve
Re: sort problem
That should work, but do take a look at solr/admin, the schema browser (or use Luke) to verify that what you get is what you expect. Oh, and please don't name it string, it'll cause you endless confusion G... Best Erick On Tue, Jan 18, 2011 at 4:16 AM, Philippe Vincent-Royol vincent.ro...@gmail.com wrote: Thanks for answers, So could i do something like that : fieldType name=string class=solr.TextField sortMissingLast=true omitNorms=true analyzer tokenizer class=solr.KeywordTokenizerFactory/ filter class=solr.LowerCaseFilterFactory/ filter class=ISOLatin1AccentFilterFactory/ /analyzer /fieldType ? thanks :) Phil 2011/1/17 Erick Erickson erickerick...@gmail.com Note two things: 1 the lowercasefilter is NOT applied to the STORED data. So the display will still have the original case although the sorting should be what you want. 2 you should NOT be sorting on a tokenized field. Use something like KeywordTokenizer followed by the lowercase filter. String types don't go through filters as I remember. Best Erick On Mon, Jan 17, 2011 at 7:57 AM, Brad Dewar bde...@stfx.ca wrote: Haha, Yes, you're not wrong. The field you are sorting on should be a fieldtype that has the lowercase filter applied. You'll probably have to re-index your data, unless you happen to already have such a field (via copyField, perhaps). Brad -Original Message- From: Salman Akram [mailto:salman.ak...@northbaysolutions.net] Sent: January-17-11 5:47 AM To: solr-user@lucene.apache.org Subject: Re: sort problem Yes. On Mon, Jan 17, 2011 at 2:44 PM, Philippe VINCENT-ROYOL vincent.ro...@gmail.com wrote: Le 17/01/11 10:32, Grijesh a écrit : Use Lowercase filter to lowering your data at both index time and search time it will make case insensitive - Thanx: Grijesh Thanks, so tell me if i m wrong... i need to modify my schema.xml to add lowercase filter and reindex my content? -- Regards, Salman Akram Senior Software Engineer - Tech Lead 80-A, Abu Bakar Block, Garden Town, Pakistan Cell: +92-321-4391210 -- Philippe VINCENT-ROYOL
missing type check when working with pint field type
Hi folks, I've noticed an unexpected behavior while working with the various built-in integer field types (int, tint, pint). It seems as the first two ones are subject to type checking, while the latter one is not. I'll give you an example based on the example schema that is shipped out with Solr. When trying to index the document doc field name=id1/field field name=foo_iinvalid_value/field field name=foo_ti1/field field name=foo_pi1/field /doc Solr responds with a NumberFormatException (the same holds when setting the value of foo_ti to invalid_value): java.lang.NumberFormatException: For input string: invalid_value Surprisingly, an attempt to index the document doc field name=id1/field field name=foo_i1/field field name=foo_ti1/field field name=foo_piinvalid_value/field /doc is successful. In the end, sorting on foo_pi leads to an exception, e.g., http://localhost:8983/solr/select?q=*:*sort=foo_pi desc raises an HTTP 500 error: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:686) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:234) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:94) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) [...] Is this a bug or did I missed something? -Sascha
Re: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
Why do you want to do this? Because toString has never been
guaranteed to be re-parsable, even in Lucene, so it's not
surprising that taking a Lucene toString() clause and submitting
it to Solr doesn't work.
Best
Erick
On Tue, Jan 18, 2011 at 4:49 AM, kun xiong xiongku...@gmail.com wrote:
-- Forwarded message --
From: kun xiong xiongku...@gmail.com
Date: 2011/1/18
Subject: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
To: solr-user@lucene.apache.org
Hi all,
I got a ParseException when I query solr with Lucene BooleanQuery
expression (toString()).
I use the default parser : LuceneQParserPlugin,which should support whole
lucene syntax,right?
Java Code:
BooleanQuery bq = new BooleanQuery();
Query q1 = new TermQuery(new Term(I_NAME_ENUM, KFC));
Query q2 = new TermQuery(new Term(I_NAME_ENUM, MCD));
bq.add(q1, Occur.SHOULD);
bq.add(q2, Occur.SHOULD);
bq.setMinimumNumberShouldMatch(1);
String solrQuery = bq.toString();
query string is : q=(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1
Exceptions :
*message* *org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... *
*description* *The request sent by the client was syntactically incorrect
(org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... ).*
*
*
Anyone could help?
Thanks
Kun
*
*
Re: missing type check when working with pint field type
I suspect you missed this comment in the schema file: *** Plain numeric field types that store and index the text value verbatim (and hence don't support range queries, since the lexicographic ordering isn't equal to the numeric ordering) *** So what's happening is that the field is being indexed as a text type and, I suspect, begin tokenized. The error you're getting is when trying to sort against a tokenized field which is undefined. At least that's my story and I'm sticking to it Best Erick On Tue, Jan 18, 2011 at 8:10 AM, Sascha Szott sz...@zib.de wrote: Hi folks, I've noticed an unexpected behavior while working with the various built-in integer field types (int, tint, pint). It seems as the first two ones are subject to type checking, while the latter one is not. I'll give you an example based on the example schema that is shipped out with Solr. When trying to index the document doc field name=id1/field field name=foo_iinvalid_value/field field name=foo_ti1/field field name=foo_pi1/field /doc Solr responds with a NumberFormatException (the same holds when setting the value of foo_ti to invalid_value): java.lang.NumberFormatException: For input string: invalid_value Surprisingly, an attempt to index the document doc field name=id1/field field name=foo_i1/field field name=foo_ti1/field field name=foo_piinvalid_value/field /doc is successful. In the end, sorting on foo_pi leads to an exception, e.g., http://localhost:8983/solr/select?q=*:*sort=foo_pi desc raises an HTTP 500 error: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:686) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:234) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:94) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) [...] Is this a bug or did I missed something? -Sascha
Solr Out of Memory Error
Hi all, I got the following error on solr with m/c configuration 4GB RAM and Intel Dual Core Processor.Can you please help me out. java.lang.OutOfMemoryError: Java heap space 2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)||java.lang. OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)|?at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:161)|?at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:139)|?at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:285)|?at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:160)|?at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)|?at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)|?at org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)|?at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|?at org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(BinaryResponseWriter.java:120)|?at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryResponseWriter.java:86)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|?at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:133)|?at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:221)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|?at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:46)|?at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:321)|?at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:253)|?at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)|?at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|?at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)|?at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|?at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|?at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)|?at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)|?at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|?at org.mortbay.jetty.Server.handle(Server.java:326)|?at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)|?at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)|Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded| 2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1024) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:361) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:271) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at
Re: Solr Out of Memory Error
Hi I haven't seen one like this before. Please provide JVM settings and Solr version. Cheers On Tuesday 18 January 2011 15:08:35 Isan Fulia wrote: Hi all, I got the following error on solr with m/c configuration 4GB RAM and Intel Dual Core Processor.Can you please help me out. java.lang.OutOfMemoryError: Java heap space 2011-01-18 18:00:27.655:WARN::Committed before 500 OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)||java.lang. OutOfMemoryError: OutOfMemoryError likely caused by the Sun VM Bug described in https://issues.apache.org/jira/browse/LUCENE-1566; try calling FSDirectory.setReadChunkSize with a a value smaller than the current chunk size (2147483647)|?at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDi rectory.java:161)|?at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.ja va:139)|?at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(Compo undFileReader.java:285)|?at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java: 160)|?at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.jav a:39)|?at org.apache.lucene.store.DataInput.readVInt(DataInput.java:86)|?at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:201)|?at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:828)|?at org.apache.lucene.index.DirectoryReader.document(DirectoryReader.java:579)| ?at org.apache.lucene.index.IndexReader.document(IndexReader.java:755)|?at org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:454)| ?at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:431)|? at org.apache.solr.response.BinaryResponseWriter$Resolver.writeDocList(Binary ResponseWriter.java:120)|?at org.apache.solr.response.BinaryResponseWriter$Resolver.resolve(BinaryRespo nseWriter.java:86)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:143)|? at org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java: 133)|?at org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java: 221)|?at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:138)|? at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:87)|?at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.j ava:46)|?at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilte r.java:321)|?at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.jav a:253)|?at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandl er.java:1157)|?at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)|? at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216 )|?at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)|? at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)|? at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)|?at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCo llection.java:230)|?at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java: 114)|?at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)|? at org.mortbay.jetty.Server.handle(Server.java:326)|?at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)|?at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java :938)|?at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)|?at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)|?at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)|?at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:2 28)|?at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:5 82)|Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded| 2011-01-18 18:00:27.656:WARN::/solr/ProdContentIndex/select java.lang.IllegalStateException: Committed at org.mortbay.jetty.Response.resetBuffer(Response.java:1024) at org.mortbay.jetty.Response.sendError(Response.java:240) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.jav a:361) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java :271) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle r.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at
Re: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
what's the alternative?
--- On Tue, 1/18/11, Erick Erickson erickerick...@gmail.com wrote:
From: Erick Erickson erickerick...@gmail.com
Subject: Re: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
To: solr-user@lucene.apache.org
Date: Tuesday, January 18, 2011, 5:24 AM
Why do you want to do this? Because toString has never been
guaranteed to be re-parsable, even in Lucene, so it's not
surprising that taking a Lucene toString() clause and submitting
it to Solr doesn't work.
Best
Erick
On Tue, Jan 18, 2011 at 4:49 AM, kun xiong xiongku...@gmail.com wrote:
-- Forwarded message --
From: kun xiong xiongku...@gmail.com
Date: 2011/1/18
Subject: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
To: solr-user@lucene.apache.org
Hi all,
I got a ParseException when I query solr with Lucene BooleanQuery
expression (toString()).
I use the default parser : LuceneQParserPlugin,which should support whole
lucene syntax,right?
Java Code:
BooleanQuery bq = new BooleanQuery();
Query q1 = new TermQuery(new Term(I_NAME_ENUM, KFC));
Query q2 = new TermQuery(new Term(I_NAME_ENUM, MCD));
bq.add(q1, Occur.SHOULD);
bq.add(q2, Occur.SHOULD);
bq.setMinimumNumberShouldMatch(1);
String solrQuery = bq.toString();
query string is : q=(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1
Exceptions :
*message* *org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... *
*description* *The request sent by the client was syntactically incorrect
(org.apache.lucene.queryParser.ParseException: Cannot parse
'(I_NAME_ENUM:kfc I_NAME_ENUM:best western)~1': Encountered FUZZY_SLOP
~1 at line 1, column 42. Was expecting one of: EOF AND ... OR ...
NOT ... + ... - ... ( ... * ... ^ ... QUOTED ... TERM ...
PREFIXTERM ... WILDTERM ... [ ... { ... NUMBER ... ).*
*
*
Anyone could help?
Thanks
Kun
*
*
using dismax
Hi, Maybe I'm missing something obvious. I'm trying to use the dismax parser and it doesn't seem like I'm using it properly. When I do this: http://localhost:8080/solr/cs/select?q=(poi_id:3) I get a row returned. When I incorporate dismax and say mm=1, no results get returned. http://localhost:8080/solr/cs/select?q=(poi_id:3)defType=dismaxmm=1 What I wanted to do when I specify mm=1 is to say at least 1 query parameter matches. What am I missing? Thanks, Tri
Re: segment gets corrupted (after background merge ?)
I ran other tests : when I execute the checkIndex on the master I got
random errors, but when I scp the file on another server (same software
exactly) no error occurs...
We will start using another server.
Just one question concerning checkIndex :
What does tokens mean ?
How is it possible that the number of tokens change while the files were
not modified at all ? (this is from the faulty server, on the other
server the tokens do not change at all)
(solr was stopped during the whole checkIndex process)
#diff 20110118_141257_checkIndex.log 20110118_142356_checkIndex.log
15c15
test: terms, freq, prox...OK [5211271 terms; 39824029 terms/docs
pairs; 58236510 tokens]
---
test: terms, freq, prox...OK [5211271 terms; 39824029 terms/docs
pairs; 58236582 tokens]
43c43
test: terms, freq, prox...OK [3947589 terms; 34468256 terms/docs
pairs; 36740496 tokens]
---
test: terms, freq, prox...OK [3947589 terms; 34468256 terms/docs
pairs; 36740533 tokens]
85c85
test: terms, freq, prox...OK [2600874 terms; 21272098 terms/docs
pairs; 10862212 tokens]
---
test: terms, freq, prox...OK [2600874 terms; 21272098 terms/docs
pairs; 10862221 tokens]
Thanks,
Le 14/01/2011 12:59, Michael McCandless a écrit :
Right, but removing a segment out from under a live IW (when you run
CheckIndex with -fix) is deadly, because that other IW doesn't know
you've removed the segment, and will later commit a new segment infos
still referencing that segment.
The nature of this particular exception from CheckIndex is very
strange... I think it can only be a bug in Lucene, a bug in the JRE or
a hardware issue (bits are flipping somewhere).
I don't think an error in the IO system can cause this particular
exception (it would cause others), because the deleted docs are loaded
up front when SegmentReader is init'd...
This is why I'd really like to see if a given corrupt index always
hits precisely the same exception if you run CheckIndex more than
once.
Mike
On Thu, Jan 13, 2011 at 10:56 PM, Lance Norskoggoks...@gmail.com wrote:
1) CheckIndex is not supposed to change a corrupt segment, only remove it.
2) Are you using local hard disks, or do run on a common SAN or remote
file server? I have seen corruption errors on SANs, where existing
files have random changes.
On Thu, Jan 13, 2011 at 11:06 AM, Michael McCandless
luc...@mikemccandless.com wrote:
Generally it's not safe to run CheckIndex if a writer is also open on the index.
It's not safe because CheckIndex could hit FNFE's on opening files,
or, if you use -fix, CheckIndex will change the index out from under
your other IndexWriter (which will then cause other kinds of
corruption).
That said, I don't think the corruption that CheckIndex is detecting
in your index would be caused by having a writer open on the index.
Your first CheckIndex has a different deletes file (_phe_p3.del, with
44824 deleted docs) than the 2nd time you ran it (_phe_p4.del, with
44828 deleted docs), so it must somehow have to do with that change.
One question: if you have a corrupt index, and run CheckIndex on it
several times in a row, does it always fail in the same way? (Ie the
same term hits the below exception).
Is there any way I could get a copy of one of your corrupt cases? I
can then dig...
Mike
On Thu, Jan 13, 2011 at 10:52 AM, Stéphane Delprat
stephane.delp...@blogspirit.com wrote:
I understand less and less what is happening to my solr.
I did a checkIndex (without -fix) and there was an error...
So a did another checkIndex with -fix and then the error was gone. The
segment was alright
During checkIndex I do not shut down the solr server, I just make sure no
client connect to the server.
Should I shut down the solr server during checkIndex ?
first checkIndex :
4 of 17: name=_phe docCount=264148
compound=false
hasProx=true
numFiles=9
size (MB)=928.977
diagnostics = {optimize=false, mergeFactor=10, os.version=2.6.26-2-amd64,
os=Linux, mergeDocStores=true, lucene.version=2.9.3 951790 - 2010-06-06
01:30:55, source=merge, os.arch=amd64, java.version=1.6.0_20,
java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_phe_p3.del]
test: open reader.OK [44824 deleted docs]
test: fields..OK [51 fields]
test: field norms.OK [51 fields]
test: terms, freq, prox...ERROR [term post_id:562 docFreq=1 != num docs
seen 0 + num docs deleted 0]
java.lang.RuntimeException: term post_id:562 docFreq=1 != num docs seen 0 +
num docs deleted 0
at
org.apache.lucene.index.CheckIndex.testTermIndex(CheckIndex.java:675)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:530)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
test: stored fields...OK [7206878 total field count; avg 32.86 fields
per doc]
test: term vectorsOK [0 total vector count; avg 0 term/freq
vector fields per doc]
FAILED
WARNING: fixIndex() would
Re: using dismax
with dismax you must specifiy fields to query upon in the qf parameter and the value for which you want to search through those fields in q. defType=luceneq=poi_id:3 defType=dismaxq=3qf=poi_id See the DisMaxQParser wiki for more On Tuesday 18 January 2011 15:50:34 Tri Nguyen wrote: Hi, Maybe I'm missing something obvious. I'm trying to use the dismax parser and it doesn't seem like I'm using it properly. When I do this: http://localhost:8080/solr/cs/select?q=(poi_id:3) I get a row returned. When I incorporate dismax and say mm=1, no results get returned. http://localhost:8080/solr/cs/select?q=(poi_id:3)defType=dismaxmm=1 What I wanted to do when I specify mm=1 is to say at least 1 query parameter matches. What am I missing? Thanks, Tri -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
sorting based on pivot aggregations
Hi I am using pivots extensively in my search,and they work well for searching and displaying. But I find the need to be able to sort by the sum of a certain pivot, after it is collapsed. So if my pivot term is:student_id,test_grade I'd want to be able to sort on the number of tests a student has taken. and also get an average. something like: :sort = sum( student_id,test_grade )/ count( student_id,test_grade ) where the values would be summed and counted over all of the entries where student_id was the same. It doesn't seem to be working now, but does seem like a feature people would like. Is there any plan to implement it? Or even better a patch that I could apply? thanks in advance steve
Re: NRT
near Near Real Time? Is that even less real time than NRT? --wunder On Jan 18, 2011, at 12:34 AM, stockii wrote: Real NRT in Solr is not implementet yet. But you can configure a near NRT-Search. http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-td1461275.html - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other under 100.000 - Solr1 for Search-Requests - commit every Minute - 4GB Xmx - Solr2 for Update-Request - delta every 2 Minutes - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/NRT-tp2276967p2278477.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: missing type check when working with pint field type
Hi Erick, I see the point. But what is pint (plong, pfloat, pdouble) actually intended for (sorting is not possible, no type checking is performed)? Seems to me as it is something very similar to the string type (both store and index the value verbatim). -Sascha On 18.01.2011 14:38, Erick Erickson wrote: I suspect you missed this comment in the schema file: *** Plain numeric field types that store and index the text value verbatim (and hence don't support range queries, since the lexicographic ordering isn't equal to the numeric ordering) *** So what's happening is that the field is being indexed as a text type and, I suspect, begin tokenized. The error you're getting is when trying to sort against a tokenized field which is undefined. At least that's my story and I'm sticking to it Best Erick On Tue, Jan 18, 2011 at 8:10 AM, Sascha Szottsz...@zib.de wrote: Hi folks, I've noticed an unexpected behavior while working with the various built-in integer field types (int, tint, pint). It seems as the first two ones are subject to type checking, while the latter one is not. I'll give you an example based on the example schema that is shipped out with Solr. When trying to index the document doc field name=id1/field field name=foo_iinvalid_value/field field name=foo_ti1/field field name=foo_pi1/field /doc Solr responds with a NumberFormatException (the same holds when setting the value of foo_ti to invalid_value): java.lang.NumberFormatException: For input string: invalid_value Surprisingly, an attempt to index the document doc field name=id1/field field name=foo_i1/field field name=foo_ti1/field field name=foo_piinvalid_value/field /doc is successful. In the end, sorting on foo_pi leads to an exception, e.g., http://localhost:8983/solr/select?q=*:*sort=foo_pi desc raises an HTTP 500 error: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:686) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:234) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:94) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) [...] Is this a bug or did I missed something? -Sascha -- Sascha Szott :: KOBV/ZIB :: sz...@zib.de :: +49 30 84185-457
Re: sorting based on pivot aggregations
So if my pivot term is:student_id,test_grade I'd want to be able to sort on the number of tests a student has taken. and also get an average. something like: :sort = sum( student_id,test_grade )/ count( student_id,test_grade ) where the values would be summed and counted over all of the entries where student_id was the same. It doesn't seem to be working now, but does seem like a feature people would like. Is there any plan to implement it? Or even better a patch that I could apply? http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function could help?
Re: partitioning documents with fields
Both solutions are working fine for me. I guess the fq performance is slower though, or? http://wiki.apache.org/solr/FilterQueryGuidance
Re: HTTP Status 400 - org.apache.lucene.queryParser.ParseException
what's the alternative? q=kfc+mdcdefType=dismaxmm=1qf=I_NAME_ENUM See more: http://wiki.apache.org/solr/DisMaxQParserPlugin
Example of DisMax query with embeded Solr (using SolrJ)
Hi, Is there an example of how to use dismax with embedded Solr?I am currently creating my query like this: QueryParser parser = new QueryParser(Version.LUCENE_CURRENT,content, new StandardAnalyzer(Version.LUCENE_CURRENT)); Query q = parser.parse(query); searcher.search(q, end); How would this look with DisMax query? Any help would be appreciated. thx!
Re: [sfield] Missing in Spatial Search
What version of Solr are you on?
On Jan 13, 2011, at 8:23 PM, Adam Estrada wrote:
According to the documentation here:
http://wiki.apache.org/solr/SpatialSearch the field that identifies the
spatial point data is sfield. See the console output below.
Jan 13, 2011 6:49:40 PM org.apache.solr.core.SolrCore execute
INFO: [] webapp=/solr path=/select
params={spellcheck=truef.jtype.facet.mincoun
t=1facet=truef.cat.facet.mincount=1f.cause.facet.mincount=1f.urgency.facet.m
incount=1rows=10start=0q=*:*f.status.facet.mincount=1facet.field=catfacet.
field=jtypefacet.field=statusfacet.field=causefacet.field=urgency?=fq={!typ
e%3Dgeofilt+pt%3D39.0914154052734,-84.517822265625+sfield%3Dcoords+d%3D300}text:
} hits=113 status=0 QTime=1
Jan 13, 2011 6:51:51 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: missing sfield for spatial
reques
t
Any ideas on this one?
Thanks in advance,
Adam
--
Grant Ingersoll
http://www.lucidimagination.com
Re: latest patches and big picture of search grouping
Hi Marc, Have you looked at the grouping stuff that has been committed? http://wiki.apache.org/solr/FieldCollapsing -Grant On Jan 17, 2011, at 5:11 AM, Marc Sturlese wrote: I need to dive into search grouping / field collapsing again. I've seen there are lot's of issues about it now. Can someone point me to the minimum patches I need to run this feature in trunk? I want to see the code of the most optimised version and what's being done in distributed search. I think I need this: https://issues.apache.org/jira/browse/SOLR-2068 https://issues.apache.org/jira/browse/SOLR-2205 https://issues.apache.org/jira/browse/SOLR-2066 But not sure if I am missing anything else. By the way, I think the current implementation of group searching is totally different that what it was before when you could choose normal or adjacent collapse. Can someone give me a quick big picture of the current implementation (I will trace the code anyway, but it's just to get an idea). Is there still a double trip? Thanks in advance. -- View this message in context: http://lucene.472066.n3.nabble.com/latest-patches-and-big-picture-of-search-grouping-tp2271383p2271383.html Sent from the Solr - User mailing list archive at Nabble.com. -- Grant Ingersoll http://www.lucidimagination.com
Re: Example of DisMax query with embeded Solr (using SolrJ)
Is there an example of how to use dismax with embedded Solr?I am currently creating my query like this: QueryParser parser = new QueryParser(Version.LUCENE_CURRENT,content, new StandardAnalyzer(Version.LUCENE_CURRENT)); Query q = parser.parse(query); searcher.search(q, end); How would this look with DisMax query? Any help would be appreciated. Every valid solr search URL can be converted into SolrQuery. Some of the parameters have special named methods, some does not. If you have a separate handler that uses dismax as a defType you can use simply: SolrQuery q = new SolrQuery(); q.setQueryType(dismax); // this roughly equivalent of q.set(qt, dismax) It does not have a special method to set query parser but you can use q.set(defType,dismax) instead. You can think SolrQuery as a something that holds key-value pairs.
Search on two core and two schema
Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Search on two core and two schema
Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontaine dfonta...@rosebud.frwrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: NRT
Thanks Otis Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Otis Gospodnetic otis_gospodne...@yahoo.com To: solr-user@lucene.apache.org Sent: Mon, January 17, 2011 11:15:23 PM Subject: Re: NRT Hi, How is NRT doing, being used in production? Which Solr is it in? Unless I missed it, I don't think there is true NRT in Solr just yet. And is there built in Spatial in that version? How is Solr 4.x doing? Well :) 3 ways to know this sort of stuff: * follow the dev list - high volume * subscribe to Sematext Blog - we publish monthly Solr Digests * check JIRA to see how many issues remain to be fixed Otis -- Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/
Re: just got 'the book' already have a question
Thanks Robert. Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Robert Muir rcm...@gmail.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 1:40:25 AM Subject: Re: just got 'the book' already have a question On Mon, Jan 17, 2011 at 11:10 PM, Dennis Gearon gear...@sbcglobal.net wrote: First of all, seems like a good book, Solr-14-Enterprise-Search-Server.pdf Question, is it possible to choose locale at search time? So if my customer is querying across cultural/national/linguistic boundaries and I have the data for him different languages in the same index, can I sort based on his language? http://wiki.apache.org/solr/UnicodeCollation#Sorting_text_for_multiple_languages
Re: Search on two core and two schema
On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.frwrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Does Solr supports indexing search for Hebrew.
Thanks Ofer :-) Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Ofer Fort o...@tra.cx To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 4:55:53 AM Subject: Re: Does Solr supports indexing search for Hebrew. take a look at : http://github.com/synhershko/HebMorph with more info at http://www.code972.com/blog/hebmorph/ On Tue, Jan 18, 2011 at 11:04 AM, prasad deshpande prasad.deshpand...@gmail.com wrote: Hello, With reference to below links I haven't found Hebrew support in Solr. http://wiki.apache.org/solr/LanguageAnalysis http://lucene.apache.org/java/3_0_3/api/all/index.html If I want to index and search Hebrew files/data then how would I achieve this? Thanks, Prasad
Re: segment gets corrupted (after background merge ?)
OK thanks for bringing closure!
The tokens output is the total number of indexed tokens (ie, as if
you had a counter that counted all tokens produced by analysis as the
indexer consumes them).
My guess is the faulty server's hardware problem also messed up this count?
Mike
On Tue, Jan 18, 2011 at 9:57 AM, Stéphane Delprat
stephane.delp...@blogspirit.com wrote:
I ran other tests : when I execute the checkIndex on the master I got random
errors, but when I scp the file on another server (same software exactly) no
error occurs...
We will start using another server.
Just one question concerning checkIndex :
What does tokens mean ?
How is it possible that the number of tokens change while the files were not
modified at all ? (this is from the faulty server, on the other server the
tokens do not change at all)
(solr was stopped during the whole checkIndex process)
#diff 20110118_141257_checkIndex.log 20110118_142356_checkIndex.log
15c15
test: terms, freq, prox...OK [5211271 terms; 39824029 terms/docs
pairs; 58236510 tokens]
---
test: terms, freq, prox...OK [5211271 terms; 39824029 terms/docs
pairs; 58236582 tokens]
43c43
test: terms, freq, prox...OK [3947589 terms; 34468256 terms/docs
pairs; 36740496 tokens]
---
test: terms, freq, prox...OK [3947589 terms; 34468256 terms/docs
pairs; 36740533 tokens]
85c85
test: terms, freq, prox...OK [2600874 terms; 21272098 terms/docs
pairs; 10862212 tokens]
---
test: terms, freq, prox...OK [2600874 terms; 21272098 terms/docs
pairs; 10862221 tokens]
Thanks,
Le 14/01/2011 12:59, Michael McCandless a écrit :
Right, but removing a segment out from under a live IW (when you run
CheckIndex with -fix) is deadly, because that other IW doesn't know
you've removed the segment, and will later commit a new segment infos
still referencing that segment.
The nature of this particular exception from CheckIndex is very
strange... I think it can only be a bug in Lucene, a bug in the JRE or
a hardware issue (bits are flipping somewhere).
I don't think an error in the IO system can cause this particular
exception (it would cause others), because the deleted docs are loaded
up front when SegmentReader is init'd...
This is why I'd really like to see if a given corrupt index always
hits precisely the same exception if you run CheckIndex more than
once.
Mike
On Thu, Jan 13, 2011 at 10:56 PM, Lance Norskoggoks...@gmail.com wrote:
1) CheckIndex is not supposed to change a corrupt segment, only remove
it.
2) Are you using local hard disks, or do run on a common SAN or remote
file server? I have seen corruption errors on SANs, where existing
files have random changes.
On Thu, Jan 13, 2011 at 11:06 AM, Michael McCandless
luc...@mikemccandless.com wrote:
Generally it's not safe to run CheckIndex if a writer is also open on
the index.
It's not safe because CheckIndex could hit FNFE's on opening files,
or, if you use -fix, CheckIndex will change the index out from under
your other IndexWriter (which will then cause other kinds of
corruption).
That said, I don't think the corruption that CheckIndex is detecting
in your index would be caused by having a writer open on the index.
Your first CheckIndex has a different deletes file (_phe_p3.del, with
44824 deleted docs) than the 2nd time you ran it (_phe_p4.del, with
44828 deleted docs), so it must somehow have to do with that change.
One question: if you have a corrupt index, and run CheckIndex on it
several times in a row, does it always fail in the same way? (Ie the
same term hits the below exception).
Is there any way I could get a copy of one of your corrupt cases? I
can then dig...
Mike
On Thu, Jan 13, 2011 at 10:52 AM, Stéphane Delprat
stephane.delp...@blogspirit.com wrote:
I understand less and less what is happening to my solr.
I did a checkIndex (without -fix) and there was an error...
So a did another checkIndex with -fix and then the error was gone. The
segment was alright
During checkIndex I do not shut down the solr server, I just make sure
no
client connect to the server.
Should I shut down the solr server during checkIndex ?
first checkIndex :
4 of 17: name=_phe docCount=264148
compound=false
hasProx=true
numFiles=9
size (MB)=928.977
diagnostics = {optimize=false, mergeFactor=10,
os.version=2.6.26-2-amd64,
os=Linux, mergeDocStores=true, lucene.version=2.9.3 951790 - 2010-06-06
01:30:55, source=merge, os.arch=amd64, java.version=1.6.0_20,
java.vendor=Sun Microsystems Inc.}
has deletions [delFileName=_phe_p3.del]
test: open reader.OK [44824 deleted docs]
test: fields..OK [51 fields]
test: field norms.OK [51 fields]
test: terms, freq, prox...ERROR [term post_id:562 docFreq=1 != num
docs
seen 0 + num docs deleted 0]
java.lang.RuntimeException: term post_id:562 docFreq=1 != num docs seen
0 +
num docs deleted 0
at
Re: using dismax
If you're trying to get to a dismax parser (named dismax in solrconfig.xml), you need to specify qt=dismax. NOTE: the Wiki is a bit confusing on this point, the fact that the dismax parser is *named* dismax in the solrconfig.xml file is coincidence, you could name it erick and specify qt=erick and get there. Markus' comments are if you're specifying a dismax parser entirely on the request line, not getting to the one you may have defined in solrconfig.xml. Both are entirely valid ways of using dismax Best Erick On Tue, Jan 18, 2011 at 9:50 AM, Tri Nguyen tringuye...@yahoo.com wrote: Hi, Maybe I'm missing something obvious. I'm trying to use the dismax parser and it doesn't seem like I'm using it properly. When I do this: http://localhost:8080/solr/cs/select?q=(poi_id:3) I get a row returned. When I incorporate dismax and say mm=1, no results get returned. http://localhost:8080/solr/cs/select?q=(poi_id:3)defType=dismaxmm=1 What I wanted to do when I specify mm=1 is to say at least 1 query parameter matches. What am I missing? Thanks, Tri
Re: missing type check when working with pint field type
These are legacy types that aren't, frankly, very useful in recent Solr. So you can probably safely ignore them. BTW, you probably want to go with Trie fields (tint, tfloat, etc) as a first choice unless you have a definite reason not to. Hope this helps Erick On Tue, Jan 18, 2011 at 10:35 AM, Sascha Szott sz...@zib.de wrote: Hi Erick, I see the point. But what is pint (plong, pfloat, pdouble) actually intended for (sorting is not possible, no type checking is performed)? Seems to me as it is something very similar to the string type (both store and index the value verbatim). -Sascha On 18.01.2011 14:38, Erick Erickson wrote: I suspect you missed this comment in the schema file: *** Plain numeric field types that store and index the text value verbatim (and hence don't support range queries, since the lexicographic ordering isn't equal to the numeric ordering) *** So what's happening is that the field is being indexed as a text type and, I suspect, begin tokenized. The error you're getting is when trying to sort against a tokenized field which is undefined. At least that's my story and I'm sticking to it Best Erick On Tue, Jan 18, 2011 at 8:10 AM, Sascha Szottsz...@zib.de wrote: Hi folks, I've noticed an unexpected behavior while working with the various built-in integer field types (int, tint, pint). It seems as the first two ones are subject to type checking, while the latter one is not. I'll give you an example based on the example schema that is shipped out with Solr. When trying to index the document doc field name=id1/field field name=foo_iinvalid_value/field field name=foo_ti1/field field name=foo_pi1/field /doc Solr responds with a NumberFormatException (the same holds when setting the value of foo_ti to invalid_value): java.lang.NumberFormatException: For input string: invalid_value Surprisingly, an attempt to index the document doc field name=id1/field field name=foo_i1/field field name=foo_ti1/field field name=foo_piinvalid_value/field /doc is successful. In the end, sorting on foo_pi leads to an exception, e.g., http://localhost:8983/solr/select?q=*:*sort=foo_pi desc raises an HTTP 500 error: java.lang.StringIndexOutOfBoundsException: String index out of range: 0 at java.lang.String.charAt(String.java:686) at org.apache.lucene.search.FieldCache$7.parseInt(FieldCache.java:234) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:457) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldCacheImpl$IntCache.createValue(FieldCacheImpl.java:447) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:224) at org.apache.lucene.search.FieldCacheImpl.getInts(FieldCacheImpl.java:430) at org.apache.lucene.search.FieldComparator$IntComparator.setNextReader(FieldComparator.java:332) at org.apache.lucene.search.TopFieldCollector$OneComparatorNonScoringCollector.setNextReader(TopFieldCollector.java:94) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:249) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) [...] Is this a bug or did I missed something? -Sascha -- Sascha Szott :: KOBV/ZIB :: sz...@zib.de :: +49 30 84185-457
Re: using dismax
Erick, The wt parameter does not specifiy the parser but the request handler to use. Except the confusion between parser and request handler you're entirely right. Cheers On Tuesday 18 January 2011 17:37:41 Erick Erickson wrote: If you're trying to get to a dismax parser (named dismax in solrconfig.xml), you need to specify qt=dismax. NOTE: the Wiki is a bit confusing on this point, the fact that the dismax parser is *named* dismax in the solrconfig.xml file is coincidence, you could name it erick and specify qt=erick and get there. Markus' comments are if you're specifying a dismax parser entirely on the request line, not getting to the one you may have defined in solrconfig.xml. Both are entirely valid ways of using dismax Best Erick On Tue, Jan 18, 2011 at 9:50 AM, Tri Nguyen tringuye...@yahoo.com wrote: Hi, Maybe I'm missing something obvious. I'm trying to use the dismax parser and it doesn't seem like I'm using it properly. When I do this: http://localhost:8080/solr/cs/select?q=(poi_id:3) I get a row returned. When I incorporate dismax and say mm=1, no results get returned. http://localhost:8080/solr/cs/select?q=(poi_id:3)defType=dismaxmm=1 What I wanted to do when I specify mm=1 is to say at least 1 query parameter matches. What am I missing? Thanks, Tri -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
Re: Search on two core and two schema
Okay .. and .. now .. you're trying to do what? perhaps you could give us an example, w/ real data .. sample queries - results. because actually i cannot imagine what you want to achieve, sorry On Tue, Jan 18, 2011 at 5:24 PM, Damien Fontaine dfonta...@rosebud.frwrote: On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.fr wrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Search on two core and two schema
I want execute this query : Schema 1 : field name=id type=string indexed=true stored=true required=true / field name=title type=string indexed=true stored=true required=true / field name=UUID_location type=string indexed=true stored=true required=true / Schema 2 : field name=UUID_location type=string indexed=true stored=true required=true / field name=label type=string indexed=true stored=true required=true / field name=type type=string indexed=true stored=true required=true / Query : select?facet=truefl=titleq=title:*facet.field=UUID_locationrows=10qt=standard Result : ?xml version=1.0 encoding=UTF-8? response lst name=responseHeader int name=status0/int int name=QTime0/int lst name=params str name=facettrue/str str name=fltitle/str str name=qtitle:*/str str name=facet.fieldUUID_location/str str name=qtstandard/str /lst /lst result name=response numFound=1889 start=0 doc str name=titletitre 1/str /doc doc str name=titleTitre 2/str /doc /result lst name=facet_counts lst name=facet_queries/ lst name=facet_fields lst name=UUID_location int name=Japan998/int int name=China891/int /lst /lst lst name=facet_dates/ /lst /response Le 18/01/2011 17:55, Stefan Matheis a écrit : Okay .. and .. now .. you're trying to do what? perhaps you could give us an example, w/ real data .. sample queries - results. because actually i cannot imagine what you want to achieve, sorry On Tue, Jan 18, 2011 at 5:24 PM, Damien Fontainedfonta...@rosebud.frwrote: On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.fr wrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Search on two core and two schema
Solr can't do that. Two cores are two seperate cores, you have to do two seperate queries, and get two seperate result sets. Solr is not an rdbms. On 1/18/2011 12:24 PM, Damien Fontaine wrote: I want execute this query : Schema 1 : field name=id type=string indexed=true stored=true required=true / field name=title type=string indexed=true stored=true required=true / field name=UUID_location type=string indexed=true stored=true required=true / Schema 2 : field name=UUID_location type=string indexed=true stored=true required=true / field name=label type=string indexed=true stored=true required=true / field name=type type=string indexed=true stored=true required=true / Query : select?facet=truefl=titleq=title:*facet.field=UUID_locationrows=10qt=standard Result : ?xml version=1.0 encoding=UTF-8? response lst name=responseHeader int name=status0/int int name=QTime0/int lst name=params str name=facettrue/str str name=fltitle/str str name=qtitle:*/str str name=facet.fieldUUID_location/str str name=qtstandard/str /lst /lst result name=response numFound=1889 start=0 doc str name=titletitre 1/str /doc doc str name=titleTitre 2/str /doc /result lst name=facet_counts lst name=facet_queries/ lst name=facet_fields lst name=UUID_location int name=Japan998/int int name=China891/int /lst /lst lst name=facet_dates/ /lst /response Le 18/01/2011 17:55, Stefan Matheis a écrit : Okay .. and .. now .. you're trying to do what? perhaps you could give us an example, w/ real data .. sample queries - results. because actually i cannot imagine what you want to achieve, sorry On Tue, Jan 18, 2011 at 5:24 PM, Damien Fontainedfonta...@rosebud.frwrote: On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.fr wrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
salvaging uncommitted data
Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Indexing and Searching Chinese
Hi, all, Now I cannot search the index when querying with Chinese keywords. Before using Solr, I ever used Lucene for some time. Since I need to crawl some Chinese sites, I use ChineseAnalyzer in the code to run Lucene. I know Solr is a server for Lucene. However, I have no idea know how to configure the analyzer in Solr? I appreciate so much for your help! Best, LB
Re: Search on two core and two schema
Le 18/01/2011 18:31, Jonathan Rochkind a écrit : Solr can't do that. Two cores are two seperate cores, you have to do two seperate queries, and get two seperate result sets. Solr is not an rdbms. Yes Solr can't do that but if i want this : 1. Core 1 call Core 2 to get the label 2. Core 1 use the Core 2 results in this own search 3. Core 1 map this own results and Core 2 results I believe that a plugin, patch ... is necessary. How do i proceed ? I must program a new handler ? On 1/18/2011 12:24 PM, Damien Fontaine wrote: I want execute this query : Schema 1 : field name=id type=string indexed=true stored=true required=true / field name=title type=string indexed=true stored=true required=true / field name=UUID_location type=string indexed=true stored=true required=true / Schema 2 : field name=UUID_location type=string indexed=true stored=true required=true / field name=label type=string indexed=true stored=true required=true / field name=type type=string indexed=true stored=true required=true / Query : select?facet=truefl=titleq=title:*facet.field=UUID_locationrows=10qt=standard Result : ?xml version=1.0 encoding=UTF-8? response lst name=responseHeader int name=status0/int int name=QTime0/int lst name=params str name=facettrue/str str name=fltitle/str str name=qtitle:*/str str name=facet.fieldUUID_location/str str name=qtstandard/str /lst /lst result name=response numFound=1889 start=0 doc str name=titletitre 1/str /doc doc str name=titleTitre 2/str /doc /result lst name=facet_counts lst name=facet_queries/ lst name=facet_fields lst name=UUID_location int name=Japan998/int int name=China891/int /lst /lst lst name=facet_dates/ /lst /response Le 18/01/2011 17:55, Stefan Matheis a écrit : Okay .. and .. now .. you're trying to do what? perhaps you could give us an example, w/ real data .. sample queries - results. because actually i cannot imagine what you want to achieve, sorry On Tue, Jan 18, 2011 at 5:24 PM, Damien Fontainedfonta...@rosebud.frwrote: On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.fr wrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Search on two core and two schema
Schemas are very differents, i can't group them. In contrast to what you're saying above, you may rethink the option of combining both type of documents in a single core. It's a perfectly valid approach to combine heteregenous documents in a single core in Solr. (and use a specific field -say 'type'- to distinguish between them when needed) Geert-Jan 2011/1/18 Jonathan Rochkind rochk...@jhu.edu Solr can't do that. Two cores are two seperate cores, you have to do two seperate queries, and get two seperate result sets. Solr is not an rdbms. On 1/18/2011 12:24 PM, Damien Fontaine wrote: I want execute this query : Schema 1 : field name=id type=string indexed=true stored=true required=true / field name=title type=string indexed=true stored=true required=true / field name=UUID_location type=string indexed=true stored=true required=true / Schema 2 : field name=UUID_location type=string indexed=true stored=true required=true / field name=label type=string indexed=true stored=true required=true / field name=type type=string indexed=true stored=true required=true / Query : select?facet=truefl=titleq=title:*facet.field=UUID_locationrows=10qt=standard Result : ?xml version=1.0 encoding=UTF-8? response lst name=responseHeader int name=status0/int int name=QTime0/int lst name=params str name=facettrue/str str name=fltitle/str str name=qtitle:*/str str name=facet.fieldUUID_location/str str name=qtstandard/str /lst /lst result name=response numFound=1889 start=0 doc str name=titletitre 1/str /doc doc str name=titleTitre 2/str /doc /result lst name=facet_counts lst name=facet_queries/ lst name=facet_fields lst name=UUID_location int name=Japan998/int int name=China891/int /lst /lst lst name=facet_dates/ /lst /response Le 18/01/2011 17:55, Stefan Matheis a écrit : Okay .. and .. now .. you're trying to do what? perhaps you could give us an example, w/ real data .. sample queries - results. because actually i cannot imagine what you want to achieve, sorry On Tue, Jan 18, 2011 at 5:24 PM, Damien Fontainedfonta...@rosebud.fr wrote: On my first schema, there are informations about a document like title, lead, text etc and many UUID(each UUID is a taxon's ID) My second schema contains my taxonomies with auto-complete and facets. Le 18/01/2011 17:06, Stefan Matheis a écrit : Search on two cores but combine the results afterwards to present them in one group, or what exactly are you trying to do Damien? On Tue, Jan 18, 2011 at 5:04 PM, Damien Fontainedfonta...@rosebud.fr wrote: Hi, I would like make a search on two core with differents schemas. Sample : Schema Core1 - ID - Label - IDTaxon ... Schema Core2 - IDTaxon - Label - Hierarchy ... Schemas are very differents, i can't group them. Have you an idea to realize this search ? Thanks, Damien
Re: Does Solr supports indexing search for Hebrew.
Whoops, picked the wrong email to reply thanks to. Wasn't actually in this thread. Dennis Gearon - Original Message From: Dennis Gearon gear...@sbcglobal.net To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 8:25:04 AM Subject: Re: Does Solr supports indexing search for Hebrew. Thanks Ofer :-) Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Ofer Fort o...@tra.cx To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 4:55:53 AM Subject: Re: Does Solr supports indexing search for Hebrew. take a look at : http://github.com/synhershko/HebMorph with more info at http://www.code972.com/blog/hebmorph/ On Tue, Jan 18, 2011 at 11:04 AM, prasad deshpande prasad.deshpand...@gmail.com wrote: Hello, With reference to below links I haven't found Hebrew support in Solr. http://wiki.apache.org/solr/LanguageAnalysis http://lucene.apache.org/java/3_0_3/api/all/index.html If I want to index and search Hebrew files/data then how would I achieve this? Thanks, Prasad
Solr UUID field for externally generated UUIDs
I would like to use the following field declaration to store my own, COMB UUIDs, (same length and format, a kind of cross between version 1 and version 4). If I leave out default value in the declaration, would that work? I.E.: fieldType name=id_uuid class=solr.UUIDField indexed=true required=true/ The above won't generate a UUID on it's own, right? Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die.
Re: what would cause large numbers of executeWithRetry INFO messages?
sorry, never did find a solution to that. if you do happen to figure it out, pls post a reply to this thread. thanks -- View this message in context: http://lucene.472066.n3.nabble.com/what-would-cause-large-numbers-of-executeWithRetry-INFO-messages-tp1453417p2281087.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: what would cause large numbers of executeWithRetry INFO messages?
Hi, This is a slave polling the master for its index version but it seems the master fails to respond. From the javadoc: public class NoHttpResponseException extends IOException Signals that the target server failed to respond with a valid HTTP response. Cheers, I see a large number (~1000) of the following executeWithRetry messages in my apache catalina log files every day (see bolded snippet below). They seem to appear at random intervals. Since they are not flagged as errors or warnings, I have been ignoring them for now. However, I started wondering if INFO message is a red-herring and thinking there might be an actual problem somewhere. Does anyone know what would cause this type of message? Are they normal? I have not seen anything in my google searches for solr that contain this message Details: 1. My CPU usage seems fine as does my heap; we have lots of cpu capacity and heap space 2. The log is from a searcher but I know that the intervals do not correspond to replication (every 15 min on the hour) 3. the INFO lines appear in all searcher logs (we have a number of searchers) 4. the data is around 10m records per searcher and occupies around 14gb 5. I am not noticing any problems performing queries on the solr (so no trace info to give you); performance and queries seem fine Log snippet: Sep 10, 2010 2:17:59 AM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave in sync with master. Sep 10, 2010 2:18:20 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server xxx.admin.inf failed to respond Sep 10, 2010 2:18:20 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Sep 10, 2010 2:18:20 AM org.apache.solr.handler.SnapPuller fetchLatestIndex INFO: Slave in sync with master. any info appreciated. thx
Re: what would cause large numbers of executeWithRetry INFO messages?
Oh, and this should not have the INFO level in my opinion. Other log lines indicating a problem with the master (such as a time out or unreachable host) are not flagged as INFO. Maybe you could file a Jira ticket? Don't forget to specifiy your Solr version. Also, please check the master log file if there something funny around the time the slave throws this error. Since they are not flagged as errors or warnings, I have been ignoring them for now. However, I started wondering if INFO message is a red-herring and thinking there might be an actual problem somewhere.
Indexing and Searching Chinese with SolrNet
Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: Indexing and Searching Chinese
Bing Li, You can configure different analyzers in your Solr's schema.xml. Have a look at the example Solr schema.xml to see how that's done. http://search-lucene.com/?q=%2Bchinese+analyzer+schemafc_project=Solrfc_type=wiki There is also SmartCN Analyzer in Lucene that you could configure in Solr. Have a look: http://search-lucene.com/?q=smartcn http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters is a good page to know. :) Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Bing Li lbl...@gmail.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:39:13 PM Subject: Indexing and Searching Chinese Hi, all, Now I cannot search the index when querying with Chinese keywords. Before using Solr, I ever used Lucene for some time. Since I need to crawl some Chinese sites, I use ChineseAnalyzer in the code to run Lucene. I know Solr is a server for Lucene. However, I have no idea know how to configure the analyzer in Solr? I appreciate so much for your help! Best, LB
Re: Indexing and Searching Chinese with SolrNet
Why creating two threads for the same problem? Anyway, is your servlet container capable of accepting UTF-8 in the URL? Also, is SolrNet capable of handling those characters? To confirm, try a tool like curl. Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: Indexing and Searching Chinese with SolrNet
Bing Li, Go to your Solr Admin page and use the Analysis functionality there to enter some Chinese text and see how it's getting analyzed at index and at search time. This will tell you what is (or isn't) going on. Here it looks like you just defined index-time analysis, so you should see your index-time analysis look very different from your query-time analysis. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Bing Li lbl...@gmail.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 1:30:37 PM Subject: Indexing and Searching Chinese with SolrNet Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: NRT
It's FFRT (pronounced ...) - Far From Real Time. To help the o.p., there is a page on Solr Wiki about what one can do with Solr and NRT search today. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Walter Underwood wun...@wunderwood.org To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 10:11:56 AM Subject: Re: NRT near Near Real Time? Is that even less real time than NRT? --wunder On Jan 18, 2011, at 12:34 AM, stockii wrote: Real NRT in Solr is not implementet yet. But you can configure a near NRT-Search. http://lucene.472066.n3.nabble.com/Tuning-Solr-caches-with-high-commit-rates-NRT-td1461275.html - --- System One Server, 12 GB RAM, 2 Solr Instances, 7 Cores, 1 Core with 31 Million Documents other under 100.000 - Solr1 for Search-Requests - commit every Minute - 4GB Xmx - Solr2 for Update-Request - delta every 2 Minutes - 4GB Xmx -- View this message in context: http://lucene.472066.n3.nabble.com/NRT-tp2276967p2278477.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: salvaging uncommitted data
Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: Indexing and Searching Chinese with SolrNet
Dear Jelsma, My servlet container is Tomcat 7. I think it should accept Chinese characters. But I am not sure how to configure it. From the console of Tomcat, I saw that the Chinese characters in the query are not displayed normally. However, it is fine in the Solr Admin page. I am not sure either if SolrNet supports Chinese. If not, how can I interact with Solr on .NET? Thanks so much! LB On Wed, Jan 19, 2011 at 2:34 AM, Markus Jelsma markus.jel...@openindex.iowrote: Why creating two threads for the same problem? Anyway, is your servlet container capable of accepting UTF-8 in the URL? Also, is SolrNet capable of handling those characters? To confirm, try a tool like curl. Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: Indexing and Searching Chinese with SolrNet
Hi, Yes but Tomcat might need to be configured to accept, see the wiki for more information on this subject. http://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config Cheers, Dear Jelsma, My servlet container is Tomcat 7. I think it should accept Chinese characters. But I am not sure how to configure it. From the console of Tomcat, I saw that the Chinese characters in the query are not displayed normally. However, it is fine in the Solr Admin page. I am not sure either if SolrNet supports Chinese. If not, how can I interact with Solr on .NET? Thanks so much! LB On Wed, Jan 19, 2011 at 2:34 AM, Markus Jelsma markus.jel...@openindex.iowrote: Why creating two threads for the same problem? Anyway, is your servlet container capable of accepting UTF-8 in the URL? Also, is SolrNet capable of handling those characters? To confirm, try a tool like curl. Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: salvaging uncommitted data
i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: Indexing and Searching Chinese with SolrNet
Dear Jelsma, After configuring the Tomcat URIEncoding, Chinese characters can be processed correctly. I appreciate so much for your help! Best, LB On Wed, Jan 19, 2011 at 3:02 AM, Markus Jelsma markus.jel...@openindex.iowrote: Hi, Yes but Tomcat might need to be configured to accept, see the wiki for more information on this subject. http://wiki.apache.org/solr/SolrTomcat#URI_Charset_Config Cheers, Dear Jelsma, My servlet container is Tomcat 7. I think it should accept Chinese characters. But I am not sure how to configure it. From the console of Tomcat, I saw that the Chinese characters in the query are not displayed normally. However, it is fine in the Solr Admin page. I am not sure either if SolrNet supports Chinese. If not, how can I interact with Solr on .NET? Thanks so much! LB On Wed, Jan 19, 2011 at 2:34 AM, Markus Jelsma markus.jel...@openindex.iowrote: Why creating two threads for the same problem? Anyway, is your servlet container capable of accepting UTF-8 in the URL? Also, is SolrNet capable of handling those characters? To confirm, try a tool like curl. Dear all, After reading some pages on the Web, I created the index with the following schema. .. fieldtype name=text class=solr.TextField positionIncrementGap=100 analyzer type=index tokenizer class=solr.ChineseTokenizerFactory/ /analyzer /fieldtype .. It must be correct, right? However, when sending a query though SolrNet, no results are returned. Could you tell me what the reason is? Thanks, LB
Re: salvaging uncommitted data
Udi, It's hard for me to tell from here, but it looks like your writes are really not going in at all, in which case there may be nothing (much) to salvage. The EBS volume is mounted? And fast (try listing a bigger dir or doing something that involves some non-trivial disk IO)? No errors anywhere in the log on commit? How exactly are you invoking the commit? There is a wait option there... Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 2:04:56 PM Subject: Re: salvaging uncommitted data i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: Any way to query by offset?
Too bad for me I guess! I was hoping there was a hidden field, perhaps, offset one could query on. That one thing would have made this possible to do by simply querying on it. On Jan 18, 2011, at 7:06 AM, Erick Erickson wrote: Ahhh, I see. I don't know of any way to do what you want. Best Erick On Mon, Jan 17, 2011 at 7:25 PM, 5 Diamond IT i...@smallbusinessconsultingexperts.com wrote: I want to start at row 1000, 2000, and 3000 and retrieve those 3 rows ONLY from the result set of whatever search was used. Yes, I can do 3 queries, start=1000 and limit 1, etc., but, want ONE query to get those 3 rows from the result set. It's the poor mans way of doing price buckets the way I want them to be. So, what I need that they do not provide is the ability to find those 3 rows out of the result set in one query. Was hoping for a function, a parser that supported this perhaps, some hidden field I am not aware of I could simply match on, any trick that would work.
Re: salvaging uncommitted data
the ebs volume is operational and i cannot see any error in dmesg etc. the only errors in catalina.out are the lock related ones (even though i removed the lock file) and when i do a commit everything looks fine in the log. i am using the following for the commit: curl http://localhost:8983/solr/update -s -H Content-type:text/xml; charset=utf-8 -d commit/ btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? udi On Tue, Jan 18, 2011 at 11:24 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, It's hard for me to tell from here, but it looks like your writes are really not going in at all, in which case there may be nothing (much) to salvage. The EBS volume is mounted? And fast (try listing a bigger dir or doing something that involves some non-trivial disk IO)? No errors anywhere in the log on commit? How exactly are you invoking the commit? There is a wait option there... Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 2:04:56 PM Subject: Re: salvaging uncommitted data i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: Solr UUID field for externally generated UUIDs
: fieldType name=id_uuid class=solr.UUIDField indexed=true : required=true/ : : The above won't generate a UUID on it's own, right? correct. -Hoss
Re: Single value vs multi value setting in tokenized field
: problem, disk space is cheap. What I wanted to know was whether it is best : to make the single field multiValued=true or not. That is, should my : 'content' field hold data like: ... : or would it be better to make it a concatenated, single value field like: functionally, the only distinctions are: * the positionIncrementGap comes into play in a multiValued field, which affects positions in phrase queries * for stored fields, the multiple vlaues are retuned distinctly instead of as one concatenated string. there shouldn't be any noticible performance difference between indexing either way. -Hoss
Re: salvaging uncommitted data
btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? The writes'll be gone if they haven't been committed yet and the process fails. org.apache.lucene.store.LockObtainFailedException: Lock obtain timed If it's removed then you on restart of the process, this should go away. However you may see a corrupted index exception. On Tue, Jan 18, 2011 at 11:31 AM, Udi Nir u...@threadsy.com wrote: the ebs volume is operational and i cannot see any error in dmesg etc. the only errors in catalina.out are the lock related ones (even though i removed the lock file) and when i do a commit everything looks fine in the log. i am using the following for the commit: curl http://localhost:8983/solr/update -s -H Content-type:text/xml; charset=utf-8 -d commit/ btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? udi On Tue, Jan 18, 2011 at 11:24 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, It's hard for me to tell from here, but it looks like your writes are really not going in at all, in which case there may be nothing (much) to salvage. The EBS volume is mounted? And fast (try listing a bigger dir or doing something that involves some non-trivial disk IO)? No errors anywhere in the log on commit? How exactly are you invoking the commit? There is a wait option there... Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 2:04:56 PM Subject: Re: salvaging uncommitted data i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: sorting based on pivot aggregations
Hello and Thanks for the reply. I've been over that page, and it doesn't seem like it helps with the pivoting aspect. That is if I am sorting via an existing pivot 'sum(student_id,test_grade)' I want my groups of student_id sorted by the sum of test_grade with that student_id. The data is all there and I can retrieve it, I just need solr to do some sorting magic with it... thanks in advance again... On Jan 18, 2011, at 10:42 AM, Ahmet Arslan wrote: So if my pivot term is:student_id,test_grade I'd want to be able to sort on the number of tests a student has taken. and also get an average. something like: :sort = sum( student_id,test_grade )/ count( student_id,test_grade ) where the values would be summed and counted over all of the entries where student_id was the same. It doesn't seem to be working now, but does seem like a feature people would like. Is there any plan to implement it? Or even better a patch that I could apply? http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function could help?
Re: Local param tag voodoo ?
Hi,
You get an error because LocalParams need to be in the beginning of a
parameter's value. So no parenthesis first. The second query should not give an
error because it's a valid query.
Anyway, i assume you're looking for :
http://wiki.apache.org/solr/SimpleFacetParameters#Multi-
Select_Faceting_and_LocalParams
Cheers,
Hey,
here are my needs :
- a query that has tagged and untagged contents
- facets that ignore the tagged contents
I tryed :
q=({!tag=toExclude} ignored) taken into account
q={tag=toExclude v='ignored'} take into account
Both resulted in a error.
Is this possible or do I have to try another way ?
Re: salvaging uncommitted data
i have not restarted the process yet. if i restart it, will i lose any data that is in memory? if so, is there a way around it? is there a way to know if there is any data waiting to be written? (if not, i will just restart...) thanks. On Tue, Jan 18, 2011 at 12:23 PM, Jason Rutherglen jason.rutherg...@gmail.com wrote: btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? The writes'll be gone if they haven't been committed yet and the process fails. org.apache.lucene.store.LockObtainFailedException: Lock obtain timed If it's removed then you on restart of the process, this should go away. However you may see a corrupted index exception. On Tue, Jan 18, 2011 at 11:31 AM, Udi Nir u...@threadsy.com wrote: the ebs volume is operational and i cannot see any error in dmesg etc. the only errors in catalina.out are the lock related ones (even though i removed the lock file) and when i do a commit everything looks fine in the log. i am using the following for the commit: curl http://localhost:8983/solr/update -s -H Content-type:text/xml; charset=utf-8 -d commit/ btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? udi On Tue, Jan 18, 2011 at 11:24 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, It's hard for me to tell from here, but it looks like your writes are really not going in at all, in which case there may be nothing (much) to salvage. The EBS volume is mounted? And fast (try listing a bigger dir or doing something that involves some non-trivial disk IO)? No errors anywhere in the log on commit? How exactly are you invoking the commit? There is a wait option there... Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 2:04:56 PM Subject: Re: salvaging uncommitted data i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
Re: salvaging uncommitted data
if i restart it, will i lose any data that is in memory? if so, is there a way around it? Usually I've restarted the process, and on restart Solr using the unlockOnStartuptrue/unlockOnStartup in solrconfig.xml will automatically remove the lock file (actually I think it may be removed automatically when the process dies). You'll lose the data. is there a way to know if there is any data waiting to be written? (if not, i will just restart...) There is via the API, offhand via the Solr dashboard, I don't know. On Tue, Jan 18, 2011 at 12:35 PM, Udi Nir u...@threadsy.com wrote: i have not restarted the process yet. if i restart it, will i lose any data that is in memory? if so, is there a way around it? is there a way to know if there is any data waiting to be written? (if not, i will just restart...) thanks. On Tue, Jan 18, 2011 at 12:23 PM, Jason Rutherglen jason.rutherg...@gmail.com wrote: btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? The writes'll be gone if they haven't been committed yet and the process fails. org.apache.lucene.store.LockObtainFailedException: Lock obtain timed If it's removed then you on restart of the process, this should go away. However you may see a corrupted index exception. On Tue, Jan 18, 2011 at 11:31 AM, Udi Nir u...@threadsy.com wrote: the ebs volume is operational and i cannot see any error in dmesg etc. the only errors in catalina.out are the lock related ones (even though i removed the lock file) and when i do a commit everything looks fine in the log. i am using the following for the commit: curl http://localhost:8983/solr/update -s -H Content-type:text/xml; charset=utf-8 -d commit/ btw where will i find the writes that have not been committed? are they all in memory or are they in some temp files somewhere? udi On Tue, Jan 18, 2011 at 11:24 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, It's hard for me to tell from here, but it looks like your writes are really not going in at all, in which case there may be nothing (much) to salvage. The EBS volume is mounted? And fast (try listing a bigger dir or doing something that involves some non-trivial disk IO)? No errors anywhere in the log on commit? How exactly are you invoking the commit? There is a wait option there... Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 2:04:56 PM Subject: Re: salvaging uncommitted data i have not stopped writing so i am getting this error all the time. the commit actually seems to go through with no errors but it does not seem to write anything to the index files (i can see this because they are old and i cannot see new stuff in search results). my index folder is on an amazon ebs volume which is a block device and looks like a local disk. thanks! udi On Tue, Jan 18, 2011 at 10:49 AM, Otis Gospodnetic otis_gospodne...@yahoo.com wrote: Udi, Hm, don't know off the top of my head, but sounds like an interesting problem. Are you getting this error while still writing to the index or did you stop all writing? Do you get this error when you issue a commit or? Is the index on the local disk or? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Udi Nir u...@threadsy.com To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 12:29:47 PM Subject: salvaging uncommitted data Hi, I have a solr server that is failing to acquire a lock with the exception below. I think that the server has a lot of uncommitted data (I am not sure how to verify this) and if so I would like to salvage it. Any suggestions how to proceed? (btw i tried removing the lock file but it did not help) Thanks, Udi Jan 18, 2011 5:17:06 PM org.apache.solr.common.SolrException log SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out : NativeFSLock@ /vol-unifi-solr/data/index/lucene-043c34f1f06a280de60b3d4e8e05601 6-write.lock at org.apache.lucene.store.Lock.obtain(Lock.java:85) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1545) at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1402) at org.apache.solr.update.SolrIndexWriter.init(SolrIndexWriter.java:19 0)
[POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
As devs of Lucene/Solr, due to the way ASF mirrors, etc. works, we really don't have a good sense of how people get Lucene and Solr for use in their application. Because of this, there has been some talk of dropping Maven support for Lucene artifacts (or at least make them external). Before we do that, I'd like to conduct an informal poll of actual users out there and see how you get Lucene or Solr. Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) Please put an X in the box that applies to you. Multiple selections are OK (for instance, if one project uses a mirror and another uses Maven) Please do not turn this thread into a discussion on Maven and it's (de)merits, I simply want to know, informally, where people get their JARs from. In other words, no discussion is necessary (we already have that going on d...@lucene.apache.org which you are welcome to join.) Thanks, Grant
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
And here's mine: On Jan 18, 2011, at 4:04 PM, Grant Ingersoll wrote: Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [x] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [x] I/we build them from source via an SVN/Git checkout.
Re: DataImportHandler: full import of a single entity
Ahmet Arslan iorixxx at yahoo.com writes: I've got a DataImportHandler set up with 5 entities. I would like to do a full import on just one entity. Is that possible? Yes, there is a parameter named entity for that. solr/dataimport?command=full-importentity=myEntity That seems to delete the entire index and replace it with only the contents of that one entity. Is there no way to leave the index alone for the other entities and just redo that one?
RE: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) James Dyer E-Commerce Systems Ingram Content Group (615) 213-4311 -Original Message- From: Grant Ingersoll [mailto:gsing...@apache.org] Sent: Tuesday, January 18, 2011 3:04 PM To: java-u...@lucene.apache.org; solr-user@lucene.apache.org Subject: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors? As devs of Lucene/Solr, due to the way ASF mirrors, etc. works, we really don't have a good sense of how people get Lucene and Solr for use in their application. Because of this, there has been some talk of dropping Maven support for Lucene artifacts (or at least make them external). Before we do that, I'd like to conduct an informal poll of actual users out there and see how you get Lucene or Solr. Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) Please put an X in the box that applies to you. Multiple selections are OK (for instance, if one project uses a mirror and another uses Maven) Please do not turn this thread into a discussion on Maven and it's (de)merits, I simply want to know, informally, where people get their JARs from. In other words, no discussion is necessary (we already have that going on d...@lucene.apache.org which you are welcome to join.) Thanks, Grant
RE: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) -Original Message- From: Grant Ingersoll [mailto:gsing...@apache.org] Sent: Tuesday, January 18, 2011 3:04 PM To: java-u...@lucene.apache.org; solr-user@lucene.apache.org Subject: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [x] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. -Glen Newton -- -
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
RE: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[x] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. On Tue, Jan 18, 2011 at 1:24 PM, Glen Newton glen.new...@gmail.com wrote: Where do you get your Lucene/Solr downloads from? [x] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. -Glen Newton -- -
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
RE: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[x] ASF Mirrors (linked in our release announcements or via the Lucene website) [x] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [x] I/we build them from source via an SVN/Git checkout.
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) -- Beatriz Nombela Escobar bea...@gmail.com
Re: DataImportHandler: full import of a single entity
That seems to delete the entire index and replace it with only the contents of that one entity. Is there no way to leave the index alone for the other entities and just redo that one? Yes, there is a parameter named clean for that. solr/dataimport?command=full-importentity=myEntityclean=false
RE: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [x] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [x] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project)
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Depending on the project, I either pull from ASF Mirrors or Source. However, I do reference Maven repository when writing Java code that is built by Maven. And it's often a pain getting it to work! On Jan 18, 2011, at 4:23 PM, Ryan Aylward wrote: [X] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) - Eric Pugh | Principal | OpenSource Connections, LLC | 434.466.1467 | http://www.opensourceconnections.com Co-Author: Solr 1.4 Enterprise Search Server available from http://www.packtpub.com/solr-1-4-enterprise-search-server Free/Busy: http://tinyurl.com/eric-cal
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
Where do you get your Lucene/Solr downloads from? [] ASF Mirrors (linked in our release announcements or via the Lucene website) [X] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout.
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
On 18.01.2011, at 22:04, Grant Ingersoll wrote: As devs of Lucene/Solr, due to the way ASF mirrors, etc. works, we really don't have a good sense of how people get Lucene and Solr for use in their application. Because of this, there has been some talk of dropping Maven support for Lucene artifacts (or at least make them external). Before we do that, I'd like to conduct an informal poll of actual users out there and see how you get Lucene or Solr. Where do you get your Lucene/Solr downloads from? [X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) regards, Lukas Kahwe Smith m...@pooteeweet.org
Re: [POLL] Where do you get Lucene/Solr from? Maven? ASF Mirrors?
[X] ASF Mirrors (linked in our release announcements or via the Lucene website) [] Maven repository (whether you use Maven, Ant+Ivy, Buildr, etc.) [X] I/we build them from source via an SVN/Git checkout. [] Other (someone in your company mirrors them internally or via a downstream project) Koji -- http://www.rondhuit.com/en/
Re: Solr UUID field for externally generated UUIDs
THX, Chris! Dennis Gearon Signature Warning It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501tag=nl.e036' EARTH has a Right To Life, otherwise we all die. - Original Message From: Chris Hostetter hossman_luc...@fucit.org To: solr-user@lucene.apache.org Sent: Tue, January 18, 2011 11:35:38 AM Subject: Re: Solr UUID field for externally generated UUIDs : fieldType name=id_uuid class=solr.UUIDField indexed=true : required=true/ : : The above won't generate a UUID on it's own, right? correct. -Hoss
