frequent index updates

2011-03-21 Thread Prav Buz
Hi, I'm wondering what are the best way to do this for this scenario: Index will have about 250 - 400 million items. Index needs to be updated every 10/20 minutes and no. of records updated could be upto 5-6 million in each. Could you please guide me on how the indexing is done when there are

Memory leak for Multiple fields faceted searching

2011-03-21 Thread newsam
Hi, Here is my environment: 32bits windows server 2008 (4cpu and 12G RAM), 320million docs, index size 12G. When we use query=*:* and 10 fields for faceted searching, it works. However, 15 feilds or more faceted searching caused the following exception: java.lang.OutOfMemoryError: Java heap

Logic operator with dismax

2011-03-21 Thread Gastone Penzo
Hi. i have a problem with logic operator OR in dismax query search. some days ago the query worked well. now it returns me anything (0 documents) i explain: the query is: http://localhost:8983/solr/select/?q= 1324OR4322OR2324OR%20hello+worlddefType=dismaxqf=code%20title the schema has the

Re: Logic operator with dismax

2011-03-21 Thread Savvas-Andreas Moysidis
Hello, The Dismax search handler doesn't have the concept of a logical operator in terms of OR-AND but rather uses a feature called Min-Should-Match (or mm). This parameter specifies the absolute number or percentage of the entered terms that you need them to match. To have an OR-like effect you

Re: frequent index updates

2011-03-21 Thread Savvas-Andreas Moysidis
Hello, This thread might help: http://search-lucene.com/m/09PHV1E0ZxQ1/Possibilities+of+near+real+time+search+with+solr/v=threaded On 21 March 2011 09:33, Prav Buz buz.p...@gmail.com wrote: Hi, I'm wondering what are the best way to do this for this scenario: Index will have about 250 -

Spellchecking Escaped Queries

2011-03-21 Thread Colin Vipurs
I'm having an issue performing a spellcheck on some information and search of the archive isn't helping. I'm indexing the word p!nk (yes, that's a bang in there), and have a replacement filter setup so that the ! becomes i. Looking at the analyzer the right thing is happening with both the

Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Marc SCHNEIDER
Hello, I'd like to know the fastest way (code lines) to update a field of a document. So my idea was: 1) Get a SolrDocument 2) Add all fields of the SolrDocument to a new SolrInputDocument 3) Update the field in SolrInputDocument 4) Add SolrInputDocument to the server and commit it Is there a

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Péter Király
Hi Marc, as far as I know the best way to do it is working from the original source, because it is possible, that not all fields are stores, and the original content of the not stored fields is not inside the Solr document. Péter 2011/3/21 Marc SCHNEIDER marc.schneide...@gmail.com: Hello,

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Ahmet Arslan
Is there a fastest way to do that? I mean transforming a SolrDocument into a SolrInputDocument? Does these static methods help? org.apache.solr.client.solrj.util.ClientUtils.toSolrDocument() org.apache.solr.client.solrj.util.ClientUtils.toSolrInputDocument()

Re: Memory leak for Multiple fields faceted searching

2011-03-21 Thread Darx Oman
How much memory you specify for java heap? -Xmx...

Re: Adding the suggest component

2011-03-21 Thread Erick Erickson
OK, I think you're jumping ahead and trying to do too many things at once. What did you download? Source? The distro? The error you posted usually happens for me when I haven't compiled the example target from source. So I'd guess you don't have the proper targets built. This assumes you

Re: Logic operator with dismax

2011-03-21 Thread Jayendra Patil
Dismax does not support boolean queries, you may try using Extended Dismax for the boolean support. https://issues.apache.org/jira/browse/SOLR-1553 Regards, Jayendra On Mon, Mar 21, 2011 at 8:24 AM, Savvas-Andreas Moysidis savvas.andreas.moysi...@googlemail.com wrote: Hello, The Dismax search

Re: working with collection : Where is default schema.xml

2011-03-21 Thread Erick Erickson
More details please? Solr doesn't generate any schemas, so I'm completely lost what you're after here. Best Erick On Fri, Mar 18, 2011 at 7:59 PM, geag34 sac@gmail.com wrote: Hello. I am working with collection. When I do a new collection I would like a custom schema.xml to be generated.

Re: How to upgrade to Solr4.0 to use Result Grouping?

2011-03-21 Thread Erick Erickson
Get the release and re-index? You can get a trunk version either through SVN or from the nightly build at https://builds.apache.org/hudson/view/S-Z/view/Solr/ Note that 3.1 also has result grouping, and is currently being released, so you might want to consider that version. Best Erick On Sun,

Re: Memory leak for Multiple fields faceted searching

2011-03-21 Thread Erick Erickson
Memory consumption grows as the number of unique terms grows. So in addition to Darx's question, how many unique terms are in all the fields you're faceting on? Best Erick 2011/3/21 newsam new...@zju.edu.cn: Hi, Here is my environment: 32bits windows server 2008 (4cpu and 12G RAM),

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Marc SCHNEIDER
Hi Ahmet, That did it, thanks a lot! Marc. On Mon, Mar 21, 2011 at 3:03 PM, Ahmet Arslan iori...@yahoo.com wrote: Is there a fastest way to do that? I mean transforming a SolrDocument into a SolrInputDocument? Does these static methods help?

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Erick Erickson
The problem is that unless you *stored* every field in the Solr index, getting the document back will not have all the original information. Getting a Solr document only returns the stored fields. And updating is really a delete followed by an add. So let's say you have field A that hasn't been

Re: keeping data consistent between Database and Solr

2011-03-21 Thread Shawn Heisey
On 3/15/2011 12:54 PM, onlinespend...@gmail.com wrote: That's pretty interesting to use the autoincrementing document ID as a way to keep track of what has not been indexed in Solr. And you overwrite this document ID even when you modify an existing document. Very cool. I suppose the number

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Marc SCHNEIDER
Hi Erick, Thanks for your answer. I'm a quite newbie to Solr so I'm a little bit confused. Do you mean that (using Solrj in my case) I should add all fields (stored and not stored) before adding the document to the index? Thanks, Marc. On Mon, Mar 21, 2011 at 3:27 PM, Erick Erickson

Re: Transform a SolrDocument into a SolrInputDocument

2011-03-21 Thread Gora Mohanty
On Mon, Mar 21, 2011 at 8:33 PM, Marc SCHNEIDER marc.schneide...@gmail.com wrote: Hi Erick, Thanks for your answer. I'm a quite newbie to Solr so I'm a little bit confused. Do you mean that (using Solrj in my case) I should add all fields (stored and not stored) before adding the document to

Re: keeping data consistent between Database and Solr

2011-03-21 Thread onlinespend...@gmail.com
On Mon, Mar 21, 2011 at 10:57 AM, Shawn Heisey s...@elyograg.org wrote: On 3/15/2011 12:54 PM, onlinespend...@gmail.com wrote: That's pretty interesting to use the autoincrementing document ID as a way to keep track of what has not been indexed in Solr. And you overwrite this document ID

more beware on 1.4.1 lucene QP NOT

2011-03-21 Thread Jonathan Rochkind
So Solr 1.4.1, lucene query parser, generally has no problem with a query that's simply one clause NOT defType=lucene q=NOT foo No problem! Also no problem: q=foo AND NOT bar But, problem when the operand to an OR is a NOT: q=foo OR (NOT bar) Does NOT work correctly. Can be worked around

Re: more beware on 1.4.1 lucene QP NOT

2011-03-21 Thread Gora Mohanty
On Mon, Mar 21, 2011 at 9:29 PM, Jonathan Rochkind rochk...@jhu.edu wrote: So Solr 1.4.1, lucene query parser, generally has no problem with a query that's simply one clause NOT defType=lucene q=NOT foo No problem! Also no problem: q=foo AND NOT bar But, problem when the operand to an

Re: DIH Issue(newbie to solr)

2011-03-21 Thread neha
Hi Gora, The absolute path solved the issue, but in a weird situation though. The index files are being written to /example/solr/data/index instead of .example/test/solr/data/index. I run start jetty with ./test/solr/ as home would you know why this is happening. PS: While starting with

Re: Segments and Memory Correlate?

2011-03-21 Thread danomano
yes we are having physical memory issues, but do does anyone know for a fact if there is direct correlation between be segment counts and RAM - memory? i.e. when the system begins a search on a segment, does it load that segments full index. Or does the system load All index data for all

Re: DIH Issue(newbie to solr)

2011-03-21 Thread Gora Mohanty
On Mon, Mar 21, 2011 at 9:45 PM, neha pneha...@yahoo.com wrote: Hi Gora, The absolute path solved the issue, but in a weird situation though. Great. That should be most of your issue solved. The

Re: DIH Issue(newbie to solr)

2011-03-21 Thread neha
Thanks Gora...it works.. Thanks again..:) Neha -- View this message in context: http://lucene.472066.n3.nabble.com/DIH-Issue-newbie-to-solr-tp2695407p2710864.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: DIH Issue(newbie to solr)

2011-03-21 Thread neha
Thanks Gora it works..!!! Thanks again. One last question, the documents get indexed well and all but when I issue full-import command it still says Total Requests made to DataSource 0 Would you know why this could happen. Neha -- View this message in context:

sending a parsed query to solr (xml-query-parser, syntaxtree)

2011-03-21 Thread karsten-solr
Hi, I am working on a migration from verity k2 to solr. At this point I have a parser for the Verity Query Language (our used subset) which generates a syntax tree. I transfer this in a couple of filters and one query. This fragmentation is the reason, why I can not use my parser inside

Re: How to upgrade to Solr4.0 to use Result Grouping?

2011-03-21 Thread Ruixiang Zhang
Hi Erick Thanks for your reply. 3.1 should be OK for me. When will it be release and where is the link to download it? Thanks Richard On Mon, Mar 21, 2011 at 7:20 AM, Erick Erickson erickerick...@gmail.comwrote: Get the release and re-index? You can get a trunk version either through SVN or

Re: How to upgrade to Solr4.0 to use Result Grouping?

2011-03-21 Thread Yonik Seeley
On Mon, Mar 21, 2011 at 10:20 AM, Erick Erickson erickerick...@gmail.com wrote: Get the release and re-index? You can get a trunk version either through SVN or from the nightly build at https://builds.apache.org/hudson/view/S-Z/view/Solr/ Note that 3.1 also has result grouping Result

Solr 1.4.1 missing CSV queryResponseWriter?

2011-03-21 Thread danomano
Hi folks, I was running 1.4.0, but today I'm trying to switch to 1.4.1 (which appears to work with Zoie), however when I try to retrieve data in CSV format, wt=csv does not appear to work anymore. Looking at the solrconfig.xml I see: (I tried pulling out the CSV entry, but that failed (class not

Re: Solr 1.4.1 missing CSV queryResponseWriter?

2011-03-21 Thread Yonik Seeley
CSV response format is only available in 3.1 (soon to be released) and later. http://wiki.apache.org/solr/CSVResponseWriter -Yonik http://www.lucenerevolution.org -- Lucene/Solr User Conference, May 25-26, San Francisco On Mon, Mar 21, 2011 at 3:02 PM, danomano dshopk...@earthlink.net wrote:

Re: How to upgrade to Solr4.0 to use Result Grouping?

2011-03-21 Thread Ruixiang Zhang
Thanks. Then where is the link to download the latest 4.0? Richard On Mon, Mar 21, 2011 at 11:57 AM, Yonik Seeley yo...@lucidimagination.comwrote: On Mon, Mar 21, 2011 at 10:20 AM, Erick Erickson erickerick...@gmail.com wrote: Get the release and re-index? You can get a trunk version

Re: How to upgrade to Solr4.0 to use Result Grouping?

2011-03-21 Thread Ruixiang Zhang
Is it the file apache-solr-4.0-2011-03-18_08-07-35-src.tgzhttps://builds.apache.org/hudson/job/Solr-trunk/lastSuccessfulBuild/artifact/artifacts/apache-solr-4.0-2011-03-18_08-07-35-src.tgz on page https://builds.apache.org/hudson/job/Solr-trunk/lastSuccessfulBuild/artifact/artifacts/? Thanks

Implementing Facets

2011-03-21 Thread McGibbney, Lewis John
Hi list, I am working with a Ajax-Solr GUI but I am getting the following error from Firebug when launching the web app on Tomcat 7.0.11. The web app uses Solr version 1.4.1 HTTP Status 400 - undefined field links/h1HR size=1 noshade=noshadepbtype/b Status report/ppbmessage/b uundefined

email - DIH

2011-03-21 Thread Matias Alonso
Hi, I’m using Data Import Handler for index emails. The problem is that nota ll the emails was indexed When I do a full import. Someone have any idea? Regards, -- Matias.

Re: Implementing Facets

2011-03-21 Thread Ahmet Arslan
--- On Mon, 3/21/11, McGibbney, Lewis John lewis.mcgibb...@gcu.ac.uk wrote: From: McGibbney, Lewis John lewis.mcgibb...@gcu.ac.uk Subject: Implementing Facets To: solr-user@lucene.apache.org solr-user@lucene.apache.org Date: Monday, March 21, 2011, 9:50 PM Hi list, I am working with a

Embedded Solr

2011-03-21 Thread Greg Georges
Hello all, I am using Solr in a Java architecture right now, and the results are great. The app development team has asked me if it is possible to embed Solr, but the request is to embed it into a C++ app and mac app using objective C. I do not have much knowledge on embedded Solr. Does it

RE: Implementing Facets

2011-03-21 Thread McGibbney, Lewis John
Hi Ahmet, Yes this is the case. I have changed it to reflect your suggestion thank you for this. After reloading the app I still get the error, here is the full stack trace from catalina.out INFO: [] Registered new searcher Searcher@8af0b0 main 21-Mar-2011 20:28:53

Re: sending a parsed query to solr (xml-query-parser, syntaxtree)

2011-03-21 Thread Erik Hatcher
On Mar 21, 2011, at 14:19 , karsten-s...@gmx.de wrote: Hi, I am working on a migration from verity k2 to solr. At this point I have a parser for the Verity Query Language (our used subset) which generates a syntax tree. I transfer this in a couple of filters and one query. This

RE: Implementing Facets

2011-03-21 Thread Ahmet Arslan
Yes this is the case. I have changed it to reflect your suggestion thank you for this. After reloading the app I still get the error, here is the full stack trace from catalina.out It seems you have one more typo: str name=f.topics.facet.limit.50/str = str name=f.topics.facet.limit50/str

debugQuery doesnot work with Solr-1.3.0 , is it true?

2011-03-21 Thread Anurag
i am trying to put this query on Solr-1.3.0 server which is integrated with Nutch-1.0. http://localhost:8983/solr/select/?q=solrversion=2.2start=0rows=10indent=ondebugQuery=on But the results, i am getting doesnot have explaination for the score as it should include tf, idf etc Should i

Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Jan-Eirik B . Nævdal
Hi Have this problem I tried to solve with filter queries but I think I`m stuck now, and don't see a solution how to solve my problem. My problem is that i want a result page that shows those documents that matches these filter query fq=openingtime:[* TO NOW] , fq=closingtime[NOW TO *] for the

Re: Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Jonathan Rochkind
You can put an actual OR in the fq (an fq, by default, is in the solr-lucene query parser language). Might that achieve what you want? fq= -openingtime:[* TO *] OR openingtime:[* TO NOW] fq= -closingtime:[* TO *] OR closingtime:[NOW TO *] Does that, or some variation of it, do what you need?

Re: debugQuery doesnot work with Solr-1.3.0 , is it true?

2011-03-21 Thread Ahmet Arslan
--- On Mon, 3/21/11, Anurag anurag.it.jo...@gmail.com wrote: From: Anurag anurag.it.jo...@gmail.com Subject: debugQuery doesnot work with Solr-1.3.0 , is it true? To: solr-user@lucene.apache.org Date: Monday, March 21, 2011, 11:18 PM i am trying to put this query on Solr-1.3.0 server which

Re: Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Jan-Eirik B . Nævdal
Unfortunalty have I tried the OR approach in the fq. with the positive filter query first i get document 4, with negative filter query first i get none result, This request gives me 1 reply

Re: working with collection : Where is default schema.xml

2011-03-21 Thread geag34
Ok thank. It is my fault. I have created collection with a lucidimagination perl script. I will errase the schema.xml. Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/working-with-collection-Where-is-default-schema-xml-tp2700455p2712496.html Sent from the Solr -

Re: Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Markus Jelsma
Hi, In a _high load_ environment with queryResultCache and filterCache enabled, you need to be very careful about using NOW in its highest precision. It adds - and likely evicts - a non retrievable entry in both caches, wasting a lot of resources. It's best to always reduce precision when

Re: Embedded Solr

2011-03-21 Thread Bill Bell
Yes it needs java to run Bill Bell Sent from mobile On Mar 21, 2011, at 2:30 PM, Greg Georges greg.geor...@biztree.com wrote: Hello all, I am using Solr in a Java architecture right now, and the results are great. The app development team has asked me if it is possible to embed Solr, but

Re: Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Ahmet Arslan
How about using below as a q= parameter? With df=OR +openingtime:[* TO NOW] +closingtime[NOW TO *] (+*:* -openingtime[* TO *] -closingtime[* TO *]) http://search-lucene.com/m/TN2a3WgvyX/ --- On Tue, 3/22/11, Jan-Eirik B. Nævdal jan-eirik.naev...@iterate.no wrote: From: Jan-Eirik B.

RE: Problems with creating a query that matches all the documents I want to display

2011-03-21 Thread Jonathan Rochkind
If the OR actually worked to do what it's trying to say, would it be what you wanted? Because I can't believe I didn't recognize this is an instance of the very thing I posted on this list this morning, where the solr-lucene query parser has problems with some kinds of 'pure negative' queries.