Optional filter queries

2012-01-03 Thread Allistair Crossley
Evening all, A subset of my documents have a field, filterMinutes, that some other documents do not. filterMinutes stores a number. I often issue a query that contains a filter query range, e.g. q=filterMinutes:[* TO 50] I am finding that adding this query excludes all documents that do not

Same index is ranking differently on 2 machines

2011-03-09 Thread Allistair Crossley
Hi, I am seeing an issue I do not understand and hope that someone can shed some light on this. The issue is that for a particular search we are seeing a particular result rank in position 3 on one machine and position 8 on the production machine. The position 3 is our desired and roughly

Re: Same index is ranking differently on 2 machines

2011-03-09 Thread Allistair Crossley
attempts to make scores from different queries (or even different indexes) comparable Regards, Jayendra On Wed, Mar 9, 2011 at 4:22 PM, Allistair Crossley a...@roxxor.co.uk wrote: Hi, I am seeing an issue I do not understand and hope that someone can shed some light on this. The issue

Re: Same index is ranking differently on 2 machines

2011-03-09 Thread Allistair Crossley
That's what I think, glad I am not going mad. I've spent 1/2 a day comparing the config files, checking out from SVN again and ensuring the databases are identical. I cannot see what else I can do to make them equivalent. Both servers checkout directly from SVN, I am convinced the files are

Re: Same index is ranking differently on 2 machines

2011-03-09 Thread Allistair Crossley
On Wed, Mar 9, 2011 at 4:38 PM, Allistair Crossley a...@roxxor.co.uk wrote: Thanks. Good to know, but even so my problem remains - the end score should not be different and is causing a dramatically different ranking of a document (3 versus 7 is dramatic for my client). This must be down

Re: [Adding] Entities when indexing a DB

2010-12-15 Thread Allistair Crossley
mission.id and event.id if the same value will be overwriting the indexed document. your ids need to be unique across all documents. i usually have a field id_original that i map the table id to, and then for id per entity i usually prefix it with the entity name in the value mapped to the

Re: search over two independent tables

2010-10-14 Thread Allistair Crossley
your first example is correct document entity name=newsfeed/entity entity name=message/entity /document i have the same config for indexing 5 different tables what you don't have from what i can see is a field name mapped to each column, e.g. field column=nf_text / i always have to provide

Re: search over two independent tables

2010-10-14 Thread Allistair Crossley
handy thing is to facet.field=type and search (without a filter) as then you'll get back counts for your Newsfeed Message results too. On Oct 14, 2010, at 5:44 AM, Allistair Crossley wrote: your first example is correct document entity name=newsfeed/entity entity name=message/entity /document

Re: search over two independent tables

2010-10-14 Thread Allistair Crossley
table (actualy Newsfeeds) Le 14/10/2010 11:49, Allistair Crossley a écrit : actually your intention is unclear ... are you wanting to run a single search and get back results from BOTH newsfeed and message? or do you want one or the other? if you want one or the other you could use my strategy

Re: check if field CONTAINS a value, as opposed to IS of a value

2010-10-14 Thread Allistair Crossley
i think you need to look at ngram tokenizing On Oct 14, 2010, at 7:55 AM, PeterKerk wrote: I try to determine if a certain word occurs within a field. http://localhost:8983/solr/db/select/?indent=onfacet=truefl=id,titleq=introtext:hi this works if an EXACT match was found on field

Re: check if field CONTAINS a value, as opposed to IS of a value

2010-10-14 Thread Allistair Crossley
actuall no you don't .. if you want hi in a sentence of hi there this is me this is just normal tokenizing and should work .. check your field type/analysers On Oct 14, 2010, at 7:59 AM, Allistair Crossley wrote: i think you need to look at ngram tokenizing On Oct 14, 2010, at 7:55 AM

Re: What is the maximum number of documents that can be indexed ?

2010-10-14 Thread Allistair Crossley
i think you answered the question by yourself ... these questions usually get the response that there is no answer. solr/lucence scale and distribute to whatever hardware you want to throw them. you probably want to turn the question around - what is the maximum number of documents that your

Re: search over two independent tables

2010-10-14 Thread Allistair Crossley
super On Oct 14, 2010, at 8:00 AM, Anthony Maudry wrote: Sorry for the late answer. It works now thanks to you, Allistair. I needed to use your uid field, common to the two entities but built in different ways. here is the result in a sample of the data-config.xml file ...

Re: What is the maximum number of documents that can be indexed ?

2010-10-14 Thread Allistair Crossley
me also. great book, just wanted a bit more on complex DIH :) On Oct 14, 2010, at 10:38 AM, Jason Brown wrote: Not related to the opening thread - but wante to thank Eric for his book. Clarified a lot of stuff and very useful. -Original Message- From: Eric Pugh

Re: which schema.xml to modify ?

2010-10-14 Thread Allistair Crossley
you will find it in the distribution at example/solr/config On Oct 14, 2010, at 3:04 PM, Ibrahim Diop wrote: Hi All, I'm a new solr user and I just want to know which schema.xml file to modify for this tutorial : http://lucene.apache.org/solr/tutorial.html Thanks, Ibrahim.

Re: Synchronizing Solr with a PostgreDB

2010-10-14 Thread Allistair Crossley
i would not cross-reference solr results with your database to merge unless you want to spank your database. nor would i load solr with all your data. what i have found is that the search results page is generally a small subset of data relating to the fuller document/result. therefore i store

Getting an ngram fieldtype to work

2010-10-08 Thread Allistair Crossley
Morning all, I would like to ngram a company name field in our index. I have read about the costs of doing so in the great David Smiley Solr 1.4 book and just to get started I have followed his example in setting up an ngram field type as follows: fieldType name=text_substring

Re: Getting an ngram fieldtype to work

2010-10-08 Thread Allistair Crossley
thing I would try is to go to the analysis page, enter your test data, and report back what each analysis stage prints out: http://localhost:8983/solr/admin/analysis.jsp -- Jan Høydahl, search solution architect Cominvent AS - www.cominvent.com On 8. okt. 2010, at 14.19, Allistair Crossley

Re: Getting an ngram fieldtype to work

2010-10-08 Thread Allistair Crossley
, Markus Jelsma wrote: How come your query analyser spits out grams? It isn't configured to do so or you posted an older field definition. Anyway, do you actually search on your new field? On Friday, October 08, 2010 02:46:08 pm Allistair Crossley wrote: Hi, Yep, I was just looking

Re: Getting an ngram fieldtype to work

2010-10-08 Thread Allistair Crossley
the Prescott result. So ngramming is working but it is not working when the query is something far to the right of the indexed value. Is this another user-error or have I missed something else here? Cheers On Oct 8, 2010, at 9:02 AM, Allistair Crossley wrote: Oh my. I am basically being

Re: Strategy for re-indexing

2010-10-08 Thread Allistair Crossley
/2010 10:49 AM, Allistair Crossley wrote: Hi, I was interested in gaining some insight into how you guys schedule updates for your Solr index (I have a single index). Right now during development I have added deltaQuery specifications to data import entities to control the number of rows

Re: multi level faceting

2010-10-04 Thread Allistair Crossley
I think that is just sending 2 fq facet queries through. In Solr PHP I would do that with, e.g. $params['facet'] = true; $params['facet.fields'] = array('Size'); $params['fq'] = array('sex' = array('Men', 'Women')); but yes i think you'd have to send through what the current facet query is and

Re: DIH sub-entity not indexing

2010-10-04 Thread Allistair Crossley
/DataImportHandler#Interactive_Development_Mo de Ephraim Ofir -Original Message- From: Allistair Crossley [mailto:a...@roxxor.co.uk] Sent: Monday, October 04, 2010 3:09 PM To: solr-user@lucene.apache.org Subject: Re: DIH sub-entity not indexing Thanks Ephraim. I tried your suggestion

Re: DIH sub-entity not indexing

2010-10-04 Thread Allistair Crossley
/lucene-solr-user/201008.mbox/%3 c9f8b39cb3b7c6d4594293ea29ccf438b01702...@icq-mail.icq.il.office.aol.com %3E Ephraim Ofir -Original Message- From: Allistair Crossley [mailto:a...@roxxor.co.uk] Sent: Monday, October 04, 2010 2:49 PM To: solr-user@lucene.apache.org Subject: Re: DIH

DIH sub-entity not indexing

2010-10-04 Thread Allistair Crossley
Hello list, I've been successful with DIH to a large extent but a seemingly simple extra column I need is posing problems. In a nutshell I have 2 entities let's say - Listing habtm Contact. I have copied the relevant parts of the configs below. I have run my SQL for the sub-entity Contact and

Re: DIH sub-entity not indexing

2010-10-04 Thread Allistair Crossley
Very clever thinking indeed. Well, that's certainly revealed the problem ... ${listing.id} is empty on my sub-entity query ... And this because I prefix the indexed ID with a letter field column=id name=id template=L${listing.id} / This appears to modify the internal value of $listing.id for

Re: solr-user

2010-10-04 Thread Allistair Crossley
I updated the SolrJ JAR requirements to be clearer on the wiki page given how many of these SolrJ emails I saw coming through since joining the list. I just created a test java class and imported the removed JARs until I found out the minimal set required. On Oct 4, 2010, at 8:27 AM, Erick

Re: DIH sub-entity not indexing

2010-10-04 Thread Allistair Crossley
=contacts column=full_name / /entity /entity /entity Am I missing the obvious? On Oct 4, 2010, at 8:22 AM, Allistair Crossley wrote: Hello list, I've been successful with DIH to a large extent but a seemingly simple extra column I need is posing problems

Re: solrj

2010-10-04 Thread Allistair Crossley
i rewrote the top jar section at http://wiki.apache.org/solr/Solrj and the following code then runs fine. import java.net.MalformedURLException; import org.apache.solr.client.solrj.SolrQuery; import org.apache.solr.client.solrj.SolrServer; import

Re: Any way to append new text to an existing indexed field?

2010-10-01 Thread Allistair Crossley
i would say question and answer are 2 different entities. if you are using the data import handler, i would personally create them as separate entities with their own queries to the database using the deltaQuery method to pick up only new rows. i guess it depends if you need question + answers

Re: Any way to append new text to an existing indexed field?

2010-10-01 Thread Allistair Crossley
, 10/1/10, Allistair Crossley a...@roxxor.co.uk wrote: From: Allistair Crossley a...@roxxor.co.uk Subject: Re: Any way to append new text to an existing indexed field? To: solr-user@lucene.apache.org Date: Friday, October 1, 2010, 7:46 AM i would say question and answer are 2 different

Re: any working SolrJ code example for Solr 1.4.1

2010-10-01 Thread Allistair Crossley
no example anyone gives you will solve your class not found exception .. you need to ensure the relevant jars (in dist) are included in your solr instance's lib folder i guess? On Oct 1, 2010, at 10:50 AM, Xin Li wrote: Hi, there, Just picked up SolrJ few days ago. I have my Solr Server

Re: any working SolrJ code example for Solr 1.4.1

2010-10-01 Thread Allistair Crossley
issue. If that's the case, I will grab Perl solution, and come back to SolrJ later. Thanks, Xin -Original Message- From: Allistair Crossley [mailto:a...@roxxor.co.uk] Sent: Friday, October 01, 2010 11:52 AM To: solr-user@lucene.apache.org Subject: Re: any working SolrJ code

Re: SolrJ

2010-09-30 Thread Allistair Crossley
it's in the dist folder with the name provided by the wiki page you refer to On Sep 30, 2010, at 3:01 PM, Christopher Gross wrote: Where can I get SolrJ? The wiki makes reference to it, and says that it is a part of the Solr builds that you download, but I can't find it in the jars that come

Missing facet values for zero counts

2010-09-29 Thread Allistair Crossley
Hello list, I am implementing a directory using Solr. The user is able to search with a free-text query or 2 filters (provided as pick-lists) for country. A directory entry only has one country. I am using Solr facets for country and I use the facet counts generated initially by a *:* search

Re: Missing facet values for zero counts

2010-09-29 Thread Allistair Crossley
Hi, For us this is a usability concern. You either don't show Sweden in a pick-list called Country and some users go away thinking you don't *ever* support Sweden (not true). OR you allow a user to execute an empty result search - but at least they know you do support Sweden. It is we believe

Re: Solr rate limiting / DoS attacks

2010-09-29 Thread Allistair Crossley
This kind of thing is not limited to Solr and you normally wouldn't solve it in software - it's more a network concern. I'd be looking at a web server solution such as Apache mod_evasive combined with a good firewall for more conventional DOS attacks. Just hide your Solr install behind the