Re: How to pass additional information into Similarity.scorePayload(...)

2008-02-15 Thread Cedric Ho
Hi Paul, Do you mean the following? e.g. to index this: first second third paragraphBorder forth fifth six originally it would be indexed as: (first,0) (second,1) (third,2) (forth,3) (fifth,4) (six,5) now it will be: (first,0) (second,0) (third,0) (forth,1) (fifth,1) (six,1) Then those Query

Re: Design questions

2008-02-15 Thread Adrian Smith
Hi, I have a similar sitaution. I also considered using $. But for the sake of not running into (potential) problems with Tokenisers, I just defined a string in a config file which for sure is never going to occur in a document and will never be searched for, e.g. dfgjkjrkruigduhfkdgjrugr

Re: How to pass additional information into Similarity.scorePayload(...)

2008-02-15 Thread Grant Ingersoll
On Feb 14, 2008, at 3:44 AM, Cedric Ho wrote: Anyway, I know my requirement is a bit strange, so it's ok if I can't do this in Lucene. I'll settle with using a ThreadLocal to store the [10/5/2] weighting and retrieve it in the Similarity.scorePayload(...) function. I don't think you are

RE: Design questions

2008-02-15 Thread spring
Document doc = new Document() for (int i = 0; i pages.length; i++) { doc.add(new Field(text, pages[i], Field.Store.NO, Field.Index.TOKENIZED)); doc.add(new Field(text, $$, Field.Store.NO, Field.Index.UN_TOKENIZED)); } UN_TOKENIZED. Nice idea! I will check this out. 2) if

RE: Design questions

2008-02-15 Thread spring
Document doc = new Document() for (int i = 0; i pages.length; i++) { doc.add(new Field(text, pages[i], Field.Store.NO, Field.Index.TOKENIZED)); doc.add(new Field(text, $$, Field.Store.NO, Field.Index.UN_TOKENIZED)); } UN_TOKENIZED. Nice idea! I will check this

from 2.2.0 to 2.3.0

2008-02-15 Thread Sergey Kabashnyuk
Hi I want to update from lucene 2.2.0 to 2.3.0 but I have assert it TermInfosWriter line 143. What is it mean assert compareToLastTerm(fieldNumber, termText, termTextStart, termTextLength) 0 || (isIndex termTextLength == 0 lastTermTextLength == 0); Can you help me? Sergey Kabashnyuk

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Michael McCandless
Can you provide the full stack trace, and any details of what you were doing when you tripped that assert? Another thing to try is to run the org.apache.lucene.index.CheckIndex tool (new in 2.3) to check your 2.2 index. It's possible your 2.2 index has some undetected corruption which

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Sergey Kabashnyuk
stack trace java.lang.AssertionError at org.apache.lucene.index.TermInfosWriter.add(TermInfosWriter.java:143) at org.apache.lucene.index.DocumentsWriter.appendPostings(DocumentsWriter.java:2290) at

RE: Design questions

2008-02-15 Thread spring
Well, it seems that this may be a solution for me too. But I'm afraid that someone one day will change this string. And then my app will not work anymore... -Original Message- From: Adrian Smith [mailto:[EMAIL PROTECTED] Sent: Freitag, 15. Februar 2008 13:02 To:

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Michael McCandless
CheckIndex tool is in the 2.3 JAR. Just make sure the JAR is on your CLASSPATH then run: java org.apache.lucene.index.CheckIndex indexDirectory However, from that stack trace, this is not a corruption issue in your existing index. My guess is you would hit this same exception if you

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Michael McCandless
OK I found one case that can cause the assert to trip incorrectly. Are you using empty string field names? I'll open a Jira issue fix it on the 2.3 branch (so this is fixed in 2.3.1). Thanks for reporting this! Mike Sergey Kabashnyuk wrote: stack trace java.lang.AssertionError

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Sergey Kabashnyuk
On Fri, 15 Feb 2008 16:31:07 +0200, Michael McCandless [EMAIL PROTECTED] wrote: OK I found one case that can cause the assert to trip incorrectly. Are you using empty string field names? Not. May be empty content of the field. I have another question. I see you use asserts in code 2.3.0

Which parser or query should I be using

2008-02-15 Thread Mitchell, Erica
trying to send this again... i keep getting delivery failures so i've attached my code snippet instead My index from Luke looks like this where the columns are for Rank/Field/Text 2 guid 2 1 guid pi1 1 statement Some policy statement for pi3, needs some text with a 2 in it 1 guid

Re: from 2.2.0 to 2.3.0

2008-02-15 Thread Michael McCandless
Sergey Kabashnyuk wrote: On Fri, 15 Feb 2008 16:31:07 +0200, Michael McCandless [EMAIL PROTECTED] wrote: OK I found one case that can cause the assert to trip incorrectly. Are you using empty string field names? Not. May be empty content of the field. Hmmm, ok. Empty content is fine.

Re: Design questions

2008-02-15 Thread Erick Erickson
You need to watch both the positionincrementgap (which, as I remember, gets added for each new field of the same name you add to the document). Make it 0 rather than whatever it is currently. You may have to create a new analyzer by subclassing your favorite analyzer and overriding the

RE: Design questions

2008-02-15 Thread spring
You need to watch both the positionincrementgap (which, as I remember, gets added for each new field of the same name you add to the document). Make it 0 rather than whatever it is currently. You may have to create a new analyzer by subclassing your favorite analyzer and overriding the

Boost value corrupted read eof exception

2008-02-15 Thread Allahbaksh Mohammedali Asadullah
Hi, I am getting read past eof exception when I am adding Document. The things works fine but while I read the Boost value I get it as default value i.e 1.0. I tried using RAMDirectory then also I am getting the same error. Can someone throw some light on this. Warm Regards, Allahbaksh

Re: Boost value corrupted read eof exception

2008-02-15 Thread Michael McCandless
Which version of Lucene are you using? Can you post the full stack trace for the exception you're hitting? Mike Allahbaksh Mohammedali Asadullah wrote: Hi, I am getting read past eof exception when I am adding Document. The things works fine but while I read the Boost value I get it as

Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread lmctndi
Hi all, [ Sorry for the cryptic title but I can't think of a better one] I am trying to integrate Lucene as a search engine for my database and would appreciate some help on a problem. First, some info about the development platform, I am developing on Windows using Hibernate for DB

Re: Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread Chris Lu
Hi, lmctndi, You can liberate yourself from this complicated schema, and use some SQLs to select out the content and put the data into one Lucene index. You may need to avoid Hibernate Search here for your case, which limit you to one way of organizing Lucene indexes. I do think this is where

Re: Boost value corrupted read eof exception

2008-02-15 Thread Michael McCandless
I had to comment out the search lines. But then when I run this I don't hit any exception. Tested on Linux Windows XP. Mike Allahbaksh Mohammedali Asadullah wrote: Hi, I am using lucene 2.3. The exception is properly handled when I run the program by lucene but when I debug it it

Boost value corrupted read eof exception

2008-02-15 Thread Allahbaksh Mohammedali Asadullah
Hi, I am using lucene 2.3. The exception is properly handled when I run the program by lucene but when I debug it it hit the exception at addDocuement. Regards, Allahbaksh import java.io.IOException; import java.io.StringReader; import org.apache.lucene.search.Hits; import

RE: Boost value corrupted read eof exception

2008-02-15 Thread Allahbaksh Mohammedali Asadullah
Hi, Thanks for your valuable time.Could you get the same boost value what you had stored. In my case I always get boost value as 1 ie. Default value. When I debug it goes through IOException though it does not show any Exception on the console in eclipse. Regards, Allahbaksh -Original

Re: Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread Erick Erickson
To expand a bit on Chris's first point: Take off your DB hat and put on your search hat G. It sounds like you have simply moved your database tables into Lucene and want to search across them. My rule is that whenever you find yourself trying to make Lucene act like a DB, you need to pause and

Re: Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread lmctndi
Thanks for your reply. Your idea prompts more questions: I understand what you are saying but don't know how to implement it. How do you go about joining all rows of all the tables belonging to one person and to index them so that I can actually use +Fname:john +County:USA as a query? Erick

Re: Boost value corrupted read eof exception

2008-02-15 Thread Chris Hostetter
Reading between the lines, I suspect two things are happening here, both of which seems like missunderstandings ... 1) i think the IOException seen while debugging is refering to an exception which is handled internally in Lucene and is not actually a problem -- but Allahbaksh is seeing it

Re: Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread Chris Lu
Sorry, sent the previous draft email by mistake. Here is the correct one. Sounds a typical SQL pivot problem. select Id, SIN, data.* from IdCard, (SELECT ID MAX(CASE WHEN name = 'Fname' THEN Value END) AS Fname, MAX(CASE WHEN name = 'Lname' THEN Value END) AS Lname,

Re: How to pass additional information into Similarity.scorePayload(...)

2008-02-15 Thread Paul Elschot
Hi Cedric, I think I'm beginning to get the point of the [10/5/2], and why you called that requirement a bit strange, see below. To use both normal position info and paragraph position info you'll need two separate, one normal, and one for the paragraphs. To use the normal field to determine

Re: Searching for multiple criteria (accross 2 tables)

2008-02-15 Thread Jake Mannix
What the other posters are referring to is that you will have to probably write some java code to do lucene indexing: you can get access to your model objects (with all their dependent data) in java. - since you are using hibernate, this shouild be easy- then create lucene documents from your

Re: DefaultIndexAccessor

2008-02-15 Thread vivek sar
Mark, There seems to be some issue with DefaultMultiIndexAccessor.java. I got following NPE exception, 2008-02-13 07:10:28,021 ERROR [http-7501-Processor6] ReportServiceImpl - java.lang.NullPointerException at

Re: DefaultIndexAccessor

2008-02-15 Thread Mark Miller
Hey vivek, sorry to hear you are having problems. I am trying to figure out how you may be seeing this problem. The IndexAccessor cannot return null because you would get an IllegalStateException not a NullPointerException. Also, the released MultiSearcher cannot be null because the Exception

Re: DefaultIndexAccessor

2008-02-15 Thread Mark Miller
Okay, sorry about this one vivek. Added to the unit tests to expose this. When I took out the MultiSearcher caching, I kept the concept of sharing a single MultiIndexAccessor. Unfortunately, this meant that multiple threads were sharing the same Searcher to Accessor Map that was used to track

Re: DefaultIndexAccessor

2008-02-15 Thread vivek sar
Mark, Here is the scenario when I saw this exception, 1) A search was run which uses MultiSearcher. This search took more than 3 mins to complete (due to index size and multiple indices) 2) Just a minute after the search was started, we started writing (in a separate thread) to one of the

Re: DefaultIndexAccessor

2008-02-15 Thread Mark Miller
Here is the fix: https://issues.apache.org/jira/browse/LUCENE-1026 vivek sar wrote: Mark, There seems to be some issue with DefaultMultiIndexAccessor.java. I got following NPE exception, 2008-02-13 07:10:28,021 ERROR [http-7501-Processor6] ReportServiceImpl -

Re: DefaultIndexAccessor

2008-02-15 Thread vivek sar
Mark, Thanks for the quick fix. Actually, it is possible that there might had been simultaneous queries using the MultiSearcher. I assumed it was thread-safe, thus was re-using the same instance. I'll update my application code as well. Thanks, -vivek On Feb 15, 2008 5:56 PM, Mark Miller [EMAIL

Re: How to pass additional information into Similarity.scorePayload(...)

2008-02-15 Thread Cedric Ho
Thanks ~ Yes it seems this would be quite difficult to achieve with Lucene. Nevermind, I'll try to figure out a workaround for it. Thanks for helping =) Cedric On Feb 16, 2008 5:30 AM, Paul Elschot [EMAIL PROTECTED] wrote: Hi Cedric, I think I'm beginning to get the point of the [10/5/2],

RE: Boost value corrupted read eof exception

2008-02-15 Thread Allahbaksh Mohammedali Asadullah
Hi Chris and Mike, Thanks for your help. Chis you got me right for both points. Sorry that I could not put my queries properly. As I can't set and get boost value I have to do my implementation some other way. Please help me in the same I have a three column table which I am trying to save