Re: Null or no analyzer
Erik Hatcher writes: however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? I think one could introduce a special syntax in query parser for keyword fields. Query parser wouldn't analyze them at all in this case. Something like field#Keyword or field#keyword containing blanks I haven't thought through all consequences for field#(keywordA keywordB otherfield:noKeyword) but I think it should be doable. Doesn't make query parser simpler, on the other hand. Morus - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
I don't like the idea of users having to know how a field was indexed though. That seems to defeat the purpose of a general-purpose QueryParser. Erik On Oct 21, 2004, at 2:38 AM, Morus Walter wrote: Erik Hatcher writes: however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? I think one could introduce a special syntax in query parser for keyword fields. Query parser wouldn't analyze them at all in this case. Something like field#Keyword or field#keyword containing blanks I haven't thought through all consequences for field#(keywordA keywordB otherfield:noKeyword) but I think it should be doable. Doesn't make query parser simpler, on the other hand. Morus - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
Erik Hatcher wrote: I don't like the idea of users having to know how a field was indexed though. That seems to defeat the purpose of a general-purpose QueryParser. Erik I agree that, but maybe lucene should provide some subclasses of QueryParser that should deal this problems. I'm just a lucene user, not a lucene developer, but I have had to implement a Extension for MultifieldQueryParser to fix some not wanted behaviour that I already discussed in the mailing list. These problems that user face with creating the right qeury strings, (with the special case of untokenized fileds) togheter with MultifieldQueryParser problems, MultiSearcher problems ... I think that all together suggest the idea of creating a QueryParser class hierarchy. What do you think about that? All the best, Sergiu On Oct 21, 2004, at 2:38 AM, Morus Walter wrote: Erik Hatcher writes: however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? I think one could introduce a special syntax in query parser for keyword fields. Query parser wouldn't analyze them at all in this case. Something like field#Keyword or field#keyword containing blanks I haven't thought through all consequences for field#(keywordA keywordB otherfield:noKeyword) but I think it should be doable. Doesn't make query parser simpler, on the other hand. Morus - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
On Oct 21, 2004, at 5:38 AM, sergiu gordea wrote: Erik Hatcher wrote: I don't like the idea of users having to know how a field was indexed though. That seems to defeat the purpose of a general-purpose QueryParser. Erik I agree that, but maybe lucene should provide some subclasses of QueryParser that should deal this problems. I'm just a lucene user, not a lucene developer, but I have had to implement a Extension for MultifieldQueryParser to fix some not wanted behaviour that I already discussed in the mailing list. These problems that user face with creating the right qeury strings, (with the special case of untokenized fileds) togheter with MultifieldQueryParser problems, MultiSearcher problems ... I think that all together suggest the idea of creating a QueryParser class hierarchy. What do you think about that? Query parsing/expansion is the holy grail. There are so many ways to do this sort of thing that I'm mostly of the opinion it is a per-project customization to get it tuned for the needs of that project. Nutch has done some nice things with query parsing/expansion and extensibility. I'm all for a more extensible base to work from, no question. I'm personally not fond of MultiFieldQueryParser - I much prefer aggregate fields that are indexed (not stored) to be used for queries. Blindly expanding queries across fields doesn't seem that useful to me. Erik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
Erik Hatcher wrote: On Oct 21, 2004, at 5:38 AM, sergiu gordea wrote: Erik Hatcher wrote: I don't like the idea of users having to know how a field was indexed though. That seems to defeat the purpose of a general-purpose QueryParser. Erik I agree that, but maybe lucene should provide some subclasses of QueryParser that should deal this problems. I'm just a lucene user, not a lucene developer, but I have had to implement a Extension for MultifieldQueryParser to fix some not wanted behaviour that I already discussed in the mailing list. These problems that user face with creating the right qeury strings, (with the special case of untokenized fileds) togheter with MultifieldQueryParser problems, MultiSearcher problems ... I think that all together suggest the idea of creating a QueryParser class hierarchy. What do you think about that? Query parsing/expansion is the holy grail. There are so many ways to do this sort of thing that I'm mostly of the opinion it is a per-project customization to get it tuned for the needs of that project. Nutch has done some nice things with query parsing/expansion and extensibility. I'm all for a more extensible base to work from, no question. I'm personally not fond of MultiFieldQueryParser - I much prefer aggregate fields that are indexed (not stored) to be used for queries. Blindly expanding queries across fields doesn't seem that useful to me. In my case is very usefull. Because my search has constaints like 1) has xxx file format attachment 2) has xxx type 3) was created by xxx 4) search in attachmets or not so ... I cannot make this customization without indexing in more fields and searching in more fields. Creating the queryString by adding filed:keyword pair is just a hardly maintainable way of reinventing the wheel. So .. in may case, MultifieldQueryParser is very useful, because I haave to add some boolean clauses after I create the base query. Last month I just refactored the method that created the search query for our extended search functionality. It was a method with 200 lines of structural code (no query parser used). using Boolean clauses and MultifieldQueryParser helped me a lot ... and the result was a method with fewer, easily maintainable lines of code. Of course ... this is needed in my project, but I think that almost all lucene indexes contain more then 2-3 fileds. So ... once again MultifieldQueryParser is an elegent solution. Sergiu Erik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Null or no analyzer
Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. Morus -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 19, 2004 11:23 AM To: Lucene Users List Subject: Null or no analyzer Hi All I have a question regarding selection of Analyzer's during query parsing i have three field in my index db_id, full_text, subject all three are indexed, however while indexing I specified to lucene to index db_id and subject but not tokenize them I want to give a single search box in my application to enable searching for documents some query can look lile motor cross rally this will get fed to QueryParser to do the relevent parsing however if the user enters Jhon Kerry subject:Elections 2004 I want to make sure that No analyzer is used fro the subject field ? how can that be done. this is because I expect the users to know the subject from a List of controlled vocabularies and also I am searching for documents that have the exact subject I tried using the PerFieldAnalyzerWrapper, but how do I get hold a Analyzer that does nothing but pass the text trough to the Searcher ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Null or no analyzer
AFIK if the term Election 2004 will be between quotation marks this should work fine. Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
On Oct 20, 2004, at 9:55 AM, Aviran wrote: AFIK if the term Election 2004 will be between quotation marks this should work fine. No, it won't. The Analyzer will analyze it, and the WhitespaceAnalyzer would split it into two tokens [Election] and [2004]. This is a tricky situation with no clear *best* way to do this sort of thing. However, given what I've seen of this thread so far I'd recommend using the PerFieldAnalyzerWrapper and associate the fields indexed as Field.Keyword with a KeywordAnalyzer. There have been some variants of this posted on the list - it is not included in the API, however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? Erik Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Null or no analyzer
Hi Erik I think the best solutuion is to have a NullAnalayzer class that allows a simple pass through The query parser then can be passed with a PerFieldAnalayzer that knows when to select NullAnalayzer or some other based on the Field:data... Field2:pp format this is something that the query parser is already geared up to do regards Rupinder -Original Message- From: Erik Hatcher [mailto:[EMAIL PROTECTED] Sent: 20 October 2004 16:29 To: Lucene Users List Subject: Re: Null or no analyzer On Oct 20, 2004, at 9:55 AM, Aviran wrote: AFIK if the term Election 2004 will be between quotation marks this should work fine. No, it won't. The Analyzer will analyze it, and the WhitespaceAnalyzer would split it into two tokens [Election] and [2004]. This is a tricky situation with no clear *best* way to do this sort of thing. However, given what I've seen of this thread so far I'd recommend using the PerFieldAnalyzerWrapper and associate the fields indexed as Field.Keyword with a KeywordAnalyzer. There have been some variants of this posted on the list - it is not included in the API, however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? Erik Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
Erik Hatcher wrote: On Oct 20, 2004, at 9:55 AM, Aviran wrote: AFIK if the term Election 2004 will be between quotation marks this should work fine. No, it won't. The Analyzer will analyze it, and the WhitespaceAnalyzer would split it into two tokens [Election] and [2004]. This is a tricky situation with no clear *best* way to do this sort of thing. However, given what I've seen of this thread so far I'd recommend using the PerFieldAnalyzerWrapper and associate the fields indexed as Field.Keyword with a KeywordAnalyzer. There have been some variants of this posted on the list - it is not included in the API, however perhaps it should be. Or perhaps there are other options to solve this recurring dilemma folks have with Field.Keyword indexed fields and QueryParser? Erik I still don't understand what is wrong with the Idea of indexing the title in a separate field and searching with a Phrase query +title:Elections 2004 ? I think that the real problem is that the title is not tokenized and the title contains more then Elections 2004 I think it is worthing to give a try to this solution. Or maybe I don't understand the problem correctly ... All the best, Sergiu Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Null or no analyzer
hi the basic problem here is that there are data source which contain a) id, b) text c) title d) authors AND d) subject heading text, title and authors need to be tokenized the subject heading can be one or more words, anyone searching such datasource is expected to know the subject headings , if the user is trying to find all articles that have the phrases Jhon Kerry and Goerge Bush as well as that are classified as Election 2004 it is possible that there are other documents that are classified as Nation Service Records or Tax Returns etc... so the object is to find documents that have the above mentioned phrases as well as one one of the subject classifiers, so as to pull out the most meaning full documents the subject classifiers pretain to domain knowledge, and it is possible that 2 or more subject classification headings are composed of the same set of words, but the sequence in which they appear can drastically alter the meaning hence tokenizing the subject field is not exactly a healthy solution. also such search tools are meant for people who know / understand this classification system Taxonomy of animals can be taken as one such example, hope this helps define the problem I still don't understand what is wrong with the Idea of indexing the title in a separate field and searching with a Phrase query +title:Elections 2004 ? I think that the real problem is that the title is not tokenized and the title contains more then Elections 2004 I think it is worthing to give a try to this solution. Or maybe I don't understand the problem correctly ... All the best, Sergiu Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Null or no analyzer
Rupinder Singh Mazara wrote: hi the basic problem here is that there are data source which contain a) id, b) text c) title d) authors AND d) subject heading text, title and authors need to be tokenized the subject heading can be one or more words, the subject must be also tokennized, otherwise you cannot get any results that doesn't match the Term exaclty so ... for example, let's asume you have the folowing titles: George Trash Elections George Trash if you search for George Trash and your title is not tokenized you will get just the second document (I hope I'm not making any mistake when I say that, anyway it can be easily tested). anyone searching such datasource is expected to know the subject headings , if the user is trying to find all articles that have the phrases Jhon Kerry and Goerge Bush as well as that are classified as Election 2004 it is possible that there are other documents that are classified as Nation Service Records or Tax Returns etc... how is there represented in the GUI as a select box? or input field? if it is select box, if you have the concept of unique domain concept .. you can use a a not tokenized string, or even a numerical representation, but I think it is not your case. In the case of input fields .. again I suggest you to tokenize the string so the object is to find documents that have the above mentioned phrases as well as one one of the subject classifiers, so as to pull out the most meaning full documents no problem ... once again .. use +subject:my searched subject the subject classifiers pretain to domain knowledge, and it is possible that 2 or more subject classification headings are composed of the same set of words, but the sequence in which they appear can drastically alter the meaning hence tokenizing the subject field is not exactly a healthy solution. the tokenization doesn't change the word order, in the case you use a PhraseQuery you will get the correct results +title:George Bush doesn't return documents with the title Bush George also such search tools are meant for people who know / understand this classification system :)) This is a general truth the the result are better when the people know what they are searching for :) Taxonomy of animals can be taken as one such example, hope this helps define the problem I cannot see anything special in your problem. Before strating to implement a complex solution probably will be better to give it a chance to the simple one ... I ensure you that you won't loose anything, and even if you decide to implement complex solutions you will have a lot of reusable code. so ... Have fun, Sergiu PS: if you can provide an example with a false positive please ... provide us the case I still don't understand what is wrong with the Idea of indexing the title in a separate field and searching with a Phrase query +title:Elections 2004 ? I think that the real problem is that the title is not tokenized and the title contains more then Elections 2004 I think it is worthing to give a try to this solution. Or maybe I don't understand the problem correctly ... All the best, Sergiu Aviran http://aviran.mordos.com -Original Message- From: Morus Walter [mailto:[EMAIL PROTECTED] Sent: Wednesday, October 20, 2004 2:25 AM To: Lucene Users List Subject: RE: Null or no analyzer Aviran writes: You can use WhiteSpaceAnalyzer Can he? If Elections 2004 is one token in the subject field (keyword), this will fail, since WhiteSpeceAnalyzer will tokenize that to `Elections' and `2004'. So I guess he has to write an identity analyzer himself unless there is one provided (which doesn't seem to be the case). The only alternatives are not using query parser or extending query parser for a key word syntax, as far as I can see. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Null or no analyzer
You can use WhiteSpaceAnalyzer Aviran http://aviran.mordos.com -Original Message- From: Rupinder Singh Mazara [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 19, 2004 11:23 AM To: Lucene Users List Subject: Null or no analyzer Hi All I have a question regarding selection of Analyzer's during query parsing i have three field in my index db_id, full_text, subject all three are indexed, however while indexing I specified to lucene to index db_id and subject but not tokenize them I want to give a single search box in my application to enable searching for documents some query can look lile motor cross rally this will get fed to QueryParser to do the relevent parsing however if the user enters Jhon Kerry subject:Elections 2004 I want to make sure that No analyzer is used fro the subject field ? how can that be done. this is because I expect the users to know the subject from a List of controlled vocabularies and also I am searching for documents that have the exact subject I tried using the PerFieldAnalyzerWrapper, but how do I get hold a Analyzer that does nothing but pass the text trough to the Searcher ? - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]