Hi,
I have a custom analyzer
It will use different tokenizers depending on the field.
There is a field in my system called Item_Code, that field is using the
KeywordTokenizer.
Any other field uses the WhitespaceTokenizer followed up with the
Snowballfilter
<code>
Public Overloads Overrides Function TokenStream(ByVal fieldName As
String, ByVal reader As TextReader) As TokenStream
If fieldName = "item_code" Then
Return New Lucene.Net.Analysis.KeywordTokenizer(reader)
Else
Dim x = New Lucene.Net.Analysis.WhitespaceTokenizer(reader)
Return New Lucene.Net.Analysis.Snowball.SnowballFilter(New
SynonymFilter(x), "English")
End If
End Function
</code>
-----Original Message-----
From: Jokin Cuadrado [mailto:[email protected]]
Sent: Tuesday, May 12, 2009 4:11 AM
To: [email protected]
Subject: Re: Snowball Filter and Quotes
Could you post the code where you construct your analyzer? you use the
whitespacefilter, but what tokenizer are you using?
On Tue, May 12, 2009 at 2:58 AM, Heath Aldrich <[email protected]>
wrote:
> Sorry in advance if this should be in the dev list...
>
>
>
> I have a index generator that uses the Snowball filter.
>
> It also uses the Whitespace filter so as to not remove anything but
> white space.
>
>
>
> When I look at the raw data in Luke, it seems like all the quotes in
my
> data have been stripped out.
>
> Just trying to find out if anyone else has seen this, and if anyone
> knows if the Snowball filter is responsible.
>
>
>
> Thanks in advance.
>
> Heath
>
>
--
Jokin
