On 7/8/06, Charlie <[EMAIL PROTECTED]> wrote:
> David Balmain wrote:
> > On 7/7/06, Charlie <[EMAIL PROTECTED]> wrote:
> >> And also it is needed to make the new Chinese analyzer work together
> >> with the original standard analyzer
> >
> > I answered this on the rails list but just in case;
> >
> > # Create a PerFieldAnalyzer (AKA PerFieldAnalyzerWrapper) which
> > # defaults to Standard
> > analyzer = PerFieldAnalyzer.new(StandardAnalyzer.new)
> >
> > # Add a special character analyzer for the chinese field or
> > # whatever field it is that has chinese characters. This splits the
> > # data into single characters.
> > analyzer["chinese"] = RegExpAnalyzer.new(/./, false)
>
> Thank you Dave,I looked up the API and found that
> PerFieldAnalyzerWrapper is useful for field analyze,especially for the
> coresponding SQL: select * from students where title like '%Charlie%'
> and location_id = 1, where location_id =1 query can be got through
> PerFieldAnalyzerWrapper.
>
> I have just now downloaded and read the book of Lucene In Action,and in
> Chapter 4,it tolds that the standardanalyzer will also split the CJK
> language into tokens although there is no spaces among them,for
> example:'中文字符' will be splitted into tokens of '中' '文' '字' '符',that is
> just what I want. But I still can not search any results from ferret. I
> use the MySQL as the database with all the encoding of UTF-8,and
> also,all of my rails sources is saved in the form of UTF-8,then when I
> input the search box of the above characters of
> '中文字符', I will got zero searched results,can you please help with that
> situation? Very Grateful!
Hi Charlie,
The StandardAnalyzer in Ferret works a little differently to the
StandardAnalyzer in Lucene. That's why you need to use the
RegExpAnalyzer I gave you.
analyzer = PerFieldAnalyzer.new(StandardAnalyzer.new)
analyzer["chinese"] = RegExpAnalyzer.new(/./, false)
You also need to make sure that this is the analyzer that is being
used by the query parser. If you are using the Index::Index class it
will handle it for you. Try this in irb;
$ irb -KU
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'ferret'
=> true
irb(main):003:0> include Ferret::Index
=> Object
irb(main):004:0> include Ferret::Analysis
=> Object
irb(main):005:0> analyzer = PerFieldAnalyzer.new(StandardAnalyzer.new)
=> #<Ferret::Analysis::PerFieldAnalyzer:0xb7b2332c>
irb(main):006:0> analyzer["chinese"] = RegExpAnalyzer.new(/./, false)
=> #<Ferret::Analysis::RegExpAnalyzer:0xb7c8bdd4>
irb(main):007:0> index = Index.new(:analyzer => analyzer)
=> #<Ferret::Index::Index:0xb7bbda30>
irb(main):008:0> index << {:english => "the quick brown fox jumped
over the lazy dog", :chinese => '中文字符'}
=> #<Ferret::Index::Index:0xb7bbda30>
irb(main):009:0> index << {:chinese => "the quick brown fox jumped
over the lazy dog", :english => '中文字符'}
=> #<Ferret::Index::Index:0xb7bbda30>
irb(main):010:0> index.search_each("chinese:中") {|doc, score| puts
"found in #{doc}"}
found in 0
=> 1
irb(main):011:0> index.search_each("english:中") {|doc, score| puts
"found in #{doc}"}
=> 0
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
