On 7/19/06, Julio Cesar Ody <[EMAIL PROTECTED]> wrote: > Just sharing my experience and asking another question. > > I tried the analyzer suggested here: > http://www.ruby-forum.com/topic/72086#101764. It works fine if you > specify the search field you want to use (anyway, it seems that's how > it's suppose to work). > > # CODE > analyzer = > Ferret::Analysis::PerFieldAnalyzer.new(Ferret::Analysis::StandardAnalyzer.new) > analyzer["chinese"] = Ferret::Analysis::RegExpAnalyzer.new(/./, false) > > index = Index::Index.new(:path => '/var/index', :analyzer => analyzer, > :default_field => "*") > > ... > > index.search_each("chinese: #{val}") do |doc, score| #val is a chinese char > puts "#{doc} - #{score}" > end > # END CODE > > This works OK. However, if you try searching like this: > > # CODE > index.search_each(val) do |doc, score| #val is a chinese char > puts "#{doc} - #{score}" > end > # END CODE > > I get in my lighttpd error log: > > /var/www/localhost/htdocs/cgi-bin/search_chinese.ruby:19:in > `search_each': : Error occured at <analysis.c>:701 (StandardError) > Error: exception 2 not handled: Error decoding input string. Check > that you have the locale set correctly > from /var/www/localhost/htdocs/cgi-bin/search_chinese.ruby:19 > > Which MAKES SENSE, since the docs I created before are created like this: > > doc = { "author" => "englishchars", "title" => "more regular chars", > "chinese" => "新闻"} > index << doc > > and I think search_each is going through all the fields (since I > explicitly said it should when I issued :default_field => "*" up > there), finding english chars, and trying to match them against the > chinese ones I supplied as a search query.
Actually, it's not because of there is a comparison between Chinese and English characters. That shouldn't cause an error. The error is being thrown because val can't be decoded using the StandardAnalyzer. Again, you need to check that val is correctly encoded and you have your locale set correctly.The only times tokenizing happens are when you add documents to the index and when you run a query through the query parser. Apart from that, all operations on strings are done at the byte level. I hope that makes sense. > So alright, I can use the suggested analyzer. But my question is: is > there a way to use an analyzer that would work with both character > types (english, and asian) simply by not returning matches them as > opposed to giving me an error? > > Thanks a ton for any help. The answer to this question is that it already should work correctly. Just make sure the locale is set correctly when the search method is called and that whatever you pass as a query to the search method is correctly encoded according to the locale. Cheers, Dave _______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
