I believe that Lucene only indexes Unicode (or as Henry Ford might say "you can search any encoding as long as it's Unicode"). Therefore, you have to translate Big5 and GB1312 to Unicode before you put them in the index. Your code that reads in the html needs to be smart enough to notice how the html is encoded and make the translation before it creates the Document. Likewise, your search terms need to be Unicode.
Btw--you will also need to use something like Che Dong's CJK Analyzer. I don't believe the standard analyzer's will handle Asian languages in any kind of reasonable fashion. Disclaimer: I am new to Lucene (about 6 months) and to processing Asian languages. In my defense, I do have a number of happy users using Lucene to search Japanese and Chinese XML files for things of interest. -----Original Message----- From: Tuan Jean Tee [mailto:[EMAIL PROTECTED] Sent: Monday, March 15, 2004 11:07 PM To: [EMAIL PROTECTED] Subject: Can lucene index both Big5 and GB2312 encoding character? Can I find out if I have both Big5 and GB2312 encoded HTML files in two separate directories, and when I build the index, does Lucene able to distinguish the character set? or Lucene only work with single encoding. Thank you. IMPORTANT - This email and any attachments are confidential and may be privileged in which case neither is intended to be waived. If you have received this message in error, please notify us and remove it from your system. It is your responsibility to check any attachments for viruses and defects before opening or sending them on. Where applicable, liability is limited by the Solicitors Scheme approved under the Professional Standards Act 1994 (NSW). Minter Ellison collects personal information to provide and market our services. For more information about use, disclosure and access, see our privacy policy at www.minterellison.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
