Thanks for the clarification, I'll play around with it and head back if things don't work according to plan.
Thanks again, e -----Original Message----- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 24, 2007 5:53 PM To: java-user@lucene.apache.org Subject: Re: Multiple Languages with Lucene (Arabic & English) You'll also find lots of discussion about indexing multiple languages if you search the mail archive for things like multiple language. I think one thing you're missing is that Lucene indexes data however you tell it to. You have both total control over and total responsibility for how things are indexed. You could, say, determine that a document was English, and index all of the fields like title_english text_english etc.. and then write that document out to your index. Say the next document was Arabic. You could index title_arabic text_arabic and write that document to your index. Now if you're searching in Arabic, you search on the title_arabic and text_arabic fields and you'd never get any hits on English documents. Not saying you want to do it this way, but you need to be clear that Lucene well do what you tell it to. Nothing more and nothing less <G>. Best Erick On 7/24/07, Grant Ingersoll <[EMAIL PROTECTED]> wrote: > > > On Jul 24, 2007, at 3:21 AM, Elie Choueiri wrote: > > > Hi > > > > > > > > I'm new to searching and am trying to use Lucene to search English > > & Arabic > > documents. I've got a bunch of questions (hopefully you'll find some > > interesting!) and am hoping someone's gone through some of them and > > has some > > answers for me! > > > > > > > > First, do I have to worry about the Arabic Analyzer overwriting the > > index > > files of the English analyzer? (Or vice versa?) > > > > i.e. When I index documents a second time, will data be overwritten? > > > > That depends whether you tell Lucene to create a new index or not. > See the IndexWriter API for your options. > > > > > > > I could just store the index files for different languages in a > > different > > location, but it's good to know and I'd rather not if I don't have > > to :) > > > > > > > > Also, on the same note, if I'm indexing documents that contain both > > Arabic > > and English, will the index files created by the English (or Arabic) > > analyzer contain garbage or become corrupted because of the language > > difference? > > > > I don't know if it will be corrupted, but probably won't be all that > useful, either. You may find the PerFieldAnalyzerWrapper to be helpful. > > > > > > > It is possible to index (using an English/Latin/Standard analyzer) > > a file > > that contains both english and arabic words, and expect the > > searches in > > English using the same analyzer to be valid, right? > > I should think so. I don't recall running across this case too much, > but do remember the reverse, Arabic files w/ some English and the > Arabic analyzer usually just skipped over the English leaving it > intact, thus searching those English terms in the Arabic index worked > just fine. > > > > > > > > > In an Arabic document with a single English word (the name of a > > corporation, > > for example) will the English word even be indexed and located by a > > search? > > I could test something like this with a small subset of documents, > > but I > > doubt the actual usefulness of a test with such a tiny (relatively > > speaking!) amount of data.. I know we can tell Lucene to store the > > full copy > > of the document, but does that affect the index itself? > > > > > > > > Finally, and here's the tricky one, are searches that contain both > > English > > and Arabic words valid? My limited understanding of the way search > > engines > > work tells me the search analyzes the context of words as well as > > statistical data to decide the relevance of hits, is this still > > valid for > > multi-lingual searches? > > > > > > They are valid, just not sure how useful, but that is for your app to > decide. I guess if your users know both Arabic and English, it > probably isn't a big deal. Lucene just tries to match up what is in > the query w/ what is in the index, so if you have validly analyzed > tokens in both the query and the index then Lucene should find them. > > HTH, > Grant > > -------------------------- > Grant Ingersoll > Center for Natural Language Processing > http://www.cnlp.org/tech/lucene.asp > > Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > -----Original Message----- From: Grant Ingersoll [mailto:[EMAIL PROTECTED] Sent: Tuesday, July 24, 2007 3:11 PM To: java-user@lucene.apache.org Subject: Re: Multiple Languages with Lucene (Arabic & English) On Jul 24, 2007, at 3:21 AM, Elie Choueiri wrote: > Hi > > > > I'm new to searching and am trying to use Lucene to search English > & Arabic > documents. I've got a bunch of questions (hopefully you'll find some > interesting!) and am hoping someone's gone through some of them and > has some > answers for me! > > > > First, do I have to worry about the Arabic Analyzer overwriting the > index > files of the English analyzer? (Or vice versa?) > > i.e. When I index documents a second time, will data be overwritten? > That depends whether you tell Lucene to create a new index or not. See the IndexWriter API for your options. > > > I could just store the index files for different languages in a > different > location, but it's good to know and I'd rather not if I don't have > to :) > > > > Also, on the same note, if I'm indexing documents that contain both > Arabic > and English, will the index files created by the English (or Arabic) > analyzer contain garbage or become corrupted because of the language > difference? > I don't know if it will be corrupted, but probably won't be all that useful, either. You may find the PerFieldAnalyzerWrapper to be helpful. > > > It is possible to index (using an English/Latin/Standard analyzer) > a file > that contains both english and arabic words, and expect the > searches in > English using the same analyzer to be valid, right? I should think so. I don't recall running across this case too much, but do remember the reverse, Arabic files w/ some English and the Arabic analyzer usually just skipped over the English leaving it intact, thus searching those English terms in the Arabic index worked just fine. > > > > In an Arabic document with a single English word (the name of a > corporation, > for example) will the English word even be indexed and located by a > search? > I could test something like this with a small subset of documents, > but I > doubt the actual usefulness of a test with such a tiny (relatively > speaking!) amount of data.. I know we can tell Lucene to store the > full copy > of the document, but does that affect the index itself? > > > > Finally, and here's the tricky one, are searches that contain both > English > and Arabic words valid? My limited understanding of the way search > engines > work tells me the search analyzes the context of words as well as > statistical data to decide the relevance of hits, is this still > valid for > multi-lingual searches? > > They are valid, just not sure how useful, but that is for your app to decide. I guess if your users know both Arabic and English, it probably isn't a big deal. Lucene just tries to match up what is in the query w/ what is in the index, so if you have validly analyzed tokens in both the query and the index then Lucene should find them. HTH, Grant -------------------------- Grant Ingersoll Center for Natural Language Processing http://www.cnlp.org/tech/lucene.asp Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
