Hello Doug,
I feel interest on the task you gave to me and I
started to look into the code in analysis package.
I found something strange when I input some CJK
characters to NutchDocumentTokenizer:
public static void main(String[] args) throws
Exception {
BufferedReader in = new BufferedReader(new
InputStreamReader(System.in));
while (true) {
System.out.print("Text: ");
String line = in.readLine();
line = "\u8521"; //*** I add this
line.***
Tokenizer tokenizer = new
NutchDocumentTokenizer(new StringReader(line));
Token token;
System.out.print("Tokens: ");
while ((token = tokenizer.next()) != null) {
System.out.print(token.termText());
System.out.print(" ");
}
System.out.println();
}
}
Nothing is output. And program does not even go into
next() routine.
Is it a bug? Because I am absolute beginer of Javacc,
I can not figure out where the problem is immediately.
To create Bi-gram for CJK, is it ok that just change
the next() method? When it see CJK character, it looks
one character ahead and concatenate two characters
together for create a new lucene Token.
Best regards,
Jun
======= At 2004-06-03, 22:08:44 you wrote: =======
>Jun,
>
>Welcome!
>
>First, there are probably bugs in Nutch's handling of
character sets,
>both in the HTML parser, and in the search page
output. I would start
>by looking at this.
>
>Also, currently we attempt to tokenize CJK ideogram
into separate words.
> That could be buggy too! And it would be better
to index these as
>bigrams. Please look at the classes in
net.nutch.analysis. This is the
>tokenizer that's used for documents and queries. I
don't think it would
>be too hard to get this to generate bigrams for
adjacent CJK ideogram
>pairs. Does that make sense?
>
>Longer term we should consider automatic language
detection and use of a
>dictionary-based segmenter for Chinese. Does that
sound like a
>reasonable plan?
>
>Cheers,
>
>Doug
>
>Jun Cai wrote:
>> Hello nutch-developers,
>>
>> I would like to participate the Nutch development.
My
>> contribution could be Chinese language
>> Internationalization. My future interest would be
>> focus crawling by recursively training the crawled
>> document and user query refinement.
>>
>> Breif introduction of myself:
>>
>> Just finish my master study in University of
Saarland,
>> Germany supervised by Prof. Gerhard Weikum. Thesis
is
>> about bootstrapping ontology from Web documents.
>>
>> Best regards,
>>
>> Jun Cai
>> [EMAIL PROTECTED]
>> 2004-06-03
>>
>>
>>
>>
>>
>>
>> __________________________________
>> Do you Yahoo!?
>> Friends. Fun. Try the all-new Yahoo! Messenger.
>> http://messenger.yahoo.com/
>>
>>
>>
-------------------------------------------------------
>> This SF.Net email is sponsored by the new
InstallShield X.
>> From Windows to Linux, servers to mobile,
InstallShield X is the one
>> installation-authoring solution that does it all.
Learn more and
>> evaluate today!
http://www.installshield.com/Dev2Dev/0504
>> _______________________________________________
>> Nutch-developers mailing list
>> [EMAIL PROTECTED]
>>
https://lists.sourceforge.net/lists/listinfo/nutch-developers
>
>
>-------------------------------------------------------
>This SF.Net email is sponsored by the new
InstallShield X.
>From Windows to Linux, servers to mobile,
InstallShield X is the one
>installation-authoring solution that does it all.
Learn more and
>evaluate today!
http://www.installshield.com/Dev2Dev/0504
>_______________________________________________
>Nutch-developers mailing list
>[EMAIL PROTECTED]
>https://lists.sourceforge.net/lists/listinfo/nutch-developers
>.
= = = = = = = = = = = = = = = = = = = =
Jun Cai
[EMAIL PROTECTED]
2004-06-06
__________________________________
Do you Yahoo!?
Friends. Fun. Try the all-new Yahoo! Messenger.
http://messenger.yahoo.com/
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
>From Windows to Linux, servers to mobile, InstallShield X is the one
installation-authoring solution that does it all. Learn more and
evaluate today! http://www.installshield.com/Dev2Dev/0504
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
