Hi Kurosaka-san,

I'd written an article on my blog several month ago about SinkTokenizer
and TeeTokenFilter.

See:
http://lucene.jugem.jp/?eid=172

Sorry, but all written in Japanese...

Koji


Teruhiko Kurosaka wrote:
> Hello,
> I'm interested in knowing how these tokenizers work together.
> The API doc for TeeTokenizer
> http://lucene.apache.org/java/2_3_1/api/org/apache/lucene/analysis/TeeTokenFilter.html
>
> has this sample code:
> SinkTokenizer sink1 = new SinkTokenizer(null);
> SinkTokenizer sink2 = new SinkTokenizer(null);
>
> TokenStream source1 = new TeeTokenFilter(new TeeTokenFilter(new 
> WhitespaceTokenizer(reader1), sink1), sink2);
> TokenStream source2 = new TeeTokenFilter(new TeeTokenFilter(new 
> WhitespaceTokenizer(reader2), sink1), sink2);
>
> TokenStream final3 = new EntityDetect(sink1);
> TokenStream final4 = new URLDetect(sink2);
>
> with an explanation that reads "sink1 and sink2 will both get tokens from 
> both reader1 and reader2 after whitespace tokenizer",
> but I don't understand how the input from reader1 and reader2 are mixed 
> together.
> Will sink1 first reaturn the reader1 text, and reader2?
> Or are they mixed randomly?
>
> -Kuro
>  
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [EMAIL PROTECTED]
> For additional commands, e-mail: [EMAIL PROTECTED]
>
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

Reply via email to