Yes I Do.
On Tue, Jan 7, 2014 at 3:59 PM, Robert Muir <rcm...@gmail.com> wrote:
> Benson, do you want to open an issue to fix this constructor to not
> take Reader? (there might be one already, but lets make a new one).
>
> These things are supposed to be reused, and have setReader for that
> purpose. i think its confusing and contributes to bugs that you have
> to have logic in e.g. the ctor THEN ALSO in reset().
>
> if someone does it correctly in the ctor, but they only test "one
> time", they might think everything is working..
>
> On Tue, Jan 7, 2014 at 3:23 PM, Benson Margulies <ben...@basistech.com>
> wrote:
> > For the record of other people who implement tokenizers:
> >
> > Say that your tokenizer has a constructor, like:
> >
> > public MyTokenizer(Reader reader, ....) {
> > super(reader);
> > myWrappedInputDevice = new MyWrappedInputDevice(reader);
> > }
> >
> > Not a good idea. Tokenizer carefully manages the data flow from the
> > constructor arg to the 'input' field. The correct form is:
> >
> > public MyTokenizer(Reader reader, ....) {
> > super(reader);
> > myWrappedInputDevice = new MyWrappedInputDevice(this.input);
> > }
> >
> >
> >
> > On Tue, Jan 7, 2014 at 2:59 PM, Robert Muir <rcm...@gmail.com> wrote:
> >
> >> See Tokenizer.java for the state machine logic. In general you should
> >> not have to do anything if the tokenizer is well-behaved (e.g. close
> >> calls super.close() and so on).
> >>
> >>
> >>
> >> On Tue, Jan 7, 2014 at 2:50 PM, Benson Margulies <bimargul...@gmail.com
> >
> >> wrote:
> >> > In 4.6.0,
> >> org.apache.lucene.analysis.BaseTokenStreamTestCase#checkResetException
> >> >
> >> > fails if incrementToken fails to throw if there's a missing reset.
> >> >
> >> > How am I supposed to organize this in a Tokenizer? A quick look at
> >> > CharTokenizer did not reveal any code for the purpose.
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> >> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
