Thank you,
that was the reason.
> From: j...@basetechnology.com
> To: java-user@lucene.apache.org
> Subject: Re: WhitespaceTokenizer, incrementToke() ArrayOutOfBoundException
> Date: Mon, 15 Apr 2013 10:25:26 -0400
>
> I didn't read your code, but do you have the "reset" that is now mandatory
> and throws AIOOBE if not present?
>
> -- Jack Krupansky
>
> -----Original Message-----
> From: andi rexha
> Sent: Monday, April 15, 2013 10:21 AM
> To: java-user@lucene.apache.org
> Subject: WhitespaceTokenizer, incrementToke() ArrayOutOfBoundException
>
> Hi,
> I have tryed to get all the tokens from a TokenStream in the same way as I
> was doing in the 3.x version of Lucene, but now (at least with
> WhitespaceTokenizer) I get an exception:
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: -1
> at java.lang.Character.codePointAtImpl(Character.java:2405)
> at java.lang.Character.codePointAt(Character.java:2369)
> at
> org.apache.lucene.analysis.util.CharacterUtils$Java5CharacterUtils.codePointAt(CharacterUtils.java:164)
> at
> org.apache.lucene.analysis.util.CharTokenizer.incrementToken(CharTokenizer.java:166)
>
>
>
> The code is quite simple, and I thought that it could have worked, but
> obviously it doesn't (unless I have made some mistakes).
>
> Here is the code, in case you spot some bugs on it (although it is trivial):
> String str = "this is a test";
> Reader reader = new StringReader(str);
> TokenStream tokenStream = new WhitespaceTokenizer(Version.LUCENE_42,
> reader); //tokenStreamAnalyzer.tokenStream("test", reader);
> CharTermAttribute attribute =
> tokenStream.getAttribute(CharTermAttribute.class);
> while (tokenStream.incrementToken()) {
> System.out.println(new String(attribute.buffer(), 0,
> attribute.length()));
> }
>
> Hope you have any idea of why it is happening.
> Regards,
> Andi
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
