-user@lucene.apache.org
Subject: Re: WhiteSpaceTokenizer
Thanks, Jack. I haven't added myself to the contributor list yet, will do
that and then login and comment on that ticket. One quick comment:
wouldn't it be more reasonable to throw exception it a token length is more
than 255, if relaxin
11:38 PM
> To: java-user@lucene.apache.org
> Subject: WhiteSpaceTokenizer
>
> The length of token has to be shorter than 255, otherwise there will
> be unpredictable behaviors for this tokenizer. I see 255 is set as a
> private final in the src code, but there is no documentation to
:
https://issues.apache.org/jira/browse/LUCENE-5785
-- Jack Krupansky
-Original Message-
From: Sheng
Sent: Thursday, August 14, 2014 11:38 PM
To: java-user@lucene.apache.org
Subject: WhiteSpaceTokenizer
The length of token has to be shorter than 255, otherwise there will
be unpredictable
The length of token has to be shorter than 255, otherwise there will
be unpredictable behaviors for this tokenizer. I see 255 is set as a
private final in the src code, but there is no documentation to explicitly
address that. Can we either make that number configurable (if not an
option, I'd like
:53 AM
To: java-user@lucene.apache.org
Subject: RE: WhitespaceTokenizer, incrementToke() ArrayOutOfBoundException
Hi,
It was always mandatory! In Lucene 2.x/3.x some Tokenizers just returned
bogus, undefined stuff if not correctly reset before usage, especially when
Tokenizers are "reused&q
213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de
> -Original Message-
> From: Jack Krupansky [mailto:j...@basetechnology.com]
> Sent: Monday, April 15, 2013 4:25 PM
> To: java-user@lucene.apache.org
> Subject: Re: WhitespaceTokenizer, incrementToke()
> ArrayOutOfBound
Thank you,
that was the reason.
> From: j...@basetechnology.com
> To: java-user@lucene.apache.org
> Subject: Re: WhitespaceTokenizer, incrementToke() ArrayOutOfBoundException
> Date: Mon, 15 Apr 2013 10:25:26 -0400
>
> I didn't read your code, but do you have the "
I didn't read your code, but do you have the "reset" that is now mandatory
and throws AIOOBE if not present?
-- Jack Krupansky
-Original Message-
From: andi rexha
Sent: Monday, April 15, 2013 10:21 AM
To: java-user@lucene.apache.org
Subject: WhitespaceTokenizer
Hi,
I have tryed to get all the tokens from a TokenStream in the same way as I was
doing in the 3.x version of Lucene, but now (at least with WhitespaceTokenizer)
I get an exception:
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException:
