rlaehdals commented on issue #14645:
URL: https://github.com/apache/lucene/issues/14645#issuecomment-3264925098
Thank you for your clarification, and I apologize for any confusion. I
really appreciate your guidance on this.
I have looked into the %token_size_limit option. If this option is not set,
the behavior is as follows for two methods:
``` java
/** Returns the maximum size of the scanner buffer, which limits the size of
tokens. */
private int zzMaxBufferLen() {
return Integer.MAX_VALUE;
}
/** Whether the scanner buffer can grow to accommodate a larger token. */
private boolean zzCanGrow() {
return true;
}
```
By modifying this value, you can control the maximum token size. Setting the
limit to be equal to the buffer size prevents any further buffer expansion.
``` sh
%token_size_limit ZZ_BUFFERSIZE
```
Then, the methods would become:
``` java
/** Returns the maximum size of the scanner buffer, which limits the size of
tokens. */
private int zzMaxBufferLen() {
return ZZ_BUFFERSIZE;
}
/** Whether the scanner buffer can grow to accommodate a larger token. */
private boolean zzCanGrow() {
return zzBuffer.length < ZZ_BUFFERSIZE;
}
```
If the input token is larger than the buffer and the scanner attempts to
expand, it cannot exceed the token limit, which may result in the following
exceptions:
``` java
throw new java.io.EOFException("Scan buffer limit reached
["+zzBuffer.length+"]");
```
So in summary, while it is possible to completely remove the skeleton, it
should be carefully considered since doing so would cause exceptions to be
thrown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
