[
https://issues.apache.org/jira/browse/LUCENE-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631414#comment-14631414
]
Steve Rowe edited comment on LUCENE-6682 at 7/17/15 2:48 PM:
-------------------------------------------------------------
In {{setMaxTokenLength()}} we should call {{setBufferSize()}} only if the
length has changed.
Also, {{setMaxTokenLength()}} maxes out the buffer size at 1M chars (i.e.
UTF-16 code units), but {{getMaxTokenLength()}} will lie and report the
requested length, even though it exceeded 1M chars, and the buffer length,
maxed out at 1M chars, is what really controls max token length. Not cool. We
should instead throw an exception when the requested {{maxTokenLength}} exceeds
1M.
Here's a patch that fixes both:
{code:java}
Index:
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
===================================================================
---
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
(revision 1691570)
+++
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
(working copy)
@@ -88,6 +88,8 @@
"<HANGUL>"
};
+ public static final int MAX_TOKEN_LENGTH_LIMIT = 1024 * 1024;
+
private int skippedPositions;
private int maxTokenLength = StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH;
@@ -97,9 +99,13 @@
public void setMaxTokenLength(int length) {
if (length < 1) {
throw new IllegalArgumentException("maxTokenLength must be greater than
zero");
+ } else if (length > MAX_TOKEN_LENGTH_LIMIT) {
+ throw new IllegalArgumentException("maxTokenLength may not exceed " +
MAX_TOKEN_LENGTH_LIMIT);
}
- this.maxTokenLength = length;
- scanner.setBufferSize(Math.min(length, 1024 * 1024)); // limit buffer size
to 1M chars
+ if (length != maxTokenLength) {
+ maxTokenLength = length;
+ scanner.setBufferSize(length);
+ }
}
/** @see #setMaxTokenLength */
{code}
was (Author: steve_rowe):
In {{setMaxTokenLength()}} we should call {{setBufferSize()}} only if the
length has changed.
Also, {{setMaxTokenLength()}} maxes out the buffer size at 1M chars (i.e.
UTF-16 code units), but {{getMaxTokenLength()}} will lie and reported the
requested length, even though it exceeded 1M chars, and the buffer length,
maxed out at 1M chars, is what really controls max token length. Not cool. We
should instead throw an exception when the requested {{maxTokenLength}} exceeds
1M.
Here's a patch that fixes both:
{code:java}
Index:
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
===================================================================
---
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
(revision 1691570)
+++
lucene/analysis/common/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
(working copy)
@@ -88,6 +88,8 @@
"<HANGUL>"
};
+ public static final int MAX_TOKEN_LENGTH_LIMIT = 1024 * 1024;
+
private int skippedPositions;
private int maxTokenLength = StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH;
@@ -97,9 +99,13 @@
public void setMaxTokenLength(int length) {
if (length < 1) {
throw new IllegalArgumentException("maxTokenLength must be greater than
zero");
+ } else if (length > MAX_TOKEN_LENGTH_LIMIT) {
+ throw new IllegalArgumentException("maxTokenLength may not exceed " +
MAX_TOKEN_LENGTH_LIMIT);
}
- this.maxTokenLength = length;
- scanner.setBufferSize(Math.min(length, 1024 * 1024)); // limit buffer size
to 1M chars
+ if (length != maxTokenLength) {
+ maxTokenLength = length;
+ scanner.setBufferSize(length);
+ }
}
/** @see #setMaxTokenLength */
{code}
> StandardTokenizer performance bug: buffer is unnecessarily copied when
> maxTokenLength doesn't change
> ----------------------------------------------------------------------------------------------------
>
> Key: LUCENE-6682
> URL: https://issues.apache.org/jira/browse/LUCENE-6682
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Steve Rowe
>
> From Piotr Idzikowski on java-user mailing list
> [http://markmail.org/message/af26kr7fermt2tfh]:
> {quote}
> I am developing own analyzer based on StandardAnalyzer.
> I realized that tokenizer.setMaxTokenLength is called many times.
> {code:java}
> protected TokenStreamComponents createComponents(final String fieldName,
> final Reader reader) {
> final StandardTokenizer src = new StandardTokenizer(getVersion(),
> reader);
> src.setMaxTokenLength(maxTokenLength);
> TokenStream tok = new StandardFilter(getVersion(), src);
> tok = new LowerCaseFilter(getVersion(), tok);
> tok = new StopFilter(getVersion(), tok, stopwords);
> return new TokenStreamComponents(src, tok) {
> @Override
> protected void setReader(final Reader reader) throws IOException {
> src.setMaxTokenLength(StandardAnalyzer.this.maxTokenLength);
> super.setReader(reader);
> }
> };
> }
> {code}
> Does it make sense if length stays the same? I see it finally calls this
> one( in StandardTokenizerImpl ):
> {code:java}
> public final void setBufferSize(int numChars) {
> ZZ_BUFFERSIZE = numChars;
> char[] newZzBuffer = new char[ZZ_BUFFERSIZE];
> System.arraycopy(zzBuffer, 0, newZzBuffer, 0,
> Math.min(zzBuffer.length, ZZ_BUFFERSIZE));
> zzBuffer = newZzBuffer;
> }
> {code}
> So it just copies old array content into the new one.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
