[
https://issues.apache.org/jira/browse/LUCENE-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096946#comment-15096946
]
Paul Elschot commented on LUCENE-6973:
--------------------------------------
Ok, somehow I can't cleanly apply the patch, neither with patch, nor with git,
nor with svn.
But as AFAICT in TeeSinkTokenFilter this uses a ArrayDeque instead of an
ArrayList and it drops the accept method. That was in the way for
PrefillTokenStream too, which is why I needed to extract the superclass. I'm
fine with dropping that method.
I get the best result with svn, which gives this output::
{noformat}
svn patch ~/patches/LUCENE-6973.patch
U lucene/CHANGES.txt
Skipped missing target:
'lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/DateRecognizerFilter.java'
A
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/DateRecognizerFilterFactory.java
Skipped missing target:
'lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TokenRangeFilter.java'
A
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TokenRangeFilterFactory.java
Skipped missing target:
'lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TokenTypeFilter.java'
A
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TokenTypeFilterFactory.java
D
lucene/analysis/common/src/java/org/apache/lucene/analysis/sinks/DateRecognizerSinkFilter.java
U
lucene/analysis/common/src/java/org/apache/lucene/analysis/sinks/TeeSinkTokenFilter.java
D
lucene/analysis/common/src/java/org/apache/lucene/analysis/sinks/TokenRangeSinkFilter.java
D
lucene/analysis/common/src/java/org/apache/lucene/analysis/sinks/TokenTypeSinkFilter.java
U
lucene/analysis/common/src/java/org/apache/lucene/analysis/sinks/package-info.java
U
lucene/analysis/common/src/resources/META-INF/services/org.apache.lucene.analysis.util.TokenFilterFactory
Skipped missing target:
'lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/DateRecognizerTokenFilterTest.java'
Skipped missing target:
'lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TokenRangeFilterTest.java'
Skipped missing target:
'lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TokenTypeFilterTest.java'
D
lucene/analysis/common/src/test/org/apache/lucene/analysis/sinks/DateRecognizerSinkTokenizerTest.java
U
lucene/analysis/common/src/test/org/apache/lucene/analysis/sinks/TestTeeSinkTokenFilter.java
D
lucene/analysis/common/src/test/org/apache/lucene/analysis/sinks/TokenRangeSinkTokenizerTest.java
D
lucene/analysis/common/src/test/org/apache/lucene/analysis/sinks/TokenTypeSinkTokenizerTest.java
U
lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java
Summary of conflicts:
Skipped paths: 6
{noformat}
I'm probably doing something wrong here, but please go ahead as you see fit.
> Improve TeeSinkTokenFilter
> --------------------------
>
> Key: LUCENE-6973
> URL: https://issues.apache.org/jira/browse/LUCENE-6973
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Shai Erera
> Assignee: Shai Erera
> Priority: Minor
> Fix For: 5.5, Trunk
>
> Attachments: LUCENE-6973.patch, LUCENE-6973.patch
>
>
> {{TeeSinkTokenFilter}} can be improved in several ways, as it's written today:
> The most major one is removing {{SinkFilter}} which just doesn't work and is
> confusing. E.g., if you set a {{SinkFilter}} which filters tokens, the
> attributes on the stream such as {{PositionIncrementAttribute}} are not
> updated. Also, if you update any attribute on the stream, you affect other
> {{SinkStreams}} ... It's best if we remove this confusing class, and let
> consumers reuse existing {{TokenFilters}} by chaining them to the sink stream.
> After we do that, we can make all the cached states a single (immutable)
> list, which is shared between all the sink streams, so we don't need to keep
> many references around, and also deal with {{WeakReference}}.
> Besides that there are some other minor improvements to the code that will
> come after we clean up this class.
> From a backwards-compatibility standpoint, I don't think that {{SinkFilter}}
> is actually used anywhere (since it just ... confusing and doesn't work as
> expected), and therefore I believe it won't affect anyone. If however someone
> did implement a {{SinkFilter}}, it should be trivial to convert it to a
> {{TokenFilter}} and chain it to the {{SinkStream}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
