Hi David, What is exactly your problem? Even the old 2.4 CachingTokenFilter did not expose its internal structures, so overriding would not change its internal implementation. The only change now is, that *all* TokenFilters in core have final implementations, which is a consequence of the new TokenStream API and the migration path to it. So it should not be possible to override next()/next(Token)/incrementToken() in all TokenStreams, as extensibility of the whole API is because of simply adding new TokenFilters into the chain, that do what you want to add. Let users override incrementToken() would possibly break a lot of things (see LUCENE-1753)
To fix your specific problems, it may be an idea to add a method (isCachingTokens) in future to TokenStreams that default to false and is true for CachingTokenFilter and TeeSinkTokenStream.SinkTokenStream. Highlighter would be able to detect, if it can reset() (better name would be rewind) the TokenStream. In this case you could simply provide another TokenFilter subclass with isCachingTokens=true and random access to the AttributeSource.States. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: David Kaelbling [mailto:dkaelbl...@blackducksoftware.com] > Sent: Thursday, August 27, 2009 10:40 PM > To: java-dev@lucene.apache.org > Subject: CachingTokenFilter extensibility and LUCENE-1685 > > Hi, > > Looking at Lucene 2.9 trunk, CachingTokenFilter seems much less extensible > than before. In previous releases I subclassed it so I could back the > cache with an array and provide random access to the stream. I can't see > how to do this any more, and the > WeightedSpanTermExtractor.getReaderForField() is still hardwired to > require a CachingTokenFilter-derived object. > > Am I missing something? Having two copies of the token stream, one for > random access and one hidden inside the CachingTokenFilter, does not sound > efficient :-) > > Thanks, > David > > -- > David Kaelbling > Senior Software Engineer > Black Duck Software, Inc. > > dkaelbl...@blackducksoftware.com > T +1.781.810.2041 > F +1.781.891.5145 > > http://www.blackducksoftware.com > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
