[
https://issues.apache.org/jira/browse/LUCENE-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546031
]
Michael Busch commented on LUCENE-1058:
---------------------------------------
I think the ideas here make sense, e. g. to have a buffering
TokenFilter that doesn't buffer all tokens but enables the
user to control which tokens to buffer.
What is still not clear to me is why we have to introduce a
new API for this and a new kind of analyzer? To allow creating
an no-value field seems strange. Can't we achieve all this
by using the Field(String, TokenStream) API without the
analyzer indirection?
The javadocs should make clear that the IndexWriter processes
fields in the same order the user added them. So if a user
adds TokenStream ts1 and thereafter ts2, they can be sure
that ts1 is processed first. With that knowledge ts1 can
buffer certain tokens that ts2 uses then. Adding even more
fields that use the same tokens is straightforward.
> New Analyzer for buffering tokens
> ---------------------------------
>
> Key: LUCENE-1058
> URL: https://issues.apache.org/jira/browse/LUCENE-1058
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Reporter: Grant Ingersoll
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 2.3
>
> Attachments: LUCENE-1058.patch, LUCENE-1058.patch, LUCENE-1058.patch,
> LUCENE-1058.patch, LUCENE-1058.patch
>
>
> In some cases, it would be handy to have Analyzer/Tokenizer/TokenFilters that
> could siphon off certain tokens and store them in a buffer to be used later
> in the processing pipeline.
> For example, if you want to have two fields, one lowercased and one not, but
> all the other analysis is the same, then you could save off the tokens to be
> output for a different field.
> Patch to follow, but I am still not sure about a couple of things, mostly how
> it plays with the new reuse API.
> See
> http://www.gossamer-threads.com/lists/lucene/java-dev/54397?search_string=BufferingAnalyzer;#54397
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
