[
https://issues.apache.org/jira/browse/LUCENE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630309#comment-13630309
]
Robert Muir commented on LUCENE-4931:
-------------------------------------
I'm trying to understand the problem, its only if:
# you aren't reusing field instances
# you aren't just using KeywordTokenizer in your Analyzer for "string fields"
(so they also work with queryparser etc)
# you have an ancient JVM
I dont like the idea of adding the private analyzer back to documentsinverter
if we can avoid it.
I also don't understand how its not an issue for e.g. IntField too, shouldnt it
be the same there?
> Make oal.document.Field reuse its internal StringTokenStream
> ------------------------------------------------------------
>
> Key: LUCENE-4931
> URL: https://issues.apache.org/jira/browse/LUCENE-4931
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
> Affects Versions: 4.0, 4.1, 4.2, 4.2.1
> Reporter: Uwe Schindler
>
> Followup from LUCENE-4930:
> Field.java has a private StringTokenStream which is used as TokenStream
> implementation for StringField (single value String tokens). Unfortunately
> this TokenStream is created on every new document/field while indexing,
> making the cost of creating the TS a significant time. With very old Java
> versions this also involves a lock in ReferenceQueue.poll() when called from
> addAttribute().
> In Lucene 3.x, DocInverterPerThread has a private thread-local
> AttributeSource for reusing, but because this was factored out to Field.java,
> we can no longer use CloseableThreadLocal (because Field are not Closeable).
> We should maybe move the special One-Token TokenStream back to
> DocInverterPerThread and just let Field.java delegate there. I know this
> would let us move back to 3.x where we had special handling of single token
> Fields in the indexer....
> Another approach would be to make Field.java use a static KeywordAnalyzer (it
> needs then be moved to core) or we add a ThreadLocal to Field.java (which may
> be expensive). Unfortunately this makes it hard to maintain, as the
> thread-localness is also needed to be bound to the IndexWriter instance.
> Because you could have 2 IndexWriters open at same time and add documents to
> both of them from one thread... This brings us back to my previous solution.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
