[
https://issues.apache.org/jira/browse/LUCENE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14906171#comment-14906171
]
Alex Chow edited comment on LUCENE-6814 at 9/24/15 10:33 AM:
-------------------------------------------------------------
Thanks for taking a look at this, [~mikemccand]!
It'd be great if this can get added to 4.10 so elasticsearch 1.x can pull it in
too.
I'm curious why there shouldn't there be some trimming in `end()` as well? Or
is a `TokenStream` meant to be used only once (no multiple `reset()`,
`incrementToken()`, `end()` on the same `TokenStream`)?
Elasticsearch seems to never reinstantiate Tokenizers and just reuses them for
each field in an index, though I may be wrong. Or elasticsearch is using
TokenStream the wrong way?
was (Author: chow):
Thanks for taking a look at this, [~mikemccand]!
I'm curious why there shouldn't there be some trimming in `end()` as well? Or
is a `TokenStream` meant to be used only once (no multiple `reset()`,
`incrementToken()`, `end()` on the same `TokenStream`)?
Elasticsearch seems to never reinstantiate Tokenizers and just reuses them for
each field in an index, though I may be wrong. Or elasticsearch is using
TokenStream the wrong way?
> PatternTokenizer should free heap after it's done
> -------------------------------------------------
>
> Key: LUCENE-6814
> URL: https://issues.apache.org/jira/browse/LUCENE-6814
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6814.patch, LUCENE-6814.patch
>
>
> Caught by Alex Chow in this Elasticsearch issue:
> https://github.com/elastic/elasticsearch/issues/13721
> Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free
> its heap usage after tokenizing is done. We can either stop reusing, or ask
> it to {{.trimToSize}} when we are done ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
