[
https://issues.apache.org/jira/browse/LUCENE-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14907054#comment-14907054
]
Alex Chow commented on LUCENE-6814:
-----------------------------------
bq. Hmm, how many {{PatternTokenizer}} instances do you have? With 24 bulk
indexing threads you should (I think?) only have at most 24 instances * 4 MB
max should be ~100 MB.
{{PatternTokenizer}} instances also scale with the number of indices
(technically shards?) if you use your own analyzer based on a
{{PatternAnalyzer}}. Custom analyzers get instantiated per index since they
have their own mappings. We have 168 indices, so that's a _lot_. One heap dump
I was poking around had 3432 instances, but it'd probably have grown more since
some analyzers haven't had to create their {{TokenStreamComponents}} yet.
> PatternTokenizer indefinitely holds heap equal to max document it has ever
> tokenized
> ------------------------------------------------------------------------------------
>
> Key: LUCENE-6814
> URL: https://issues.apache.org/jira/browse/LUCENE-6814
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: Trunk, 5.4
>
> Attachments: LUCENE-6814.patch, LUCENE-6814.patch
>
>
> Caught by Alex Chow in this Elasticsearch issue:
> https://github.com/elastic/elasticsearch/issues/13721
> Today, PatternTokenizer reuses a single StringBuilder, but it doesn't free
> its heap usage after tokenizing is done. We can either stop reusing, or ask
> it to {{.trimToSize}} when we are done ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
