[
https://issues.apache.org/jira/browse/JCR-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836699#action_12836699
]
Marcel Reutegger commented on JCR-2505:
---------------------------------------
> do you mean lucene tokenizer ?
yes.
actually it's not that bad and only affects Jackrabbit versions 1.4 and lower.
That's also the version where I discovered the issue. More recent versions use
lucene-core 2.3.2 or 2.4.1, which only use one TokenStream at a time and even
try to re-use the tokenizer.
See LUCENE-969.
There's still room for improvement. The JackrabbitAnalyzer does not implement
reusableTokenStream(). That is, each value of a multi-valued property will
instantiate a new Tokenizer.
The proposed patch fixes this. With the patch applied, the JCRAPITest suite
takes about 10% less time to execute on my machine.
> High memory usage on node with multi-valued string properties
> -------------------------------------------------------------
>
> Key: JCR-2505
> URL: https://issues.apache.org/jira/browse/JCR-2505
> Project: Jackrabbit Content Repository
> Issue Type: Improvement
> Components: jackrabbit-core
> Reporter: Marcel Reutegger
> Priority: Minor
> Attachments: JCR-2505.patch
>
>
> Multi-valued string properties are tokenized per value, which may consume
> quite some memory when there are lots of small values in on a property. The
> memory footprint is 2k per value, because each value is tokenized with a
> separate tokenizer instance. That tokenizer uses a stream buffer of 2k bytes.
> Instead the values should be concatenated (whitespace separated) and then
> tokenized in one go.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
