[
https://issues.apache.org/jira/browse/LUCENE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988707#comment-13988707
]
Robert Muir commented on LUCENE-5638:
-------------------------------------
I benchmarked this patch against trunk with luceneutil's TestAnalyzerPerf.java.
I ran 3 runs of it against both trunk and patch (because its quite noisy), but
I think it shows a trend:
especially for ones like Shingles doing lots of saveState()/restoreState() and
ones like n-grams calling clearAttributes() many times.
Times are reported in milliseconds to analyze en-wiki linedocs file: lower is
better.
||Standard||Run 1||Run 2||Run 3||
|trunk|22983.37|23680.52|27390.50|
|patch|21303.90|21419.21|21351.63|
||LowerCase||Run 1||Run 2||Run 3||
|trunk|18032.37|17795.05|17533.72|
|patch|16688.78|17267.51|16650.72|
||EdgeNGrams||Run 1||Run 2||Run 3||
|trunk|40774.93|38842.67|53023.12|
|patch|30798.36|31389.50|31799.47|
||Shingles||Run 1||Run 2||Run 3||
|trunk|75702.71|75594.85|82081.14|
|patch|49572.20|58220.35|48895.09|
||WordDelimiter||Run 1||Run 2||Run 3||
|trunk|40101.05|38491.25|46736.42|
|patch|30995.33|31743.91|30488.61|
About the patch: there are more Tokenizers in the analysis/ module referring to
DEFAULT_ATTRIBUTE_FACTORY directly today (because they all have ctors to pass
in a custom factory). So we should fix those too.
> Default Attributes are expensive
> --------------------------------
>
> Key: LUCENE-5638
> URL: https://issues.apache.org/jira/browse/LUCENE-5638
> Project: Lucene - Core
> Issue Type: Bug
> Components: modules/analysis
> Reporter: Robert Muir
> Attachments: LUCENE-5638.patch
>
>
> Changes like LUCENE-5634 make it clear that the default AttributeFactory
> stuff has a very high cost: weakmaps/reflection/etc.
> Additionally I think clearAttributes() is more expensive than it should be:
> it has to traverse a linked-list, calling clear() per token.
> Operations like cloning (save/restoreState) have a high cost tll.
> Maybe we can have a better Default? In other words, rename
> DEFAULT_ATTRIBUTE_FACTORY to REFLECTION_ATTRIBUTE_FACTORY, and instead have a
> faster default factory that just has one AttributeImpl with the "core ones"
> that 95% of users are dealing with (TOKEN_ATTRIBUTE_FACTORY?): anything
> outside of that falls back to reflection.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
