[
https://issues.apache.org/jira/browse/LUCENE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078707#comment-17078707
]
Bruno Roustant commented on LUCENE-9286:
----------------------------------------
To complete the perf benchmark, I ran luceneutil on both wikimedium500k and
wikimediumall. I see a perf slowdown of 4%-5% in PKLookup with FST off-heap
(and only on PKLookup).
Given that when it was introduced this direct addressing node improved the
PKLookup perf of at least twice this slowdown, and given that this fix improves
greatly the FSTEnum traversal speed and the memory for large automata, I
consider this slowdown is ok.
I'm going to merge the PR tomorrow.
> FST arc.copyOf clones BitTables and this can lead to excessive memory use
> -------------------------------------------------------------------------
>
> Key: LUCENE-9286
> URL: https://issues.apache.org/jira/browse/LUCENE-9286
> Project: Lucene - Core
> Issue Type: Bug
> Affects Versions: 8.5
> Reporter: Dawid Weiss
> Assignee: Bruno Roustant
> Priority: Major
> Attachments: screen-[1].png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> I see a dramatic increase in the amount of memory required for construction
> of (arguably large) automata. It currently OOMs with 8GB of memory consumed
> for bit tables. I am pretty sure this didn't require so much memory before
> (the automaton is ~50MB after construction).
> Something bad happened in between. Thoughts, [~broustant], [~sokolov]?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
