[
https://issues.apache.org/jira/browse/LUCENE-7647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834748#comment-15834748
]
Adrien Grand commented on LUCENE-7647:
--------------------------------------
I ran a merge that cantains 1M documents from the wikipedia benchmark including
deleted docs, in order to test the worst case. Here is what the info stream
reports about stored fields before/after the change:
Before:
{noformat}
SM 0 [2017-01-23T15:03:34.956Z; Lucene Merge Thread #0]: 41827 msec to merge
stored fields [996093 docs]
SM 0 [2017-01-23T15:06:49.785Z; Lucene Merge Thread #0]: 41722 msec to merge
stored fields [996093 docs]
SM 0 [2017-01-23T15:14:09.943Z; Lucene Merge Thread #0]: 42138 msec to merge
stored fields [996093 docs]
{noformat}
After:
{noformat}
SM 0 [2017-01-23T15:17:33.241Z; Lucene Merge Thread #0]: 42050 msec to merge
stored fields [996093 docs]
SM 0 [2017-01-23T15:20:00.656Z; Lucene Merge Thread #0]: 42320 msec to merge
stored fields [996093 docs]
SM 0 [2017-01-23T15:22:04.047Z; Lucene Merge Thread #0]: 42520 msec to merge
stored fields [996093 docs]
{noformat}
I think this is either noise a an acceptable slow down. That makes sense since
we always decompress about 16K of data. Initialization of the Inflater is
likely much less costly than decompressing that amount of data.
> CompressingStoredFieldsFormat should reclaim memory more aggressively
> ---------------------------------------------------------------------
>
> Key: LUCENE-7647
> URL: https://issues.apache.org/jira/browse/LUCENE-7647
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Adrien Grand
> Attachments: LUCENE-7647.patch
>
>
> When stored fields are configured with {{BEST_COMPRESSION}}, we rely on
> garbage collection to reclaim Deflater/Inflater instances. However these
> classes use little JVM memory but may use significant native memory, so if
> may happen that the OS runs out of native memory before the JVM collects
> these unreachable Deflater/Inflater instances. We should look into reclaiming
> native memory more aggressively.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
