I do not think the real problem gets solved by fixing it here, this is just one particular "instance" of the problem and it will pop up somewhere else. If you have a lot of threads, or you have very high thread churn, gc()/oom is going to hurt with or without this patch, just slightly later. It is hard to balance between memory needs and pressure at gc() and we have no easy knobs to control it at application level.
Would a globally bound object pool, per index cache be an option with new as fallback option? It is another sync, but this should be acceptable and probably useful for other places? With smart defaults would be completely transparent for end user, but would enable making these decisions user responsibility (zero sized pool == always create object… ) On May 11, 2013, at 10:25 PM, "Adrien Grand (JIRA)" <[email protected]> wrote: > > [ > https://issues.apache.org/jira/browse/LUCENE-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655364#comment-13655364 > ] > > Adrien Grand commented on LUCENE-4995: > -------------------------------------- > > bq. Are we sure this is the right thing to do? > > I have no idea at all. On the one hand, it looks reasonable to me to have a > reusable per-thread buffer to handle decompression, but on the other hand, it > makes me unhappy that its size is unbounded: if an index has a few 1M > documents on S segments and T threads, the JVM will have to reserve S*T*1M of > heap just to be able to handle decompression. > >> Remove the strong reference of CompressingStoredFieldsReader on the >> decompression buffer >> ---------------------------------------------------------------------------------------- >> >> Key: LUCENE-4995 >> URL: https://issues.apache.org/jira/browse/LUCENE-4995 >> Project: Lucene - Core >> Issue Type: Bug >> Reporter: Adrien Grand >> Assignee: Adrien Grand >> Attachments: LUCENE-4995.patch >> >> >> CompressingStoredFieldsReader has a strong reference on the buffer it uses >> for decompression. Although it makes the reader able to reuse this buffer, >> this can trigger high memory usage in case some documents are very large. >> Creating this buffer on demand would help give memory back to the JVM. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
