[
https://issues.apache.org/jira/browse/LUCENE-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12837811#action_12837811
]
Michael McCandless commented on LUCENE-2283:
--------------------------------------------
bq. a yourkit snapshot showed that the PerDocs for an IndexWriter were using
~40M of memory
What was IW's ramBufferSizeMB when you saw this?
bq. however i have reports that eventually the memory is completely exhausted
resulting in out of memory errors.
Hmm, that makes me nervous, because I think in this case the use should be
bounded.
bq. As a side note, closing the index writer at commit time would sometimes
fail, resulting in some following updates to fail because the index writer was
locked and couldn't be reopened until the old index writer was garbage
collected, so i don't want to go back to this for commits.
That doesn't sound good! Can you post some details on this (eg an exception)?
But, anyway, keeping the same IW open and just calling commit is (should be)
fine.
bq. As far as a fix goes, wouldn't it be better to have the RAMFile's used for
stored fields pull and return byte buffers from the byte block pool on the
DocumentsWriter?
Yes, that's a great solution -- a single pool. But that's a somewhat bigger
change. I guess we can pass a byte[] allocator to RAMFile. It'd have to be a
new pool, too (DW's byte blocks are 32KB not the 1KB that RAMFile uses).
> Possible Memory Leak in StoredFieldsWriter
> ------------------------------------------
>
> Key: LUCENE-2283
> URL: https://issues.apache.org/jira/browse/LUCENE-2283
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.4.1
> Reporter: Tim Smith
> Assignee: Michael McCandless
> Fix For: 3.1
>
>
> StoredFieldsWriter creates a pool of PerDoc instances
> this pool will grow but never be reclaimed by any mechanism
> furthermore, each PerDoc instance contains a RAMFile.
> this RAMFile will also never be truncated (and will only ever grow) (as far
> as i can tell)
> When feeding documents with large number of stored fields (or one large
> dominating stored field) this can result in memory being consumed in the
> RAMFile but never reclaimed. Eventually, each pooled PerDoc could grow very
> large, even if large documents are rare.
> Seems like there should be some attempt to reclaim memory from the PerDoc[]
> instance pool (or otherwise limit the size of RAMFiles that are cached) etc
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
