[
https://issues.apache.org/jira/browse/LUCENE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569652#comment-13569652
]
Kristofer Karlsson commented on LUCENE-4740:
--------------------------------------------
I just had an idea for a potential optimization:
Use a timerthread/executor/whatever to delay the unmap operation for X seconds
after marking it as closed.
This should give the clones enough time to figure out that it should fail
nicely instead of crash the jvm.
You could even have the clones make sure to check the state of the master every
X/2 seconds before performing any operation:
{noformat}
byte readByte() {
if (this.closed) {
throw ...;
}
long currentTime = System.currentTimeMillis();
if (currentTime - lastCheck >= X/2) {
lastCheck = currentTime;
if (master.isClosed() {
this.close();
throw ...;
}
}
return curBuf.readByte();
}
{noformat}
Not sure how much overhead this would mean in practice, but it would at least
avoid synchronization most of the time.
An alternative is to schedule a periodic job for each clone, checking the
master state and updating the clone state. That might lead to memory leaks
unless weakreferences are used, so maybe that's not actually an improvement.
> Weak references cause extreme GC churn
> --------------------------------------
>
> Key: LUCENE-4740
> URL: https://issues.apache.org/jira/browse/LUCENE-4740
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 3.6.1
> Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16
> cores
> -Xmx16G
> Reporter: Kristofer Karlsson
> Fix For: 4.2, 5.0
>
> Attachments: LUCENE-4740.patch, LUCENE-4740.patch
>
>
> We are running a set of independent search machines, running our custom
> software using lucene as a search library. We recently upgraded from lucene
> 3.0.3 to 3.6.1 and noticed a severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling
> because it's spending so much time in GC. We noticed about 212 million
> WeakReference, originating from WeakIdentityMap, originating from
> MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from
> MMapIndexInput, and as a side-effect, disabling support for explictly
> unmapping the mmapped data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
