[
https://issues.apache.org/jira/browse/LUCENE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568611#comment-13568611
]
Uwe Schindler commented on LUCENE-4740:
---------------------------------------
bq. After doing some more thinking and micro-benchmarking, I think the problem
occurs when you create clones at a faster rate than the GC can cope with.
I agree that might be aproblem and you may be facing it. How mayn requests per
second do you have on your server?
This behaviour is Java's weak reference GC behaviour, it has nothing to do with
WeakIdentityMap. The default WeakHashMap from JDK has the same problems.
bq. Agree that the weakreferences for classes is probably a very minor part of
it, and very unlikely part of the problem here.
That is very common, the JDK uses the same mechanism like in AttributeSource at
several places. It is definitely not part of the problem.
The problem here is the weak map that has a very high throughput of puts (every
query produces at least one IndexInput clone, possibly more). The high
throughput already lead to the change to WeakIdentityMap recently, because a
synchronized WeakHashMap was not able to handle the large number of concurrent
puts (Lucene 3.6.0 regression).
I am currently thinking of making the whole thing work without weak references
and instead have some "hard reference" from the clone to master (it is already
there, MappedByteBuffer.duplicate() returns a duplicate buffer that has a
reference to the master). The problem with this is, that you need a check on
every access of the IndexInput if the buffer is still valid. If it is only some
null check, we may add it, but its risky for performance too.
My ide was that the master creates some boolean[1] and passes this boolen[1]
array to all childs. When the master closes, it does set the b[0] to false. All
childs would do a check on b[0]... Not sure how this affects performance.
> Weak references cause extreme GC churn
> --------------------------------------
>
> Key: LUCENE-4740
> URL: https://issues.apache.org/jira/browse/LUCENE-4740
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 3.6.1
> Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16
> cores
> Reporter: Kristofer Karlsson
> Priority: Critical
> Attachments: LUCENE-4740.patch
>
>
> We are running a set of independent search machines, running our custom
> software using lucene as a search library. We recently upgraded from lucene
> 3.0.3 to 3.6.1 and noticed a severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling
> because it's spending so much time in GC. We noticed about 212 million
> WeakReference, originating from WeakIdentityMap, originating from
> MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from
> MMapIndexInput, and as a side-effect, disabling support for explictly
> unmapping the mmapped data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
