[
https://issues.apache.org/jira/browse/LUCENE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568634#comment-13568634
]
Kristofer Karlsson commented on LUCENE-4740:
--------------------------------------------
bq. I agree that might be aproblem and you may be facing it. How mayn requests
per second do you have on your server?
Not that many - about 8000 per minute on yesterdays peak, which is about 133
per second. However, each requests leads to several complex lucene queries,
though I don't have any numbers on the actual lucene query throughput.
bq. This behaviour is Java's weak reference GC behaviour, it has nothing to do
with WeakIdentityMap. The default WeakHashMap from JDK has the same problems.
Agreed.
bq. My ide was that the master creates some boolean[1] and passes this
boolen[1] array to all childs. When the master closes, it does set the b[0] to
false. All childs would do a check on b[0]... Not sure how this affects
performance.
Yes, I thought about this too, and I am not sure the performance penalty would
be that problematic (but it would need to be measured). And if possibly, users
of the inputs should avoid doing small individual byte gets, and instead try to
consume chunks of bytes to avoid the overhead.
I would prefer an AtomicBoolean since it uses a volatile field. As far as I
know, you can't make contents of arrays volatile.
In any case, wouldn't it would possible to skip the whole master/slave
relationship and make everyone equal, just sharing the closed state flag?
Though then running close() on a clone would close everything, which is
possibly not what you want to happen.
> Weak references cause extreme GC churn
> --------------------------------------
>
> Key: LUCENE-4740
> URL: https://issues.apache.org/jira/browse/LUCENE-4740
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 3.6.1
> Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16
> cores
> Reporter: Kristofer Karlsson
> Priority: Critical
> Attachments: LUCENE-4740.patch
>
>
> We are running a set of independent search machines, running our custom
> software using lucene as a search library. We recently upgraded from lucene
> 3.0.3 to 3.6.1 and noticed a severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling
> because it's spending so much time in GC. We noticed about 212 million
> WeakReference, originating from WeakIdentityMap, originating from
> MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from
> MMapIndexInput, and as a side-effect, disabling support for explictly
> unmapping the mmapped data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
