[
https://issues.apache.org/jira/browse/LUCENE-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13568074#comment-13568074
]
Kristofer Karlsson commented on LUCENE-4740:
--------------------------------------------
After doing some more thinking and micro-benchmarking, I think the problem
occurs when you create clones at a faster rate than the GC can cope with.
{noformat}
public class WeakTest extends TestCase {
private static final int CPUS = Runtime.getRuntime().availableProcessors();
public void testFoo() throws Exception {
final WeakIdentityMap<Object, Boolean> map =
WeakIdentityMap.newConcurrentHashMap();
ExecutorService executorService = Executors.newCachedThreadPool();
for (int k = 0; k < CPUS; k++) {
executorService.submit(new Runnable() {
public void run() {
while (true) {
map.put(new Object(), Boolean.TRUE);
}
}
});
executorService.submit(new Runnable() {
public void run() {
while (true) {
System.gc();
}
}
});
}
while (true) {
System.out.println("Map size: " + map.size());
Thread.sleep(1000);
}
}
}
{noformat}
I tried running this with -Xmx100m. This makes the map grow indefinitely.
I know this is not a very reliable test, since the JVM only considers
System.gc() a hint, but it definitely seems like it's not very good at freeing
weak references during high load.
If I add Thread.sleep(1) in the put-worker, I can make it grow slower, but it
still seems to grow over time.
(Running java 7 on my home computer)
{noformat}
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
{noformat}
> Weak references cause extreme GC churn
> --------------------------------------
>
> Key: LUCENE-4740
> URL: https://issues.apache.org/jira/browse/LUCENE-4740
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/store
> Affects Versions: 3.6.1
> Environment: Linux debian squeeze 64 bit, Oracle JDK 6, 32 GB RAM, 16
> cores
> Reporter: Kristofer Karlsson
> Priority: Critical
>
> We are running a set of independent search machines, running our custom
> software using lucene as a search library. We recently upgraded from lucene
> 3.0.3 to 3.6.1 and noticed a severe degradation of performance.
> After doing some heap dump digging, it turns out the process is stalling
> because it's spending so much time in GC. We noticed about 212 million
> WeakReference, originating from WeakIdentityMap, originating from
> MMapIndexInput.
> Our problem completely went away after removing the clones weakhashmap from
> MMapIndexInput, and as a side-effect, disabling support for explictly
> unmapping the mmapped data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
