[
https://issues.apache.org/jira/browse/LUCENE-4930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630542#comment-13630542
]
Christian Ziech commented on LUCENE-4930:
-----------------------------------------
{quote}
The issue is not class unloading in your own application while it is running.
The VM will never do this. It will only unload classes, when the ClassLoader is
released. This happens e.g. when you redeploy your webapplication in your Jetty
or Tomcat container or (and this is the most important reason) when you reload
Solr cores: If you have a custom analyzer JAR file in your plugins directory
that uses custom attributes (like lucene-kuromoji.jar Japanese Analyzer), your
would have a memory leak. Solr loads plugins in its own classloader. If you
restart a core it reinitializes its plugins and releases the old classloader.
If the AttributeSource would refer to these classes, they could never be
unloaded. The same happens if you have a webapp that uses a lucene-core.jar
file from outside the webapp (e.g. from Ubuntu repository in /usr/lib), but has
own analyzers shipped in the webapp. In that case, the classes could not be
unloaded on webapp shutdown.
The WeakIdentityMap prevents this big resource leak (permgen issue). If you
wonder: The values in the map also have a WeakReference, because the key's weak
reference and the Map.Entry is only removed when you actually call get() on the
map. If you unload the webapp, nobody calls get() anymore, so all Map.Entry
would refer to the classes and are never removed.
One optimization might be possible: As the number of classes in this map is
very low and the important thing is to release the class reference when no
longer needed, we could add an option to WeakIdentityMap to make reap() a
no-op. This would keep the WeakReference and Map.Entrys in the map, but the
classes could get freed. The small overhead (you can count the number of
entries on your fingers) would be minimal and the lost WeakReferences in the
map would be no problem.
Another approach would be to make DefaultAttributeSource have a lookup table
(without weak keys) on all Lucene-Internal attributes (which are the only ones
actually used by IndexWriter). I would prefer this approach.
{quote}
I totally understood the problem with the unloading of the keys (but I think I
worded it badly) - I just did not expect it to be grave since every reload
would only leave behind two dead weak references and the related map entry.
A possibly better option than making reap a no-op could be to only reap on put.
I mean one usually invokes get() but once that event of unloading an interface
actually happens and something new needs to be added one would reap the old
keys (in worst case perhaps one unloading later).
I also tried to think of a good way to have one AttributeFactory per class
loader (I mean you only really have a problem if the class loader that loads
the interface class is a child of the class loader that did load the the
AttributeFactory class) but couldn't find one.
> Lucene's use of WeakHashMap at index time prevents full use of cores on some
> multi-core machines, due to contention
> -------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-4930
> URL: https://issues.apache.org/jira/browse/LUCENE-4930
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 4.2
> Environment: Dell blade system with 16 cores
> Reporter: Karl Wright
> Attachments: thread_dump.txt
>
>
> Our project is not optimally using full processing power during under
> indexing load on Lucene 4.2.0. The reason is the
> AttributeSource.addAttribute() method, which goes through a WeakHashMap
> synchronizer, which is apparently single-threaded for a significant amount of
> time. Have a look at the following trace:
> "pool-1-thread-28" prio=10 tid=0x00007f47fc104800 nid=0x672b waiting for
> monitor entry [0x00007f47d19ed000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:98)
> - waiting to lock <0x00000005c5cd9988> (a
> java.lang.ref.ReferenceQueue$Lock)
> at
> org.apache.lucene.util.WeakIdentityMap.reap(WeakIdentityMap.java:189)
> at org.apache.lucene.util.WeakIdentityMap.get(WeakIdentityMap.java:82)
> at
> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.getClassForInterface(AttributeSource.java:74)
> at
> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory.createAttributeInstance(AttributeSource.java:65)
> at
> org.apache.lucene.util.AttributeSource.addAttribute(AttributeSource.java:271)
> at
> org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:107)
> at
> org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:254)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:256)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:376)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1473)
> at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1148)
> at
> org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1129)
> …
> We’ve had to make significant changes to the way we were indexing in order to
> not hit this issue as much, such as indexing using TokenStreams which we
> reuse, when it would have been more convenient to index with just tokens.
> (The reason is that Lucene internally creates TokenStream objects when you
> pass a token array to IndexableField, and doesn’t reuse them, and the
> addAttribute() causes massive contention as a result.) However, as you can
> see from the trace above, we’re still running into contention due to other
> addAttribute() method calls that are buried deep inside Lucene.
> I can see two ways forward. Either not use WeakHashMap or use it in a more
> efficient way, or make darned sure no addAttribute() calls are done in the
> main code indexing execution path. (I think it would be easy to fix
> DocInverterPerField in that way, FWIW. I just don’t know what we’ll run into
> next.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
