[
https://issues.apache.org/jira/browse/LUCENE-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171726#comment-13171726
]
Uwe Schindler edited comment on LUCENE-3653 at 12/18/11 1:05 AM:
-----------------------------------------------------------------
Based on my work for another issue I added removal of synchronization for the
caches in VirtualMethod and AttributeSource. As those caches are generally read
(should be without synchronization) and seldom populated, ConcurrentHashMap is
the best choice, as gets are without locks.
The problem is that those reflection caches need a WeakHashMap, but there is no
ConcurrentWeakHashMap. Based on the work on issue LUCENE-3531, I reactivated my
WeakIdentityMap and made the backing map exchangeable (there are 2 static
factory methods allowing ConcurrentHashMap or simple HashMaps as backing map).
For the reflection caches, identity is also fine (Class.equals/hashcode is
based on class identity). The reflection caches also have no problem doing the
heavy reflection work twice if two threads are requesting the same class at the
same time - work is done twice and stored twice in map, but who cares?
This patch improves concurrency on creation of TokenStreams and instantiation
of all classes using VirtualMethod for backwards compatibility.
was (Author: thetaphi):
Based on my work for another issue I added removal of synchronization for
the caches in VirtualMethod and AttributeSource. As those caches are generally
read (should be without synchronization) and seldom populated,
ConcurrentHashMap is the best choice, as gets are without locks.
The problem is that those reflection caches need a WeakHashMap, but there is no
ConcurrentWeakHashMap. Based on the work on issue LUCENE-3531, I reactivated my
WeakIdentityMap and made the backing map exchangeable (there are 2 static
factory methods allowing ConcurrentHashMap or simple HasMaps as backing map).
For the reflection caches, identity is also fine (Class.equals/hashcode is
based on identity). The reflection caches also have no problem, doing the work
twice if two threads are requesting the same class at the same time - work is
done twice and stored twice in map, but who cares?
This patch improves concurrency on creation of TokenStreams and all classes
using VirtualMethod for backwards compatibility.
> Lucene Search not scalling
> --------------------------
>
> Key: LUCENE-3653
> URL: https://issues.apache.org/jira/browse/LUCENE-3653
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Gerrit Jansen van Vuuren
> Attachments: App.java,
> LUCENE-3653-VirtualMethod+AttributeSource.patch, profile_1_a.png,
> profile_1_b.png, profile_1_c.png, profile_1_d.png, profile_2_a.png,
> profile_2_b.png, profile_2_c.png
>
>
> I've noticed that when doing thousands of searches in a single thread the
> average time is quite low i.e. a few milliseconds. When adding more
> concurrent searches doing exactly the same search the average time increases
> drastically.
> I've profiled the search classes and found that the whole of lucene blocks on
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
> public synchronized int getImplementationDistance
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2
> seconds when doing 500 concurrent searches on the same in memory index. Note:
> That the index is not being updates at all, so not refresh methods are called
> at any stage.
> Some questions:
> Why do we need synchronization here?
> There must be a non-lockable solution for these, they basically cause
> lucene to be ok for single thread applications but disastrous for any
> concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of
> these classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
