[
https://issues.apache.org/jira/browse/LUCENE-3653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172289#comment-13172289
]
Gerrit Jansen van Vuuren edited comment on LUCENE-3653 at 12/19/11 2:08 PM:
----------------------------------------------------------------------------
Just to clear up on some comments:
Guys I'm testing this on a production deploy and the use case is simple. Parser
Query, Search, Get Doc fields. Contention was seen without profiling and
profiling was used to find where the contention in the application.
|I don't care. They should be reusing tokenstreams in their application.
Token streams is not the only place of contention discussed here. Please look
at the test attached (App.java). It does not recreate the token streams on each
search, only (by mistake) for each thread. But still this only has a very small
impact. Searches are repeated from each thread reusing the same token stream.
|We don't need to make lucene more complicated to (possibly) speed up someones
broken code.
Please write an example and show me your findings. I've done the same already.
Have a look at App.java
|I want to see the QPS change, I don't trust the benchmarks or the various
fancy jvm tools being used on this issue.
I think its all ghosts.
Yourkit is a standard Profiling tool. I didn't just wake up and say hey I'll
profile lucene. I only started after I noticed the contention on a production
application where we are trying to use lucene in memory.
|Lots of commits here, a title that says 'lucene doesn't scale, lots of arguing
that there are locking problems, but not one luceneutil benchmark run
??, You don't need to be a scientist nor use a profiling tool to see that all
threads will block on a synchronized block or method on each request if that
method is called during each search request, from that it doesn't take much to
know that this does not scale.
Again, this is not a go at lucene, the heading 'Lucene Search not scalling' is
causing toooo much problems here I see, I could have chosen a better heading
and if we can change it to something else please do so. I think lucene is
really great, thats why I'm trying to use it. Thanks for the support, ideas and
patches so far.
was (Author: gerritjvv):
Just to clear up on some comments:
Guys I'm testing this on a production deploy and the use case is simple. Parser
Query, Search, Get Doc fields. Contention was seen without profiling and
profiling was used to find where the contention in the application.
|I don't care. They should be reusing tokenstreams in their application.
Token streams is not the only place of contention discussed here. Please look
at the test attached (App.java). It does not recreate the token streams on each
search, only (by mistake) for each thread. But still this only has a very small
impact. Searches are repeated from each thread reusing the same token stream.
|We don't need to make lucene more complicated to (possibly) speed up someones
broken code.
Please write an example and show me your findings. I've done the same already.
Have a look at App.java
|I want to see the QPS change, I don't trust the benchmarks or the various
fancy jvm tools being used on this issue. I think its all ghosts.
Yourkit is a standard Profiling tool. I didn't just wake up and say hey I'll
profile lucene. I only started after I noticed the contention on a production
application where we are trying to use lucene in memory.
|Lots of commits here, a title that says 'lucene doesn't scale, lots of arguing
that there are locking problems, but not one luceneutil benchmark run
??, You don't need to be a scientist nor use a profiling tool to see that all
threads will block on a synchronized block or method on each request if that
method is called during each search request, from that it doesn't take much to
know that this does not scale.
Again, this is not a go at lucene, the heading 'Lucene Search not scalling' is
causing toooo much problems here I see, I could have chosen a better heading
and if we can change it to something else please do so. I think lucene is
really great, thats why I'm trying to use it. Thanks for the support, ideas and
patches so far.
> Lucene Search not scalling
> --------------------------
>
> Key: LUCENE-3653
> URL: https://issues.apache.org/jira/browse/LUCENE-3653
> Project: Lucene - Java
> Issue Type: Improvement
> Reporter: Gerrit Jansen van Vuuren
> Attachments: App.java,
> LUCENE-3653-VirtualMethod+AttributeSource.patch,
> LUCENE-3653-VirtualMethod+AttributeSource.patch,
> LUCENE-3653-VirtualMethod+AttributeSource.patch, LUCENE-3653-no-sync.png,
> LUCENE-3653-sync-.png, LUCENE-3653.patch,
> LUCENE-3653.patch-BiasedLockingStartupDelay_1.png,
> LUCENE-3653.patch-BiasedLockingStartupDelay_2.png,
> LUCENE-3653.patch-BiasedLockingStartupDelay_3.png,
> Threads-LUCENE-3653.patch.png, lucene-unsync.diff, profile_1_a.png,
> profile_1_b.png, profile_1_c.png, profile_1_d.png, profile_2_a.png,
> profile_2_b.png, profile_2_c.png
>
>
> I've noticed that when doing thousands of searches in a single thread the
> average time is quite low i.e. a few milliseconds. When adding more
> concurrent searches doing exactly the same search the average time increases
> drastically.
> I've profiled the search classes and found that the whole of lucene blocks on
> org.apache.lucene.index.SegmentCoreReaders.getTermsReader
> org.apache.lucene.util.VirtualMethod
> public synchronized int getImplementationDistance
> org.apache.lucene.util.AttributeSourcew.getAttributeInterfaces
> These cause search times to increase from a few milliseconds to up to 2
> seconds when doing 500 concurrent searches on the same in memory index. Note:
> That the index is not being updates at all, so not refresh methods are called
> at any stage.
> Some questions:
> Why do we need synchronization here?
> There must be a non-lockable solution for these, they basically cause
> lucene to be ok for single thread applications but disastrous for any
> concurrent implementation.
> I'll do some experiments by removing the synchronization from the methods of
> these classes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
