Notice that while DisjunctionScorer.advance and
DisjuntionScorer.advanceAfterCurrent appear to be called
in 2.9, in 2.4, I am only seeing DisjuntionScorer.advanceAfterCurrent
called.
Can someone explain that?
Mark Miller wrote:
> Something is very odd about this if they both cover the same search and
> the environ for both is identical. Even if one search was done twice,
> and we divide the numbers for the new api by 2 - its still *very* odd.
>
> With 2.4, ScorerDocQueue.topDoc is called half a million times.
> With 2.9, its called over 4 million times.
>
> Huh?
>
> Thomas Becker wrote:
>
>> No it's only a single segment. But two calls. One doing a getHitsCount first
>> and
>> the other doing the actual search. I'll paste both methods below if someone's
>> interested.
>>
>> Will dig into lucene's sources and compare 2.4 search behaviour for my case
>> with
>> 2.9 tomorrow. It was about time to get more into lucene-core sources anyhow.
>> :)
>>
>> See you tomorrow guys and thanks a lot again! It's a pleasure.
>>
>> public int getHitsCount(String query, Filter filter) throws
>> LuceneServiceException {
>> log.debug("getHitsCount('{}, {}')", query, filter);
>> if (StringUtils.isBlank(query)) {
>> log.warn("getHitsCount: empty lucene query");
>> return 0;
>> }
>> long startTimeMillis = System.currentTimeMillis();
>> int count = 0;
>>
>> if (indexSearcher == null) {
>> return 0;
>> }
>>
>> BooleanQuery.setMaxClauseCount(MAXCLAUSECOUNT);
>> Query q = null;
>> try {
>> q = createQuery(query);
>> TopScoreDocCollector tsdc =
>> TopScoreDocCollector.create(1, true);
>> indexSearcher.search(q, filter, tsdc);
>> count = tsdc.getTotalHits();
>> log.info("getHitsCount: count = {}",count);
>> } catch (ParseException ex) {
>> throw new LuceneServiceException("invalid lucene
>> query:" + query, ex);
>> } catch (IOException e) {
>> throw new LuceneServiceException(" indexSearcher could
>> be corrupted", e);
>> } finally {
>> long durationMillis = System.currentTimeMillis() -
>> startTimeMillis;
>> if (durationMillis > slowQueryLimit) {
>> log.warn("getHitsCount: Slow query: {} ms,
>> query={}", durationMillis, query);
>> }
>> log.debug("getHitsCount: query took {} ms",
>> durationMillis);
>> }
>> return count;
>> }
>>
>> public List<Document> search(String query, Filter filter, Sort sort,
>> int from,
>> int size) throws LuceneServiceException {
>> log.debug("{} search('{}', {}, {}, {}, {})", new Object[] {
>> indexAlias, query,
>> filter, sort, from, size });
>> long startTimeMillis = System.currentTimeMillis();
>>
>> List<Document> docs = new ArrayList<Document>();
>> if (indexSearcher == null) {
>> return docs;
>> }
>> Query q = null;
>> try {
>> if (query == null) {
>> log.warn("search: lucene query is null...");
>> return docs;
>> }
>> q = createQuery(query);
>> BooleanQuery.setMaxClauseCount(MAXCLAUSECOUNT);
>> if (size < 0 || size > maxNumHits) {
>> // set hard limit for numHits
>> size = maxNumHits;
>> if (log.isDebugEnabled())
>> log.debug("search: Size set to
>> hardlimit: {} for query: {} with filter:
>> {}", new Object[] { size, query, filter });
>> }
>> TopFieldCollector collector =
>> TopFieldCollector.create(sort, size + from,
>> true, false, false, true);
>> indexSearcher.search(q, filter, collector);
>> if(size > collector.getTotalHits())
>> size = collector.getTotalHits();
>> if (size > 100000)
>> log.info("search: size: {} bigger than 100.000
>> for query: {} with filter:
>> {}", new Object[] { size, query, filter });
>> TopDocs td = collector.topDocs(from, size);
>> ScoreDoc[] scoreDocs = td.scoreDocs;
>> for (ScoreDoc scoreDoc : scoreDocs) {
>> docs.add(indexSearcher.doc(scoreDoc.doc));
>> }
>> } catch (ParseException e) {
>> log.warn("search: ParseException: {}", e.getMessage());
>> if (log.isDebugEnabled())
>> log.warn("search: ParseException: ", e);
>> return Collections.emptyList();
>> } catch (IOException e) {
>> log.warn("search: IOException: ", e);
>> return Collections.emptyList();
>> } finally {
>> long durationMillis = System.currentTimeMillis() -
>> startTimeMillis;
>> if (durationMillis > slowQueryLimit) {
>> log.warn("search: Slow query: {} ms, query={},
>> indexUsed={}",
>> new Object[] { durationMillis,
>> query,
>> indexSearcher.getIndexReader().directory() });
>> }
>> log.debug("search: query took {} ms", durationMillis);
>> }
>> return docs;
>> }
>>
>>
>> Uwe Schindler wrote:
>>
>>
>>>>> http://ankeschwarzer.de/tmp/lucene_29_newapi_mmap_singlereq.png
>>>>>
>>>>> Have to verify that the last one is not by accident more than one
>>>>>
>>>>>
>>>> request.
>>>>
>>>>
>>>>> Will
>>>>> do the run again and then post the required info.
>>>>>
>>>>>
>>>> The last figure shows, that IndexSearcher.searchWithFilter was called
>>>> twice
>>>> in contrast to the first figure, where IndexSearcher.search was called
>>>> only
>>>> once.
>>>>
>>>>
>>> I forgot, searchWithFilter it is called per segment in 2.9. If it was only
>>> one search, you must have two segments and therefore no optimized index for
>>> this to be correct?
>>>
>>> Uwe
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>>
>>>
>>>
>>
>>
>
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
