Re: Scale function inefficiency perhaps?

[Greg Bowyer]

I am going to weigh in on this, this function turns up in my profiler 
dumps as being expensive (attached). My profiler output is from 
lucene_3_5, so exacting details might deviate.

With that said, whilst it looks bad, I dont think for the case of 
scaling a field its _that_ bad, essentially (as far as I can tell 
through debugging) it boils down to iterating items from the field 
cache, which is not the most expensive thing going. With that said, I 
was personally thinking if it could be stored in the index, if you are 
suggesting that its stored in the codecs with lucene4 that would be great.

In my case, I am wondering if it has a bad interaction with the grouping 
feature introduced into 3.5; I think that the specific DocValues 
instance it iterates over for groups might be something that is not as 
cheap to iterate as the fieldCache and as a result causes the expensive 
calls I see. I am currently looking at SOLR-3109, as I think the seen 
expense of the Scale Function might be making that problem far worse.

If people want I can get harder better profiling output or debugging here.

-- Greg

On 09/02/12 12:52, Chris Hostetter wrote:
: can anyone confirm?

everything you said sounded right to me ... scale(...) as a general
purpose function probably can't be optimized very heaivly.  but it would
probably be possible to write an optimization specific to
scale(fieldname).

I seem to recall someone saying something at some point about the idea of
having the codec record stats like the min/max value of a field (per
segment) ... assuming i'm not smoking crack, so you wouldn't have to do
any caching, you could have an optimized "scalefield(f)" function that
could fail hard if the field didn't use the expected codec type, otherwise
it just asked all the atomic readers what the min/max where to find the
overall min/max.


-Hoss

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org


Call tree (all threads together)
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────┬─────────────────┐
│                                          Name                               
                                         │     Time (ms)      │  Own Time 
(ms)  │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────┼─────────────────┤
│ 
┌─org.apache.solr.handler.component.QueryComponent.process(ResponseBuilder) 
                                         │    993,965   63 %  │             
 1  │
│ │                                                                         
                                           │                    │           
      │
│ ├─┬─org.apache.solr.search.grouping.CommandHandler.execute()        
                                                 │    623,143   40 %  │     
         0  │
│ │ │                                                                     
                                             │                    │         
        │
│ │ ├─┬─org.apache.lucene.search.IndexSearcher.search(Query, 
Filter, Collector)                                        │    320,197   20 % 
 │              0  │
│ │ │ │                                                                 
                                               │                    │       
          │
│ │ │ ├─┬─org.apache.lucene.search.IndexSearcher.search(Weight, 
Filter, Collector)                                     │    320,154   20 %  
│              0  │
│ │ │ │ │                                                             
                                                 │                    │     
            │
│ │ │ │ 
├─┬─org.apache.lucene.search.FilteredQuery.getFilteredScorer(IndexReader,
 Similarity, Weight, Weight, Filter)  │    267,698   17 %  │              0 
 │
│ │ │ │ │ │                                                         
                                                   │                    │   
              │
│ │ │ │ │ 
└─┬─org.apache.solr.search.function.BoostedQuery$BoostedWeight.scorer(IndexReader,
 boolean, boolean)         │    267,698   17 %  │              0  │
│ │ │ │ │   │                                                       
                                                   │                    │   
              │
│ │ │ │ │   
├─┬─org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>(BoostedQuery,
 Similarity, ...)        │    267,484   17 %  │              0  │
│ │ │ │ │   │ │                                                   
                                                     │                    │ 
                │
│ │ │ │ │   │ 
└─┬─org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>(BoostedQuery,
 Similarity, ...)      │    267,484   17 %  │              0  │
│ │ │ │ │   │   │                                                 
                                                     │                    │ 
                │
│ │ │ │ │   │   
└─┬─org.apache.solr.search.function.MultiFloatFunction.getValues(Map, 
IndexReader)                     │    267,484   17 %  │              0  │
│ │ │ │ │   │     │                                               
                                                     │                    │ 
                │
│ │ │ │ │   │     
└─┬─org.apache.solr.search.Double2Parser$Function.getValues(Map, 
IndexReader)                        │    267,484   17 %  │              0  
│
│ │ │ │ │   │       │                                             
                                                     │                    │ 
                │
│ │ │ │ │   │       
└─┬─org.apache.solr.search.function.TopValueSource.getValues(Map, 
IndexReader)                     │    267,484   17 %  │              0  │
│ │ │ │ │   │         │                                           
                                                     │                    │ 
                │
│ │ │ │ │   │         
└───org.apache.solr.search.function.ScaleFloatFunction.getValues(Map, 
IndexReader)               │    267,484   17 %  │        267,484  │
│ │ │ │ │   │                                                       
                                                   │                    │   
              │
│ │ │ │ │   
└───org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(IndexReader,
 boolean, boolean)              │        213    0 %  │              0  │
│ │ │ │ │                                                             
                                                 │                    │     
            │
│ │ │ │ 
└───org.apache.lucene.search.FilteredQuery$2.score(Collector)           
                                       │     52,456    3 %  │            
621  │
│ │ │ │                                                                 
                                               │                    │       
          │
│ │ │ 
└───org.apache.lucene.search.IndexSearcher.createNormalizedWeight(Query)
                                         │         42    0 %  │             
 0  │
│ │ │                                                                     
                                             │                    │         
        │
│ │ 
├─┬─org.apache.solr.search.grouping.CommandHandler.computeDocSet(Query, 
Filter, List)                              │    300,359   19 %  │           
  61  │
│ │ │ │                                                                 
                                               │                    │       
          │
│ │ │ ├─┬─org.apache.lucene.search.IndexSearcher.search(Query, 
Filter, Collector)                                      │    300,276   19 %  
│              0  │
│ │ │ │ │                                                             
                                                 │                    │     
            │
│ │ │ │ 
├─┬─org.apache.lucene.search.IndexSearcher.search(Weight, Filter, 
Collector)                                   │    300,271   19 %  │         
     0  │
│ │ │ │ │ │                                                         
                                                   │                    │   
              │
│ │ │ │ │ 
├─┬─org.apache.lucene.search.FilteredQuery.getFilteredScorer(IndexReader,
 Similarity, Weight, Weight, ...)   │    259,201   17 %  │              3  
│
│ │ │ │ │ │ │                                                     
                                                     │                    │ 
                │
│ │ │ │ │ │ 
└─┬─org.apache.solr.search.function.BoostedQuery$BoostedWeight.scorer(IndexReader,
 boolean, boolean)       │    259,197   17 %  │              2  │
│ │ │ │ │ │   │                                                   
                                                     │                    │ 
                │
│ │ │ │ │ │   
├─┬─org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>(BoostedQuery,
 Similarity, ...)      │    259,003   17 %  │              0  │
│ │ │ │ │ │   │ │                                               
                                                       │                    
│                 │
│ │ │ │ │ │   │ 
└─┬─org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>(BoostedQuery,
 Similarity, ...)    │    259,003   17 %  │              0  │
│ │ │ │ │ │   │   │                                             
                                                       │                    
│                 │
│ │ │ │ │ │   │   
└─┬─org.apache.solr.search.function.MultiFloatFunction.getValues(Map, 
IndexReader)                   │    259,003   17 %  │              0  │
│ │ │ │ │ │   │     │                                           
                                                       │                    
│                 │
│ │ │ │ │ │   │     
└─┬─org.apache.solr.search.Double2Parser$Function.getValues(Map, 
IndexReader)                      │    259,003   17 %  │              0  │
│ │ │ │ │ │   │       │                                         
                                                       │                    
│                 │
│ │ │ │ │ │   │       
└─┬─org.apache.solr.search.function.TopValueSource.getValues(Map, 
IndexReader)                   │    259,003   17 %  │              0  │
│ │ │ │ │ │   │         │                                       
                                                       │                    
│                 │
│ │ │ │ │ │   │         
└───org.apache.solr.search.function.ScaleFloatFunction.getValues(Map, 
IndexReader)             │    259,003   17 %  │        259,003  │
│ │ │ │ │ │   │                                                   
                                                     │                    │ 
                │
│ │ │ │ │ │   
└───org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(IndexReader,
 boolean, boolean)            │        191    0 %  │              0  │
│ │ │ │ │ │                                                         
                                                   │                    │   
              │
│ │ │ │ │ 
├───org.apache.lucene.search.FilteredQuery$2.score(Collector)           
                                     │     41,044    3 %  │            567  
│
│ │ │ │ │ │                                                         
                                                   │                    │   
              │
│ │ │ │ │ 
└───org.apache.solr.search.DocSetDelegateCollector.setNextReader(IndexReader,
 int)                           │         24    0 %  │              0  │
│ │ │ │ │                                                             
                                                 │                    │     
            │
│ │ │ │ 
└───org.apache.lucene.search.IndexSearcher.createNormalizedWeight(Query)
                                       │          4    0 %  │              
0  │
│ │ │ │                                                                 
                                               │                    │       
          │
│ │ │ 
└───org.apache.solr.search.DocSetDelegateCollector.<init>(int, int, 
Collector)                                   │         20    0 %  │         
    20  │
│ │ │                                                                     
                                             │                    │         
        │
│ │ 
└───org.apache.solr.search.SolrIndexSearcher.getProcessedFilter(DocSet, 
List)                                      │      2,586    0 %  │           
   5  │
│ │                                                                         
                                           │                    │           
      │
│ 
├───org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher$QueryResult,
 SolrIndexSearcher$QueryCommand)   │    362,135   23 %  │              0  
│
│ │                                                                         
                                           │                    │           
      │
│ ├───org.apache.solr.search.SolrIndexSearcher.getDocSet(List)        
                                                 │      4,325    0 %  │     
         0  │
│ │                                                                         
                                           │                    │           
      │
│ ├───org.apache.solr.search.Grouping.execute()                       
                                                 │      3,591    0 %  │     
         0  │
│ │                                                                         
                                           │                    │           
      │
│ 
├───org.apache.solr.search.grouping.CommandHandler.processResult(SolrIndexSearcher$QueryResult,
 ...)                 │        655    0 %  │              0  │
│ │                                                                         
                                           │                    │           
      │
│ 
├───org.apache.solr.handler.component.QueryComponent.doFieldSortValues(ResponseBuilder,
 SolrIndexSearcher)           │        101    0 %  │            101  │
│ │                                                                         
                                           │                    │           
      │
│ └───org.apache.solr.search.SolrIndexSearcher.getFirstMatch(Term)    
                                                 │         10    0 %  │     
         0  │
│                                                                             
                                         │                    │             
    │
│ org.apache.solr.handler.component.QueryComponent.prepare(ResponseBuilder)   
                                         │      4,811    0 %  │             
 0  │
│                                                                             
                                         │                    │             
    │
│ org.apache.solr.handler.component.FacetComponent.process(ResponseBuilder)   
                                         │      2,901    0 %  │             
 0  │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┴─────────────────┘

Generated by YourKit Java Profiler 10.0.0 08─Feb─2012 16:12:23

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org