[
https://issues.apache.org/jira/browse/LUCENE-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Ericson updated LUCENE-855:
--------------------------------
Attachment: FieldCacheRangeFilter.patch
Andy was correct the 2 performance tests were bogus as they did not call get()
from the bit sets. And my code does all of the work int the get() call. I
guess I should have looked a little closer at the tests before using it
I changes his tests and mine to call and IndexSearcher.search(q,filter) and
actually do the search
Here are the results
Using the MemoryCachedRangeFilter
[junit] ------------- Standard Output ---------------
[junit] Start interval: Tue Apr 09 14:32:14 PDT 2002
[junit] End interval: Sun Apr 08 14:32:14 PDT 2007
[junit] Creating RAMDirectory index...
[junit] Reader opened with 100000 documents. Creating RangeFilters...
[junit] Standard RangeFilter finished in 57533ms
[junit] MemoryCachedRangeFilter inished in 905ms
[junit] ------------- ---------------- ---------------
Using FieldCacheRangeFilter
[junit] ------------- Standard Output ---------------
[junit] Start interval: Tue Apr 09 14:30:29 PDT 2002
[junit] End interval: Sun Apr 08 14:30:29 PDT 2007
[junit] Creating RAMDirectory index...
[junit] Reader opened with 100000 documents. Creating RangeFilters...
[junit] Standard RangeFilter finished in 58822ms
[junit] FieldCacheRangeFilter inished in 102ms
[junit] ------------- ---------------- ---------------
They are much closer this time
I have fixed my BitSets to allow a user to call nextClearBit or nextSetBit
> MemoryCachedRangeFilter to boost performance of Range queries
> -------------------------------------------------------------
>
> Key: LUCENE-855
> URL: https://issues.apache.org/jira/browse/LUCENE-855
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Affects Versions: 2.1
> Reporter: Andy Liu
> Attachments: FieldCacheRangeFilter.patch,
> FieldCacheRangeFilter.patch, MemoryCachedRangeFilter.patch,
> MemoryCachedRangeFilter_1.4.patch
>
>
> Currently RangeFilter uses TermEnum and TermDocs to find documents that fall
> within the specified range. This requires iterating through every single
> term in the index and can get rather slow for large document sets.
> MemoryCachedRangeFilter reads all <docId, value> pairs of a given field,
> sorts by value, and stores in a SortedFieldCache. During bits(), binary
> searches are used to find the start and end indices of the lower and upper
> bound values. The BitSet is populated by all the docId values that fall in
> between the start and end indices.
> TestMemoryCachedRangeFilterPerformance creates a 100K RAMDirectory-backed
> index with random date values within a 5 year range. Executing bits() 1000
> times on standard RangeQuery using random date intervals took 63904ms. Using
> MemoryCachedRangeFilter, it took 876ms. Performance increase is less
> dramatic when you have less unique terms in a field or using less number of
> documents.
> Currently MemoryCachedRangeFilter only works with numeric values (values are
> stored in a long[] array) but it can be easily changed to support Strings. A
> side "benefit" of storing the values are stored as longs, is that there's no
> longer the need to make the values lexographically comparable, i.e. padding
> numeric values with zeros.
> The downside of using MemoryCachedRangeFilter is there's a fairly significant
> memory requirement. So it's designed to be used in situations where range
> filter performance is critical and memory consumption is not an issue. The
> memory requirements are: (sizeof(int) + sizeof(long)) * numDocs.
> MemoryCachedRangeFilter also requires a warmup step which can take a while to
> run in large datasets (it took 40s to run on a 3M document corpus). Warmup
> can be called explicitly or is automatically called the first time
> MemoryCachedRangeFilter is applied using a given field.
> So in summery, MemoryCachedRangeFilter can be useful when:
> - Performance is critical
> - Memory is not an issue
> - Field contains many unique numeric values
> - Index contains large amount of documents
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
