Op Thursday 04 December 2008 23:03:40 schreef robert engels: > The biggest benefit I see of using the field cache to do filter > caching, is that the same cache can be used for sorting - thereby > improving the performance and memory usage.
Would it be possible to build such Filter caching into CachingWrapperFilter instead of into QueryFilter? Both filter caching and the field value caching will need access to the underlying (segment?) readers. > > The downside I see is that if you have a common filter that is built > from many fields, you are going to use a lot more memory, as every > field used needs to be cached. With my code you would only have a > single "bitset" for the filter. But with many ranges that would mean many bitsets, and MemoryCachedRangeFilter only needs to cache the field values once for any number of ranges. It's a tradeoff. Regards, Paul Elschot > > On Dec 4, 2008, at 4:00 PM, robert engels wrote: > > Lucene-831 is far more comprehensive. > > > > I also think that by exposing access to the sub-readers it can be > > far simpler (closer to what I have provided). > > > > In the mean-time, you should be able to use the provided class with > > a few modifications. > > > > The "reload the entire cache" was a deal breaker for us, so I came > > up the attached. Works very well. > > > > On Dec 4, 2008, at 3:54 PM, Uwe Schindler wrote: > >> I am looking all the time to LUCENE-831, which is a new version of > >> FieldCache that is compatible with IndexReader.reopen() and > >> invalidates only > >> reloaded segments. In each release of Lucene I am very unhappy, > >> because it > >> is still not in. The same problem like yours is if you have a one > >> million > >> documents index that is updated by adding a few documents each > >> half hour. If > >> you use sorting by a field, whenever the index is reopened and you > >> really > >> only a very small segment is added, nevertheless the complete > >> FieldCache is > >> rebuild, very bad :(. > >> > >> > >> So I think the ultimative fix would be to hopefully apply > >> LUCENE-831 soon > >> and also use LUCENE-1461 as RangeFilter cache. > >> > >> ----- > >> Uwe Schindler > >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> eMail: [EMAIL PROTECTED] > >> ________________________________________ > >> From: robert engels [mailto:[EMAIL PROTECTED] > >> Sent: Thursday, December 04, 2008 9:39 PM > >> To: java-dev@lucene.apache.org > >> Subject: Re: [jira] Commented: (LUCENE-855) > >> MemoryCachedRangeFilter to boost > >> performance of Range queries > >> > >> I can't seem to post to Jira, so I am attaching here... > >> > >> I attached QueryFilter.java. > >> > >> In reading this patch, and other similar ones, the problem seems > >> to be that > >> if the index is modified, the cache is invalidated, causing a > >> complete > >> reload of the cache. Do I have this correct? > >> > >> The attached patch works really well in a highly interactive > >> environment, as > >> the cache is only invalidated at the segment level. > >> > >> The MyMultiReader is a subclass that allows access to the > >> underlying SegmentReaders. > >> > >> The patch cannot be applied, but I think the implementation works > >> far better > >> in many cases - it is also far less memory intensive. Scanning the > >> bitset > >> could also be optimized very easily using internal skip values. > >> > >> Maybe this is completely off-base, but the solution has worked > >> very well for > >> us. Maybe this is a completely different issue and separate > >> incident should > >> be opened ? > >> > >> is there any interest in this? > >> > >> > >> > >> ------------------------------------------------------------------ > >>--- To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > > > > ------------------------------------------------------------------- > >-- To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
