Hi Alan, I don't have any direct feedback... but I know there is an issue that you may want to be aware of (and incorporate?) - https://issues.apache.org/jira/browse/SOLR-3514
Otis -- Search Analytics - http://sematext.com/search-analytics/index.html Performance Monitoring - http://sematext.com/spm/index.html On Mon, Oct 15, 2012 at 9:37 AM, Alan Woodward <alan.woodw...@romseysoftware.co.uk> wrote: > Hi list, > > I'm having a go at improving the performance of ExternalFileField (relevant > to this thread: > http://lucene.472066.n3.nabble.com/Reloading-ExternalFileField-blocks-Solr-td4012399.html#a4013305), > and thought I'd get some feedback. What do people think of the following? > > - FileFloatSource needs to be updated in three cases: > - when new segments are added > - when segments are merged > - when the external file source is updated > > In our use-case, new documents will not have values in the external file (it > contains things like click-data, which will only appear after the document > has been in the index for a while), so we don't need to reload when new > segments are added. > > My plan is to hook the cache refresh into either newSearcher or postCommit. > I change the FileFloatSource internals to be keyed on individual > SegmentReaders rather than top-level IndexReaders, so existing float caches > don't need to be reloaded for unchanged segments; I (somehow?) detect if > segments with empty caches contain new documents (in which case we can just > give them all default values) or are the result of merges (in which case we > need to reload the external file and repopulate). > > I also plan to modify the reloadCaches update handler so that instead of just > clearing the cache (and hence slowing down the next query to hit, as the new > caches are lazy-loaded), it reloads the file in the background and then cuts > over to the new caches. > > I'll open a JIRA and post patches once I've begun the actual implementation, > but if anybody notices something that would stop this working, it would be > nice to hear about it before I start… :-) > > Thanks, > > Alan Woodward
