I'm not really sure, if this approach is possible for working in changes every - let's say - 30 seconds!?
-------- Original-Nachricht -------- > Datum: Thu, 17 Jan 2008 05:35:13 +0100 > Von: "Marcus Falk" <[EMAIL PROTECTED]> > An: java-user@lucene.apache.org, java-user@lucene.apache.org > Betreff: SV: Integrating dynamic data into Lucene search/ranking > We did this in our system, indexing a constant flow of news articles, by > doing as Otis described (reopened the indexsearcher).. > > Every 3:d minute we are creating a new indexsearcher in the background > after this searcher has been created we are fireing some warm up queries > against it and after that we change the old searcher to point to the new one. > Works fine for us and we got large indexes (several millions of articles)... > > /Regards > Marcus > > > > > ________________________________ > > Från: Tobias Lohr [mailto:[EMAIL PROTECTED] > Skickat: on 2008-01-16 21:57 > Till: java-user@lucene.apache.org > Ämne: Re: Integrating dynamic data into Lucene search/ranking > > > > The index contains about a several ten thousand documents, with a field > count of about fifty. The index is going to be rebuild approx. every > day, but varies, since the searchable content doesn't change very often. > Now I face the challenge to work in more dynamic data into the index, > and even make this searchable, or more using it to sort the documents in > the search result. The change frequency of this data is very high, at > minimum every minute, but can be every few seconds. > > I know that it is impossible to have a real time up to date index and > that there will be gap in time. But as long as this is not to large, > this is ok. I've tried to do some incremental update every X seconds or > minutes. This would eventually work for small indexes, with less data to > be indexed for a single document, but I think that this is not the right > approach here. It's also not the right way, because of collecting all > the static, searchable data that didn't changed for a document, although > only the dynamic information changed, you know? > > I'm sorry, what exactly do you mean with "reopen the IndexSearcher every > N minutes"? > > You say, that I could also store the data in an external data store. Do > you mean the dynamic data? If yes, I some how need this information > within an index, in order to sort by it, right? Or do I overlook > something here? > > Thanks, > Tobias > > Otis Gospodnetic schrieb: > > Tobias, > > > > The question is a little too open, I think. Perhaps start by saying > what you've tried, what doesn't work, what you think won't work, the actual > rate of change, the size of your index and, very importantly, how quickly you > need to see index changes (adds, deletes, updates). > > > > How about this for the boostrap question: just update (delete+add) the > whole Document and reopen the IndexSeearcher every N minutes. Would that > work for you? > > > > Does only the stored data change or also searchable data? If the > former, you could choose to store that in the external data store (e.g. RDBMS, > BDB...) > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- > > From: Tobias Lohr <[EMAIL PROTECTED]> > > To: java-user@lucene.apache.org > > Sent: Tuesday, January 15, 2008 11:33:56 AM > > Subject: Integrating dynamic data into Lucene search/ranking > > > > I have a more architectural question, which is maybe sort of off topic, > > but as I want to implement it using Java and Lucene, it's the right > > forum however: > > > > I'm thinking of an approach to design a system that integrates dynamic > > information into a search (and a ranking) functionality using Lucene. > > With dynamic data I mean, data which changes very often within the > > typical index rebuild cycle, i.e. transactional data. > > > > A good example is the sorting of products in an online store by product > > availability. > > > > Does anybody know good reading resources (approaches, whitepapers, > > books etc.) for integrating such dynamic information into a > search/ranking > > functionality? > > > > (I already searched at Google, but couldn't find anything useful > > though.) > > > > Thanks in advance! > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > > -- Psssst! Schon vom neuen GMX MultiMessenger gehört? Der kann`s mit allen: http://www.gmx.net/de/go/multimessenger?did=10 --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
