Re: Realtime search and facets with very frequent commits

2010-04-06 Thread Janne Majaranta
Yeah, thanks for pointing this out. I'm not using any relevancy functions (yet). The data indexed for my app is basically log events. The most relevant events are the newest ones, so sorting by timestamp is enough. BTW, your book is great ;) -Janne 2010/3/31 Smiley, David W. dsmi...@mitre.org

Re: Realtime search and facets with very frequent commits

2010-02-18 Thread Otis Gospodnetic
-user@lucene.apache.org Sent: Thu, February 18, 2010 2:12:37 AM Subject: Re: Realtime search and facets with very frequent commits Hi, Yes, I did play with mergeFactor. I didn't play with mergePolicy. Wouldn't that affect indexing speed and possibly memory usage ? I don't have any

Re: Realtime search and facets with very frequent commits

2010-02-18 Thread Janne Majaranta
...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, February 18, 2010 2:12:37 AM Subject: Re: Realtime search and facets with very frequent commits Hi, Yes, I did play with mergeFactor. I didn't play with mergePolicy. Wouldn't that affect indexing speed and possibly memory usage

Re: Realtime search and facets with very frequent commits

2010-02-18 Thread Jason Rutherglen
...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, February 18, 2010 2:12:37 AM Subject: Re: Realtime search and facets with very frequent commits Hi, Yes, I did play with mergeFactor. I didn't play with mergePolicy. Wouldn't that affect indexing speed and possibly memory usage

Re: Realtime search and facets with very frequent commits

2010-02-18 Thread Janne Majaranta
search :: http://search-hadoop.com/ - Original Message From: Janne Majaranta janne.majara...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, February 18, 2010 2:12:37 AM Subject: Re: Realtime search and facets with very frequent commits Hi, Yes, I did

Re: Realtime search and facets with very frequent commits

2010-02-17 Thread Jan Høydahl / Cominvent
Hi, Have you tried playing with mergeFactor or even mergePolicy? -- Jan Høydahl - search architect Cominvent AS - www.cominvent.com On 16. feb. 2010, at 08.26, Janne Majaranta wrote: Hey Dipti, Basically query optimizations + setting cache sizes to a very high level. Other than that, the

Re: Realtime search and facets with very frequent commits

2010-02-17 Thread Janne Majaranta
Hi, Yes, I did play with mergeFactor. I didn't play with mergePolicy. Wouldn't that affect indexing speed and possibly memory usage ? I don't have any problems with indexing speed ( 1000 - 2000 docs / sec via the standard HTTP API ). My problem is that I need very warm caches to get fast

Re: Realtime search and facets with very frequent commits

2010-02-15 Thread dipti khullar
@lucene.apache.org Sent: Thu, February 11, 2010 12:35:20 PM Subject: Realtime search and facets with very frequent commits Hello, I have a log search like application which requires indexed log events to be searchable within a minute and uses facets and the statscomponent

Re: Realtime search and facets with very frequent commits

2010-02-15 Thread Janne Majaranta
Hey Dipti, Basically query optimizations + setting cache sizes to a very high level. Other than that, the config is about the same as the out-of-the-box config that comes with the Solr download. I haven't found a magic switch to get very fast query responses + facet counts with the frequency of

Realtime search and facets with very frequent commits

2010-02-11 Thread Janne Majaranta
Hello, I have a log search like application which requires indexed log events to be searchable within a minute and uses facets and the statscomponent. Some stats: - The log events are indexed every 10 seconds with a commitWithin of 60 seconds. - 1M events / day (~75% are updates to previous

Re: Realtime search and facets with very frequent commits

2010-02-11 Thread Jason Rutherglen
Janne, I usually just turn the caches to next to nearly off for frequent commits. Jason On Thu, Feb 11, 2010 at 9:35 AM, Janne Majaranta janne.majara...@gmail.com wrote: Hello, I have a log search like application which requires indexed log events to be searchable within a minute and uses

Re: Realtime search and facets with very frequent commits

2010-02-11 Thread Janne Majaranta
Hey Jason, Do you use faceting with frequent commits ? And by turning off the caches you mean setting autowarmcount to zero ? I did try to turn off autowarming with a 36M documents instance but getting facets over those documents takes over 10 seconds. With a warm cache it takes 200ms ...

Re: Realtime search and facets with very frequent commits

2010-02-11 Thread Yonik Seeley
On Thu, Feb 11, 2010 at 3:21 PM, Janne Majaranta janne.majara...@gmail.com wrote: Hey Jason, Do you use faceting with frequent commits ? And by turning off the caches you mean setting autowarmcount to zero ? I did try to turn off autowarming with a 36M documents instance but getting facets

Re: Realtime search and facets with very frequent commits

2010-02-11 Thread Otis Gospodnetic
/ - Original Message From: Janne Majaranta janne.majara...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, February 11, 2010 12:35:20 PM Subject: Realtime search and facets with very frequent commits Hello, I have a log search like application which requires indexed log events

Re: Realtime search and facets with very frequent commits

2010-02-11 Thread Janne Majaranta
://search-hadoop.com/ - Original Message From: Janne Majaranta janne.majara...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, February 11, 2010 12:35:20 PM Subject: Realtime search and facets with very frequent commits Hello, I have a log search like application