You'll have to do some tuning with that kind of ingestion rate, and you're talking about a significant size cluster here. At 172M documents/day or so, you're not going to store very many days per node.
Storing doesn't make much of any difference as far as search speed is concerned, the raw data is stored in separate files (*.fdt and *.fdx files) and doesn't affect _search_. They are accessed to assemble the response. Otherwise there's not a lot of info to go on here. Here are some resources: http://wiki.apache.org/solr/SolrPerformanceFactors http://searchhub.org/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/ Best, Erick On Sat, Jan 25, 2014 at 1:27 AM, Hamed Ghavamnia <ghavamni...@gmail.com> wrote: > Hello, > > I searched a lot about lucene limits and its performance, but I still don't > know how much I can count on it. I'm storing logs and indexing them with > lucene. The event per second is 2000. The format of each log is generally > 'fieldname' : 'fieldvalue'. > What search performance should I expect after a few days. Right now I'm > having around 25 seconds of query response time on around 500 million logs. > Each log is converted into a document and the field values are stored as > well as being indexed. I have around 10 fields in each log. > Is my query time normal, of am I making a huge mistake? > How much does storing fields make a difference, would it be better if I > didn't store the fields. > > Thanks. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
