Hi Vincent, My 2 cents:
We had a production environment with ~250g and ~1M docs with static + dynamic fields in Solr (afair lucene 7) with a machine having 4GB for the jvm and (afair) a little bit more maybe 6GB OS ‚cache‘. In peak times (re-index) we had 10-15k updates / minute and (partially) complex queries up to 50/sec per jvm. At those times our servers still had rotating discs. In this setup we did not experience any performance issues when we did not had bugs / misconfigurations. We were thinking of sharding / splitting indexes, but did not do it due to complexity of maintaining those later - AND especially - there was NO NEED at all. Elasticsearch/Solr started to do it out of the box at that time. Maybe Kibana/ELK or such is a thing to look at too. Cheers from Berlin, Ralf Von meinem Telefon gesendet, etwaige Rechtschreibfehler kann ich nicht ausschliessen > Am 04.01.2024 um 17:32 schrieb Michael McCandless <luc...@mikemccandless.com>: > > Hi Vincent, > > Lucene has a hard limit of ~2.1 B documents in a single index; hopefully > you hit the ~50 - 100 GB limit well before that. > > Otherwise it's very application dependent: how much latency can you > tolerate during searching, how fast are the underlying IO devices at random > and large sequential IO, the types of queries, etc. > > Lucene should not require much additional RAM as the index gets larger -- > much work has been done in recent years to move data structures off-heap. > > Mike McCandless > > http://blog.mikemccandless.com > > >> On Tue, Jan 2, 2024 at 9:49 AM <vvse...@gmail.com> wrote: >> >> Hello, >> >> is there a recommended / rule of thumb maximum size for index? >> I try to target between 50 and 100 Gb, before spreading to other servers. >> or is this just a matter of how much memory and cpu I have? >> this is a log aggregation use case. a lot of write, smaller number of reads >> obviously. >> I am using lucene 9. >> thanks, >> Vincent >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
