Hi Vincent, Lucene has a hard limit of ~2.1 B documents in a single index; hopefully you hit the ~50 - 100 GB limit well before that.
Otherwise it's very application dependent: how much latency can you tolerate during searching, how fast are the underlying IO devices at random and large sequential IO, the types of queries, etc. Lucene should not require much additional RAM as the index gets larger -- much work has been done in recent years to move data structures off-heap. Mike McCandless http://blog.mikemccandless.com On Tue, Jan 2, 2024 at 9:49 AM <vvse...@gmail.com> wrote: > Hello, > > is there a recommended / rule of thumb maximum size for index? > I try to target between 50 and 100 Gb, before spreading to other servers. > or is this just a matter of how much memory and cpu I have? > this is a log aggregation use case. a lot of write, smaller number of reads > obviously. > I am using lucene 9. > thanks, > Vincent >
