Dawid Weiss <dawid.we...@gmail.com> wrote: > Merging segments as large as this one requires not just CPU, but also > serious I/O throughput efficiency. I assume you have fast NVMe drives > on that machine, otherwise it'll be slow, no matter what. It's just a > lot of bytes going back and forth.
We have quite a lot of experience with creating fully merged 900GB indexes. On our plain-SSD (Samsung 840) equipped machine this took ~8 hours with a single CPU-core at 100%. On our 7200 RPM spinning drive machines (same CPU class) it took nearly twice as long. Back of the envelope says reading & writing 900GB in 8 hours is 2*900GB/(8*60*60s) = 64MB/s. I don't remember the interface for our SSD machine, but even with SATA II this is only ~1/5th of the possible fairly sequential IO throughput. So for us at least, NVMe drives are not needed to have single-threaded CPU as bottleneck. And +1 to the issue BTW. It does not matter too much for us now, as we have shifted to a setup where we build more indexes in parallel, but 3 years ago our process was sequential so the 8 hour delay before building the next part was a bit of an annoyance. - Toke Eskildsen --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
