Thanks for the response Shawn. We are using Windows server with pretty huge indexes (multiple TB cores). With Mmap, I have observed that the machine just completely freezes with high CPU and memory usage to a point where it becomes impossible to even connect to it. SimpleFS works out well for us in this case.
As noted in my first email, even with SimpleFS, Solr 7 completes the crawl in nearly 1/5th the time taken in Solr 8. Hence there should be something OUTSIDE the directory factory in the code which is causing this. Thanks, Rahul On Tue, May 30, 2023 at 10:47 PM Shawn Heisey <apa...@elyograg.org> wrote: > On 5/30/23 15:34, Rahul Goswami wrote: > > Environment details: - Java 11 on Windows server - Xms1536m Xmx3072m - > > Indexing client code running 15 parallel threads indexing in batches of > > 1000 - using SimpleFSDirectoryFactory (since Mmap doesn't quite work > > well on Windows for our index sizes which commonly run north of 1 TB) > > Don't change the directoryFactory. You *WANT* Solr to use MMAP for your > indexes. Not using MMAP is likely to slow things down considerably. > MMAP should work just fine on 64-bit Windows with 64-bit Java. Which of > course requires 64-bit hardware. > > 32 bit systems and software cannot properly deal with data larger than > about 2GB. > > Thanks, > Shawn >
