Hi Uwe, Thank you for the detailed explanation, that helps a lot. I am still trying to understand and confirm a few details though.
> Is my understanding correct that it still makes sense to avoid MMAPing > files with the random access pattern on the most recent Lucene and JVM > versions? > Who said this? This is simply not true! Myths.... Originally I saw this concern mentioned here: https://github.com/elastic/elasticsearch/issues/27748. There is an answer at the bottom, saying this: > We analyzed the same problem with mmapfs around six months ago and came to > the same conclusions. This is why we have introduced hybridfs (#36668) to > use NIO when we expect the access pattern to be random such that sequential > read-ahead would be painful and otherwise use mmap I did not do any research to confirm that, but that thread suggests that cache trashing issue happens only on mmap: > You can see mmapfs is consuming more cache and IO than niofs. I also noticed this <https://github.com/elastic/elasticsearch/pull/36668#discussion_r242147643>comment from Adrien <https://github.com/elastic/elasticsearch/pull/36668#discussion_r242147643> in the PR which introduces hybridfs: > let's mention FS cache usage rather than the number of mmaps as a reason > for this hybrid store type? So if I read this correctly, it seems that the main reason why Elasticsearch started to use the hybrid approach was the FS cache. Am I missing something, or maybe things have changed and it's not relevant anymore? -- Regards, Alex On Mon, Aug 22, 2022 at 8:18 AM Uwe Schindler <u...@thetaphi.de> wrote: > Hi Alexander, > > I understand that NIOFSDirectory also uses the FS cache, but doesn't > MMapDirectory tend to fill up the cache with unnecessary data for random > access pattern due to sequential read-ahead? Our concern is that it can > potentially lead to evicting hot pages used by another process on the same > host, affecting its performance. > > no this is not the case (at least not on Linux or Solaris). It is no > difference between read() and a pagefault by mmap. It will read the same > pages and put them into cache. It won't read more pages for mmap. What gets > read depends on fadvice or madvise, which Lucene does not change (the OS > decides). > > As far as I can Elasticsearch also avoids using MMap for everything by > default, e.g stored fields and term vectors are not MMAPed. > > The Elasticsearch reason is different: It does it because of the limited > number of mappings available by current kernels. Elasticsearch clusters > tend to have many indexes and to avoid too many mappings they do this. It > has nothing to do with caching. > > stored fields and term vectors are valid candidates to not mmapping them > if you have pressure on number of mappings. The access pattern is > completely different. So what Elasticserach does is a valid thing to do. If > you really want to spare mappings, use the stored fields / term vectors > approach. But then you also need to disable CFS files which is > contra-productive, as it raises the number of mappings and file handles. > > Does it make sense or am I missing something? Is my understanding correct > that it still makes sense to avoid MMAPing files with the random access > pattern on the most recent Lucene and JVM versions? > > Who said this? This is simply not true! Myths.... > > One last word: With the next Lucene version after Java 19 came out you > will be able to work around the "too many mappings" problem for huge clouds > of Elasticsearch clusters due to a new MMAP implementation choosen using > MultiRelease lucene-core.jar file. This will allow them to mmap everything > when Java 19+ is used (and the preview features of Java are enabled). This > works by having huger blocks of virtual memory (currently limited to 1 > Gigabyte per mapping) => https://github.com/apache/lucene/pull/912 > > Uwe > > > Thank you, > Alex > > > On Fri, Aug 19, 2022 at 2:42 AM Robert Muir <rcm...@gmail.com> wrote: > >> On Thu, Aug 18, 2022 at 1:47 PM Alexander Lukyanchikov >> <alexanderlukyanchi...@gmail.com> wrote: >> >> > >> > Currently we are trying to avoid switching to MMAP because there is >> another process running on the same host and extensively utilizes the FS >> cache. >> > >> >> This makes no sense, NIOFSDirectory uses the FS cache the exact same >> way as mmap. it just uses read() interface instead. >> >> A self-created problem! >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> -- > Uwe Schindler > Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de > eMail: u...@thetaphi.de > >