Hi Uwe,
Thank you for the detailed explanation, that helps a lot. I am still trying
to understand and confirm a few details though.

> Is my understanding correct that it still makes sense to avoid MMAPing
> files with the random access pattern on the most recent Lucene and JVM
> versions?
> Who said this? This is simply not true! Myths....


Originally I saw this concern mentioned here:
https://github.com/elastic/elasticsearch/issues/27748. There is an answer
at the bottom, saying this:

> We analyzed the same problem with mmapfs around six months ago and came to
> the same conclusions. This is why we have introduced hybridfs (#36668) to
> use NIO when we expect the access pattern to be random such that sequential
> read-ahead would be painful and otherwise use mmap


I did not do any research to confirm that, but that thread suggests that
cache trashing issue happens only on mmap:

> You can see mmapfs is consuming more cache and IO than niofs.


I also noticed this
<https://github.com/elastic/elasticsearch/pull/36668#discussion_r242147643>comment
from Adrien
<https://github.com/elastic/elasticsearch/pull/36668#discussion_r242147643>
in the PR which introduces hybridfs:

> let's mention FS cache usage rather than the number of mmaps as a reason
> for this hybrid store type?


So if I read this correctly, it seems that the main reason why
Elasticsearch started to use the hybrid approach was the FS cache.

Am I missing something, or maybe things have changed and it's not relevant
anymore?

--
Regards,
Alex


On Mon, Aug 22, 2022 at 8:18 AM Uwe Schindler <u...@thetaphi.de> wrote:

> Hi Alexander,
>
> I understand that NIOFSDirectory also uses the FS cache, but doesn't
> MMapDirectory tend to fill up the cache with unnecessary data for random
> access pattern due to sequential read-ahead? Our concern is that it can
> potentially lead to evicting hot pages used by another process on the same
> host, affecting its performance.
>
> no this is not the case (at least not on Linux or Solaris). It is no
> difference between read() and a pagefault by mmap. It will read the same
> pages and put them into cache. It won't read more pages for mmap. What gets
> read depends on fadvice or madvise, which Lucene does not change (the OS
> decides).
>
> As far as I can Elasticsearch also avoids using MMap for everything by
> default, e.g stored fields and term vectors are not MMAPed.
>
> The Elasticsearch reason is different: It does it because of the limited
> number of mappings available by current kernels. Elasticsearch clusters
> tend to have many indexes and to avoid too many mappings they do this. It
> has nothing to do with caching.
>
> stored fields and term vectors are valid candidates to not mmapping them
> if you have pressure on number of mappings. The access pattern is
> completely different. So what Elasticserach does is a valid thing to do. If
> you really want to spare mappings, use the stored fields / term vectors
> approach. But then you also need to disable CFS files which is
> contra-productive, as it raises the number of mappings and file handles.
>
> Does it make sense or am I missing something? Is my understanding correct
> that it still makes sense to avoid MMAPing files with the random access
> pattern on the most recent Lucene and JVM versions?
>
> Who said this? This is simply not true! Myths....
>
> One last word: With the next Lucene version after Java 19 came out you
> will be able to work around the "too many mappings" problem for huge clouds
> of Elasticsearch clusters due to a new MMAP implementation choosen using
> MultiRelease lucene-core.jar file. This will allow them to mmap everything
> when Java 19+ is used (and the preview features of Java are enabled). This
> works by having huger blocks of virtual memory (currently limited to 1
> Gigabyte per mapping) => https://github.com/apache/lucene/pull/912
>
> Uwe
>
>
> Thank you,
> Alex
>
>
> On Fri, Aug 19, 2022 at 2:42 AM Robert Muir <rcm...@gmail.com> wrote:
>
>> On Thu, Aug 18, 2022 at 1:47 PM Alexander Lukyanchikov
>> <alexanderlukyanchi...@gmail.com> wrote:
>>
>> >
>> > Currently we are trying to avoid switching to MMAP because there is
>> another process running on the same host and extensively utilizes the FS
>> cache.
>> >
>>
>> This makes no sense, NIOFSDirectory uses the FS cache the exact same
>> way as mmap. it just uses read() interface instead.
>>
>> A self-created problem!
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>

Reply via email to