Thanks Mike, appreciate the reply and the suggestions very much.
And Your article link to concurrent search is amazing.
Together with in memory and concurrent index (especially in read only mode)
these will speed up Lucene queries very much.
Happy Holidays
Best regards
On 12/14/20 10:12 AM, Michael McCandless wrote:
Hello,
Yes, that is exactly what MMapDirectory.setPreload is trying to do, but
not
promises (it is best effort). I think it asks the OS to touch all pages
in
the mapped region so they are cached in RAM, if you have enough RAM.
Make your JVM heap as low as possible to let the OS have more RAM to use
to
load your index.
Mike McCandless
https://urldefense.com/v3/__http://blog.mikemccandless.com__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJn-Lr5mA$
On Sun, Dec 13, 2020 at 4:18 PM <baris.ka...@oracle.com> wrote:
Hi,-
it would be nice to create a Lucene index in files and then effectively
load it into memory once (since i use in read-only mode). I am looking
into
if this is doable in Lucene.
i wish there were an option to load whole Lucene index into memory:
Both of below urls have links to the blog url where i quoted a very nice
section:
https://urldefense.com/v3/__https://lucene.apache.org/core/8_5_0/core/org/apache/lucene/store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJaN3djDw$
https://urldefense.com/v3/__https://lucene.apache.org/core/8_5_2/core/org/apache/lucene/store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJhxlyzBw$
This following blog mentions about such option
to run in the memory: (see the underlined sentence below)
https://urldefense.com/v3/__https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html?m=1__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJ1O4pdIg$
MMapDirectory will not load the whole index into physical memory. Why
should it do this? We just ask the operating system to map the file into
address space for easy access, by no means we are requesting more. Java
and
the O/S optionally provide the option to try loading the whole file into
RAM (if enough is available), but Lucene does not use that option (we
may
add this possibility in a later version).
My question is: is there such an option?
is the method setPreLoad for this purpose:
to load all Lucene lndex into memory?
I would like to use MMapDirectory and set my
JVM heap to 16G or a bit less (since my index is
around this much).
The Lucene 8.5.2 (8.5.0 as well) javadocs say:
public void setPreload(boolean preload)
Set to true to ask mapped pages to be loaded into physical memory on
init.
The behavior is best-effort and operating system dependent.
For example Lucene 4.0.0 does not have setPreLoad method.
https://urldefense.com/v3/__https://lucene.apache.org/core/4_0_0/core/org/apache/lucene/store/MMapDirectory.html__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOJ_Zf_dhQ$
Happy Holidays
Best regards
Ps. i know there is also BytesBuffersDirectory class for in memory
Lucene
but this requires creating Lucene Index on the fly.
This is great for only such kind of Lucene indexes that can be created
quickly on the fly.
Ekaterina has a nice article on this BytesBuffersDirectory class:
https://urldefense.com/v3/__https://medium.com/@ekaterinamihailova/in-memory-search-and-autocomplete-with-lucene-8-5-f2df1bc71c36__;!!GqivPVa7Brio!LEQH8Tyb_BBN_Kc3fEH2w-yhpvS-VwMrpuB0gctqchp3j7L7V6x9piciHOIosJjRzQ$
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
