[
https://issues.apache.org/jira/browse/LUCENE-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126867#comment-14126867
]
Uwe Schindler edited comment on LUCENE-5928 at 9/9/14 11:44 AM:
----------------------------------------------------------------
Hi,
this is not an issue of WildcardQuery. This is also not related to used heap
space. What you see differences in is in most cases a common misunderstanding
about those 2 terms:
- Virtual Memory (VIRT): This is allocated address space, *it is not allocated
memory*. On 64 bit platforms this is for free and is not limited by physical
memory (it is not even related to each other). If you use mmap, VIRT is
something like RES + up to 2 times the size of all open indexes. Internally the
whole index is seen like a swap file to the OS kernel.
- Resident Memory (RES): This is size of heap space + size of direct memory.
This is *allocated* memory, but may reside on swap, too. Please keep in mind,
that some operating systems also count memory, which was mmapped from file
system cache to the process there, because this is resident. You can see this
looking at SHR (share), which is memory shared with other processes (in that
case the kernel). For Lucene this RES memory is also not a problem, becaus ethe
file system cache is managed by the kernel and freed on request (SHR/RES goes
down then).
By executing a Wildcard like "*:*" you just access the whole term dictionary
and all positings lists, so they are accessed on disk and therefore loaded into
file system cache. When using MMap, the space in file system cache is also
shown in VIRT of the process, because the linux/windows kernel maps the file
system memory into the address space. But its does not "waste" memory.
For more information, see:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
was (Author: thetaphi):
Hi,
this is not an issue of WildcardQuery. This is also not related to used heap
space. What you see differences in is in most cases a common misunderstanding
about those 2 terms:
- Virtual Memory (VIRT): This is allocated address space, *it is not allocated
memory*. On 64 bit platforms this is for free and is not limited by physical
memory (it is not even related to each other). If you use mmap, VIRT is
something like RES + up to 2 times the size of all open indexes. Internally the
whole index is seen like a swap file to the OS kernel.
- Resident Memory (RES): This is size of heap space + size of direct memory.
This is *allocated* memory, but may reside on swap, too.
By executing a Wildcard like "*:*" you just access the whole term dictionary
and all positings lists, so they are accessed on disk and therefore loaded into
file system cache. When using MMap, the space in file system cache is also
shown in VIRT of the process, because the linux/windows kernel maps the file
system memory into the address space. But its does not "waste" memory.
For more information, see:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> WildcardQuery may has memory leak
> ---------------------------------
>
> Key: LUCENE-5928
> URL: https://issues.apache.org/jira/browse/LUCENE-5928
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 4.9
> Environment: SSD 1.5T, RAM 256G 10*10000
> Reporter: Littlestar
> Assignee: Uwe Schindler
>
> data 800G, records 15*10000*10000.
> one search thread.
> content:???
> content:*
> content:*1
> content:*2
> content:*3
> jvm heap=96G, but the jvm memusage over 130g?
> run more wildcard, use memory more....
> Does luence search/index use a lot of DirectMemory or Native Memory?
> I use -XX:MaxDirectMemorySize=4g, it does nothing better.
> Thanks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
