Thanks Mike and Jokin for your comments on the memory problem. I have
submitted the query to the Hibernate Search list although I haven't
seen a response yet.
In the meantime I did my own investigating in the code (I'd rather
have avoided this!). I'm seeing results that don't make any sense and
maybe someone here with more experience of Lucene and the way memory
is allocated by the JVM may shed light on, what to me, are quite
illogical observations.
As you may recall I had a stand-alone Lucene search and a Hibernate
Search version. Looking in the HS code did not shed any light on the
issue. I took my stand-alone Lucene code and put it in a method and
replaced the search in the HS class (the constructor of
QueryHits.java) with the call to my method. Bear in mind this method
is the same code as posted in my earlier message - it sets up the
Lucene search from scratch (i.e.: no data structures created by HS
were used). So, effectively I was calling my stand-alone code after
any setup done by Hibernate and any memory it may have allocated
(which turned out to be a few Mb).
I get OOM! Printing the free memory at this point shows bags of
memory left. Indeed, the same free memory (+/- a few Mb) as the
stand-alone Lucene version!
I then instrumented the Lucene method where the OOM is occuring
(FSDirectory.readInternal()). I cannot understand the results I am
seeing. Below is a snippet of the output of each with the code around
FSDirectory line 598 as follows:
...
do {
long tot = Runtime.getRuntime().totalMemory();
long free =Runtime.getRuntime().freeMemory();
System.out.println("LUCENE: offset="+offset+" total="+total+"
len-total="+(len-total)+" free mem="+free+" used ="+(tot-free));
int i = file.read(b, offset+total, len-total);
...
The stand-alone version:
...
LUCENE: offset=0 total=0 len-total=401 free mem=918576864 used =330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=883 free mem=918576864 used =330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=1024 free mem=918576864 used
=330080544
LUCENE: offset=0 total=0 len-total=209000000 free mem=631122912
used =617534496
LUCENE: offset=209000000 total=0 len-total=20900000 free
mem=631122912 used =617534496
LUCENE: offset=229900000 total=0 len-total=20900000 free
mem=631122912 used =617534496
LUCENE: offset=250800000 total=0 len-total=20900000 free
mem=631122912 used =617534496
...
completes successfully!
The method called via Hibernate Search:
...
LUCENE: offset=0 total=0 len-total=401 free mem=924185480 used =334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=883 free mem=924185480 used =334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=1024 free mem=924185480 used
=334892152
LUCENE: offset=0 total=0 len-total=209000000 free mem=636731528
used =622346104
Exception in thread "main" java.lang.OutOfMemoryError
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(Unknown Source)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.readInternal(FSDirectory.java:599)
... fails with exception!
Note that the HS version has slightly more free memory because I ran
it with -Xms1210M as opposed to -Xms1200M for the stand-alone to
offset any memory used by HS when it starts up.
As you can see, these are identical for all practical purposes. So
what gives?
I'm stumped, so any suggestions appreciated.
Thanks.
Quoting Michael McCandless <[email protected]>:
Unfortunately, I'm not familiar with exactly what Hibernate search does
with the Lucene APIs.
It must be doing something beyond what your standalone Lucene test case does.
Maybe ask this question on the Hibernate list?
Mike
