RE: JVM Crash in Lucene
I resolved this issue for the time-being by adding following parameter
to the command:
-XX:CompileCommand=exclude,org/apache/lucene/index/IndexReader$1,doBody
/Daniel
-Original Message-
From: Daniel Pfeifer [mailto:[EMAIL PROTECTED]
Sent: den 8 februari 2006 08:05
To: java-user@lucene.apache.org
Subject: Re: JVM Crash in Lucene
Got the same problem. Running 1.5.0_05 on Solaris 10. I've seen that
this issue has been reported on Sun's forum but no answer yet.
Another interesting thing which I noticed. We previously used the
RAMDirectory and we never got JVM-crashes when using RAMDirectory.
However, once we started using FSDirectory the JVM started to crash.
I tested adding -Xcomp parameter and the JVM has not crashed yet. But
then again, the SearchService hasn't been up long enough to be sure that
it solved the problem.
/Daniel
You also might try -Xbatch or -Xcomp to see if that fixes it (or
reproduces it faster).
Here's a great list of JVM options:
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
-Yonik
On 12/11/05, Yonik Seeley [EMAIL PROTECTED] wrote:
Sounds like it's a hotspot bug.
AFAIK, hotspot doesn't just compile a method once... it can do
optimization over time.
To work around it, have you tried pre previous version: 1.5_05?
It's possible it's a fairly new bug. We've been running with that
version and Lucene 1.4.3 without problems (on Opteron, RHEL4).
You could also try the latest Lucene 1.9 to see if that changes
enough
to avoid the bug.
-Yonik
On 12/11/05, Dan Gould [EMAIL PROTECTED] wrote:
First, thank you Chris, Yonik, and Dan for your ideas as to what
might be
causing this problem.
I tried moving things around so that the IndexReader is still open
when it
calls
TermFreqVector.getTerms()/TermFreqVector.getTermFrequencies(). It
didn't seem to make any difference.
I also tried running Java with the flags:
-Xmx2048m -XX:MaxPermSize=200m
(the box has 4GB of RAM) and it still crashes. It's hard to tell,
but the
program does seem to run for a lot longer (maybe 10 hours), but
that could
just be randomness in my tests.
The JVM always seems to crash with
Current CompileTask:
opto:1836
org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object;
(99 bytes)
which in the Lucene source is:
private static IndexReader open(final Directory directory,
final boolean
closeDirectory) throws IOException {
synchronized (directory) { // in-
inter-process
sync
return (IndexReader)new Lock.With(
directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
IndexWriter.COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
if (infos.size() == 1) { // index is
optimized
return SegmentReader.get(infos, infos.info(0),
closeDirectory);
}
IndexReader[] readers = new
IndexReader[infos.size()];
for (int i = 0; i infos.size(); i++)
readers[i] = SegmentReader.get(infos.info(i));
return new MultiReader(directory, infos,
closeDirectory,
readers);
}
}.run();
}
}
that's definitely a non-trivial bit of code, but I can't imagine
that
there's a problem that I'm seeing that no one else else.
Moreover, that
code gets run hundreds or even thousands of times before it
crashes, so I
don't image it's being HotSpot-compiled for the first time.
I'm running the 1.4.3 release and the 1.5.0_06-b05 JVM on Centos
Linux on
an Opteron.
Any further guesses?
Thank you all very much,
Dan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: JVM Crash in Lucene
Got the same problem. Running 1.5.0_05 on Solaris 10. I've seen that
this issue has been reported on Sun's forum but no answer yet.
Another interesting thing which I noticed. We previously used the
RAMDirectory and we never got JVM-crashes when using RAMDirectory.
However, once we started using FSDirectory the JVM started to crash.
I tested adding -Xcomp parameter and the JVM has not crashed yet. But
then again, the SearchService hasn't been up long enough to be sure that
it solved the problem.
/Daniel
You also might try -Xbatch or -Xcomp to see if that fixes it (or
reproduces it faster).
Here's a great list of JVM options:
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
-Yonik
On 12/11/05, Yonik Seeley [EMAIL PROTECTED] wrote:
Sounds like it's a hotspot bug.
AFAIK, hotspot doesn't just compile a method once... it can do
optimization over time.
To work around it, have you tried pre previous version: 1.5_05?
It's possible it's a fairly new bug. We've been running with that
version and Lucene 1.4.3 without problems (on Opteron, RHEL4).
You could also try the latest Lucene 1.9 to see if that changes
enough
to avoid the bug.
-Yonik
On 12/11/05, Dan Gould [EMAIL PROTECTED] wrote:
First, thank you Chris, Yonik, and Dan for your ideas as to what
might be
causing this problem.
I tried moving things around so that the IndexReader is still open
when it
calls
TermFreqVector.getTerms()/TermFreqVector.getTermFrequencies(). It
didn't seem to make any difference.
I also tried running Java with the flags:
-Xmx2048m -XX:MaxPermSize=200m
(the box has 4GB of RAM) and it still crashes. It's hard to tell,
but the
program does seem to run for a lot longer (maybe 10 hours), but
that could
just be randomness in my tests.
The JVM always seems to crash with
Current CompileTask:
opto:1836
org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object;
(99 bytes)
which in the Lucene source is:
private static IndexReader open(final Directory directory,
final boolean
closeDirectory) throws IOException {
synchronized (directory) { // in-
inter-process
sync
return (IndexReader)new Lock.With(
directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
IndexWriter.COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
if (infos.size() == 1) { // index is
optimized
return SegmentReader.get(infos, infos.info(0),
closeDirectory);
}
IndexReader[] readers = new
IndexReader[infos.size()];
for (int i = 0; i infos.size(); i++)
readers[i] = SegmentReader.get(infos.info(i));
return new MultiReader(directory, infos,
closeDirectory,
readers);
}
}.run();
}
}
that's definitely a non-trivial bit of code, but I can't imagine
that
there's a problem that I'm seeing that no one else else.
Moreover, that
code gets run hundreds or even thousands of times before it
crashes, so I
don't image it's being HotSpot-compiled for the first time.
I'm running the 1.4.3 release and the 1.5.0_06-b05 JVM on Centos
Linux on
an Opteron.
Any further guesses?
Thank you all very much,
Dan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: JVM Crash in Lucene
Sounds like it's a hotspot bug.
AFAIK, hotspot doesn't just compile a method once... it can do
optimization over time.
To work around it, have you tried pre previous version: 1.5_05?
It's possible it's a fairly new bug. We've been running with that
version and Lucene 1.4.3 without problems (on Opteron, RHEL4).
You could also try the latest Lucene 1.9 to see if that changes enough
to avoid the bug.
-Yonik
On 12/11/05, Dan Gould [EMAIL PROTECTED] wrote:
First, thank you Chris, Yonik, and Dan for your ideas as to what might be
causing this problem.
I tried moving things around so that the IndexReader is still open when it
calls TermFreqVector.getTerms()/TermFreqVector.getTermFrequencies(). It
didn't seem to make any difference.
I also tried running Java with the flags:
-Xmx2048m -XX:MaxPermSize=200m
(the box has 4GB of RAM) and it still crashes. It's hard to tell, but the
program does seem to run for a lot longer (maybe 10 hours), but that could
just be randomness in my tests.
The JVM always seems to crash with
Current CompileTask:
opto:1836
org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object;
(99 bytes)
which in the Lucene source is:
private static IndexReader open(final Directory directory, final boolean
closeDirectory) throws IOException {
synchronized (directory) { // in- inter-process
sync
return (IndexReader)new Lock.With(
directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
IndexWriter.COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
if (infos.size() == 1) { // index is optimized
return SegmentReader.get(infos, infos.info(0),
closeDirectory);
}
IndexReader[] readers = new IndexReader[infos.size()];
for (int i = 0; i infos.size(); i++)
readers[i] = SegmentReader.get(infos.info(i));
return new MultiReader(directory, infos, closeDirectory,
readers);
}
}.run();
}
}
that's definitely a non-trivial bit of code, but I can't imagine that
there's a problem that I'm seeing that no one else else. Moreover, that
code gets run hundreds or even thousands of times before it crashes, so I
don't image it's being HotSpot-compiled for the first time.
I'm running the 1.4.3 release and the 1.5.0_06-b05 JVM on Centos Linux on
an Opteron.
Any further guesses?
Thank you all very much,
Dan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: JVM Crash in Lucene
You also might try -Xbatch or -Xcomp to see if that fixes it (or
reproduces it faster).
Here's a great list of JVM options:
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
-Yonik
On 12/11/05, Yonik Seeley [EMAIL PROTECTED] wrote:
Sounds like it's a hotspot bug.
AFAIK, hotspot doesn't just compile a method once... it can do
optimization over time.
To work around it, have you tried pre previous version: 1.5_05?
It's possible it's a fairly new bug. We've been running with that
version and Lucene 1.4.3 without problems (on Opteron, RHEL4).
You could also try the latest Lucene 1.9 to see if that changes enough
to avoid the bug.
-Yonik
On 12/11/05, Dan Gould [EMAIL PROTECTED] wrote:
First, thank you Chris, Yonik, and Dan for your ideas as to what might be
causing this problem.
I tried moving things around so that the IndexReader is still open when it
calls TermFreqVector.getTerms()/TermFreqVector.getTermFrequencies(). It
didn't seem to make any difference.
I also tried running Java with the flags:
-Xmx2048m -XX:MaxPermSize=200m
(the box has 4GB of RAM) and it still crashes. It's hard to tell, but the
program does seem to run for a lot longer (maybe 10 hours), but that could
just be randomness in my tests.
The JVM always seems to crash with
Current CompileTask:
opto:1836
org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object;
(99 bytes)
which in the Lucene source is:
private static IndexReader open(final Directory directory, final boolean
closeDirectory) throws IOException {
synchronized (directory) { // in- inter-process
sync
return (IndexReader)new Lock.With(
directory.makeLock(IndexWriter.COMMIT_LOCK_NAME),
IndexWriter.COMMIT_LOCK_TIMEOUT) {
public Object doBody() throws IOException {
SegmentInfos infos = new SegmentInfos();
infos.read(directory);
if (infos.size() == 1) { // index is optimized
return SegmentReader.get(infos, infos.info(0),
closeDirectory);
}
IndexReader[] readers = new IndexReader[infos.size()];
for (int i = 0; i infos.size(); i++)
readers[i] = SegmentReader.get(infos.info(i));
return new MultiReader(directory, infos, closeDirectory,
readers);
}
}.run();
}
}
that's definitely a non-trivial bit of code, but I can't imagine that
there's a problem that I'm seeing that no one else else. Moreover, that
code gets run hundreds or even thousands of times before it crashes, so I
don't image it's being HotSpot-compiled for the first time.
I'm running the 1.4.3 release and the 1.5.0_06-b05 JVM on Centos Linux on
an Opteron.
Any further guesses?
Thank you all very much,
Dan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
RE: JVM Crash in Lucene
MaxPermSize is a not-always-documented jvm flag similar to Xmx. You set it like
this...
java -Xmx200M -XX:MaxPermSize=100m Myclass
It sets the maximum amount of space that will be available to the jvm for
'permanent' things (classes, etc.) and defaults to just 64MB. This can often
run out with code that generates lots of classes as it goes along (e.g. all
those proxies in jboss, something similar in big xdoclet tasks).
One important thing is how it relates to Xmx - whereas garbage collector limits
(for the young and old parts of ram) are within the overall limit set by Xmx,
Max Perm Size is separate from Xmx. This means you could chuck gigabytes of ram
at your Xmx and still get something a bit like an out of memory error by using
up the default 64mb limit for class definitions.
Here's some web pages that refer with more detail
http://www.tagtraum.com/gcviewer-vmflags.html
http://java.sun.com/docs/hotspot/gc/
http://www.unixville.com/~moazam/stories/2004/05/17/maxpermsizeAndHowItRelatesToTheOverallHeap.html
hth
Dan
-Original Message-
From: Dan Gould [mailto:[EMAIL PROTECTED]
Sent: 09 December 2005 01:49
To: java-user@lucene.apache.org
Subject: JVM Crash in Lucene
Hi--
I'm relatively new to Lucene. When I run my app, I get a JVM error.
This gets called a lot, but only fails every once in awhile (maybe 1 in
100 calls?)
I filed a report with Sun, but I don't expect to hear anything from them.
So, I was wondering if any Lucene experts have run across this or have any
ideas for workarounds (if not, I'll have to take lucene out of my system
and manage the termvectors myself):
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# SIGSEGV (0xb) at pc=0x002a95a5c300, pid=26545, tid=1085655392
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode)
# Problematic frame:
# V [libjvm.so+0x4fa300]
#
# An error report file with more information is saved as
hs_err_pid26545.log
That error file is huge, but the salient part seems to be:
Stack: [0x40a5c000,0x40b5d000), sp=0x40b59f70,
free sp
ace=1015k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0x4fa300]
V [libjvm.so+0x4fa3c2]
V [libjvm.so+0x4fa876]
V [libjvm.so+0x58051e]
V [libjvm.so+0x58017c]
V [libjvm.so+0x2b968b]
V [libjvm.so+0x2b67ba]
V [libjvm.so+0x25f3c7]
V [libjvm.so+0x2bed06]
V [libjvm.so+0x2be6b2]
V [libjvm.so+0x6163f9]
V [libjvm.so+0x610ffe]
V [libjvm.so+0x554745]
Current CompileTask:
opto:1621
org.apache.lucene.index.IndexReader$1.doBody()Ljava/lang/Object;
(99 bytes)
My code that is causing the JVM crash is:
//Get the terms and frequencies
IndexReader indexreader = null;
TermFreqVector tv = null;
try {
indexreader = indexManager.getIndexReader();
tv = indexManager.getTermVector(indexreader, item);
if(null==tv) {
if(log.isErrorEnabled()) log.error(No term vector for item
+ item.getId());
//??? better handling
return;
}
}
finally {
//XXX should close earlier
if(null!=indexreader) {
try {
indexreader.close();
}
catch(IOException ex) {
if(log.isDebugEnabled()) log.debug(IOException cleaning
up indexreader , ex);
}
}
}
//Replace the String[] with its equivalent long[]... (and tally up
total occurs)
String[] termstrings = tv.getTerms();
int[] freqs = tv.getTermFrequencies();
Thank you for your help,
Dan
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: JVM Crash in Lucene
: I'm relatively new to Lucene. When I run my app, I get a JVM error.
: This gets called a lot, but only fails every once in awhile (maybe 1 in
: 100 calls?)
i'm not that familiar with TermFreqVectors, and I have no idea what
indexManager is, but I'm suprised this works at all ... I thought calling
the getters on a TermFreqVector required it to read from the underlying
files (which would require that the reader still be open).
Is it possible that that's your problem? .. is there perhaps some strange
GC interaction issue that results in the data TermFreqVector is looking
for to go bye-bye because you closed the reader?
(it sounds silly when I read it back to myself ... but it's the best guess
i've got just reading your code)
: My code that is causing the JVM crash is:
:
: //Get the terms and frequencies
: IndexReader indexreader = null;
: TermFreqVector tv = null;
: try {
: indexreader = indexManager.getIndexReader();
: tv = indexManager.getTermVector(indexreader, item);
: if(null==tv) {
: if(log.isErrorEnabled()) log.error(No term vector for item
+ item.getId());
: //??? better handling
: return;
: }
: }
: finally {
: //XXX should close earlier
: if(null!=indexreader) {
: try {
: indexreader.close();
: }
: catch(IOException ex) {
: if(log.isDebugEnabled()) log.debug(IOException cleaning
up indexreader , ex);
: }
: }
: }
:
: //Replace the String[] with its equivalent long[]... (and tally up
total occurs)
: String[] termstrings = tv.getTerms();
: int[] freqs = tv.getTermFrequencies();
-Hoss
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: JVM Crash in Lucene
The only problems I've had with 1.5 JVM crashes and Lucene was related to stack overflow... try increasing the stack size and see of anything different happens. My crashes happened while trying to use Luke to open a 4GB index with thousands of indexed fields. -Yonik - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
