Hi Simon,
i checked my log files one more time to get the error timestamps.
I get the first Error at 14:37:
06.07.2012 14:37:52 org.apache.solr.common.SolrException log
SCHWERWIEGEND: null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:358)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:323)
Next one, and the first Java heap Space error at 17:35:
06.07.2012 17:35:36 org.apache.solr.common.SolrException log
SCHWERWIEGEND: null:java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:248)
at
org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:269)
at
org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
at
org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:307)
at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:330)
Commit failure a couple of seconds later:
06.07.2012 17:35:38 org.apache.solr.common.SolrException log
SCHWERWIEGEND: auto commit error...:java.lang.IllegalStateException:
this writer hit an OutOfMemoryError; cannot commit
at
org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2650)
follow by 10 Java heap space Exceptions, and one minute later at 17:36
the first auto-warming Exception:
06.07.2012 17:36:26 org.apache.solr.common.SolrException log
SCHWERWIEGEND: Error during auto-warming of key:pubDate:[1340971496000
TO 1341576296000]:java.lang.OutOfMemoryError: Java heap space
06.07.2012 17:36:28 org.apache.solr.common.SolrException log
SCHWERWIEGEND: Error during auto-warming of key:pubDate:[1340971495000
TO 1341576295000]:java.lang.OutOfMemoryError: Java heap space
> it really seems that you are hitting an OOM during auto warming. can
> this be the case for your failure.
> Can you raise the JVM memory and see if you still hit the spike and go
> OOM? this is very unlikely a IndexWriter problem. I'd rather look at
> your warmup queries ie. fieldcache, FieldValueCache usage. Are you
> sorting / facet on anything?
Auto warming problems began one minute after the "java
heap"-exceptions, so i think this are subsequent problems.
I configured very small caches(max. sizes between 512 and 2048) for my use case.
Warming queries looks like this, with sorting, but without facetting:
<lst>
<str name="q">ag</str>
<str name="fq">pubDate:[NOW-1DAY TO *]</str>
<str name="sort">pubDate desc</str>
</lst>
Du you think that 8GB for JVM are not enough? To raise the JVM memory
can solve the problem..
As mentioned, this server runs a long time with the same config
without problems, i am surprised that this problem was there at one
time
without heavy usage...now it's running smoothly again after restart
yesterday, so i don't know whet the problem appears again.
I try to update to 4.0 alpha today and run it with tomcat and report:)
Best regards
Vadim
2012/7/10 Simon Willnauer <simon.willna...@gmail.com>:
> it really seems that you are hitting an OOM during auto warming. can
> this be the case for your failure.
> Can you raise the JVM memory and see if you still hit the spike and go
> OOM? this is very unlikely a IndexWriter problem. I'd rather look at
> your warmup queries ie. fieldcache, FieldValueCache usage. Are you
> sorting / facet on anything?
>
> simon
>
> On Tue, Jul 10, 2012 at 4:49 PM, Vadim Kisselmann
> <v.kisselm...@gmail.com> wrote:
>> Hi Robert,
>>
>>> Can you run Lucene's checkIndex tool on your index?
>>
>> No, unfortunately not. This Solr should run without stoppage, an
>> tomcat-restart is ok, but not more:)
>> I tested newer trunk-versions a couple of months ago, but they fail
>> all with tomcat.
>> i would test 4.0-alpha in next days with tomcat and open an jira-issue
>> if it doesn't work with it.
>>
>>> do you have another exception in your logs? To my knowledge, in all
>>> cases that IndexWriter throws an OutOfMemoryError, the original
>>> OutOfMemoryError is also rethrown (not just this IllegalStateException
>>> noting that at some point, it hit OOM.
>>
>> Hmm, i checked older logs and found something new, what i have not
>> seen in VisualVM. "Java heap space"-Problems, just before OOM.
>> My JVM has 8GB -Xmx/-Xms, 16GB for OS, nothing else on this machine.
>> This Errors pop up's during normal run according logs, no optimizes,
>> high loads(max. 30 queries per minute) or something special at this time.
>>
>> SCHWERWIEGEND: null:ClientAbortException: java.net.SocketException: Broken
>> pipe
>> SCHWERWIEGEND: null:java.lang.OutOfMemoryError: Java heap space
>> SCHWERWIEGEND: auto commit error...:java.lang.IllegalStateException:
>> this writer hit an OutOfMemoryError; cannot commit
>> SCHWERWIEGEND: Error during auto-warming of
>> key:org.apache.solr.search.QueryResultKey@7cba935e:java.lang.OutOfMemoryError:
>> Java heap space
>> SCHWERWIEGEND: org.apache.solr.common.SolrException: Internal Server Error
>> SCHWERWIEGEND: null:org.apache.solr.common.SolrException: Internal Server
>> Error
>>
>> I knew this failures when i work on virtual machines with solr 1.4,
>> big indexes and ridiculous small -Xmx sizes.
>> But on real hardware, with enough RAM, fast disks/cpu's it's new for me:)
>>
>> Best regards
>> Vadim
