Hi Thomas,
Answers will be inline.
On 04/04/2011 01:12 PM, Thomas Becker wrote:
HI Matthieu,
do you have any more log information from before that crash? It looks
like it happened during a young generation garbage collection. Your
eden space (where all new objects are placed in memory) was full and
there was not much space (probably not enough) left in the old
generation for objects which survived the young generation gc. The
interesting part of your crash dump indicating this, is:
These are the only logs related to Jetty that I am aware of. We parsed
/var/log but could not find anything relevant, besides our main
application log (the one that sends data to solr). More about this below.
Heap
PSYoungGen total 417088K, used 417056K [0x00007f04eb080000,
0x00007f050a3f0000, 0x00007f0515b20000)
eden space 416960K, 100% used
[0x00007f04eb080000,0x00007f05047b0000,0x00007f05047b0000)
from space 128K, 75% used
[0x00007f05055d0000,0x00007f05055e8000,0x00007f05055f0000)
to space 47232K, 0% used
[0x00007f05075d0000,0x00007f05075e8000,0x00007f050a3f0000)
PSOldGen total 529920K, used 377402K [0x00007f0495b20000,
0x00007f04b60a0000, 0x00007f04eb080000)
object space 529920K, 71% used
[0x00007f0495b20000,0x00007f04acbae9b0,0x00007f04b60a0000)
PSPermGen total 21504K, used 20440K [0x00007f048b320000,
0x00007f048c820000, 0x00007f0495b20000)
object space 21504K, 95% used
[0x00007f048b320000,0x00007f048c7163a8,0x00007f048c820000)
As solr/lucene usage can be quite memory intensive it's likely that
your app simply ran out of memory. Maybe during indexing. So your next
steps will be to find out what the application was doing before the
crash and think about what might have caused it to run oom. If you
can't find the root cause this way, you should try to get a heap dump
and analyze what was consuming memory.
That makes sense : reading our app logs shows that malformed data was
sent to our solr server, which never answered back with any
acknowledgement of indexation. Now I guess that said sent data builds up
with each iteration, apparently leading to this crash.
Try setting these JVM options
(http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html)
in your startup script:
-XX:HeapDumpPath=./java_pid<pid>.hprof Path to directory or filename
for heap dump. /Manageable/. (Introduced in 1.4.2 update 12, 5.0
update 7.) -XX:-HeapDumpOnOutOfMemoryError Dump heap to file when
java.lang.OutOfMemoryError is thrown. /Manageable/. (Introduced in
1.4.2 update 12, 5.0 update 7.)
-XX:-PrintGCDetails Print more details at garbage collection.
/Manageable/. (Introduced in 1.4.0.) -XX:-PrintGCTimeStamps Print
timestamps at garbage collection. /Manageable/ (Introduced in 1.4.0.)
Again it might be normal and that your application simply needs a
bigger heap. Your current settings are: "jvm_args: -Xms512M
-Xmx2048M". If you don't have more memory available you might need to
rethink your indexing strategy (if the oom happened during indexing).
But it might as well be that you've got a memory leak of some kind.
You'll have to read your log files and try to get a heap dump.
As indexing is memory intensive, proper tuning of your jvm memory
settings is crucial. Normally you want a big newsize and the overall
heap space as big as you have physical memory. But that heavily
depends on your indexing strategy.
We assign 50% of the available memory to solr, the rest is left for our
main application. This was a test appliance, with less memory than what
we expect to use in production.
Hope I did not confuse you too much. Please not that this is most
certainly NOT a jetty issue.
Cheers,
Thomas
You were very clear, thanks for your help ! One more question, but I
guess that'd be more of a Solr question : would it be okay to have Jetty
be restarted right after such a crash ? We can't have the indexation
service down for too long.
On 04/04/2011 12:20, Matthieu Huin wrote:
Greetings all,
I am currently using solr as the backend behind a log aggregation and
search system my team is developing. All was well and good until I
noticed a test server crashing quite unexpectedly. We'd like to dig more
into the incident but none of us has much experience with Jetty crash
logs - not to mention that our Java is very rusty.
The crash log is joined as an attachment.
Could anyone help us with understanding what went wrong there ?
Also, would it be possible and/or wise to automatically restart the
server in case of such a crash ?
Thanks for your help. If you need any extra info about that case, do not
hesitate to ask !
Matthieu Huin
( Resending this message as I used to wrong e-mail to send it, sorry
for the inconvenience)
_______________________________________________
jetty-users mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/jetty-users
_______________________________________________
jetty-users mailing list
[email protected]
https://dev.eclipse.org/mailman/listinfo/jetty-users
