Hi, George, now that you mention it, the only thing showing up in the logs
at the time of the memory error were lines about SOLR stats. I figured
that was just an indication of normal usage. On the off chance these lines
might help with diagnosis, I'm pasting a few below:
Jul 12, 2011 9:48:19 AM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/dspace/solr/statistics/data/index,segFN=segments_tl02,version=1
282198870198,generation=1380242,filenames=[_wvbt.nrm, _ghmn.frq,
_px24.tii, _wuy1.fdt, _wvbt.prx, _75l4.tis, _75l4.tii, _wu4u.fnm,
_wvbw.fdt, _wvbu.fnm, _wvbw.tii, _px24.fnm, _wvbw.fdx, _px24.tis,
_wvbw.frq, _wvbs.fnm, _wv9i.frq, _75
l4.fdx, _wv0t.frq, _wvbr.fnm, _wsrd.tis, _px24.prx, _px24.fdt, _75l4.fdt,
_75l4.prx, _px24.fdx, _wuy1.fdx, _wu4u.fdt, _wv5l.tis, _wsrd.frq,
_wu4u.fdx, _wvbv.fnm, _wsrd.tii, _wsrd.fdx, _wvby.fnm, _wvbu.tis,
_wvbv.tis, _wsrd.fdt, _wvby.tii, _wv3m.frq, _wvbu.nrm, _wv5l.fnm,
_wvbr.prx, _wvbx.fnm, _wsrd.prx, _wv9i.fdt,
_wvbu.prx, _wv9i.fdx, _wvbt.tis, _wvby.prx, _wtdv.prx, _wuy1.prx,
_wuy1.fnm, _wvbt.fnm, _wu4u.tii, _75l4.frq, _wu4u.tis, _wvbs.tis,
_wtdv.tis, _wv9i.tis, _wvby.nrm, segments_tl02, _ghmn.fdx, _wuy1.tis,
_ghmn.fdt, _wvby.fdt, _wv9i.tii, _wvbv.fdx, _wv5l.fdx, _wvbs.tii,
_wtdv.tii, _wv3m.prx, _wvbv.fdt, _wvbw.nrm, _ghm
n.fnm, _wvbt.tii, _wvbv.prx, _px24.frq, _wvbt.fdt, _wtdv.fnm, _wvbv.nrm,
_wvbt.fdx, _wvbr.tii, _wv3m.tis, _wvbu.frq, _wv0t.fdx, _wv5l.prx,
_wvbr.tis, _wu4u.frq, _wv0t.fnm, _wv9i.fnm, _wv0t.fdt, _wvbv.tii,
_wvby.frq, _wsrd.fnm, _wvbw.prx, _wv0t.prx, _wv5l.fdt, _wvby.fdx,
_wvbv.frq, _wvbs.frq, _wvbr.frq, _wvby.tis, _
wvbw.fnm, _wvbu.tii, _wv3m.fnm, _wvbs.fdx, _wv5l.frq, _wv0t.tii,
_wvbr.fdx, _wvbs.fdt, _wvbr.fdt, _wv9i.prx, _ghmn.tii, _wv3m.fdx,
_wvbu.fdx, _wtdv.fdx, _wv3m.tii, _wvbs.nrm, _wv3m.fdt, _wvbu.fdt,
_ghmn.tis, _wvbx.frq, _wtdv.frq, _wvbx.fdx, _wv0t.tis, _wu4u.prx,
_wtdv.fdt, _wvbx.tis, _wvbx.fdt, _wuy1.frq, _ghmn.prx
, _wvbs.prx, _wvbt.frq, _75l4.fnm, _wvbw.tis, _wvbx.nrm, _wvbx.prx,
_wuy1.tii, _wv5l.tii, _wvbx.tii]
Jul 12, 2011 9:48:19 AM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1282198870198
Jul 12, 2011 9:48:19 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 6
Jul 12, 2011 9:48:19 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=6
Jul 12, 2011 9:49:43 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:49:43 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:49:44 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 0
Jul 12, 2011 9:49:44 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=0
Jul 12, 2011 9:51:01 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:01 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:06 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:06 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:11 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:11 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:15 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:15 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:19 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:19 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:33 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:33 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:51:49 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:51:49 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:52:05 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:52:05 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:52:10 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:52:10 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:52:31 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:52:31 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
Jul 12, 2011 9:52:47 AM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[null]} 0 1
Jul 12, 2011 9:52:47 AM org.apache.solr.core.SolrCore execute
INFO: [statistics] webapp=/solr path=/update params={wt=javabin&version=1}
status=0 QTime=1
#
# There is insufficient memory for the Java Runtime Environment to
continue.
# Native memory allocation (malloc) failed to allocate 32756 bytes for
ChunkPool::allocate
# An error report file with more information is saved as:
# /home/dspace/hs_err_pid7188.log
--
HARDY POTTINGER <[email protected]>
University of Missouri Library Systems
http://lso.umsystem.edu/~pottingerhj/
"No matter how far down the wrong road you've gone,
turn back." --Turkish proverb
On 7/12/11 2:11 PM, "George S Kozak" <[email protected]> wrote:
>Hardy:
>
>I was having a similar problem with my Tomcat until I upgraded to Tomcat
>7.0.14. My theory (of which I have no hard facts...so take with a grain
>of salt) is that the Solr stats use up a lot memory. I found that a lot
>of my memory problems (and other problems) went away when I upgraded
>Tomcat.
>
>George Kozak
>Digital Library Specialist
>Cornell University Library Information Technologies (CUL-IT)
>501 Olin Library
>Cornell University
>Ithaca, NY 14853
>607-255-8924
>
>
>-----Original Message-----
>From: Pottinger, Hardy J. [mailto:[email protected]]
>Sent: Tuesday, July 12, 2011 12:28 PM
>To: [email protected]
>Subject: [Dspace-tech] seeking help in tracking down why Tomcat is dying
>on us
>
>Hi, I thought I'd ask the community for help debugging this one. For the
>past week, we've been having troubles with Tomcat dying on us. The error
>message in the Tomcat logs indicates that the JVM has run out of memory
>(snippets of log files and config files below). The system has 8GB of
>RAM, and here's what we have configured for Tomcat memory settings:
>
>### > tomcat memory settings, from /etc/tomcat5/tomct5.conf
>
># recommended settings for a production DSpace environment # from Mark
>Wood ([email protected]) JAVA_OPTS="-Xmx1024M -Xms768M"
>JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=128M"
>JAVA_OPTS="$JAVA_OPTS -XX:PermSize=32M"
>
># tweak from Bill Anderson at GAtech: use the parallel garbage collector
>JAVA_OPTS="$JAVA_OPTS -XX:+UseParallelGC"
>
>
>
>### > Here is the error message we're seeing in the tomcat logs:
>
># There is insufficient memory for the Java Runtime Environment to
>continue.
># Native memory allocation (malloc) failed to allocate 32756 bytes for
>ChunkPool::allocate # An error report file with more information is saved
>as:
># /home/dspace/hs_err_pid7188.log
>
>
>### > contents of /home/dspace/hs_err_pid7188.log
>
># There is insufficient memory for the Java Runtime Environment to
>continue.
># Native memory allocation (malloc) failed to allocate 32756 bytes for
>ChunkPool::allocate # Possible reasons:
># The system is out of physical RAM or swap space
># In 32 bit mode, the process size limit was hit
># Possible solutions:
># Reduce memory load on the system
># Increase physical memory or swap space
># Check if swap backing store is full
># Use 64 bit Java on a 64 bit OS
># Decrease Java heap size (-Xmx/-Xms)
># Decrease number of Java threads
># Decrease Java thread stack sizes (-Xss)
># Set larger code cache with -XX:ReservedCodeCacheSize=
># This output file may be truncated or incomplete.
>#
># Out of Memory Error (allocation.cpp:211), pid=7188, tid=1800473488 # #
>JRE version: 6.0_26-b03 # Java VM: Java HotSpot(TM) Server VM (20.1-b02
>mixed mode linux-x86 )
>
>--------------- T H R E A D ---------------
>
>Current thread (0x6ba00800): JavaThread "C2 CompilerThread0" daemon
>[_thread_in_native, id=7204, stack(0x6b490000,0x6b511000)]
>
>Stack: [0x6b490000,0x6b511000], sp=0x6b50de80, free space=503k Native
>frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
>code)
>V [libjvm.so+0x7248b0]
>
>### > there are two similar hs_err.log files, reporting similar error
>conditions (insufficient memory), dated 07/05/2011 and 07/06/2011
>
>
>I can post snippets from other log files, but figured I'd see if anyone
>had any advice of where to look/what to look for?
>
>
>--
>HARDY POTTINGER <[email protected]> University of Missouri Library
>Systems http://lso.umsystem.edu/~pottingerhj/
>"No matter how far down the wrong road you've gone, turn back." --Turkish
>proverb
>
>
>
>
>
>--------------------------------------------------------------------------
>----
>AppSumo Presents a FREE Video for the SourceForge Community by Eric Ries,
>the creator of the Lean Startup Methodology on "Lean Startup Secrets
>Revealed." This video shows you how to validate your ideas, optimize your
>ideas and identify your business strategy.
>http://p.sf.net/sfu/appsumosfdev2dev
>_______________________________________________
>DSpace-tech mailing list
>[email protected]
>https://lists.sourceforge.net/lists/listinfo/dspace-tech
------------------------------------------------------------------------------
AppSumo Presents a FREE Video for the SourceForge Community by Eric
Ries, the creator of the Lean Startup Methodology on "Lean Startup
Secrets Revealed." This video shows you how to validate your ideas,
optimize your ideas and identify your business strategy.
http://p.sf.net/sfu/appsumosfdev2dev
_______________________________________________
DSpace-tech mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/dspace-tech
