And here is what we do:
We start SEC to monitor *the public instance only* using this
shell script:
#!/bin/bash
#
# Start the monitor of the public instance
#
# We use SEC to monitor the log files.
#
# Author: Andreas Weder, NOSE Applied Intelligence AG
#
NOTIFY_ADDR="[EMAIL PROTECTED]"
echo "." | mail -A lognet -s "Notice: Web monitor started. You will be notified
on restart." $NOTIFY_ADDR
echo "Monitor starting. Notification email has been sent."
sec.pl -conf=/home/magnolia/etc/sec.publicwatch
-input=/home/magnolia/var/log/public.log -poll_timeout=5
echo "." | mail -A lognet -s "Notice: Web monitor stopped. This may notbe
intentional." $NOTIFY_ADDR
The configuration file for sec mentioned above is this one here:
#
# SEC Script for watching the public instance
#
# Author: Andreas Weder, NOSE Applied Intelligence AG
#
type=SingleWithSuppress
ptype=regexp
pattern=[Tt]oo\smany\sopen\sfiles
desc=too many open files detected
action=shellcmd /home/magnolia/bin/serverDownAction.sh '%t' '%s'
window=600
type=SingleWithSuppress
ptype=regexp
pattern=java.lang.OutOfMemoryError
desc=public instance has run out of memory
action=shellcmd /home/magnolia/bin/serverDownAction.sh '%t' '%s'
window=600
As you can see, in case of a detected event, we call a script
called serverDownAction.sh, which restarts the servers:
#!/bin/bash
#
# Called by SEC if the public instance is down
#
# Author: Andreas Weder, NOSE Applied Intelligence AG
#
TMPFILE=/tmp/notify_mail.txt
NOTIFY_ADDR="[EMAIL PROTECTED]"
export HOME=/home/magnolia
export CATALINA_HOME=$HOME/current/public
echo "." | mail -A lognet -s "IMPORTANT: Web problem: attempting to restart
public server" $NOTIFY_ADDR
echo "*****************************************" > $TMPFILE
echo " SEC monitor has issued a restart of the " >> $TMPFILE
echo " public instance " >> $TMPFILE
echo "*****************************************" >> $TMPFILE
echo >> $TMPFILE
echo " Time: $1" >> $TMPFILE
echo " Message: $2" >> $TMPFILE
echo >> $TMPFILE
echo "Your friendly Server Monitor" >> $TMPFILE
echo >> $TMPFILE
echo >> $TMPFILE
/home/magnolia/bin/restartServers.sh >> $TMPFILE
cat $TMPFILE | mail -A lognet -s "IMPORTANT: Web restarted" $NOTIFY_ADDR
The restart script restarts both author and public instance
(this is not needed as we've only detected that the public
instance failed, but what the heck):
#!/bin/bash
echo "*********************************";
echo " Restarting all servers ";
echo "*********************************";
echo
echo "Restarting at "`date`"...";
/home/magnolia/bin/stop
sleep 10
PID=`ps -ef | grep public | grep Bootstrap | gawk '{ print $2 }'`
if [ "$PID" ];
then
kill -TERM $PID;
fi
sleep 5
/home/magnolia/bin/start
I've built-in a kill command here since our module code inside
the public instance keeps the Java VM hanging around for a couple
of minutes and I don't want to wait for that.
Start and stop should be obvious.
Hope this helps
Andreas
> -----Original Message-----
> From: [email protected] [mailto:[EMAIL PROTECTED]
> Sent: Freitag, 1. Juli 2005 14:44
> To: [email protected]
> Subject: Re: [magnolia-user] Out Of Memory on public instance
>
>
>
> Andreas,
>
> what is SEC? Could you plz give a link to that?
>
>
>
> Andreas Weder wrote:
>
> >Dear John,
> >
> >we've been using Magnolia 2.03 on Tomcat 5.0 using Java 1.4.2_06
> >for a site with roughly 25000 visits per month (that's not hits
> >or pages served, it's more than that). We did get some out of
> >memory exceptions every now and then, but the leaks were in
> >our code.
> >
> >Are you sure you've increased the upper limit of the VM to
> >something reasonable? We're at least using 512 MB for each VM.
> >Values like 128 MB are just not good enough, especially not
> >if you're activating often. It's not so much a Magnolia thing,
> >I think.
> >
> >What we did get every now and then, however, were exceptions
> >due to "too many open files". Again, this may very well be
> >caused by our code sending newsletters, but may have other
> >reasons (Jackrabbit, maybe even Magnolia).
> >
> >What I've done is that I've used SEC to monitor log files for
> >this exception and also an out of memory (just in case, it
> >never occurred again since we fixed the problems in our code).
> >If SEC detects them, the affected instance is restarted.
> >
> >Works extremely reliable ever since. And as I said: the excep-
> >tions occur only very rarely.
> >
> >Andreas
> >_________________________________________________________________
> > NOSE applied intelligence ag
> > [www] http://www.nose.ch
> > [email] [EMAIL PROTECTED]
> > andreas weder [pgp id] 0x44117CA2
> > hardturmstrasse 171 [office] +41-1-277 57 11
> > 8005 zurich [direct] +41-1-277 57 24
> > switzerland [fax] +41-1-277 57 12
> >
> >
> >
> >>-----Original Message-----
> >>From: [email protected] [mailto:[EMAIL PROTECTED]
> >>Sent: Freitag, 1. Juli 2005 14:27
> >>To: [email protected]
> >>Subject: [magnolia-user] Out Of Memory on public instance
> >>
> >>
> >>We've recently launched the public version of a site using magnolia,
> >>however we are now receiving and OutOfMemory error, normally
> about a day
> >>after restarting tomcat. There were no problems with the
> installation of
> >>Java or magnolia. It seems it can't cope with the load of traffic? It's
> >>our first implementation with magnolia, so we're rather concerned now
> >>about using it for other projects.
> >>
> >>We are running magnolia 2.02 on Tomcat 5.0.28 using java 1.5.0_02, on
> >>Windows 2003 Server cluster. Extract from log file below.
> >>
> >>Can anyone help with this?
> >>
> >>TIA
> >>
> >>John
> >>
> >>--
> >>
> >>John Olival
> >>Managing Director
> >>Novasoft Explican
> >>
> >>
> >>sample output of magnoliaPublic_log.2005-07-01.log
> >>
> >>
> >>2005-07-01 13:40:32 ApplicationDispatcher[] El Servlet.service() para
> >>servlet jsp lanzó una excepción
> >>java.lang.OutOfMemoryError: unable to create new native thread
> >> at java.lang.Thread.start0(Native Method)
> >> at java.lang.Thread.start(Thread.java:574)
> >> at
> >>org.apache.slide.jcr.core.observation.ObservationManagerFactory.<i
> >>nit>(ObservationManagerFactory.java:92)
> >> at
> >>org.apache.slide.jcr.core.RepositoryImpl.getObservationManagerFact
> >>ory(RepositoryImpl.java:376)
> >> at
> >>org.apache.slide.jcr.core.WorkspaceImpl.getObservationManager(Work
> >>spaceImpl.java:865)
> >> at
> >>org.apache.slide.jcr.core.RepositoryImpl.<init>(RepositoryImpl.java:291)
> >> at
> >>org.apache.slide.jcr.core.RepositoryFactory.getRepository(Reposito
> >>ryFactory.java:281)
> >> at
> >>org.apache.jsp.templates.jsp.novasoft.global.navHoriz_jsp._jspServ
> >>ice(navHoriz_jsp.java:110)
> >> at
> >>org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> >> at
> >>org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrap
> >>per.java:324)
> >> at
> >>org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
> >> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A
> >>pplicationFilterChain.java:237)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati
> >>onFilterChain.java:157)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationD
> >>ispatcher.java:704)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.doInclude(Applicati
> >>onDispatcher.java:590)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.include(Application
> >>Dispatcher.java:510)
> >> at
> >>org.apache.taglibs.standard.tag.common.core.ImportSupport.acquireS
> >>tring(ImportSupport.java:314)
> >> at
> >>org.apache.taglibs.standard.tag.common.core.ImportSupport.doEndTag
> >>(ImportSupport.java:179)
> >> at
> >>org.apache.jsp.templates.jsp.novasoft.templateStdLayout.stdLayout_
> >>jsp._jspx_meth_c_import_2
> >>
> >>(stdLayout_jsp.java:234)
> >> at
> >>org.apache.jsp.templates.jsp.novasoft.templateStdLayout.stdLayout_
> >>jsp._jspService(stdLayout_jsp.java:80)
> >> at
> >>org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> >> at
> >>org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrap
> >>per.java:324)
> >> at
> >>org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292)
> >> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A
> >>pplicationFilterChain.java:237)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati
> >>onFilterChain.java:157)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationD
> >>ispatcher.java:704)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.processRequest(Appl
> >>icationDispatcher.java:474)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.doForward(Applicati
> >>onDispatcher.java:409)
> >> at
> >>org.apache.catalina.core.ApplicationDispatcher.forward(Application
> >>Dispatcher.java:312)
> >> at info.magnolia.cms.Dispatcher.dispatch(Dispatcher.java:65)
> >> at
> >>info.magnolia.cms.servlets.EntryServlet.doGet(EntryServlet.java:121)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A
> >>pplicationFilterChain.java:237)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati
> >>onFilterChain.java:157)
> >> at
> >>info.magnolia.cms.Filter.MultipartRequestFilter.doFilter(Multipart
> >>RequestFilter.java:76)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A
> >>pplicationFilterChain.java:186)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati
> >>onFilterChain.java:157)
> >> at
> >>info.magnolia.cms.Filter.ContentTypeFilter.doFilter(ContentTypeFil
> >>ter.java:52)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(A
> >>pplicationFilterChain.java:186)
> >> at
> >>org.apache.catalina.core.ApplicationFilterChain.doFilter(Applicati
> >>onFilterChain.java:157)
> >> at
> >>org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp
> >>erValve.java:214)
> >> at
> >>org.apache.catalina.core.StandardValveContext.invokeNext(StandardV
> >>alveContext.java:104)
> >> at
> >>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.
> >>java:520)
> >> at
> >>org.apache.catalina.core.StandardContextValve.invokeInternal(Stand
> >>ardContextValve.java:198)
> >> at
> >>org.apache.catalina.core.StandardContextValve.invoke(StandardConte
> >>xtValve.java:152)
> >> at
> >>org.apache.catalina.core.StandardValveContext.invokeNext(StandardV
> >>alveContext.java:104)
> >> at
> >>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.
> >>java:520)
> >> at
> >>org.apache.catalina.core.StandardHostValve.invoke(StandardHostValv
> >>e.java:137)
> >> at
> >>org.apache.catalina.core.StandardValveContext.invokeNext(StandardV
> >>alveContext.java:104)
> >> at
> >>org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValv
> >>e.java:118)
> >> at
> >>org.apache.catalina.core.StandardValveContext.invokeNext(StandardV
> >>alveContext.java:102)
> >> at
> >>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.
> >>java:520)
> >> at
> >>org.apache.catalina.core.StandardEngineValve.invoke(StandardEngine
> >>Valve.java:109)
> >> at
> >>org.apache.catalina.core.StandardValveContext.invokeNext(StandardV
> >>alveContext.java:104)
> >> at
> >>org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.
> >>java:520)
> >> at
> >>org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
> >> at
> >>org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
> >> at
> >>org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:300)
> >> at
> >>org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:374)
> >> at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:743)
> >> at
> >>org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket
> >>.java:675)
> >> at
> >>org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
> >> at
> >>org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(Thre
> >>adPool.java:683)
> >> at java.lang.Thread.run(Thread.java:595)
> >>
> >>
> >>
> >>----------------------------------------------------------------
> >>for list details see
> >>http://www.magnolia.info/en/magnolia/developer.html
> >>----------------------------------------------------------------
> >>
> >>
> >>
> >>
> >
> >
> >----------------------------------------------------------------
> >for list details see
> >http://www.magnolia.info/en/magnolia/developer.html
> >----------------------------------------------------------------
> >
> >
> >
> >
>
>
> --
> Regards,
> Alex
>
>
> ----------------------------------------------------------------
> for list details see
> http://www.magnolia.info/en/magnolia/developer.html
> ----------------------------------------------------------------
>
>
----------------------------------------------------------------
for list details see
http://www.magnolia.info/en/magnolia/developer.html
----------------------------------------------------------------
