Re: [Resin-interest] Watchdog loglevel

[Jens Dueholm Christensen]

From: resin-interest-boun...@caucho.com 
[mailto:resin-interest-boun...@caucho.com] On Behalf Of Scott Ferguson

 > Well, remember that the watchdog itself doesn't normally shutdown Resin on 
 > errors. Resin exits itself and the watchdog just starts a new instance. 
 > (Resin 4.0 communicates the reason better to the watchdog through exit 
 > codes.)

 > So the problem is in the Resin instance itself. 

 > Are there hs_err* files or something similar?

No, not from this particular incident, but we have an older hr_err-file and an 
old watchdog-log that does show a restart around the same time as the hs_err 
file was created.

*Eeeek* (the same sound a small animal that's about to get squished makes!) - 
Seems the JVM crashed at that time due to:

# Problematic frame:
# C  [libawt.so+0x6f881]  IntRgbSrcMaskFill+0x1b1

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  
sun.java2d.loops.MaskFill.MaskFill(Lsun/java2d/SunGraphics2D;Lsun/java2d/SurfaceData;Ljava/awt/Composite;[BII)V
J  
sun.java2d.pipe.TextRenderer.drawGlyphList(Lsun/java2d/SunGraphics2D;Lsun/font/GlyphList;)V
J  
sun.java2d.pipe.GlyphListPipe.drawString(Lsun/java2d/SunGraphics2D;Ljava/lang/String;DD)V
J  sun.java2d.SunGraphics2D.drawString(Ljava/lang/String;FF)V
J  
org.jfree.text.TextLine.draw(Ljava/awt/Graphics2D;FFLorg/jfree/ui/TextAnchor;FFD)V

This app creates thousands upon thousands of PDF reports with jFreeChart every 
day..

We'll have to dig into this - thanks Scott.

Regards,
Jens Dueholm Christensen 
Survey IT

___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Re: [Resin-interest] Watchdog loglevel

[Scott Ferguson]

On 2/12/13 1:50 AM, Jens Dueholm Christensen wrote:


Hi

We're running Resin 3.1.11 (soon to be 3.1.12 in the next 
servicewindow) in our production environment, and a few days ago we 
had an app that was restarted several times by the watchdog -- with no 
apparent reason.


The watchdog-log contains this:

[2013/02/08 11:17:46.478] WatchdogProcess[Watchdog[results],1] 
stopping Resin


[2013/02/08 11:17:46.478] WatchdogProcess[Watchdog[results],2] 
starting Resin


[2013/02/08 11:20:37.256] WatchdogProcess[Watchdog[results],2] 
stopping Resin


[2013/02/08 11:20:37.256] WatchdogProcess[Watchdog[results],3] 
starting Resin


[2013/02/08 11:23:24.221] WatchdogProcess[Watchdog[results],3] 
stopping Resin


[2013/02/08 11:23:24.221] WatchdogProcess[Watchdog[results],4] 
starting Resin


There was a lot of regular and normal activity in our apps stdout-log 
before and inbetween all the restarts, but nothing that -- in our 
opinion -- should cause a restart by the watchdog. The JVM log has no 
mention of problems (performing CMS and young generation GC as 
expected) and load on the server was also low -- no automatic 
stacktraces were taken.




Well, remember that the watchdog itself doesn't normally shutdown Resin 
on errors. Resin exits itself and the watchdog just starts a new 
instance. (Resin 4.0 communicates the reason better to the watchdog 
through exit codes.)


So the problem is in the Resin instance itself.

Are there hs_err* files or something similar?

-- Scott


We have been running with the same resin configuration, app codebase 
and OS software-stack for a long time, so we are quite baffeled, as 
this struck us as lightning from a clear sky.


Is there any way of getting more verbose output about the watchdog and 
what it decides to do?


We tried setting  and 
restarting Resin completely (not just a restart of the JVM), but that 
did not seem to help.


Regards,

*Jens Dueholm Christensen
*Survey IT



___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest