Not sure if this might help someone, but we ran into a problem last week
that was driving us crazy, and I wanted to share the solution with you in
hopes that it might help someone else.
We're running JRun 2.3.3 on Red Hat Linux 6.1, and our site had been running
for about five months when suddenly one of the web servers in our server
farm began experiencing a strange problem. It suddenly could no longer open
any files on the server. If we restarted JRun, it would work again for a
little while (about ten minutes or so, depending on usage), and then start
all over again. We took that server out of the farm and tried several
things to fix the problem, to no avail.
Then, about two days later, another web server started doing the same thing.
This had become a crisis, so we were doing everything we could to try to fix
the problem. We called Allaire, and they responded in about an hour and a
half (not too bad, given the 24 hour window we were quoted), but they really
had no recommendation other than upgrading JRun and/or the JVM we were
using. These solutions did not work, so we were left to our own devices.
Since the problem seemed to be with file handles, we searched for a Linux
utility that would give us a list of open files. We finally found one, and
ran it on one of the servers.
What we discovered was that there was a log file used by JRun that we didn't
know about, buried deep in the JRun directory structure. This file is the
log that gets written to every time JRun instantiates an object, and is also
where all output from the "log()" method is sent. Here's where it's
located:
jrun233/jrun/jsm-default/services/jse/logs/event.log
This file had been constantly appended to since the day we launched our
servers, and had grown to over 2GB. Apparently, it had reached the limit of
what Linux would allow, so every time JRun attempted to write to the file,
the write would fail and the file handle would stay open. Therefore, it was
just a matter of time before the multiple attempts by JRun to write to the
file took up all the available file handles. Restarting JRun must have
closed all the file write attempts, and then it started all over again.
Each server hit the limit at a slightly different time, but they eventually
all would have had the same problem.
Obviously, the answer was to rotate the log file daily, which is what we are
now doing. We haven't had the problem again since.
Hope that helps someone...you guys have been extremely helpful to us on many
occasions.
Nate McKie
CTO/Principal, Asynchrony.com
www.asynchrony.com
[EMAIL PROTECTED]
------------------------------------------------------------------------------
Archives: http://www.egroups.com/group/jrun-interest/
Unsubscribe: http://www.houseoffusion.com/index.cfm?sidebar=lists&body=lists/jrun_talk
or send a message to [EMAIL PROTECTED] with 'unsubscribe' in the
body.
