Maybe I can throw a little light on this. I don't know too much about
what goes on inside jBoss or the JVM, but I can tell you what I see
happening on a Linux system.
When something is wrong with the deployable EAR, and I am not really sure
exactly what, jBoss can get into a state where it just starts creating
threads until the operating system puts a stop to it. In Linux, each
thread gets a process identifier ('pid') and threads are 'lightweight
processes' which run on the operating system scheduler. The threads share
memory and descriptors so the overhead is much less than for heavyweight
processes, which is the whole purpose of threads.
However, the system limitations applicable to heavyweight processes apply
equally to lightweight processes, in particular that a single non-root
user may never exceed half of the available system pids. By default when
building a Linux 2.2 kernel, the system has a global limit of 512 pids.
This is configurable in the linux source (include/linux/tasks.h):
#define NR_TASKS 512 /* On x86 Max about 4000 */
#define MAX_TASKS_PER_USER (NR_TASKS/2)
#define MIN_TASKS_LEFT_FOR_ROOT 4
It is possible on most platforms which run Linux to increase this number
in the source code and rebuild the kernel. (It is problematic on SPARC,
but that it beyond the scope of the discussion here.) Nevertheless, it is
important to realize that (a) there is always some finite limit which
usually has a grounding in actual hardware, (b) scheduling lots of threads
requires time and effort on the part of the kernel which is by definition
done at the expense of useful work, and (c) Linux typically runs on large
servers using only the defaults such that increasing the task count is so
rare that it requires a source edit rather than a configuration option.
As long as jBoss is not run by the root user, which is obviously extremely
dangerous and undesirable for other reasons anyway, the maximum number of
pids it can own concurrently is going to be limited to 255 on a default
kernel. Once this thread limit is reached, jBoss will essentially wedge
to the point where it no longer even responds to SIGTERM to shut down
gracefully, but requires SIGKILL to stop it. I think that jBoss sees this
situation as an "out of memory" condition as reflected by the JVM, but I
am not sure about this. Of course, if jBoss is run by the root user for
some reason, the entire machine will be wedged at this point and the only
remedy would be something as drastic as the reset switch or the watchdog.
These observations apply to JBoss-2.2.1/Tomcat-3.2.1 running on Debian
(Woody) using Java HotSpot(TM) Server VM (build 1.3.1-b24, mixed mode).
The Linux kernel is 2.2.19 SMP. The "run.sh" file has been modified to
include the switches "-Xms8m -Xmx480m" so as to allow the JVM to allocate
much more than the default 64MB, but in fact it usually only allocates
between 45MB and 65MB before wedging. The machine has 256MB physical RAM
and 1GB of swap, and in practice never swaps. Here is a snapshot with
jBoss running under moderate load:
total used free shared buffers cached
Mem: 257640 191672 65968 73588 77708 22528
-/+ buffers/cache: 91436 166204
Swap: 1048560 0 1048560
The main thread of the JVM at this point shows:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
jboss 19954 0.6 25.3 923052 65184 ? S 05:01 0:36
Notice that the operating system virtual machine size of 923MB is a little
large, but this is just address space and is fairly far divorced from
actual memory. The real resident size of the JVM is only 65MB, and the
whole machine has only about 200MB allocated and none of the 1GB swap
space in use at all. I am not sure why the JVM has such a large amount of
address space allocated, even if not actual memory, when it has been told
that it can use only about half of that. Remember that the data above
comes from a snapshot of a normally running jBoss installation. However,
maybe "out of memory" means exactly what it says.
-- Mike
On 2001-06-21 at 05:43 -0400, Jim Archer wrote:
> Were are battling this even as I type...
>
> It seem that if jBoss hits the operating systems pid (thread) limit for a
> process, it throws out of memory exceptions. We just ran some load test
> scripts and when we get too many threads, pop goes the weasil.
>
> I'm looking at two possibilities... Catch the out of memory exceptions in
> my code and then sleep for a bit and try again (not a great olution) or
> throw more hardware / operating system at it.
>
> Maybe both.
>
> Does anyone know what jBoss should do when the thread limit is hit?
>
> Thanks...
>
> Jim
>
>
> --On Thursday, June 21, 2001 10:11 AM +0100 Nicolai P Guba
> <[EMAIL PROTECTED]> wrote:
>
> > Hello
> >
> > The problem is much more drastic than I initially thought. I just
> > asked a question about closing PreparedStatements. Obviously, I had a
> > few errors. I left the server running, recompiled the app,
> > redeployed, forgot to remove one close on the stmt, then redeployed
> > and now this is what I've got.
> >
> > [Auto deploy] Deployment
> > failed:file:/usr/local/jboss-2.2.2/deploy/CAMS.jar [Auto deploy]
> > org.jboss.deployment.J2eeDeploymentException: Error while starting
> > CAMS.jar: unable to create new native thread [Auto deploy] at
> > org.jboss.deployment.J2eeDeployer.startApplication(J2eeDeployer.java:448)
> > [Auto deploy] at
> > org.jboss.deployment.J2eeDeployer.deploy(J2eeDeployer.java:178) [Auto
> > deploy] at java.lang.reflect.Method.invoke(Native Method) [Auto deploy]
> > at
> > com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628)
> > [Auto deploy] at
> > com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523)
> > [Auto deploy] at org.jboss.ejb.AutoDeployer.deploy(AutoDeployer.java:358)
> > [Auto deploy] at org.jboss.ejb.AutoDeployer.run(AutoDeployer.java:221)
> > [Auto deploy] at java.lang.Thread.run(Thread.java:484)
> >
> > *ouch*
> >
> > It appears that it doens't only occur when it doesn't validate/verify,
> > but also when there are transaction rollback exceptions.
_______________________________________________
JBoss-user mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/jboss-user
