cdegeiso [https://community.jboss.org/people/cdegeiso] created the discussion

"JBoss 5.1.0.GA Severe Latency Problem"

To view the discussion, visit: https://community.jboss.org/message/832794#832794

--------------------------------------------------------------
Hi all,

  I have an issue that I have not been able to determine whether it is a 
symptom or a cause of a severe latency problem I am experiencing with a 
deployment of a COTS package using JBoss 5.1.0.GA. We use 5.1 because that is 
what is currently supported with the application. I have heard the next release 
will have JBoss 7 support and am anxious to move on to something more solid. 

  We've been running 5.1 for a couple of years now (JBoss 4.2.1 years before 
that) and it used to run fine in JBoss 4 and for a little over a year on 5.1, 
then performance started degrading and has become unusable at times. Although 
we have been using JBoss for a long time, I am by no means a JBoss expert. I 
know enough to get it running and processing data for our application, that's 
about it. Unfortunately, due to apathy of system admins (don't ask), the task 
of figuring out what is going on has been assigned to me (an application 
administrator), so I will have to get spun up quick on it.

  First some symptoms. After a few days of the application running, latency 
starts to drift in. It will get so bad that transaction times are clocked at 
roughly 75s per transaction. CPU and memory usage are good on windows boxes and 
HP/UX database box, I can access JMX console just fine and monitor activity 
there, but some anomalies exist. One, JCA connections Available and connections 
in use do not equal our total connection pool size. For example, our connection 
pool is set to max 300 connections and I will seel something like 164 
connections in use and 52 avaliable. When we aren't having issues, these two 
numbers always equal 300 if added together. The only way we have found to 
recover is to completely shut off the JBoss servers, bump the database, and 
restart the JBoss servers. Stopping JBoss services doesn't fix it and neither 
does restarting the JBoss servers. The database has to clear out all sessions 
on it's end before JBoss will connect again. DBA states that the database is 
fully functioning during this time and has not provided any anomalies from logs 
on the oracle end.

  The second thing that occurs when we get slammed with latency is when I run a 
thread dump to see what tasks are currently processing during the latency, I 
see a lot of threads that look like this:

*Thread: Thread-6293325* : priority:5, demon:false, threadId:6370353, 
threadState:RUNNABLE

> java.net.SocketInputStream.socketRead0(Native Method)
> java.net.SocketInputStream.__AW_read(Unknown Source)
> java.net.SocketInputStream.read(Unknown Source)
> oracle.net.ns.Packet.receive(Packet.java:300)
> oracle.net.ns.DataPacket.receive(DataPacket.java:106)
> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
> oracle.net.ns.NetInputStream.__AW_read(NetInputStream.java:260)
> oracle.net.ns.NetInputStream.read(NetInputStream.java)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> oracle.jdbc.driver.T4CTTIoping.doOPING(T4CTTIoping.java:52)
> oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:4008)
> - locked <0x14948d6e> (a oracle.jdbc.driver.T4CConnection)
> oracle.jdbc.driver.PhysicalConnection$3.run(PhysicalConnection.java:7868)
> java.lang.Thread.run(Unknown Source)
> 
*Thread: Thread-6293326* : priority:5, demon:true, threadId:6370354, 
threadState:RUNNABLE
*Thread: Thread-6293327* : priority:5, demon:true, threadId:6370355, 
threadState:RUNNABLE

> java.net.SocketInputStream.socketRead0(Native Method)
> java.net.SocketInputStream.__AW_read(Unknown Source)
> java.net.SocketInputStream.read(Unknown Source)
> oracle.net.ns.Packet.receive(Packet.java:300)
> oracle.net.ns.DataPacket.receive(DataPacket.java:106)
> oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
> oracle.net.ns.NetInputStream.__AW_read(NetInputStream.java:260)
> oracle.net.ns.NetInputStream.read(NetInputStream.java)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
> oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
> oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
> oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1137)
> oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:290)
> oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:192)
> oracle.jdbc.driver.T4CTTIoping.doOPING(T4CTTIoping.java:52)
> oracle.jdbc.driver.T4CConnection.doPingDatabase(T4CConnection.java:4008)
> - locked <0x65f90b32> (a oracle.jdbc.driver.T4CConnection)
> oracle.jdbc.driver.PhysicalConnection$3.run(PhysicalConnection.java:7868)
> java.lang.Thread.run(Unknown Source)
> 
*Thread: Thread-6293328* : priority:5, demon:true, threadId:6370356, 
threadState:RUNNABLE
*Thread: Thread-6293329* : priority:5, demon:true, threadId:6370357, 
threadState:RUNNABLE
*Thread: Thread-6293330* : priority:5, demon:true, threadId:6370358, 
threadState:RUNNABLE
*Thread: Thread-6293331* : priority:5, demon:true, threadId:6370359, 
threadState:RUNNABLE
*Thread: Thread-6293332* : priority:5, demon:true, threadId:6370360, 
threadState:RUNNABLE
*Thread: Thread-6293323* : priority:5, demon:true, threadId:6370361, 
threadState:RUNNABLE
*Thread: Thread-6293333* : priority:5, demon:true, threadId:6370362, 
threadState:RUNNABLE
*Thread: Thread-6293334* : priority:5, demon:true, threadId:6370363, 
threadState:RUNNABLE

  All of the other threads have some kind of name that I can determine what 
they are used for, like:

*Thread: ajp-0.0.0.0-8005-197*
**Thread: ReportQueueAgent**
***Thread: HeartbeatThread***

  All of these threads have stack traces that I can see what they are doing and 
can be reasonably sure that they are doing something productive. The generic 
threads (what I call them) above don't really seem to do much other than open 
connections and hold onto them. When things are running smoothly, the generic 
threads are non-existent. They gradually build up over time though as latency 
kicks in. These also seem to correlate with higher user connections (the more 
users that are in, the more likely performance will tank).

  As I said earlier, I am no expert, but I feel like there is a direct 
correlation to these threads and our latency issue. While they may not be the 
cause, I would like to know how I go about getting JBoss to tell me why these 
threads are being created so I can find a way to stop it or, in the case of a 
code defect, submit it to the developer for review.

  More info pertaining to our environment. I am not sure it matters, but in 
case it does, here are some specs:

User Base: Total 4,000 users
Concurrent Users: 175 (peak), 110 (avg)
No Load balancing - App and process servers serve two completely separate 
functions, although they both run JBoss and TRIRIGA.

(All are virtual with the exception of HP/UX)

Web Server: AMD64 Dual Core processor, 4GB Ram, Windows 2003, IIS 6.0, ISAPI 
Redirector 1.2.32
App Server: AMD64 Dual Core processor, 12GB Ram, Windows 2003, JBoss 5.1.0.GA 
(hosting Tririga) - Java Heap 8GB
Process Server: AMD64 Dual Core processor, 10GB Ram, Windows 2003, JBoss 
5.1.0.GA (hosting Tririga) - Java Heap 6GB
Report Server: AMD64 Dual Core processor, 4GB Ram, Windows 2003, Crystal RAS 
2008 SP3
Database Server: 6 processors, 18GB Ram, Oracle 11gR2

Windows boxes will be upgraded to Windows 2008R2 within the next couple of 
months.

Thanks,

Chris
--------------------------------------------------------------

Reply to this message by going to Community
[https://community.jboss.org/message/832794#832794]

Start a new discussion in Beginner's Corner at Community
[https://community.jboss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2075]

_______________________________________________
jboss-user mailing list
jboss-user@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-user

Reply via email to