Bugs item #832880, was opened at 2003-10-30 00:19
Message generated for change (Comment added) made by thetooth
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=832880&group_id=22866
Category: JBossServer
Group: v3.2
>Status: Closed
Resolution: None
Priority: 9
Submitted By: Brian Hogue (thetooth)
Assigned to: Adrian Brock (ejort)
Summary: JNDI lookup does not return after/under moderate load
Initial Comment:
After upgrading from 2.4.4 to 3.2.1(Tomcat bundle) our
client applications began failing when attempting a jndi
Context lookup via, Object ref = (myHome)
jndiContext.lookup("foo/myData"); where jndiContext is
an InitialContext derived from
("java.naming.factory.initial", "org.jnp.interfaces.NamingC
ontextFactory");
("java.naming.provider.url", "myServer:1099");
("java.naming.factory.url.pkgs", "org.jboss.naming:org.jnp
.interfaces");
The application makes heavy use of SLSB's accessed
from remote clients, of which the above mentioned line
of code handles the lookup.
The failure is manifested by the above call not returning
after the server has been up for a few hours and has
experienced moderate load. The 2.4.4 server *never*
exhibited this behavior under extreme load. The only
changes with the upgrade was 3.2.1 JBoss and 1.4.2
java. The call does not seem to timeout but rather
to 'hang'. We are attempting to upgrade to 3.2.2,
hoping for a fix.
This is occuring on multiple servers; (compaq DL-380's)
running RH Linux 7.3 (Linux 2.4.18-19.7.xsmp) with
Hotspot VM (1.4.2_01). Apache is integrated via
mod_jk . The servers have 2 GB of ram with 1.5
available and dedicated to the VM.
The server socket on 1099 goes into and stays in a
SYN_RECV state. To remove the LAN/WAN as a possible
culprit we have replicated the problem (once the
problem starts occuring on a server) by trying to do the
context lookup from the server itself to itself, using
127.0.0.1 to make sure nothing is externally routed.
There are no memory consumption issues within the JVM
or at the OS level. CPU usage and load points are very
low. HTTP/servlet request servicing remains unaffected
when JBoss gets into the described state. Once JBoss is
restarted, it will behave normally for a time.
We are using the 'Standard Stateless SessionBean'
container with the ssl-invoker.
----------------------------------------------------------------------
>Comment By: Brian Hogue (thetooth)
Date: 2003-12-11 09:42
Message:
Logged In: YES
user_id=897950
I am closing this - upgrading the Linux kernal seems to have
fixed the problem. Thanks
----------------------------------------------------------------------
Comment By: Brian Hogue (thetooth)
Date: 2003-11-09 14:48
Message:
Logged In: YES
user_id=897950
Thanks for the follow-up. We do have the RmiPort set to a
fixed value, so we were watching for activity on that port.
When the server gets into the described state, we dont see
activity on the RmiPort. 1099 stays in SYN_RECV.
We just recently went to 3.2.2 and upgraded the linux kernel
to 2.4.20. Its too early to claim victory, but things are
looking better. I am inclined to believe there may be problems
with the 1.4 VM and the 2.4.18 kernel. I'll post more test
results when available.
----------------------------------------------------------------------
Comment By: Scott M Stark (starksm)
Date: 2003-11-09 12:36
Message:
Logged In: YES
user_id=175228
The 1099 port is only used to obtain the RMI stub to the
JNDI naming service. All lookups through a Context derived
from an InitialContext have nothing to do with this 1099
socket. Instead, communication is going through the RMI
socket which by default is bound to an anonymous port. You
would have to set the NamingService RmiPort attribute to fix
this to a known port in order to see what state the socket
associated the client lookup is in.
----------------------------------------------------------------------
Comment By: Brian Hogue (thetooth)
Date: 2003-10-30 07:16
Message:
Logged In: YES
user_id=897950
Amend point 1. I had taken the jboss server down *before* I
ran the rmi client-server test.
----------------------------------------------------------------------
Comment By: Brian Hogue (thetooth)
Date: 2003-10-30 07:08
Message:
Logged In: YES
user_id=897950
Thank you for the follow-up. The next time the problem
occurs I will collect all the information you suggest. This is
what I can answer now:
1). I could telnet to 1099 while it was in the afore-mentioned
state, just got the typical
Trying 67.105.52.45...^M
Connected to 67.105.52.45.^M
Escape character is '^]'.^M
[EMAIL PROTECTED]@^Yjava.rmi.MarshalledObject|½^^~Wícü>^B^@
[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]&[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@^Thttp://0.0.0.0:8083/[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL
PROTECTED]
@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@^Esr^@
[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]
[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]@[EMAIL
PROTECTED]@[EMAIL PROTECTED]@[EMAIL PROTECTED]
@[EMAIL PROTECTED] closed by foreign host.^M
Also of note, while the server is in the described condition, I
was able to run a simple (non-jboss) rmi client-server test
designed to lookup and object on 1099 and pass data to and
from it. That seems to work even when nothing is getting
through for jboss on 1099.
2.) Nothing at all special on the Checkpoint firewall or TCP-IP
config. Our TCP params are stock, if there are any in
particular you'd like to know about, let me know. There are
no overruns, errors, or drops reported by the adapters on the
boxes or the firewall. To eliminate the network variable i ran
a Jboss client test from the Jboss server itself using 127.0.0.1
as the target host to make sure there was no external routing
and got the same failure.
3.) will collect
4.) will collect
5.) No error messages in server.log (nor in /var/log/messages)
at all.
6.) will collect
7.) will collect
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-10-30 04:10
Message:
Logged In: YES
user_id=9459
A SYN_RECV means the socket has sent a SYN_ACK to the client
but hasn't recieved the third phase connect from the client.
The client will be stored in a backlog queue until the
connection
times out.
Some things to investigate:
1) Can you telnet to port 1099? What do you see?
2) Firewall and TCP/IP configuration
3) tcpdump -i eth0
4) How many threads is jboss using?
see jboss.server:service=ServerInfo on the jmx-console
or pstree -p
5) Error messages in log/server.log
6) Configure log4j on the client and enable trace logging for
org.jnp
7) Thread dumps on the client and server ctrl-\ or kill -3
${pid}
Regards,
Adrian
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=832880&group_id=22866
-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click
_______________________________________________
JBoss-Development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
