Bugs item #784343, was opened at 2003-08-06 19:15
Message generated for change (Comment added) made by ejort
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=784343&group_id=22866
Category: JBossMQ
Group: v3.2
Status: Open
Resolution: None
Priority: 5
Submitted By: Todd Wolff (wolfftw)
Assigned to: Adrian Brock (ejort)
Summary: Invalid msgType Concurrency Bug?
Initial Comment:
Using 3.2.2RC3 Jetty Build, 1.4.2jdk on Windows XP Pro
SP1 the attached test case causes an intermittent
IllegalArgumentException: Invalid msgType error, when
using the UIL2ConnectionFactory.
When using the jvm IL (java:/ConnectionFactory), the
process simply hangs after a few thousand messages,
with no error messages. It may take a few attempts to
get it to hang, and the machine should be relatively fast
(roughly 100 messages per second) to duplicate the
problem.
The sessions are all non-transacted, the messages are
non_persistent and do not expire. I don't believe I've
broken any design idioms within the test case, but if so,
please email me with the problem. I've spent several
days trying to solve this one.
See readme.txt in attached test case for test execution
instructions. Thanks and let me know if you need
anything else.
[EMAIL PROTECTED]
----------------------------------------------------------------------
>Comment By: Adrian Brock (ejort)
Date: 2003-08-12 16:11
Message:
Logged In: YES
user_id=9459
:-)
But what about the JVM IL? I don't see how that could be
caused by
a firewall problem.
See if you can get a hand running with TRACE enabled.
Regards,
Adrian
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-12 15:49
Message:
Logged In: YES
user_id=839156
Adrian,
Well ... I found the problem. Not jboss, my code, the OS, nor
the jdk. It was a race condition with the firewall software
installed on the machine. Apparently, above a certain
message rate, the stream (which filters through the firewall)
gets corrupted.
I am now getting message rates of up to 275/second with
three sending threads - AND no exceptions. Thanks so much
for your help and keep up the good work.
Todd
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-08-11 19:46
Message:
Logged In: YES
user_id=9459
Your latest log shows the stream is corrupted.
Looks like that the message type 0x80 (-128)
and the message id 0x00000200 (512)
should go together 0x80000002 or -2147483646
Unless this is a very early message that has taken a while
to process,
I would guess this is really a subscription id (seen
elsewhere in the log).
If that is true, the stream is wayoff.
Can you see if you can find that message id processed in the
log,
should be at the start.
Also can you run it again with {%t} in the
"ConversionPattern" of the log
so we can see the threads. I cannot workout from your log
which request is failing (seeing the threads will help me
determine
the queue) - send me the full log to [EMAIL PROTECTED]
It is bizarre that nobody else can reproduce this, since you can
reproduce it so consistently using the same test?
Regards,
Adrian
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-11 15:53
Message:
Logged In: YES
user_id=839156
Adrian,
Here's the new trace. Good luck.
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-08-08 15:21
Message:
Logged In: YES
user_id=9459
The jars are too big for sourceforge so I've attached the
changed
classes instead.
org.jboss.util.stream.Streams.class in jboss-common.jar
org.jboss.mq.il.uil2.SocketHandler$ReadTask.class in jbossmq.jar
Regards,
Adrian
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-08-08 15:17
Message:
Logged In: YES
user_id=9459
I'm attaching a replacement jboss-common.jar and jbossmq.jar
that will dump the stream in hex characters when there is a
failure.
It is not very clever, so if there is a problem with the
protocol,
the dump will just hang waiting for more input from the open
socket.
Regards,
Adrian
----------------------------------------------------------------------
Comment By: Scott M Stark (starksm)
Date: 2003-08-08 06:22
Message:
Logged In: YES
user_id=175228
I have made ~10 runs using JDK 1.4.2 on an xp pro sp1 2.4GHz
laptop with msg rates of 100-125 and not been able to see
this issue. We may need to create a special debug patch that
saves off the working byte[] buffer so that when a bad msg
is see we can dump out the buffer and position to see if the
index into the buffer is just getting offline or the data
really is bad. Both the message type 33, msgID: 41943041 are
completely off.
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-08 02:16
Message:
Logged In: YES
user_id=839156
I was running 1.4.1_04 when I first noticed the problem and
upgraded to 1.4.2, hoping it would go away.
So ... perhaps it is platform specific. The centrino
configuration includes a new chipset, I believe, and I'm
running XP Pro. I can try running on another win2k box
tomorrow, but it's slower ...
How many messages per second are you guys processing, so
that we can rule out the race condition?
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-08 02:07
Message:
Logged In: YES
user_id=839156
Adrian,
It's not a classpath or stray jar issue. I copy the build
directly from the jetty output directory and the run the same
test script you're running. hmmm.
I uploaded another trace, which is different than trace2, but
contains the same exception.
----------------------------------------------------------------------
Comment By: Elias Ross (genman)
Date: 2003-08-07 23:12
Message:
Logged In: YES
user_id=556458
Since my company is very concerened about the reliablity of
JMS, I went ahead and tested using my own installation. I
use JVM 1.4.1_01 (Blackdown) on Redhat Linux 8.
I was unable to reproduce the problem, even after tens of tries.
If all JBoss threads stop running, even unrelated ones then
this would suggest a JVM bug, or OS or hardware issue. As
1.4.2 is fairly new, I suggest you test and deploy with a
JVM 1.4.1 release.
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-08-07 21:29
Message:
Logged In: YES
user_id=9459
Hi,
I've been trying all evening to get either the jvm il or
UIL2 il to fail.
I've turned off all logging and run different size tests
with different
numbers of threads.
I haven't seen any failures.
The UIL2 stacktraces look like the problems we were having
with an old version of concurrent.jar (which is used elsewhere
by jbossmq), it was replaced with a fixed version for
jboss-3.2.0
Can I ask whether you have any old jars in jre/lib/ext or you
have modified the classpath to add old jars from somewhere?
For Trace2.txt can you go back further in the log so we can see
the history for the request that fails:
Read msgType: unknown message type 0, msgID: 76057
Regards,
Adrian
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-07 20:01
Message:
Logged In: YES
user_id=839156
Adrian,
To get the jvmIL to hang fairly consistently, I had to turn off
all logging. It appears to be a race condition. It runs like a
champ with debug level logging.
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-07 13:42
Message:
Logged In: YES
user_id=839156
I get two different errors when using UIL2. Here's the first
trace. BTW, it fails everytime unlike the jvm IL.
----------------------------------------------------------------------
Comment By: Todd Wolff (wolfftw)
Date: 2003-08-07 13:39
Message:
Logged In: YES
user_id=839156
Here's the thread dump using the jvm il following the hang.
BTW, I'm not using a multiprocessor machine. What I meant
by multithreaded was multiple httpclient threads, multiple
queue receivers listening on the same queue, multiple queue
senders ...
I had to run at least six times sending 3000 messages each
time to get it to hang. My machine is a Pentium M, 1.3 ghz.
It processes the 3000 messages in .489 minutes.
I will follow-up with the UIL2 trace.
----------------------------------------------------------------------
Comment By: Adrian Brock (ejort)
Date: 2003-08-07 09:19
Message:
Logged In: YES
user_id=9459
I've tried to get your testcase to crash but it is not happening
for me. Probably because I don't have a multi-processor machine.
When it hangs using the jvm IL, take a threaddump using
ctrl-break
and post the results.
Alternatively, set TRACE logging for org.jboss.mq while running
with UIL2 and post the log snippet for the last couple of
minutes before
you get the IllegalArgumentException
Regards,
Adrian
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=784343&group_id=22866
-------------------------------------------------------
This SF.Net email sponsored by: Free pre-built ASP.NET sites including
Data Reports, E-commerce, Portals, and Forums are available now.
Download today and enter to win an XBOX or Visual Studio .NET.
http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01
_______________________________________________
JBoss-Development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
