Just a few details on this.. so servicemix deadlocks every time I startup,
where servicemix needs to redeploy all SEs and all SUs... here is the
deadlock trace.. all my threads are sitting like this:
"pool-flow.seda.servicemix-bean-thread-1" prio=6 tid=0x2879c400 nid=0x1728
waiting for monitor entry [0x3512f000..0x3512
fb94]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.createQueue(SedaFlow.java:189)
- waiting to lock <0x07db2050> (a
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:179)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
at
org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:79)
at
org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:571)
then if I force it closed, since it's hanging and unresponsive... I do get a
ton of NPE's on startup and then it APPEARS to start behaving normally... so
I'm not sure if this latest 13 build has solved the problem enough or if
some code changes I made did, which again is here:
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
at
org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569)
at
and results in some illegal sendSync errors.... is it possible that my
MessageExchangeFactory is giving back a null message or something like that
which is causing the NPE?
On Sun, Jun 22, 2008 at 5:38 PM, Ryan Moquin <[EMAIL PROTECTED]> wrote:
> Hey Bruce,
>
> Out of curiousity.. did you guys do anything with this issue in the last
> day or two? I've been working on some stuff on my end that I was hoping may
> have had some effect.. when I run it on the build tagged "12" in the nightly
> builds, I have the same issue. When I try the one tagged "13" all my
> outgoing JMS requests on the DeliveryChannel deadlock the first time I start
> up Servicemix. The second time I start up it, I get a ton of errors about
> illegally calling sendSync but my threads don't deadlock that second time.
> The deadlock is way down in the bowels of servicemix... I don't see that NPE
> anymore on the latest one, so I'm just curious so I know exactly what I'm
> trying to solve at this point! :)
>
> Thanks!!
>
> Ryan
>
>
> On Sat, Jun 21, 2008 at 12:48 AM, Ryan Moquin <[EMAIL PROTECTED]>
> wrote:
>
>> I hate to say this Bruce, but unfortunately 3.2.2 is working pretty bad
>> for me and has the same problem :( It only seems to affect our production
>> server and in a few minutes after start up, I start getting the
>> NullPointerException.. slowly all my services start doing it and they all
>> stop working. If I stop and restart servicemix, then servicemix-jms
>> components are no longer routable. I'm guessing this error causes
>> Servicemix to shut them down and not deploy them anymore (Servicemix does
>> the same thing if a service unit starts up, tries to make a Joram connection
>> to a server that isn't up, servicemix will shutdown that SU and will always
>> shut it down immediately after it starts on any subsequent run).
>>
>> It seems like this problem must be related to a race condition. When
>> doing development testing, I never see this problem on my laptop even under
>> high load. One our fast test server, I see this error popup once on startup
>> and then it doesn't seem to happen again. On our even faster production
>> server, the whole thing looses it's wheels and falls apart after a few
>> minutes.
>>
>> I'm supposed to be deploying this system in a few days and of course
>> that's the only spot where I can't temporarily limp by. Is it possible you
>> could give me some hints on what the problem is and I'll debug it this
>> weekend to see if I can fix it, or at least patch it temporarily? I really
>> need to figure out way to get around this problem. Other than that, 3.2.2
>> seems to work perfectly fine.
>>
>> Here is the caused by error again in case it's any bit different than the
>> 3.2.1 one was:
>>
>> Caused by: java.lang.NullPointerException
>> at
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>> at
>> org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569)
>> at
>> com.notification.impl.JbiNotificationHandlerImpl.sendNotification(JbiNotificat
>> ionHandlerImpl.java:80)
>>
>> Also, is the delivery channel component threadsafe? I'm curious if
>> multiple threads accessing it is a problem or if I should keep access to it
>> synchronized? I'm currently synchronizing, but don't want to if I don't
>> need to.
>>
>> Thanks!
>> Ryan
>>
>>
>> On Thu, Jun 19, 2008 at 2:19 AM, Bruce Snyder <[EMAIL PROTECTED]>
>> wrote:
>>
>>> On Wed, Jun 18, 2008 at 7:56 PM, Ryan Moquin <[EMAIL PROTECTED]>
>>> wrote:
>>> > I'm using Servicemix 3.2.1, so I'll give 3.2.2 a try. I was kind of
>>> waiting
>>> > until it was released, but this problem is now cropping up on a regular
>>> > basis on a server so I'll definitely give it a shot. Hopefully this
>>> will
>>> > allow me to get this project done so I can then get that Joram write up
>>> done
>>> > since I should have ran across hopefully most of the gotchas for it at
>>> that
>>> > point.
>>>
>>> 3.2.2 will be released very soon and I know it's pretty stable. Maybe
>>> we can release it this weekend.
>>>
>>> Bruce
>>> --
>>> perl -e 'print
>>> unpack("u30","D0G)[EMAIL PROTECTED]&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>>> );'
>>>
>>> Apache ActiveMQ - http://activemq.org/
>>> Apache Camel - http://activemq.org/camel/
>>> Apache ServiceMix - http://servicemix.org/
>>>
>>> Blog: http://bruceblog.org/
>>>
>>
>>
>
