The currentRequest is a thread local variable, so if another exchange
is handled it should not damage the processing of another thread.
On Mon, Jun 23, 2008 at 8:45 AM, Ryan Moquin <[EMAIL PROTECTED]> wrote:
> Could someone on the servicemix team verify this rationale for me? I think
> I figured out my problem, which is a race condition in Servicemix (which of
> course is exploited more on a faster machine). I'm not sure what the best
> solution to work around this problem is, and I am on a short timeframe since
> of course this problem really only plagues our production environment and
> not our dev or test environment. Anyhow, I have a servicemix-bean component
> that is taking an input, performing some analysis on it, and then sending it
> back out to a JMS topic. In order to send it back out, I'm using the
> DeliveryChannel that is injected into my bean. This sharing of the delivery
> channel is what is causing the race condition, here is the reason. The
> exceptions I've been mentioning are referring to the following line:
>
> requests.put(messageExchange.getExchangeId(), currentRequest.get());
>
> This line is dependant upon the request I'm building in order to send out to
> the JMS queue (at least I believe so). The thing is that currentRequest is
> global to the current thread, so when a message comes into my
> servicemix-bean component, which I believe would be this method snippet:
>
> protected void onProviderExchange(MessageExchange exchange) throws Exception
> {
> ...
> currentRequest.set(req);
> synchronized (req) {
> ...
> }
> checkEndOfRequest(req, corId);
> currentRequest.set(null);
>
> Doesn't this mean that anytime a message is delivered (inside the
> synchronized block), there is a good chance that when my bean is trying to
> use the delivery channel to send a message, that the "currentRequest" member
> variable is being clobbered? I get an NPE .. it seems like that's most
> likely a sideeffect of the currentRequest.set(null). Such as my send inside
> the bean, clobbers the currentRequest, and then in turn, the BeanEndpoint
> clobbers my request and makes it null, which causes my send to blow up?
>
> It appears as though I maybe should be creating my own deliverychannel or
> using another method to send out from my bean? Or is this something that
> could be fixed, and if so, how soon? Or is it something I could do with
> some guidance? Or is there something else I can do? I'm going to go look
> through the docs for another option, but I really have to work around this,
> since my app is pretty much dead right now, which is the worst time.
>
> Thanks for any help!!
>
> Ryan
>
> On Sun, Jun 22, 2008 at 7:17 PM, Ryan Moquin <[EMAIL PROTECTED]> wrote:
>
>> Just a few details on this.. so servicemix deadlocks every time I startup,
>> where servicemix needs to redeploy all SEs and all SUs... here is the
>> deadlock trace.. all my threads are sitting like this:
>>
>> "pool-flow.seda.servicemix-bean-thread-1" prio=6 tid=0x2879c400 nid=0x1728
>> waiting for monitor entry [0x3512f000..0x3512
>> fb94]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.createQueue(SedaFlow.java:189)
>> - waiting to lock <0x07db2050> (a
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow)
>> at
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:179)
>> at
>> org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:162)
>> at
>> org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
>> at
>> org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
>> at
>> org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
>> at
>> org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:830)
>> at
>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
>> at
>> org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
>> at
>> org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:79)
>> at
>> org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:571)
>>
>>
>> then if I force it closed, since it's hanging and unresponsive... I do get
>> a ton of NPE's on startup and then it APPEARS to start behaving normally...
>> so I'm not sure if this latest 13 build has solved the problem enough or if
>> some code changes I made did, which again is here:
>>
>> Caused by: java.lang.NullPointerException
>> at
>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>> at
>> org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569)
>> at
>>
>> and results in some illegal sendSync errors.... is it possible that my
>> MessageExchangeFactory is giving back a null message or something like that
>> which is causing the NPE?
>>
>>
>> On Sun, Jun 22, 2008 at 5:38 PM, Ryan Moquin <[EMAIL PROTECTED]>
>> wrote:
>>
>>> Hey Bruce,
>>>
>>> Out of curiousity.. did you guys do anything with this issue in the last
>>> day or two? I've been working on some stuff on my end that I was hoping may
>>> have had some effect.. when I run it on the build tagged "12" in the nightly
>>> builds, I have the same issue. When I try the one tagged "13" all my
>>> outgoing JMS requests on the DeliveryChannel deadlock the first time I start
>>> up Servicemix. The second time I start up it, I get a ton of errors about
>>> illegally calling sendSync but my threads don't deadlock that second time.
>>> The deadlock is way down in the bowels of servicemix... I don't see that NPE
>>> anymore on the latest one, so I'm just curious so I know exactly what I'm
>>> trying to solve at this point! :)
>>>
>>> Thanks!!
>>>
>>> Ryan
>>>
>>>
>>> On Sat, Jun 21, 2008 at 12:48 AM, Ryan Moquin <[EMAIL PROTECTED]>
>>> wrote:
>>>
>>>> I hate to say this Bruce, but unfortunately 3.2.2 is working pretty bad
>>>> for me and has the same problem :( It only seems to affect our production
>>>> server and in a few minutes after start up, I start getting the
>>>> NullPointerException.. slowly all my services start doing it and they all
>>>> stop working. If I stop and restart servicemix, then servicemix-jms
>>>> components are no longer routable. I'm guessing this error causes
>>>> Servicemix to shut them down and not deploy them anymore (Servicemix does
>>>> the same thing if a service unit starts up, tries to make a Joram
>>>> connection
>>>> to a server that isn't up, servicemix will shutdown that SU and will always
>>>> shut it down immediately after it starts on any subsequent run).
>>>>
>>>> It seems like this problem must be related to a race condition. When
>>>> doing development testing, I never see this problem on my laptop even under
>>>> high load. One our fast test server, I see this error popup once on
>>>> startup
>>>> and then it doesn't seem to happen again. On our even faster production
>>>> server, the whole thing looses it's wheels and falls apart after a few
>>>> minutes.
>>>>
>>>> I'm supposed to be deploying this system in a few days and of course
>>>> that's the only spot where I can't temporarily limp by. Is it possible you
>>>> could give me some hints on what the problem is and I'll debug it this
>>>> weekend to see if I can fix it, or at least patch it temporarily? I really
>>>> need to figure out way to get around this problem. Other than that, 3.2.2
>>>> seems to work perfectly fine.
>>>>
>>>> Here is the caused by error again in case it's any bit different than the
>>>> 3.2.1 one was:
>>>>
>>>> Caused by: java.lang.NullPointerException
>>>> at
>>>> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881)
>>>> at
>>>> org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569)
>>>> at
>>>> com.notification.impl.JbiNotificationHandlerImpl.sendNotification(JbiNotificat
>>>> ionHandlerImpl.java:80)
>>>>
>>>> Also, is the delivery channel component threadsafe? I'm curious if
>>>> multiple threads accessing it is a problem or if I should keep access to it
>>>> synchronized? I'm currently synchronizing, but don't want to if I don't
>>>> need to.
>>>>
>>>> Thanks!
>>>> Ryan
>>>>
>>>>
>>>> On Thu, Jun 19, 2008 at 2:19 AM, Bruce Snyder <[EMAIL PROTECTED]>
>>>> wrote:
>>>>
>>>>> On Wed, Jun 18, 2008 at 7:56 PM, Ryan Moquin <[EMAIL PROTECTED]>
>>>>> wrote:
>>>>> > I'm using Servicemix 3.2.1, so I'll give 3.2.2 a try. I was kind of
>>>>> waiting
>>>>> > until it was released, but this problem is now cropping up on a
>>>>> regular
>>>>> > basis on a server so I'll definitely give it a shot. Hopefully this
>>>>> will
>>>>> > allow me to get this project done so I can then get that Joram write
>>>>> up done
>>>>> > since I should have ran across hopefully most of the gotchas for it at
>>>>> that
>>>>> > point.
>>>>>
>>>>> 3.2.2 will be released very soon and I know it's pretty stable. Maybe
>>>>> we can release it this weekend.
>>>>>
>>>>> Bruce
>>>>> --
>>>>> perl -e 'print
>>>>> unpack("u30","D0G)[EMAIL PROTECTED]&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>>>>> );'
>>>>>
>>>>> Apache ActiveMQ - http://activemq.org/
>>>>> Apache Camel - http://activemq.org/camel/
>>>>> Apache ServiceMix - http://servicemix.org/
>>>>>
>>>>> Blog: http://bruceblog.org/
>>>>>
>>>>
>>>>
>>>
>>
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
