Actually, after your conversation.. I went and upped the threads, which seemed like a logical solution (I didn't tell you only 10% of the CPU was being used, so it could handle more load). So I just went and doubled the threads in servicemix for the heck of it, and the problem completely went away. So your theory on the problem was correct, it was just easier to up the thread count to correct it.
Anyhow, anyone else that has servicemix processing lots of messages, with a similar problem, and has CPU room to spare. That's one solution. :) Thanks!!! Ryan On Mon, Jun 23, 2008 at 2:48 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: > No, I don't think your code is a problem at this point. > I'm not sure what happens exactly, but the fix I suggested is the only > solution i'm seeing right now. > > On Mon, Jun 23, 2008 at 8:29 PM, Ryan Moquin <[EMAIL PROTECTED]> > wrote: > > Oh, so you mean that it's not the bean sending the message to itself, > it's > > that the jms consumer is sending back a reply, but not allocating a new > > thread for it. I am sending synchronously.. so maybe that's why? If > that's > > the case.. then it sounds more like I'm simply not handling the reply > > properly then? > > > > On Mon, Jun 23, 2008 at 12:21 PM, Guillaume Nodet <[EMAIL PROTECTED]> > wrote: > > > >> it may be a little more complicated though. If all the thread pool > >> are exhausted, it could that the jms send back the done status to the > >> pojo, and everything could be executed in the same thread. Not sure > >> though ... But this is the only explanation I have so far. > >> > >> On Mon, Jun 23, 2008 at 5:52 PM, Ryan Moquin <[EMAIL PROTECTED]> > >> wrote: > >> > Ohhhh ok, that was my initial thought, but then I decided it read as > >> though > >> > I could actually get it from my bean component without modifiying the > >> > source. No problem, now that I follow you.. I'll go ahead and try it. > >> > > >> > Out of curiosity though, if my bean is sending a message to itself for > >> some > >> > reason.. isn't that a sign of a bigger problem? I'm trying to send a > >> > message to my jms component.... > >> > > >> > On Mon, Jun 23, 2008 at 10:59 AM, Guillaume Nodet <[EMAIL PROTECTED]> > >> wrote: > >> > > >> >> Sorry, I was suggesting that you modify the BeanEndpoint, recompile > >> >> the component and install it instead of the old one. > >> >> You can compile 3.2.x branch by checking it out from; > >> >> > >> http://svn.apache.org/repos/asf/servicemix/smx3/branches/servicemix-3.2 > >> >> Then run: > >> >> mvn install -Pstep1 -Dmaven.test.skip=true > >> >> mvn install -Pstep2 -Dmaven.test.skip=true > >> >> You may be able to simply hack the component and recompile it without > >> >> compiling the whole tree: > >> >> cd deployables/serviceengines/servicemix-bean > >> >> mvn install > >> >> > >> >> On Mon, Jun 23, 2008 at 4:52 PM, Ryan Moquin <[EMAIL PROTECTED] > > > >> >> wrote: > >> >> > I feel dumb asking this, but I've been looking at the source code > and > >> I'm > >> >> > not sure how to test out your suggestion? How am I able to > retrieve > >> the > >> >> > current request? It looks like it's contained in a parent object > that > >> >> isn't > >> >> > accessible to my bean? > >> >> > > >> >> > On Mon, Jun 23, 2008 at 6:01 AM, Guillaume Nodet <[EMAIL PROTECTED] > > > >> >> wrote: > >> >> > > >> >> >> Could you please test something ? > >> >> >> In the BeanEndpoint, save the old value for the > currentRequest.get() > >> >> >> in a local variable before setting it to the new value, and > instead > >> of > >> >> >> setting it to null after processing, set it back to the old value. > >> It > >> >> >> may fix the problem ... > >> >> >> > >> >> >> On Mon, Jun 23, 2008 at 11:58 AM, Guillaume Nodet < > [EMAIL PROTECTED]> > >> >> >> wrote: > >> >> >> > I'm wondering if for some reason your endpoint sends a request > to > >> >> itself > >> >> >> ? > >> >> >> > If this request is handled synchronously, it may happen that the > >> >> >> > currentRequest is set to null while handling this request, but > in > >> the > >> >> >> > same thread as the first request ? > >> >> >> > Under high load, this behavior could happen if the thread pool > is > >> >> >> > exhausted, because under such conditions, the work will be > executed > >> in > >> >> >> > the same thread... > >> >> >> > > >> >> >> > On Mon, Jun 23, 2008 at 9:52 AM, Ryan Moquin < > >> [EMAIL PROTECTED]> > >> >> >> wrote: > >> >> >> >> Oh right, because the thread obviously would be the one that > would > >> go > >> >> >> >> through onMessageExchange method and then it would be set to > null > >> >> >> after... > >> >> >> >> (bear with me, long weekend dealing with this problem and it's > 4am > >> on > >> >> a > >> >> >> >> workday.. haha). > >> >> >> >> > >> >> >> >> It's a rather large bean, composed of several classes (it does > a > >> lot > >> >> of > >> >> >> >> complicated stuff inside the bean)... but I could probably send > >> you > >> >> some > >> >> >> >> pieces of it... do you mind if I email them privately to you? > >> >> >> >> > >> >> >> >> On Mon, Jun 23, 2008 at 3:44 AM, Guillaume Nodet < > >> [EMAIL PROTECTED]> > >> >> >> wrote: > >> >> >> >> > >> >> >> >>> But the current request is only set to null *after* calling > the > >> >> pojo. > >> >> >> >>> Do you have the full code of you bean ? > >> >> >> >>> > >> >> >> >>> On Mon, Jun 23, 2008 at 9:31 AM, Ryan Moquin < > >> >> [EMAIL PROTECTED]> > >> >> >> >>> wrote: > >> >> >> >>> > Sure, I backed out the changes in my editor.. basically my > bean > >> is > >> >> >> >>> standard > >> >> >> >>> > compared to the servicemix docs (for how to handle the > exchange > >> >> >> coming > >> >> >> >>> in), > >> >> >> >>> > then my bean takes the injected deliverychannel and hands it > to > >> a > >> >> >> class > >> >> >> >>> that > >> >> >> >>> > creates a message to send out with results of some analysis > >> >> >> performed.. > >> >> >> >>> here > >> >> >> >>> > is the code to send out (it's using the DeliveryChannel > >> injected > >> >> into > >> >> >> the > >> >> >> >>> > bean by Servicemix): > >> >> >> >>> > > >> >> >> >>> > Marshaller notificationMarshaller = > >> >> jaxbContext.createMarshaller(); > >> >> >> >>> > > >> >> >> > notificationMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, > >> >> >> >>> > Boolean.TRUE); > >> >> >> >>> > notificationMarshaller.marshal(notification, writer); > >> >> >> >>> > if (destinationQname == null) { > >> >> >> >>> > destinationQname = new > QName(getDestinationNamespace(), > >> >> >> >>> > getDestinationService()); > >> >> >> >>> > } > >> >> >> >>> > MessageExchangeFactory exchangeFactory = > >> >> >> >>> > deliveryChannel.createExchangeFactory(); > >> >> >> >>> > InOnly exchange = > exchangeFactory.createInOnlyExchange(); > >> >> >> >>> > NormalizedMessage message = exchange.createMessage(); > >> >> >> >>> > message.setContent(new > StringSource(writer.toString())); > >> >> >> >>> > exchange.setInMessage(message); > >> >> >> >>> > exchange.setService(destinationQname); > >> >> >> >>> > deliveryChannel.send(exchange); > >> >> >> >>> > > >> >> >> >>> > The send is where the ConcurrentHashmap complains about an > NPE > >> .. > >> >> I > >> >> >> think > >> >> >> >>> > because the BeanComponent sets the CurrentRequest to null > >> around > >> >> the > >> >> >> time > >> >> >> >>> my > >> >> >> >>> > code is trying to send a request that must get clobbered? > >> There's > >> >> no > >> >> >> >>> other > >> >> >> >>> > explanation for it.. it looks like the NPE is caused by > >> >> >> >>> > currentRequest.get(), which is set back to null by the > >> >> BeanComponent > >> >> >> and > >> >> >> >>> > since they are in the same thread.... they interfere. > >> >> >> >>> > > >> >> >> >>> > Ryan > >> >> >> >>> > > >> >> >> >>> > On Mon, Jun 23, 2008 at 3:24 AM, Guillaume Nodet < > >> >> [EMAIL PROTECTED]> > >> >> >> >>> wrote: > >> >> >> >>> > > >> >> >> >>> >> Is there any way you could paste the code of the bean > you're > >> >> using > >> >> >> so > >> >> >> >>> >> that i can have a quick look. You can skip any non-jbi > related > >> >> >> >>> >> processing... > >> >> >> >>> >> > >> >> >> >>> >> On Mon, Jun 23, 2008 at 9:18 AM, Ryan Moquin < > >> >> >> [EMAIL PROTECTED]> > >> >> >> >>> >> wrote: > >> >> >> >>> >> > No, I'm just simply using the deliverychannel to send > out, > >> >> which > >> >> >> is > >> >> >> >>> >> > mentioned in the servicemix docs as something that you > can > >> do. > >> >> I > >> >> >> >>> assume > >> >> >> >>> >> > that means it's recommended :) I did see that I can > inject > >> a > >> >> >> >>> >> > ComponentContext and so I'm currently changing my > outbound > >> >> >> messages to > >> >> >> >>> >> that > >> >> >> >>> >> > to see if I can get around this problem. I think I just > >> can't > >> >> use > >> >> >> the > >> >> >> >>> >> > DeliveryContext in the same Pojo (same thread). > >> >> >> >>> >> > > >> >> >> >>> >> > On Mon, Jun 23, 2008 at 3:11 AM, Guillaume Nodet < > >> >> >> [EMAIL PROTECTED]> > >> >> >> >>> >> wrote: > >> >> >> >>> >> > > >> >> >> >>> >> >> The NPE in the PojoChannel.send() method means that the > >> >> >> >>> >> >> currentRequest.get() is null. > >> >> >> >>> >> >> Do you perform any threading operation in the pojo ? If > >> yes, > >> >> you > >> >> >> >>> >> >> should try to remove that, as the component is already > able > >> to > >> >> >> handle > >> >> >> >>> >> >> concurrent requests. > >> >> >> >>> >> >> > >> >> >> >>> >> >> On Sat, Jun 21, 2008 at 6:48 AM, Ryan Moquin < > >> >> >> [EMAIL PROTECTED] > >> >> >> >>> > > >> >> >> >>> >> >> wrote: > >> >> >> >>> >> >> > I hate to say this Bruce, but unfortunately 3.2.2 is > >> working > >> >> >> pretty > >> >> >> >>> >> bad > >> >> >> >>> >> >> for > >> >> >> >>> >> >> > me and has the same problem :( It only seems to > affect > >> our > >> >> >> >>> production > >> >> >> >>> >> >> > server and in a few minutes after start up, I start > >> getting > >> >> the > >> >> >> >>> >> >> > NullPointerException.. slowly all my services start > doing > >> it > >> >> >> and > >> >> >> >>> they > >> >> >> >>> >> all > >> >> >> >>> >> >> > stop working. If I stop and restart servicemix, then > >> >> >> >>> servicemix-jms > >> >> >> >>> >> >> > components are no longer routable. I'm guessing this > >> error > >> >> >> causes > >> >> >> >>> >> >> > Servicemix to shut them down and not deploy them > anymore > >> >> >> >>> (Servicemix > >> >> >> >>> >> does > >> >> >> >>> >> >> > the same thing if a service unit starts up, tries to > make > >> a > >> >> >> Joram > >> >> >> >>> >> >> connection > >> >> >> >>> >> >> > to a server that isn't up, servicemix will shutdown > that > >> SU > >> >> and > >> >> >> >>> will > >> >> >> >>> >> >> always > >> >> >> >>> >> >> > shut it down immediately after it starts on any > >> subsequent > >> >> >> run). > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > It seems like this problem must be related to a race > >> >> condition. > >> >> >> >>> When > >> >> >> >>> >> >> doing > >> >> >> >>> >> >> > development testing, I never see this problem on my > >> laptop > >> >> even > >> >> >> >>> under > >> >> >> >>> >> >> high > >> >> >> >>> >> >> > load. One our fast test server, I see this error > popup > >> once > >> >> on > >> >> >> >>> >> startup > >> >> >> >>> >> >> and > >> >> >> >>> >> >> > then it doesn't seem to happen again. On our even > faster > >> >> >> >>> production > >> >> >> >>> >> >> server, > >> >> >> >>> >> >> > the whole thing looses it's wheels and falls apart > after > >> a > >> >> few > >> >> >> >>> >> minutes. > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > I'm supposed to be deploying this system in a few days > >> and > >> >> of > >> >> >> >>> course > >> >> >> >>> >> >> that's > >> >> >> >>> >> >> > the only spot where I can't temporarily limp by. Is > it > >> >> >> possible > >> >> >> >>> you > >> >> >> >>> >> >> could > >> >> >> >>> >> >> > give me some hints on what the problem is and I'll > debug > >> it > >> >> >> this > >> >> >> >>> >> weekend > >> >> >> >>> >> >> to > >> >> >> >>> >> >> > see if I can fix it, or at least patch it temporarily? > I > >> >> >> really > >> >> >> >>> need > >> >> >> >>> >> to > >> >> >> >>> >> >> > figure out way to get around this problem. Other than > >> that, > >> >> >> 3.2.2 > >> >> >> >>> >> seems > >> >> >> >>> >> >> to > >> >> >> >>> >> >> > work perfectly fine. > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > Here is the caused by error again in case it's any bit > >> >> >> different > >> >> >> >>> than > >> >> >> >>> >> the > >> >> >> >>> >> >> > 3.2.1 one was: > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > Caused by: java.lang.NullPointerException > >> >> >> >>> >> >> > at > >> >> >> >>> >> >> > > >> >> >> >>> > >> >> > java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881) > >> >> >> >>> >> >> > at > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > >> >> >> >>> >> > >> >> >> >>> > >> >> >> > >> >> > >> > org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569) > >> >> >> >>> >> >> > at > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > >> >> >> >>> >> > >> >> >> >>> > >> >> >> > >> >> > >> > com.notification.impl.JbiNotificationHandlerImpl.sendNotification(JbiNotificat > >> >> >> >>> >> >> > ionHandlerImpl.java:80) > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > Also, is the delivery channel component threadsafe? > I'm > >> >> >> curious if > >> >> >> >>> >> >> multiple > >> >> >> >>> >> >> > threads accessing it is a problem or if I should keep > >> access > >> >> to > >> >> >> it > >> >> >> >>> >> >> > synchronized? I'm currently synchronizing, but don't > >> want > >> >> to > >> >> >> if I > >> >> >> >>> >> don't > >> >> >> >>> >> >> > need to. > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > Thanks! > >> >> >> >>> >> >> > Ryan > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > On Thu, Jun 19, 2008 at 2:19 AM, Bruce Snyder < > >> >> >> >>> [EMAIL PROTECTED] > >> >> >> >>> >> > > >> >> >> >>> >> >> > wrote: > >> >> >> >>> >> >> > > >> >> >> >>> >> >> >> On Wed, Jun 18, 2008 at 7:56 PM, Ryan Moquin < > >> >> >> >>> [EMAIL PROTECTED] > >> >> >> >>> >> > > >> >> >> >>> >> >> >> wrote: > >> >> >> >>> >> >> >> > I'm using Servicemix 3.2.1, so I'll give 3.2.2 a > try. > >> I > >> >> was > >> >> >> >>> kind > >> >> >> >>> >> of > >> >> >> >>> >> >> >> waiting > >> >> >> >>> >> >> >> > until it was released, but this problem is now > >> cropping > >> >> up > >> >> >> on a > >> >> >> >>> >> >> regular > >> >> >> >>> >> >> >> > basis on a server so I'll definitely give it a > shot. > >> >> >> Hopefully > >> >> >> >>> >> this > >> >> >> >>> >> >> will > >> >> >> >>> >> >> >> > allow me to get this project done so I can then get > >> that > >> >> >> Joram > >> >> >> >>> >> write > >> >> >> >>> >> >> up > >> >> >> >>> >> >> >> done > >> >> >> >>> >> >> >> > since I should have ran across hopefully most of > the > >> >> gotchas > >> >> >> for > >> >> >> >>> it > >> >> >> >>> >> at > >> >> >> >>> >> >> >> that > >> >> >> >>> >> >> >> > point. > >> >> >> >>> >> >> >> > >> >> >> >>> >> >> >> 3.2.2 will be released very soon and I know it's > pretty > >> >> >> stable. > >> >> >> >>> Maybe > >> >> >> >>> >> >> >> we can release it this weekend. > >> >> >> >>> >> >> >> > >> >> >> >>> >> >> >> Bruce > >> >> >> >>> >> >> >> -- > >> >> >> >>> >> >> >> perl -e 'print > >> >> >> >>> >> >> >> > >> >> >> >>> > >> unpack("u30","D0G)[EMAIL PROTECTED]&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*" > >> >> >> >>> >> >> >> );' > >> >> >> >>> >> >> >> > >> >> >> >>> >> >> >> Apache ActiveMQ - http://activemq.org/ > >> >> >> >>> >> >> >> Apache Camel - http://activemq.org/camel/ > >> >> >> >>> >> >> >> Apache ServiceMix - http://servicemix.org/ > >> >> >> >>> >> >> >> > >> >> >> >>> >> >> >> Blog: http://bruceblog.org/ > >> >> >> >>> >> >> >> > >> >> >> >>> >> >> > > >> >> >> >>> >> >> > >> >> >> >>> >> >> > >> >> >> >>> >> >> > >> >> >> >>> >> >> -- > >> >> >> >>> >> >> Cheers, > >> >> >> >>> >> >> Guillaume Nodet > >> >> >> >>> >> >> ------------------------ > >> >> >> >>> >> >> Blog: http://gnodet.blogspot.com/ > >> >> >> >>> >> >> > >> >> >> >>> >> > > >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> > >> >> >> >>> >> -- > >> >> >> >>> >> Cheers, > >> >> >> >>> >> Guillaume Nodet > >> >> >> >>> >> ------------------------ > >> >> >> >>> >> Blog: http://gnodet.blogspot.com/ > >> >> >> >>> >> > >> >> >> >>> > > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> > >> >> >> >>> -- > >> >> >> >>> Cheers, > >> >> >> >>> Guillaume Nodet > >> >> >> >>> ------------------------ > >> >> >> >>> Blog: http://gnodet.blogspot.com/ > >> >> >> >>> > >> >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > Cheers, > >> >> >> > Guillaume Nodet > >> >> >> > ------------------------ > >> >> >> > Blog: http://gnodet.blogspot.com/ > >> >> >> > > >> >> >> > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> Cheers, > >> >> >> Guillaume Nodet > >> >> >> ------------------------ > >> >> >> Blog: http://gnodet.blogspot.com/ > >> >> >> > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Cheers, > >> >> Guillaume Nodet > >> >> ------------------------ > >> >> Blog: http://gnodet.blogspot.com/ > >> >> > >> > > >> > >> > >> > >> -- > >> Cheers, > >> Guillaume Nodet > >> ------------------------ > >> Blog: http://gnodet.blogspot.com/ > >> > > > > > > -- > Cheers, > Guillaume Nodet > ------------------------ > Blog: http://gnodet.blogspot.com/ >
