Glad you solved your problem ... On Tue, Jun 24, 2008 at 5:10 PM, Ryan Moquin <[EMAIL PROTECTED]> wrote: > Actually, after your conversation.. I went and upped the threads, which > seemed like a logical solution (I didn't tell you only 10% of the CPU was > being used, so it could handle more load). So I just went and doubled the > threads in servicemix for the heck of it, and the problem completely went > away. So your theory on the problem was correct, it was just easier to up > the thread count to correct it. > > Anyhow, anyone else that has servicemix processing lots of messages, with a > similar problem, and has CPU room to spare. That's one solution. :) > > Thanks!!! > Ryan > > On Mon, Jun 23, 2008 at 2:48 PM, Guillaume Nodet <[EMAIL PROTECTED]> wrote: > >> No, I don't think your code is a problem at this point. >> I'm not sure what happens exactly, but the fix I suggested is the only >> solution i'm seeing right now. >> >> On Mon, Jun 23, 2008 at 8:29 PM, Ryan Moquin <[EMAIL PROTECTED]> >> wrote: >> > Oh, so you mean that it's not the bean sending the message to itself, >> it's >> > that the jms consumer is sending back a reply, but not allocating a new >> > thread for it. I am sending synchronously.. so maybe that's why? If >> that's >> > the case.. then it sounds more like I'm simply not handling the reply >> > properly then? >> > >> > On Mon, Jun 23, 2008 at 12:21 PM, Guillaume Nodet <[EMAIL PROTECTED]> >> wrote: >> > >> >> it may be a little more complicated though. If all the thread pool >> >> are exhausted, it could that the jms send back the done status to the >> >> pojo, and everything could be executed in the same thread. Not sure >> >> though ... But this is the only explanation I have so far. >> >> >> >> On Mon, Jun 23, 2008 at 5:52 PM, Ryan Moquin <[EMAIL PROTECTED]> >> >> wrote: >> >> > Ohhhh ok, that was my initial thought, but then I decided it read as >> >> though >> >> > I could actually get it from my bean component without modifiying the >> >> > source. No problem, now that I follow you.. I'll go ahead and try it. >> >> > >> >> > Out of curiosity though, if my bean is sending a message to itself for >> >> some >> >> > reason.. isn't that a sign of a bigger problem? I'm trying to send a >> >> > message to my jms component.... >> >> > >> >> > On Mon, Jun 23, 2008 at 10:59 AM, Guillaume Nodet <[EMAIL PROTECTED]> >> >> wrote: >> >> > >> >> >> Sorry, I was suggesting that you modify the BeanEndpoint, recompile >> >> >> the component and install it instead of the old one. >> >> >> You can compile 3.2.x branch by checking it out from; >> >> >> >> >> http://svn.apache.org/repos/asf/servicemix/smx3/branches/servicemix-3.2 >> >> >> Then run: >> >> >> mvn install -Pstep1 -Dmaven.test.skip=true >> >> >> mvn install -Pstep2 -Dmaven.test.skip=true >> >> >> You may be able to simply hack the component and recompile it without >> >> >> compiling the whole tree: >> >> >> cd deployables/serviceengines/servicemix-bean >> >> >> mvn install >> >> >> >> >> >> On Mon, Jun 23, 2008 at 4:52 PM, Ryan Moquin <[EMAIL PROTECTED] >> > >> >> >> wrote: >> >> >> > I feel dumb asking this, but I've been looking at the source code >> and >> >> I'm >> >> >> > not sure how to test out your suggestion? How am I able to >> retrieve >> >> the >> >> >> > current request? It looks like it's contained in a parent object >> that >> >> >> isn't >> >> >> > accessible to my bean? >> >> >> > >> >> >> > On Mon, Jun 23, 2008 at 6:01 AM, Guillaume Nodet <[EMAIL PROTECTED] >> > >> >> >> wrote: >> >> >> > >> >> >> >> Could you please test something ? >> >> >> >> In the BeanEndpoint, save the old value for the >> currentRequest.get() >> >> >> >> in a local variable before setting it to the new value, and >> instead >> >> of >> >> >> >> setting it to null after processing, set it back to the old value. >> >> It >> >> >> >> may fix the problem ... >> >> >> >> >> >> >> >> On Mon, Jun 23, 2008 at 11:58 AM, Guillaume Nodet < >> [EMAIL PROTECTED]> >> >> >> >> wrote: >> >> >> >> > I'm wondering if for some reason your endpoint sends a request >> to >> >> >> itself >> >> >> >> ? >> >> >> >> > If this request is handled synchronously, it may happen that the >> >> >> >> > currentRequest is set to null while handling this request, but >> in >> >> the >> >> >> >> > same thread as the first request ? >> >> >> >> > Under high load, this behavior could happen if the thread pool >> is >> >> >> >> > exhausted, because under such conditions, the work will be >> executed >> >> in >> >> >> >> > the same thread... >> >> >> >> > >> >> >> >> > On Mon, Jun 23, 2008 at 9:52 AM, Ryan Moquin < >> >> [EMAIL PROTECTED]> >> >> >> >> wrote: >> >> >> >> >> Oh right, because the thread obviously would be the one that >> would >> >> go >> >> >> >> >> through onMessageExchange method and then it would be set to >> null >> >> >> >> after... >> >> >> >> >> (bear with me, long weekend dealing with this problem and it's >> 4am >> >> on >> >> >> a >> >> >> >> >> workday.. haha). >> >> >> >> >> >> >> >> >> >> It's a rather large bean, composed of several classes (it does >> a >> >> lot >> >> >> of >> >> >> >> >> complicated stuff inside the bean)... but I could probably send >> >> you >> >> >> some >> >> >> >> >> pieces of it... do you mind if I email them privately to you? >> >> >> >> >> >> >> >> >> >> On Mon, Jun 23, 2008 at 3:44 AM, Guillaume Nodet < >> >> [EMAIL PROTECTED]> >> >> >> >> wrote: >> >> >> >> >> >> >> >> >> >>> But the current request is only set to null *after* calling >> the >> >> >> pojo. >> >> >> >> >>> Do you have the full code of you bean ? >> >> >> >> >>> >> >> >> >> >>> On Mon, Jun 23, 2008 at 9:31 AM, Ryan Moquin < >> >> >> [EMAIL PROTECTED]> >> >> >> >> >>> wrote: >> >> >> >> >>> > Sure, I backed out the changes in my editor.. basically my >> bean >> >> is >> >> >> >> >>> standard >> >> >> >> >>> > compared to the servicemix docs (for how to handle the >> exchange >> >> >> >> coming >> >> >> >> >>> in), >> >> >> >> >>> > then my bean takes the injected deliverychannel and hands it >> to >> >> a >> >> >> >> class >> >> >> >> >>> that >> >> >> >> >>> > creates a message to send out with results of some analysis >> >> >> >> performed.. >> >> >> >> >>> here >> >> >> >> >>> > is the code to send out (it's using the DeliveryChannel >> >> injected >> >> >> into >> >> >> >> the >> >> >> >> >>> > bean by Servicemix): >> >> >> >> >>> > >> >> >> >> >>> > Marshaller notificationMarshaller = >> >> >> jaxbContext.createMarshaller(); >> >> >> >> >>> > >> >> >> >> >> notificationMarshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, >> >> >> >> >>> > Boolean.TRUE); >> >> >> >> >>> > notificationMarshaller.marshal(notification, writer); >> >> >> >> >>> > if (destinationQname == null) { >> >> >> >> >>> > destinationQname = new >> QName(getDestinationNamespace(), >> >> >> >> >>> > getDestinationService()); >> >> >> >> >>> > } >> >> >> >> >>> > MessageExchangeFactory exchangeFactory = >> >> >> >> >>> > deliveryChannel.createExchangeFactory(); >> >> >> >> >>> > InOnly exchange = >> exchangeFactory.createInOnlyExchange(); >> >> >> >> >>> > NormalizedMessage message = exchange.createMessage(); >> >> >> >> >>> > message.setContent(new >> StringSource(writer.toString())); >> >> >> >> >>> > exchange.setInMessage(message); >> >> >> >> >>> > exchange.setService(destinationQname); >> >> >> >> >>> > deliveryChannel.send(exchange); >> >> >> >> >>> > >> >> >> >> >>> > The send is where the ConcurrentHashmap complains about an >> NPE >> >> .. >> >> >> I >> >> >> >> think >> >> >> >> >>> > because the BeanComponent sets the CurrentRequest to null >> >> around >> >> >> the >> >> >> >> time >> >> >> >> >>> my >> >> >> >> >>> > code is trying to send a request that must get clobbered? >> >> There's >> >> >> no >> >> >> >> >>> other >> >> >> >> >>> > explanation for it.. it looks like the NPE is caused by >> >> >> >> >>> > currentRequest.get(), which is set back to null by the >> >> >> BeanComponent >> >> >> >> and >> >> >> >> >>> > since they are in the same thread.... they interfere. >> >> >> >> >>> > >> >> >> >> >>> > Ryan >> >> >> >> >>> > >> >> >> >> >>> > On Mon, Jun 23, 2008 at 3:24 AM, Guillaume Nodet < >> >> >> [EMAIL PROTECTED]> >> >> >> >> >>> wrote: >> >> >> >> >>> > >> >> >> >> >>> >> Is there any way you could paste the code of the bean >> you're >> >> >> using >> >> >> >> so >> >> >> >> >>> >> that i can have a quick look. You can skip any non-jbi >> related >> >> >> >> >>> >> processing... >> >> >> >> >>> >> >> >> >> >> >>> >> On Mon, Jun 23, 2008 at 9:18 AM, Ryan Moquin < >> >> >> >> [EMAIL PROTECTED]> >> >> >> >> >>> >> wrote: >> >> >> >> >>> >> > No, I'm just simply using the deliverychannel to send >> out, >> >> >> which >> >> >> >> is >> >> >> >> >>> >> > mentioned in the servicemix docs as something that you >> can >> >> do. >> >> >> I >> >> >> >> >>> assume >> >> >> >> >>> >> > that means it's recommended :) I did see that I can >> inject >> >> a >> >> >> >> >>> >> > ComponentContext and so I'm currently changing my >> outbound >> >> >> >> messages to >> >> >> >> >>> >> that >> >> >> >> >>> >> > to see if I can get around this problem. I think I just >> >> can't >> >> >> use >> >> >> >> the >> >> >> >> >>> >> > DeliveryContext in the same Pojo (same thread). >> >> >> >> >>> >> > >> >> >> >> >>> >> > On Mon, Jun 23, 2008 at 3:11 AM, Guillaume Nodet < >> >> >> >> [EMAIL PROTECTED]> >> >> >> >> >>> >> wrote: >> >> >> >> >>> >> > >> >> >> >> >>> >> >> The NPE in the PojoChannel.send() method means that the >> >> >> >> >>> >> >> currentRequest.get() is null. >> >> >> >> >>> >> >> Do you perform any threading operation in the pojo ? If >> >> yes, >> >> >> you >> >> >> >> >>> >> >> should try to remove that, as the component is already >> able >> >> to >> >> >> >> handle >> >> >> >> >>> >> >> concurrent requests. >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> On Sat, Jun 21, 2008 at 6:48 AM, Ryan Moquin < >> >> >> >> [EMAIL PROTECTED] >> >> >> >> >>> > >> >> >> >> >>> >> >> wrote: >> >> >> >> >>> >> >> > I hate to say this Bruce, but unfortunately 3.2.2 is >> >> working >> >> >> >> pretty >> >> >> >> >>> >> bad >> >> >> >> >>> >> >> for >> >> >> >> >>> >> >> > me and has the same problem :( It only seems to >> affect >> >> our >> >> >> >> >>> production >> >> >> >> >>> >> >> > server and in a few minutes after start up, I start >> >> getting >> >> >> the >> >> >> >> >>> >> >> > NullPointerException.. slowly all my services start >> doing >> >> it >> >> >> >> and >> >> >> >> >>> they >> >> >> >> >>> >> all >> >> >> >> >>> >> >> > stop working. If I stop and restart servicemix, then >> >> >> >> >>> servicemix-jms >> >> >> >> >>> >> >> > components are no longer routable. I'm guessing this >> >> error >> >> >> >> causes >> >> >> >> >>> >> >> > Servicemix to shut them down and not deploy them >> anymore >> >> >> >> >>> (Servicemix >> >> >> >> >>> >> does >> >> >> >> >>> >> >> > the same thing if a service unit starts up, tries to >> make >> >> a >> >> >> >> Joram >> >> >> >> >>> >> >> connection >> >> >> >> >>> >> >> > to a server that isn't up, servicemix will shutdown >> that >> >> SU >> >> >> and >> >> >> >> >>> will >> >> >> >> >>> >> >> always >> >> >> >> >>> >> >> > shut it down immediately after it starts on any >> >> subsequent >> >> >> >> run). >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > It seems like this problem must be related to a race >> >> >> condition. >> >> >> >> >>> When >> >> >> >> >>> >> >> doing >> >> >> >> >>> >> >> > development testing, I never see this problem on my >> >> laptop >> >> >> even >> >> >> >> >>> under >> >> >> >> >>> >> >> high >> >> >> >> >>> >> >> > load. One our fast test server, I see this error >> popup >> >> once >> >> >> on >> >> >> >> >>> >> startup >> >> >> >> >>> >> >> and >> >> >> >> >>> >> >> > then it doesn't seem to happen again. On our even >> faster >> >> >> >> >>> production >> >> >> >> >>> >> >> server, >> >> >> >> >>> >> >> > the whole thing looses it's wheels and falls apart >> after >> >> a >> >> >> few >> >> >> >> >>> >> minutes. >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > I'm supposed to be deploying this system in a few days >> >> and >> >> >> of >> >> >> >> >>> course >> >> >> >> >>> >> >> that's >> >> >> >> >>> >> >> > the only spot where I can't temporarily limp by. Is >> it >> >> >> >> possible >> >> >> >> >>> you >> >> >> >> >>> >> >> could >> >> >> >> >>> >> >> > give me some hints on what the problem is and I'll >> debug >> >> it >> >> >> >> this >> >> >> >> >>> >> weekend >> >> >> >> >>> >> >> to >> >> >> >> >>> >> >> > see if I can fix it, or at least patch it temporarily? >> I >> >> >> >> really >> >> >> >> >>> need >> >> >> >> >>> >> to >> >> >> >> >>> >> >> > figure out way to get around this problem. Other than >> >> that, >> >> >> >> 3.2.2 >> >> >> >> >>> >> seems >> >> >> >> >>> >> >> to >> >> >> >> >>> >> >> > work perfectly fine. >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Here is the caused by error again in case it's any bit >> >> >> >> different >> >> >> >> >>> than >> >> >> >> >>> >> the >> >> >> >> >>> >> >> > 3.2.1 one was: >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Caused by: java.lang.NullPointerException >> >> >> >> >>> >> >> > at >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> >> >> java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:881) >> >> >> >> >>> >> >> > at >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> org.apache.servicemix.bean.BeanEndpoint$PojoChannel.send(BeanEndpoint.java:569) >> >> >> >> >>> >> >> > at >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >> >> >> >> >> com.notification.impl.JbiNotificationHandlerImpl.sendNotification(JbiNotificat >> >> >> >> >>> >> >> > ionHandlerImpl.java:80) >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Also, is the delivery channel component threadsafe? >> I'm >> >> >> >> curious if >> >> >> >> >>> >> >> multiple >> >> >> >> >>> >> >> > threads accessing it is a problem or if I should keep >> >> access >> >> >> to >> >> >> >> it >> >> >> >> >>> >> >> > synchronized? I'm currently synchronizing, but don't >> >> want >> >> >> to >> >> >> >> if I >> >> >> >> >>> >> don't >> >> >> >> >>> >> >> > need to. >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Thanks! >> >> >> >> >>> >> >> > Ryan >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > On Thu, Jun 19, 2008 at 2:19 AM, Bruce Snyder < >> >> >> >> >>> [EMAIL PROTECTED] >> >> >> >> >>> >> > >> >> >> >> >>> >> >> > wrote: >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> >> On Wed, Jun 18, 2008 at 7:56 PM, Ryan Moquin < >> >> >> >> >>> [EMAIL PROTECTED] >> >> >> >> >>> >> > >> >> >> >> >>> >> >> >> wrote: >> >> >> >> >>> >> >> >> > I'm using Servicemix 3.2.1, so I'll give 3.2.2 a >> try. >> >> I >> >> >> was >> >> >> >> >>> kind >> >> >> >> >>> >> of >> >> >> >> >>> >> >> >> waiting >> >> >> >> >>> >> >> >> > until it was released, but this problem is now >> >> cropping >> >> >> up >> >> >> >> on a >> >> >> >> >>> >> >> regular >> >> >> >> >>> >> >> >> > basis on a server so I'll definitely give it a >> shot. >> >> >> >> Hopefully >> >> >> >> >>> >> this >> >> >> >> >>> >> >> will >> >> >> >> >>> >> >> >> > allow me to get this project done so I can then get >> >> that >> >> >> >> Joram >> >> >> >> >>> >> write >> >> >> >> >>> >> >> up >> >> >> >> >>> >> >> >> done >> >> >> >> >>> >> >> >> > since I should have ran across hopefully most of >> the >> >> >> gotchas >> >> >> >> for >> >> >> >> >>> it >> >> >> >> >>> >> at >> >> >> >> >>> >> >> >> that >> >> >> >> >>> >> >> >> > point. >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> >> 3.2.2 will be released very soon and I know it's >> pretty >> >> >> >> stable. >> >> >> >> >>> Maybe >> >> >> >> >>> >> >> >> we can release it this weekend. >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> >> Bruce >> >> >> >> >>> >> >> >> -- >> >> >> >> >>> >> >> >> perl -e 'print >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> unpack("u30","D0G)[EMAIL PROTECTED]&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*" >> >> >> >> >>> >> >> >> );' >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> >> Apache ActiveMQ - http://activemq.org/ >> >> >> >> >>> >> >> >> Apache Camel - http://activemq.org/camel/ >> >> >> >> >>> >> >> >> Apache ServiceMix - http://servicemix.org/ >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> >> Blog: http://bruceblog.org/ >> >> >> >> >>> >> >> >> >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> -- >> >> >> >> >>> >> >> Cheers, >> >> >> >> >>> >> >> Guillaume Nodet >> >> >> >> >>> >> >> ------------------------ >> >> >> >> >>> >> >> Blog: http://gnodet.blogspot.com/ >> >> >> >> >>> >> >> >> >> >> >> >>> >> > >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> >> >> >> >> >>> >> -- >> >> >> >> >>> >> Cheers, >> >> >> >> >>> >> Guillaume Nodet >> >> >> >> >>> >> ------------------------ >> >> >> >> >>> >> Blog: http://gnodet.blogspot.com/ >> >> >> >> >>> >> >> >> >> >> >>> > >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> >>> -- >> >> >> >> >>> Cheers, >> >> >> >> >>> Guillaume Nodet >> >> >> >> >>> ------------------------ >> >> >> >> >>> Blog: http://gnodet.blogspot.com/ >> >> >> >> >>> >> >> >> >> >> >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > -- >> >> >> >> > Cheers, >> >> >> >> > Guillaume Nodet >> >> >> >> > ------------------------ >> >> >> >> > Blog: http://gnodet.blogspot.com/ >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Cheers, >> >> >> >> Guillaume Nodet >> >> >> >> ------------------------ >> >> >> >> Blog: http://gnodet.blogspot.com/ >> >> >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Cheers, >> >> >> Guillaume Nodet >> >> >> ------------------------ >> >> >> Blog: http://gnodet.blogspot.com/ >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> Cheers, >> >> Guillaume Nodet >> >> ------------------------ >> >> Blog: http://gnodet.blogspot.com/ >> >> >> > >> >> >> >> -- >> Cheers, >> Guillaume Nodet >> ------------------------ >> Blog: http://gnodet.blogspot.com/ >> >
-- Cheers, Guillaume Nodet ------------------------ Blog: http://gnodet.blogspot.com/
