On Jun 7, 2013, at 2:01 PM, John Bellassai <[email protected]> wrote: > Hi Daniel. I was under the impression that a new Document object is > generated and only accessible within each invocation of handleMessage() and > thus would not be susceptible to thread-safety issues, but I'll take your > advice over my very limited understanding of the inner workings of CXF ;).
It SHOULD be caching the documents and only creating a new one if there is a required change (like the URL). I think. > I like your idea of writing it to a CachedOutputStream inside the lock. I > will give that a try and look into submitting a patch. Sounds good! Dan > > Thanks again! > > --John > > > On Jun 7, 2013, at 12:21 PM, Daniel Kulp <[email protected]> wrote: > >> >> On Jun 7, 2013, at 12:30 PM, John Bellassai <[email protected]> wrote: >> >>> Hi, >>> >>> We've been seeing an issue in production for the past few months where >>> after running smoothly for a couple weeks, our application hangs and stops >>> responding to new requests until we bounce the container (Tomcat 7/CXF >>> 2.5.3). >>> >>> Some thread/heap dump analysis for a few of these hang events have shown a >>> running theme. It seems that all of Tomcat's HTTPS handler threads except >>> one are waiting on a specific lock to become available. The problem is in >>> the org.apache.cxf.frontend.WSDLGetInterceptor class, specifically in the >>> synchronized block in the handleMessage method. >>> >>> What we are experiencing seems to not technically be a deadlock because one >>> thread is legitimately holding the lock and is still runnable and writing >>> the WSDL to the client, but for some reason (network issues perhaps), it is >>> not making very quick progress while other threads continue to pile up. >>> Eventually all of Tomcat's handler threads are in use and are waiting for >>> this lock so from the outside, the server does not respond to new requests. >>> >>> In examining the code for this interceptor I wonder if the synchronized >>> block needs to be as coarse as it is currently. In other words, would it >>> be possible to lock only while creating the WSDL Document object, then >>> actually write to the XMLStreamWriter outside of the synchronized block >>> such that all threads can make progress even if one client happens to be >>> slow or experiencing network issues? >> >> I don't think that would be possible for two reasons: >> >> 1) Another thread could then modify the document while it's being written >> out. I'm not exactly sure what would happen in that case. >> >> 2) Traversing a DOM object is also not thread safe: >> http://xerces.apache.org/xerces2-j/faq-dom.html#faq-1 >> Thus, you don't want 2 threads traversing it at the same time. >> >> One potential fix that might work would be to write it to a >> CachedOutputStream within the lock. That should be fairly fast. Then >> outside the lock, write that to the network stream. Would you care to >> give that a try and maybe submit a patch? >> >> >> -- >> Daniel Kulp >> [email protected] - http://dankulp.com/blog >> Talend Community Coder - http://coders.talend.com >> > -- Daniel Kulp [email protected] - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
