I went ahead and opened a ticket yesterday and provided a patch: https://issues.apache.org/jira/browse/CXF-5078.
I'm not sure what the development cycle for CXF looks like, but do you think chances are good that this patch could be included in the next bunch of releases? If so I will wait for the next release, but if not, I will probably need to provide a build to our customers which includes this functionality as a new WSDLGetInterceptor class which places itself before the existing one in the interceptor chain. At the very least I was hoping one of the CXF devs could have a look at it to make sure I'm not doing something stupid. --John On Jun 7, 2013, at 2:55 PM, Daniel Kulp <[email protected]> wrote: > > On Jun 7, 2013, at 2:01 PM, John Bellassai <[email protected]> wrote: >> Hi Daniel. I was under the impression that a new Document object is >> generated and only accessible within each invocation of handleMessage() and >> thus would not be susceptible to thread-safety issues, but I'll take your >> advice over my very limited understanding of the inner workings of CXF ;). > > It SHOULD be caching the documents and only creating a new one if there is a > required change (like the URL). I think. > >> I like your idea of writing it to a CachedOutputStream inside the lock. I >> will give that a try and look into submitting a patch. > > Sounds good! > Dan > > >> >> Thanks again! >> >> --John >> >> >> On Jun 7, 2013, at 12:21 PM, Daniel Kulp <[email protected]> wrote: >> >>> >>> On Jun 7, 2013, at 12:30 PM, John Bellassai <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> We've been seeing an issue in production for the past few months where >>>> after running smoothly for a couple weeks, our application hangs and stops >>>> responding to new requests until we bounce the container (Tomcat 7/CXF >>>> 2.5.3). >>>> >>>> Some thread/heap dump analysis for a few of these hang events have shown a >>>> running theme. It seems that all of Tomcat's HTTPS handler threads except >>>> one are waiting on a specific lock to become available. The problem is in >>>> the org.apache.cxf.frontend.WSDLGetInterceptor class, specifically in the >>>> synchronized block in the handleMessage method. >>>> >>>> What we are experiencing seems to not technically be a deadlock because >>>> one thread is legitimately holding the lock and is still runnable and >>>> writing the WSDL to the client, but for some reason (network issues >>>> perhaps), it is not making very quick progress while other threads >>>> continue to pile up. Eventually all of Tomcat's handler threads are in >>>> use and are waiting for this lock so from the outside, the server does not >>>> respond to new requests. >>>> >>>> In examining the code for this interceptor I wonder if the synchronized >>>> block needs to be as coarse as it is currently. In other words, would it >>>> be possible to lock only while creating the WSDL Document object, then >>>> actually write to the XMLStreamWriter outside of the synchronized block >>>> such that all threads can make progress even if one client happens to be >>>> slow or experiencing network issues? >>> >>> I don't think that would be possible for two reasons: >>> >>> 1) Another thread could then modify the document while it's being written >>> out. I'm not exactly sure what would happen in that case. >>> >>> 2) Traversing a DOM object is also not thread safe: >>> http://xerces.apache.org/xerces2-j/faq-dom.html#faq-1 >>> Thus, you don't want 2 threads traversing it at the same time. >>> >>> One potential fix that might work would be to write it to a >>> CachedOutputStream within the lock. That should be fairly fast. Then >>> outside the lock, write that to the network stream. Would you care to >>> give that a try and maybe submit a patch? >>> >>> >>> -- >>> Daniel Kulp >>> [email protected] - http://dankulp.com/blog >>> Talend Community Coder - http://coders.talend.com >>> >> > > -- > Daniel Kulp > [email protected] - http://dankulp.com/blog > Talend Community Coder - http://coders.talend.com >
