Can you move up to 2.3.5? We did add some synchronized blocks around the ReflectionServiceFactoryBean to make the singleton cases work a lot better. That may fix this issue.
Dan On Monday, July 25, 2011 12:35:11 PM Algirdas Veitas wrote: > More information....after doing some additional testing, it looks like the > exception is thrown in different areas of the CXF code, but the > ConcurrentModificationException is thrown when trying to manipulate the > > private List<AbstractServiceConfiguration> serviceConfigurations = > new ArrayList<AbstractServiceConfiguration>(); > > that belongs to ReflectionServiceFactoryBean....see stack traces below for > additional samples: > > Caused by: java.util.ConcurrentModificationException > at > java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) > at java.util.AbstractList$Itr.next(AbstractList.java:343) > at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createInputWrapp > edMessageParts(ReflectionServiceFactoryBean.java:1619) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createOperation( > ReflectionServiceFactoryBean.java:934) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.createOperation(JaxWsSe > rviceFactoryBean.java:616) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createInterface( > ReflectionServiceFactoryBean.java:907) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServiceFrom > Class(ReflectionServiceFactoryBean.java:433) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(J > axWsServiceFactoryBean.java:680) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServic > eModel(ReflectionServiceFactoryBean.java:505) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(Reflectio > nServiceFactoryBean.java:242) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFact > oryBean.java:202) at > org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(Abst > ractWSDLBasedEndpointFactory.java:101) at > org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:90) > at > org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean > .java:117 > > Caused by: java.util.ConcurrentModificationException > at > java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) > at java.util.AbstractList$Itr.next(AbstractList.java:343) > at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.addFault(Reflect > ionServiceFactoryBean.java:2032) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeFaults > (ReflectionServiceFactoryBean.java:2011) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createMessagePar > ts(ReflectionServiceFactoryBean.java:1575) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createOperation( > ReflectionServiceFactoryBean.java:928) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.createOperation(JaxWsSe > rviceFactoryBean.java:616) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createInterface( > ReflectionServiceFactoryBean.java:907) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServiceFrom > Class(ReflectionServiceFactoryBean.java:433) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(J > axWsServiceFactoryBean.java:680) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initializeServic > eModel(ReflectionServiceFactoryBean.java:505) at > org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(Reflectio > nServiceFactoryBean.java:242) at > org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFact > oryBean.java:202) at > org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(Abst > ractWSDLBasedEndpointFactory.java:101) at > org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:90) > at > org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean > .java:117) at > org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java > :124) > > On Mon, Jul 25, 2011 at 11:30 AM, Algirdas Veitas <[email protected]>wrote: > > whoops...in the previous post...we mentioned 2.1.3 but we are using > > 2.3.1 > > > > On Mon, Jul 25, 2011 at 10:36 AM, Algirdas Veitas <[email protected]>wrote: > >> Hi Daniel, > >> > >> Unfortunately, we are unable to reproduce this problem in an isolated > >> environment. But at this point, it does look like to be an issue with > >> multiple threads using the same instance of the client. Because we > >> are > >> unable to reproduce, our current strategy is to create a separate > >> client per SOAP call. > >> > >> Using the following documentation as a guideline > >> http://cxf.apache.org/docs/jax-ws-configuration.html, we came up with > >> the following: > >> > >> Here was the original configuration for the client: > >> > >> > >> <jaxws:client id="xService" serviceClass="com.XService" > >> address="#xServiceUrl" > >> </jaxws:client> > >> > >> Here is what we just tried: > >> > >> <bean id="xServiceProxyFactory" > >> > >> class="org.apache.cxf.jaxws. > >> > >> JaxWsProxyFactoryBean"> > >> > >> <property name="serviceClass" value="com.XService"/> > >> <property name="address" ref="xServiceUrl"/> > >> > >> </bean> > >> > >> <bean id="xService" scope="prototype" class="com.XService" > >> > >> factory-bean="xServiceProxyFactory" > >> > >> factory-method="create"/> > >> > >> Notice that the xService bean is a prototype, so we get a new instance > >> for each request made, to avoid the original issue we were seeing. > >> > >> Now in our code we have a class that looks like this: > >> > >> public class xServiceHelperImpl implements BeanFactoryAware > >> { > >> > >> @Autowired > >> @Qualifier(value="xServiceProxyFactory") > >> private JaxWsProxyFactoryBean proxyFactoryBean; > >> > >> @Override > >> public void setBeanFactory(BeanFactory beanFactory) throws > >> > >> BeansException { > >> > >> this.beanFactory=beanFactory; > >> > >> } > >> > >> private xService getXService() { > >> > >> xService xService = > >> > >> (xService)this.beanFactory.getBean("xService"); > >> > >> return xService; > >> > >> } > >> > >> } > >> > >> And we have a JUnit test that spawns multiple threads that basically > >> calls "getXService()" and then invokes a web service call. > >> > >> When there is just 1 thread configured in the test, everything works > >> fine. > >> > >> When there are 2 threads configured in the test, we start getting > >> errors, it is a ConcurrentModificationExcpeiton, but in a different > >> place then where we saw before :(....here is the stack trace > >> > >> org.springframework.beans.factory.BeanCreationException: Error > >> creating > >> bean with name 'xService' defined in class path resource > >> [spring/myCxfClient.xml]: Instantiation of bean failed; nested > >> exception is > >> org.springframework.beans.factory.BeanDefinitionStoreException: > >> Factory method [public java.lang.Object > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] threw exception; > >> nested exception is java.util.ConcurrentModificationException > >> > >> at > >> > >> org.springframework.beans.factory.support.ConstructorResolver.instanti > >> ateUsingFactoryMethod(ConstructorResolver.java:581) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactor > >> y.java:983) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:879) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory.createBean(AbstractAutowireCapableBeanFactory.java:456) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea > >> n(AbstractBeanFactory.java:310) > >> > >> at > >> > >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean( > >> AbstractBeanFactory.java:190) > >> > >> at > >> com.XServiceHelperImpl.getXService(xServiceHelperImpl.java:21 > >> 7) > >> at com.XServiceHelperImpl.getX(xServiceHelperImpl.java:63) > >> at > >> > >> com..ITXServiceHelperImplTest$ITUserReadOnlyClient.run(ITUserServiceHe > >> lperImplTest.java:101) > >> > >> at java.lang.Thread.run(Thread.java:662) > >> > >> Caused by: > >> org.springframework.beans.factory.BeanDefinitionStoreException: > >> Factory method [public java.lang.Object > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] threw exception; > >> nested exception is java.util.ConcurrentModificationException > >> > >> at > >> > >> org.springframework.beans.factory.support.SimpleInstantiationStrategy. > >> instantiate(SimpleInstantiationStrategy.java:157) > >> > >> at > >> > >> org.springframework.beans.factory.support.ConstructorResolver.instanti > >> ateUsingFactoryMethod(ConstructorResolver.java:570) > >> > >> ... 10 more > >> > >> Caused by: java.util.ConcurrentModificationException > >> > >> at > >> > >> java.util.AbstractList$Itr.checkForComodification(AbstractList.java:37 > >> 2) > >> > >> at java.util.AbstractList$Itr.next(AbstractList.java:343) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createInpu > >> tWrappedMessageParts(ReflectionServiceFactoryBean.java:1619) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createOper > >> ation(ReflectionServiceFactoryBean.java:934) > >> > >> at > >> > >> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.createOperation(J > >> axWsServiceFactoryBean.java:616) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createInte > >> rface(ReflectionServiceFactoryBean.java:907) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildServi > >> ceFromClass(ReflectionServiceFactoryBean.java:433) > >> > >> at > >> > >> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromC > >> lass(JaxWsServiceFactoryBean.java:680) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initialize > >> ServiceModel(ReflectionServiceFactoryBean.java:505) > >> > >> at > >> > >> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(Ref > >> lectionServiceFactoryBean.java:242) > >> > >> at > >> > >> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServi > >> ceFactoryBean.java:202) > >> > >> at > >> > >> org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoin > >> t(AbstractWSDLBasedEndpointFactory.java:101) > >> > >> at > >> > >> org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.jav > >> a:90) > >> > >> at > >> > >> org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFacto > >> ryBean.java:117) > >> > >> at > >> > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBea > >> n.java:124) > >> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > >> ava:39) > >> > >> at > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > >> orImpl.java:25) > >> > >> at java.lang.reflect.Method.invoke(Method.java:597) > >> at > >> > >> org.springframework.beans.factory.support.SimpleInstantiationStrategy. > >> instantiate(SimpleInstantiationStrategy.java:145) > >> > >> ... 11 more > >> > >> Are we not configuring our client correctly? Just to reiterate we are > >> using 2.1.3. > >> > >> Thanks, > >> Al > >> > >> On Fri, Jul 22, 2011 at 5:09 PM, Daniel Kulp <[email protected]> wrote: > >>> On Friday, July 22, 2011 4:16:39 PM Algirdas Veitas wrote: > >>> > Small world :) > >>> > > >>> > RE: Hmm... with #3, you shouldn't even be hitting this. The > >>> > >>> HEADER_LIST > >>> > >>> > should > >>> > be completely created per request since it wouldn't be pulled > >>> > from the request > >>> > context. However, if the HEADER_LIST is added to the request > >>> > >>> context at > >>> > >>> > ANY time, even using #3 after that would cause an issue. If you > >>> > aren't touching the HEADER_LIST anywhere, then I'm not really > >>> > sure what is > >>> > >>> going > >>> > >>> > on. > >>> > > >>> > Yeah, we are not manipulating the HEADER_LIST anywhere in our > >>> > code, so > >>> > >>> am > >>> > >>> > not sure what is going on either.... > >>> > > >>> > RE: 2) After returning from any method where you had set a > >>> > header, > >>> > >>> call: * > >>> > >>> > context*.getMessageContext().remove(Header.HEADER_LIST); > >>> > > >>> > How do I get a handle to the "context", in our situation where > >>> > we are > >>> > >>> using > >>> > >>> > generated code? > >>> > >>> Sorry. That would be the actual proxy object. For example: > >>> > >>> MyServiceInterface proxy = service.getMyServicePort(); > >>> ((BindingProvider)proxy).getRequestContext()..... > >>> > >>> With JAX-WS, all the proxys implement not only the generated > >>> interface, > >>> but > >>> also the BindingProvider interface. > >>> > >>> > >>> Dan > >>> > >>> > Thanks, > >>> > Al > >>> > > >>> > On Fri, Jul 22, 2011 at 4:04 PM, Daniel Kulp <[email protected]> wrote: > >>> > > On Friday, July 22, 2011 3:58:06 PM Algirdas Veitas wrote: > >>> > > > Hi Daniel, > >>> > > > > >>> > > > Nice meeting you as well! > >>> > > > >>> > > Just discovered you and I went to Northeastern at roughly > >>> > > the same > >>> > >>> time. > >>> > >>> > > We > >>> > > may have met before. :-) > >>> > > > >>> > > > One follow up question....the way we are adding headers > >>> > > > is #3. Our generated client code treats our headers > >>> > > > like an ordinary parameter in > >>> > > > >>> > > the > >>> > > > >>> > > > method signature. Will your solution work for this > >>> > > > strategy as > >>> > > > well? > >>> > > > >>> > > Hmm... with #3, you shouldn't even be hitting this. The > >>> > > HEADER_LIST should > >>> > > be completely created per request since it wouldn't be > >>> > > pulled from > >>> > >>> the > >>> > >>> > > request > >>> > > context. However, if the HEADER_LIST is added to the > >>> > > request > >>> > >>> context > >>> > >>> > > at > >>> > > > >>> > > ANY time, even using #3 after that would cause an issue. If > >>> > > you > >>> > >>> aren't > >>> > >>> > > touching the HEADER_LIST anywhere, then I'm not really sure > >>> > > what is > >>> > > going > >>> > > on. > >>> > > Strange. Definitely give it a try though. > >>> > > > >>> > > > >>> > > Dan > >>> > > > >>> > > > Al > >>> > > > > >>> > > > On Fri, Jul 22, 2011 at 3:45 PM, Daniel Kulp > >>> > > > <[email protected]> > >>> > >>> wrote: > >>> > > > > We talked briefly about this at lunch today (nice to > >>> > > > > meet you > >>> > > > > Al!) but wanted > >>> > > > > to follow up here..... > >>> > > > > > >>> > > > > > >>> > > > > The per-proxy request context can definitely come > >>> > > > > into play > >>> > > > > here. If any of > >>> > > > > the threads add a header list to the RequestContext > >>> > > > > via the > >>> > > > > method > >>> > > > > mentioned > >>> > >>> > > > > in #4 of: > >>> http://cxf.apache.org/faq#FAQ-HowcanIaddsoapheaderstotherequest%2Fre > >>> spon > >>> > >>> > > > > se%3F > >>> > > > > > >>> > > > > then those headers would be sent on all methods > >>> > > > > called on the > >>> > > > > proxy. > >>> > > > > > >>> > > > > What's worse, I THINK that List is just copied into > >>> > > > > the real > >>> > > > > message > >>> > > > > context > >>> > > > > so any header processing done during the processing > >>> > > > > of the > >>> > > > > message > >>> > > > >>> > > would > >>> > > > >>> > > > > affect that list, thus affecting all threads as well > >>> > > > > as future > >>> > > > > method > >>> > > > > calls on > >>> > > > > the same thread. Thinking about this, it may > >>> > > > > make sense to > >>> > > > > clone > >>> > > > > the > >>> > > > > list > >>> > > > > at the very start of processing messages to make > >>> > > > > sure the header list > >>> > > > >>> > > in > >>> > > > >>> > > > > the > >>> > > > > request context isn't modified. That might be > >>> > > > > worth filing a > >>> > > > > jira > >>> > > > >>> > > for. > >>> > > > >>> > > > > For now, I would suggest doing 2 things: > >>> > > > > > >>> > > > > 1) Use the thread local request contexts: > >>> > > > > > >>> > > > > <jaxws:client id="xService" > >>> > > > > > >>> > > > > serviceClass="com.XService" > >>> > > > > address="#xServiceUrl" > > >>> > > > > > >>> > > > > <jaxws:properties> > >>> > > > > > >>> > > > > <entry > >>> > > > > key="thread.local.request.con > >>> > > > > text" > >>> > > > > value="true" /> > >>> > > > > > >>> > > > > </jaxws:properties> > >>> > > > > > >>> > > > > </jaxws:client> > >>> > > > > > >>> > > > > 2) After returning from any method where you had set > >>> > > > > a header, > >>> > > > > call: > >>> > > > > context.getMessageContext().remove(Header.HEADER_LIS > >>> > > > > T); > >>> > > > > > >>> > > > > > >>> > > > > That should keep things in a good state. > >>> > > > > > >>> > > > > Dan > >>> > > > > > >>> > > > > On Friday, July 22, 2011 10:25:10 AM Algirdas Veitas wrote: > >>> > > > > > Hi, > >>> > > > > > > >>> > > > > > We are getting a ConcurrentModificationException > >>> > > > > > exception > >>> > > > > > in our > >>> > > > > > CXF > >>> > > > > > client, when it is processing SOAPHeaders. We > >>> > > > > > did find > >>> > > > > > this > >>> > > > > > following link that describes what is going on > >>> > >>> http://cxf.547215.n5.nabble.com/jira-Created-CXF-2762-Cannot-deploy- > >>> csta > >>> > >>> > > > > -web > >>> > > > > > >>> > > > > > -service-td586662.html#a586663, and we will > >>> > > > > > address that > >>> > > > > > (Header > >>> > > > > > object > >>> > > > > > needs to be unique across threads), but > >>> > > > > > something else > >>> > > > > > strange is > >>> > > > > > happening: > >>> > > > > > > >>> > > > > > To start here is our client setup > >>> > > > > > > >>> > > > > > <!-- Web Service Clients --> > >>> > > > > > > >>> > > > > > <jaxws:client id="xService" > >>> > > > > > > >>> > > > > > serviceClass="com.XService" > >>> > > > > > address="#xServiceUrl" /> > >>> > > > > > > >>> > > > > > This WSDL that is associate with this client has > >>> > > > > > some > >>> > > > > > methods that > >>> > > > > > DO > >>> > > > > > >>> > > > > have a > >>> > > > > > >>> > > > > > Header and some DO NOT. > >>> > > > > > > >>> > > > > > And here is the shortened stack trace: > >>> > > > > > > >>> > > > > > <http://newportave.jira.com/wiki/display/21+Jul+ > >>> > > > > > 2011+19/11%3 > >>> > > > > > A46%2C87 4> > >>> > > > > > java.util.ConcurrentModificationException > >>> > > > > > at > >>> > >>> java.util.AbstractList$Itr.checkForComodification(AbstractList.java: > >>> 372) > >>> > >>> > > > > > at > >>> > > > > > java.util.AbstractList$Itr.next(AbstractList.ja > >>> > > > > > va:343) > >>> > > > > > at > >>> > >>> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.writeSoap > >>> Enve > >>> > >>> > > > > lope > >>> > > > > > >>> > > > > > Start(SoapOutInterceptor.java:139) at > >>> > >>> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.handleMes > >>> sage > >>> > >>> > > > > (Soa > >>> > > > > > >>> > > > > > pOutInterceptor.java:81) at > >>> > >>> org.apache.cxf.binding.soap.interceptor.SoapOutInterceptor.handleMes > >>> sage > >>> > >>> > > > > (Soa > >>> > > > > > >>> > > > > > pOutInterceptor.java:61) at > >>> > >>> org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseIntercep > >>> torC > >>> > >>> > > > > hain > >>> > > > > > >>> > > > > > .java:255) at > >>> > > > > > >>> > > > > org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl > >>> > > > > .java:516) > >>> > > > > > >>> > > > > > Here is the rub: This exception is being thrown > >>> > > > > > when we are > >>> > > > > > calling > >>> > > > > > a > >>> > > > > > >>> > > > > method > >>> > > > > > >>> > > > > > that DOES NOT have a Header, but on line 139 in > >>> > > > > > SoapOutInterceptor, > >>> > > > > > it > >>> > > > > > looks like we are trying to process an list that > >>> > > > > > has atleast > >>> > > > > > 1 > >>> > > > > > header. > >>> > > > > > >>> > > > > Am > >>> > > > > > >>> > > > > > pretty certain (this is occurring only in a > >>> > > > > > specific > >>> > > > > > environment) > >>> > > > > > that > >>> > > > > > other methods on this service were made previous > >>> > > > > > to this > >>> > > > > > error that > >>> > > > > > did > >>> > > > > > include a SOAP Header. Am speculating, but is > >>> > > > > > the SOAP > >>> > > > > > Header being > >>> > > > > > >>> > > > > cached > >>> > > > > > >>> > > > > > in the request context and the request context > >>> > > > > > is scoped per > >>> > > > > > client > >>> > > > > > instance as per the FAQ ( > >>> > > > > > http://cxf.apache.org/faq.html#FAQ-AreJAXWSclien > >>> > > > > > tproxiesthre > >>> > > > > > adsafe%3 F)? > >>> > > > > > > >>> > > > > > > >>> > > > > > Thanks, > >>> > > > > > Al > >>> > > > > > >>> > > > > -- > >>> > > > > Daniel Kulp > >>> > > > > [email protected] > >>> > > > > http://dankulp.com/blog > >>> > > > > Talend - http://www.talend.com > >>> > > > >>> > > -- > >>> > > Daniel Kulp > >>> > > [email protected] > >>> > > http://dankulp.com/blog > >>> > > Talend - http://www.talend.com > >>> > >>> -- > >>> Daniel Kulp > >>> [email protected] > >>> http://dankulp.com/blog > >>> Talend - http://www.talend.com -- Daniel Kulp [email protected] http://dankulp.com/blog Talend - http://www.talend.com
