On Wednesday, July 27, 2011 7:57:58 PM Gary Gregory wrote: > Hi Dan, > > Any thoughts on the timing for 2.4.2?
We need a release of WSS4J 1.6.2 first as we now rely on some of the new functionality and fixes that are in there. I THINK Colm is planning on doing that sometime next week. Once that is released, I'll do the 2.4.2 release of CXF. So I would say on or before Aug 15th would be a fairly good estimate. Dan > > Gary > > On Jul 27, 2011, at 15:34, "Daniel Kulp" <[email protected]> wrote: > > Just looked at the code.... > > > > > > The JaxWsProxyFactoryBean doesn't have the synchronized flag on the > > create method like the one on the superclass does. :-( > > > > You can likely workaround this by creating your own subclass of > > JaxWsProxyFactoryBean that just has a single method of: > > > > > > public synchronized Object create() { > > > > return super.create(); > > > > } > > > > and using that instead. I'll get this fixes for 2.4.2. > > > > Dan > > > > On Tuesday, July 26, 2011 10:17:10 PM Algirdas Veitas wrote: > >> OK, we migrated to 2.3.5 and are still using a "client per SOAP call" > >> strategy. We are still seeing intermittent issues. We have tried to > >> configure our client as a spring bean with scope prototype AND call > >> the > >> create() method in our code directly. In both cases, we are getting > >> intermittent NPE's in JaxWsProxyFactoryBean, but on different lines. > >> Unfortunately, upgrading to 2.4.1 didn't resolve the issue either :( > >> > >> We have a simple test case that has 2 threads....and here is the stack > >> trace when our client in configured as a spring bean (prototype) > >> > >> org.springframework.beans.factory.BeanCreationException: Error > >> creating bean with name 'XService' defined in class path resource > >> [spring/myCxfClient.xml]: Instantiation of bean failed; nested > >> exception is > >> org.springframework.beans.factory.BeanDefinitionStoreException: > >> Factory method [public java.lang.Object > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] threw exception; > >> nested exception is java.lang.NullPointerException > >> Index:0, about to call ws > >> > >> at > >> > >> org.springframework.beans.factory.support.ConstructorResolver.instanti > >> ateUsi ngFactoryMethod(ConstructorResolver.java:581) at > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory > >> .instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.jav > >> a:983) at > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory > >> .createBeanInstance(AbstractAutowireCapableBeanFactory.java:879) at > >> org.springframework.beans.factory.support.AbstractAutowireCapableBean > >> Factory .doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at > >> org.springframework.beans.factory.support.AbstractAutowireCapableBeanF > >> actory .createBean(AbstractAutowireCapableBeanFactory.java:456) at > >> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBea > >> n(Abst ractBeanFactory.java:310) at > >> org.springframework.beans.factory.support.AbstractBeanFactory.getBean( > >> Abstra ctBeanFactory.java:190) at > >> com.xHelperImpl.getXService(XServiceHelperImpl.java:203) > >> > >> at > >> > >> com.helper.XHelperImplTest$ITUserReadOnlyClient.run(ITUserServiceHelpe > >> rImplT est.java:98) at java.lang.Thread.run(Thread.java:662) > >> Caused by: > >> org.springframework.beans.factory.BeanDefinitionStoreException: > >> Factory method [public java.lang.Object > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] threw exception; > >> nested exception is java.lang.NullPointerException > >> > >> at > >> > >> org.springframework.beans.factory.support.SimpleInstantiationStrategy. > >> instan tiate(SimpleInstantiationStrategy.java:157) at > >> org.springframework.beans.factory.support.ConstructorResolver.instanti > >> ateUsi ngFactoryMethod(ConstructorResolver.java:570) ... 10 more > >> Caused by: java.lang.NullPointerException > >> > >> at > >> > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.needWrapperClassInterceptor > >> (JaxWs ProxyFactoryBean.java:152) at > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBea > >> n.java > >> > >> :135) at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) > >> :at > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > >> orImpl .java:25) at java.lang.reflect.Method.invoke(Method.java:597) > >> > >> at > >> > >> org.springframework.beans.factory.support.SimpleInstantiationStrategy. > >> instan tiate(SimpleInstantiationStrategy.java:145) ... 11 more > >> > >> > >> Here is our configuration.... > >> > >> <bean id="xServiceProxyFactory" > >> class="org.apache.cxf.jaxws.JaxWsProxyFactoryBean"> > >> > >> <property name="serviceClass" value="com.XService"/> > >> <property name="address" ref="xServiceUrl"/> > >> > >> </bean> > >> > >> <bean id="xService" scope="prototype" class="com.XService" > >> factory-bean="xServiceProxyFactory" factory-method="create"/> > >> > >> > >> When we call create() on the proxy factory in our code (and not use a > >> prototype scoped bean), we see a similiar error (but not as frequently > >> as the above) error: > >> > >> java.lang.NullPointerException > >> > >> at > >> > >> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBea > >> n.java > >> > >> :135) at > >> :com.xServiceHelperImpl.getXService(XServiceHelperImpl.java:210) at > >> > >> com.helper.XHelperImplTest$ITUserReadOnlyClient.run(ITUserServiceHelpe > >> rImplT est.java:98) > >> > >> > >> Al > >> > >> On Mon, Jul 25, 2011 at 4:49 PM, Daniel Kulp <[email protected]> wrote: > >>> On Monday, July 25, 2011 12:53:37 PM Algirdas Veitas wrote: > >>>> Will give that a shot, but it looks like this is the same issue as > >>>> https://issues.apache.org/jira/browse/CXF-3075, and according the > >>>> entry > >>> > >>> was > >>> > >>>> fixed in 2.3.1. Were there more changes since then? > >>> > >>> For the client creation, there is: > >>> https://issues.apache.org/jira/browse/CXF-3558 > >>> > >>> which is in 2.3.5. > >>> > >>> Ideally, you'd move right to 2.4.1. It's definitely easier to debug > >>> things from my side there. > >>> > >>> > >>> Dan > >>> > >>>> Al > >>>> > >>>> On Mon, Jul 25, 2011 at 12:48 PM, Daniel Kulp <[email protected]> wrote: > >>>>> Can you move up to 2.3.5? We did add some synchronized blocks > >>>>> around the ReflectionServiceFactoryBean to make the singleton > >>>>> cases work a lot better. That may fix this issue. > >>>>> > >>>>> Dan > >>>>> > >>>>> On Monday, July 25, 2011 12:35:11 PM Algirdas Veitas wrote: > >>>>>> More information....after doing some additional testing, it > >>>>>> looks > >>>>>> like > >>>>> > >>>>> the > >>>>> > >>>>>> exception is thrown in different areas of the CXF code, but > >>>>>> the > >>>>>> ConcurrentModificationException is thrown when trying to > >>>>>> manipulate > >>>>>> the > >>>>>> > >>>>>> private List<AbstractServiceConfiguration> > >>>>>> serviceConfigurations = > >>>>>> > >>>>>> new > >>>>>> ArrayList<AbstractServiceConfiguration>(); > >>>>>> > >>>>>> that belongs to ReflectionServiceFactoryBean....see stack > >>>>>> traces > >>>>>> below > >>>>> > >>>>> for > >>>>> > >>>>>> additional samples: > >>>>>> > >>>>>> Caused by: java.util.ConcurrentModificationException > >>>>>> > >>>>>> at > >>>>>> > >>>>>> java.util.AbstractList$Itr.checkForComodification(AbstractLi > >>>>>> st.java: 372) > >>>>>> > >>>>>> at > >>>>>> java.util.AbstractList$Itr.next(AbstractList.java:3 > >>>>>> 43) > >>>>>> at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createIn > >>> putW > >>> > >>>>> rapp > >>>>> > >>>>>> edMessageParts(ReflectionServiceFactoryBean.java:1619) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createOp > >>> erat > >>> > >>>>> ion( > >>>>> > >>>>>> ReflectionServiceFactoryBean.java:934) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.createOperation > >>> (Jax > >>> > >>>>> WsSe > >>>>> > >>>>>> rviceFactoryBean.java:616) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createIn > >>> terf > >>> > >>>>> ace( > >>>>> > >>>>>> ReflectionServiceFactoryBean.java:907) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildSer > >>> vice > >>> > >>>>> From > >>>>> > >>>>>> Class(ReflectionServiceFactoryBean.java:433) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFro > >>> mCla > >>> > >>>>> ss(J > >>>>> > >>>>>> axWsServiceFactoryBean.java:680) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initiali > >>> zeSe > >>> > >>>>> rvic > >>>>> > >>>>>> eModel(ReflectionServiceFactoryBean.java:505) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(R > >>> efle > >>> > >>>>> ctio > >>>>> > >>>>>> nServiceFactoryBean.java:242) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsSer > >>> vice > >>> > >>>>> Fact > >>>>> > >>>>>> oryBean.java:202) at > >>> > >>> org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpo > >>> int( > >>> > >>>>> Abst > >>>>> > >>>>>> ractWSDLBasedEndpointFactory.java:101) at > >>> > >>> org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java: > >>>>> 90) > >>>>> > >>>>>> at > >>> > >>> org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFac > >>> tory > >>> > >>>>> Bean > >>>>> > >>>>>> .java:117 > >>>>>> > >>>>>> Caused by: java.util.ConcurrentModificationException > >>>>>> > >>>>>> at > >>>>>> > >>>>>> java.util.AbstractList$Itr.checkForComodification(AbstractLi > >>>>>> st.java: 372) > >>>>>> > >>>>>> at > >>>>>> java.util.AbstractList$Itr.next(AbstractList.java:3 > >>>>>> 43) > >>>>>> at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.addFault > >>> (Ref > >>> > >>>>> lect > >>>>> > >>>>>> ionServiceFactoryBean.java:2032) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initiali > >>> zeFa > >>> > >>>>> ults > >>>>> > >>>>>> (ReflectionServiceFactoryBean.java:2011) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createMe > >>> ssag > >>> > >>>>> ePar > >>>>> > >>>>>> ts(ReflectionServiceFactoryBean.java:1575) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createOp > >>> erat > >>> > >>>>> ion( > >>>>> > >>>>>> ReflectionServiceFactoryBean.java:928) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.createOperation > >>> (Jax > >>> > >>>>> WsSe > >>>>> > >>>>>> rviceFactoryBean.java:616) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.createIn > >>> terf > >>> > >>>>> ace( > >>>>> > >>>>>> ReflectionServiceFactoryBean.java:907) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.buildSer > >>> vice > >>> > >>>>> From > >>>>> > >>>>>> Class(ReflectionServiceFactoryBean.java:433) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFro > >>> mCla > >>> > >>>>> ss(J > >>>>> > >>>>>> axWsServiceFactoryBean.java:680) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.initiali > >>> zeSe > >>> > >>>>> rvic > >>>>> > >>>>>> eModel(ReflectionServiceFactoryBean.java:505) at > >>> > >>> org.apache.cxf.service.factory.ReflectionServiceFactoryBean.create(R > >>> efle > >>> > >>>>> ctio > >>>>> > >>>>>> nServiceFactoryBean.java:242) at > >>> > >>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsSer > >>> vice > >>> > >>>>> Fact > >>>>> > >>>>>> oryBean.java:202) at > >>> > >>> org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpo > >>> int( > >>> > >>>>> Abst > >>>>> > >>>>>> ractWSDLBasedEndpointFactory.java:101) at > >>> > >>> org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java: > >>>>> 90) > >>>>> > >>>>>> at > >>> > >>> org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFac > >>> tory > >>> > >>>>> Bean > >>>>> > >>>>>> .java:117) at > >>> > >>> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryB > >>> ean. > >>> > >>>>> java > >>>>> > >>>>>> :124) > >>>>>> > >>>>>> On Mon, Jul 25, 2011 at 11:30 AM, Algirdas Veitas > >>>>>> <[email protected] > >>>>>> > >>>>>> wrote: > >>>>>>> whoops...in the previous post...we mentioned 2.1.3 but > >>>>>>> we are > >>>>>>> using > >>>>>>> 2.3.1 > >>>>>>> > >>>>>>> On Mon, Jul 25, 2011 at 10:36 AM, Algirdas Veitas > >>>>> > >>>>> <[email protected]>wrote: > >>>>>>>> Hi Daniel, > >>>>>>>> > >>>>>>>> Unfortunately, we are unable to reproduce this problem > >>>>>>>> in an > >>>>>>>> isolated environment. But at this point, it does look > >>>>>>>> like > >>>>>>>> to be an issue with multiple threads using the same > >>>>>>>> instance > >>>>>>>> of the client. Because we are > >>>>>>>> unable to reproduce, our current strategy is to create > >>>>>>>> a > >>>>>>>> separate > >>>>>>>> client per SOAP call. > >>>>>>>> > >>>>>>>> Using the following documentation as a guideline > >>>>>>>> http://cxf.apache.org/docs/jax-ws-configuration.html, > >>>>>>>> we came > >>>>>>>> up with the following: > >>>>>>>> > >>>>>>>> Here was the original configuration for the client: > >>>>>>>> > >>>>>>>> > >>>>>>>> <jaxws:client id="xService" > >>>>>>>> serviceClass="com.XService" > >>>>>>>> address="#xServiceUrl" > >>>>>>>> </jaxws:client> > >>>>>>>> > >>>>>>>> Here is what we just tried: > >>>>>>>> > >>>>>>>> <bean id="xServiceProxyFactory" > >>>>>>>> > >>>>>>>> class="org.apache.cxf.jaxws. > >>>>>>>> > >>>>>>>> JaxWsProxyFactoryBean"> > >>>>>>>> > >>>>>>>> <property name="serviceClass" > >>>>>>>> value="com.XService"/> > >>>>>>>> <property name="address" > >>>>>>>> ref="xServiceUrl"/> > >>>>>>>> > >>>>>>>> </bean> > >>>>>>>> > >>>>>>>> <bean id="xService" scope="prototype" > >>>>>>>> class="com.XService" > >>>>>>>> > >>>>>>>> factory-bean="xServiceProx > >>>>>>>> yFactory > >>>>>>>> " > >>>>>>>> > >>>>>>>> factory-method="create"/> > >>>>>>>> > >>>>>>>> Notice that the xService bean is a prototype, so we > >>>>>>>> get a new > >>>>>>>> instance for each request made, to avoid the original > >>>>>>>> issue > >>>>>>>> we were seeing. > >>>>>>>> > >>>>>>>> Now in our code we have a class that looks like this: > >>>>>>>> > >>>>>>>> public class xServiceHelperImpl implements > >>>>>>>> BeanFactoryAware > >>>>>>>> { > >>>>>>>> > >>>>>>>> @Autowired > >>>>>>>> @Qualifier(value="xServiceProxyFactory") > >>>>>>>> private JaxWsProxyFactoryBean > >>>>>>>> proxyFactoryBean; > >>>>>>>> > >>>>>>>> @Override > >>>>>>>> public void setBeanFactory(BeanFactory > >>>>>>>> beanFactory) > >>>>>>>> throws > >>>>>>>> > >>>>>>>> BeansException { > >>>>>>>> > >>>>>>>> this.beanFactory=beanFactory; > >>>>>>>> > >>>>>>>> } > >>>>>>>> > >>>>>>>> private xService getXService() { > >>>>>>>> > >>>>>>>> xService xService = > >>>>>>>> > >>>>>>>> (xService)this.beanFactory.getBean("xService"); > >>>>>>>> > >>>>>>>> return xService; > >>>>>>>> > >>>>>>>> } > >>>>>>>> > >>>>>>>> } > >>>>>>>> > >>>>>>>> And we have a JUnit test that spawns multiple threads > >>>>>>>> that > >>>>>>>> basically > >>>>>>>> calls "getXService()" and then invokes a web service > >>>>>>>> call. > >>>>>>>> > >>>>>>>> When there is just 1 thread configured in the test, > >>>>>>>> everything > >>>>>>>> works > >>>>>>>> fine. > >>>>>>>> > >>>>>>>> When there are 2 threads configured in the test, we > >>>>>>>> start > >>>>>>>> getting > >>>>>>>> errors, it is a ConcurrentModificationExcpeiton, but > >>>>>>>> in a > >>>>>>>> different > >>>>>>>> place then where we saw before :(....here is the stack > >>>>>>>> trace > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.BeanCreationExceptio > >>>>>>>> n: Error > >>>>>>>> creating > >>>>>>>> bean with name 'xService' defined in class path > >>>>>>>> resource > >>>>>>>> [spring/myCxfClient.xml]: Instantiation of bean > >>>>>>>> failed; nested > >>>>>>>> exception is > >>>>>>>> org.springframework.beans.factory.BeanDefinitionStoreE > >>>>>>>> xception > >>>>>>>> > >>>>>>>> Factory method [public java.lang.Object > >>>>>>>> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] > >>>>>>>> threw > >>>>>>>> exception; nested exception is > >>>>>>>> java.util.ConcurrentModificationException > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.ConstructorR > >>>>>>>> esolver. > >>>>>>>> instanti > >>>>>>>> ateUsingFactoryMethod(ConstructorResolver.java:581) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractAuto > >>>>>>>> wireCapa > >>>>>>>> bleBeanF > >>>>>>>> actory.instantiateUsingFactoryMethod(AbstractAutowireC > >>>>>>>> apableB > >>>>>>>> eanFactor y.java:983) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractAuto > >>>>>>>> wireCapa > >>>>>>>> bleBeanF > >>>>>>>> actory.createBeanInstance(AbstractAutowireCapableBeanF > >>>>>>>> actory. > >>>>>>>> java:879) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractAuto > >>>>>>>> wireCapa > >>>>>>>> bleBeanF > >>>>>>>> actory.doCreateBean(AbstractAutowireCapableBeanFactory > >>>>>>>> .java:4 > >>>>>>>> 85) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractAuto > >>>>>>>> wireCapa > >>>>>>>> bleBeanF > >>>>>>>> actory.createBean(AbstractAutowireCapableBeanFactory.j > >>>>>>>> ava:456 > >>>>>>>> ) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractBean > >>>>>>>> Factory. > >>>>>>>> doGetBea n(AbstractBeanFactory.java:310) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.AbstractBean > >>>>>>>> Factory. > >>>>>>>> getBean( AbstractBeanFactory.java:190) > >>>>>>>> > >>>>>>>> at > >>>>>>>> com.XServiceHelperImpl.getXService(xServiceHel > >>>>>>>> perImpl. > >>>>>>>> java:21 > >>>>>>>> 7) > >>>>>>>> at > >>>>>>>> com.XServiceHelperImpl.getX(xServiceHelperImpl > >>>>>>>> .java:6 > >>>>>>>> 3) > >>>>>>>> at > >>>>>>>> > >>>>>>>> com..ITXServiceHelperImplTest$ITUserReadOnlyClient.run > >>>>>>>> (ITUserS > >>>>>>>> erviceHe lperImplTest.java:101) > >>>>>>>> > >>>>>>>> at java.lang.Thread.run(Thread.java:662) > >>>>>>>> > >>>>>>>> Caused by: > >>>>>>>> org.springframework.beans.factory.BeanDefinitionStoreE > >>>>>>>> xception > >>>>>>>> > >>>>>>>> Factory method [public java.lang.Object > >>>>>>>> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create()] > >>>>>>>> threw > >>>>>>>> exception; nested exception is > >>>>>>>> java.util.ConcurrentModificationException > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.SimpleInstan > >>>>>>>> tiationS > >>>>>>>> trategy. > >>>>>>>> instantiate(SimpleInstantiationStrategy.java:157) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.ConstructorR > >>>>>>>> esolver. > >>>>>>>> instanti > >>>>>>>> ateUsingFactoryMethod(ConstructorResolver.java:570) > >>>>>>>> > >>>>>>>> ... 10 more > >>>>>>>> > >>>>>>>> Caused by: java.util.ConcurrentModificationException > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> java.util.AbstractList$Itr.checkForComodification(Abst > >>>>>>>> ractList > >>>>>>>> .java:37 2) > >>>>>>>> > >>>>>>>> at > >>>>>>>> java.util.AbstractList$Itr.next(AbstractList.j > >>>>>>>> ava:343 > >>>>>>>> ) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.cr > >>>>>>>> eateInpu > >>>>>>>> tWrappedMessageParts(ReflectionServiceFactoryBean.java > >>>>>>>> > >>>>>>>> :1619) > >>>>>>>> : > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.cr > >>>>>>>> eateOper ation(ReflectionServiceFactoryBean.java:934) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.c > >>>>>>>> reateOpe > >>>>>>>> ration(J axWsServiceFactoryBean.java:616) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.cr > >>>>>>>> eateInte rface(ReflectionServiceFactoryBean.java:907) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.bu > >>>>>>>> ildServi > >>>>>>>> ceFromClass(ReflectionServiceFactoryBean.java:433) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.b > >>>>>>>> uildServ > >>>>>>>> iceFromC lass(JaxWsServiceFactoryBean.java:680) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.in > >>>>>>>> itialize > >>>>>>>> ServiceModel(ReflectionServiceFactoryBean.java:505) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.service.factory.ReflectionServiceFactor > >>>>>>>> yBean.cr > >>>>>>>> eate(Ref lectionServiceFactoryBean.java:242) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.c > >>>>>>>> reate(Ja > >>>>>>>> xWsServi ceFactoryBean.java:202) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.frontend.AbstractWSDLBasedEndpointFacto > >>>>>>>> ry.creat > >>>>>>>> eEndpoin t(AbstractWSDLBasedEndpointFactory.java:101) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.frontend.ClientFactoryBean.create(Clien > >>>>>>>> tFactory > >>>>>>>> Bean.jav a:90) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.frontend.ClientProxyFactoryBean.create( > >>>>>>>> ClientPr > >>>>>>>> oxyFacto ryBean.java:117) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxW > >>>>>>>> sProxyFa > >>>>>>>> ctoryBea n.java:124) > >>>>>>>> > >>>>>>>> at > >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke0( > >>>>>>>> Native > >>>>>>>> Method) > >>>>>>>> at > >>>>>>>> > >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMeth > >>>>>>>> odAccess > >>>>>>>> orImpl.j ava:39) > >>>>>>>> > >>>>>>>> at > >>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(Delega > >>>>>>>> tingMeth > >>>>>>>> odAccess orImpl.java:25) > >>>>>>>> > >>>>>>>> at > >>>>>>>> java.lang.reflect.Method.invoke(Method.java:5 > >>>>>>>> 97) > >>>>>>>> at > >>>>>>>> > >>>>>>>> org.springframework.beans.factory.support.SimpleInstan > >>>>>>>> tiationS > >>>>>>>> trategy. > >>>>>>>> instantiate(SimpleInstantiationStrategy.java:145) > >>>>>>>> > >>>>>>>> ... 11 more > >>>>>>>> > >>>>>>>> Are we not configuring our client correctly? Just to > >>>>>>>> reiterate we are using 2.1.3. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Al > >>>>>>>> > >>>>>>>> On Fri, Jul 22, 2011 at 5:09 PM, Daniel Kulp > >>>>>>>> <[email protected]> > >>>>> > >>>>> wrote: > >>>>>>>>> On Friday, July 22, 2011 4:16:39 PM Algirdas Veitas wrote: > >>>>>>>>>> Small world :) > >>>>>>>>>> > >>>>>>>>>> RE: Hmm... with #3, you shouldn't even be > >>>>>>>>>> hitting this. > >>>>>>>>>> > >>>>>>>>>> The > >>>>>>>>> > >>>>>>>>> HEADER_LIST > >>>>>>>>> > >>>>>>>>>> should > >>>>>>>>>> be completely created per request since it > >>>>>>>>>> wouldn't be > >>>>>>>>>> pulled > >>>>>>>>>> from the request > >>>>>>>>>> context. However, if the HEADER_LIST is > >>>>>>>>>> added to the > >>>>>>>>>> request > >>>>>>>>> > >>>>>>>>> context at > >>>>>>>>> > >>>>>>>>>> ANY time, even using #3 after that would cause > >>>>>>>>>> an issue. > >>>>>>>>>> > >>>>>>>>>> If you > >>>>>>>>>> > >>>>>>>>>> aren't touching the HEADER_LIST anywhere, then > >>>>>>>>>> I'm not > >>>>>>>>>> really > >>>>>>>>>> sure what is > >>>>>>>>> > >>>>>>>>> going > >>>>>>>>> > >>>>>>>>>> on. > >>>>>>>>>> > >>>>>>>>>> Yeah, we are not manipulating the HEADER_LIST > >>>>>>>>>> anywhere > >>>>>>>>>> in our > >>>>>>>>>> code, so > >>>>>>>>> > >>>>>>>>> am > >>>>>>>>> > >>>>>>>>>> not sure what is going on either.... > >>>>>>>>>> > >>>>>>>>>> RE: 2) After returning from any method where you > >>>>>>>>>> had set > >>>>>>>>>> a > >>>>>>>>>> header, > >>>>>>>>> > >>>>>>>>> call: * > >>>>>>>>> > >>>>>>>>>> context*.getMessageContext().remove(Header.HEADE > >>>>>>>>>> R_LIST); > >>>>>>>>>> > >>>>>>>>>> How do I get a handle to the "context", in our > >>>>>>>>>> situation > >>>>>>>>>> where > >>>>>>>>>> we are > >>>>>>>>> > >>>>>>>>> using > >>>>>>>>> > >>>>>>>>>> generated code? > >>>>>>>>> > >>>>>>>>> Sorry. That would be the actual proxy object. > >>>>>>>>> For > >>>>>>>>> example: > >>>>>>>>> > >>>>>>>>> MyServiceInterface proxy = > >>>>>>>>> service.getMyServicePort(); > >>>>>>>>> ((BindingProvider)proxy).getRequestContext()..... > >>>>>>>>> > >>>>>>>>> With JAX-WS, all the proxys implement not only the > >>>>>>>>> generated > >>>>>>>>> interface, > >>>>>>>>> but > >>>>>>>>> also the BindingProvider interface. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Dan > >>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Al > >>>>>>>>>> > >>>>>>>>>> On Fri, Jul 22, 2011 at 4:04 PM, Daniel Kulp > >>>>>>>>>> <[email protected]> > >>>>> > >>>>> wrote: > >>>>>>>>>>> On Friday, July 22, 2011 3:58:06 PM Algirdas Veitas wrote: > >>>>>>>>>>>> Hi Daniel, > >>>>>>>>>>>> > >>>>>>>>>>>> Nice meeting you as well! > >>>>>>>>>>> > >>>>>>>>>>> Just discovered you and I went to > >>>>>>>>>>> Northeastern at > >>>>>>>>>>> roughly > >>>>>>>>>>> the same > >>>>>>>>> > >>>>>>>>> time. > >>>>>>>>> > >>>>>>>>>>> We > >>>>>>>>>>> may have met before. :-) > >>>>>>>>>>> > >>>>>>>>>>>> One follow up question....the way we are > >>>>>>>>>>>> adding > >>>>>>>>>>>> headers > >>>>>>>>>>>> is #3. Our generated client code treats > >>>>>>>>>>>> our > >>>>>>>>>>>> headers > >>>>>>>>>>>> like an ordinary parameter in > >>>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>>> method signature. Will your solution > >>>>>>>>>>>> work for > >>>>>>>>>>>> this > >>>>>>>>>>>> strategy as > >>>>>>>>>>>> well? > >>>>>>>>>>> > >>>>>>>>>>> Hmm... with #3, you shouldn't even be > >>>>>>>>>>> hitting this. > >>>>>>>>>>> > >>>>>>>>>>> The > >>>>>>>>>>> > >>>>>>>>>>> HEADER_LIST should > >>>>>>>>>>> be completely created per request since it > >>>>>>>>>>> wouldn't > >>>>>>>>>>> be > >>>>>>>>>>> pulled from > >>>>>>>>> > >>>>>>>>> the > >>>>>>>>> > >>>>>>>>>>> request > >>>>>>>>>>> context. However, if the HEADER_LIST is > >>>>>>>>>>> added to > >>>>>>>>>>> the > >>>>>>>>>>> request > >>>>>>>>> > >>>>>>>>> context > >>>>>>>>> > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> ANY time, even using #3 after that would > >>>>>>>>>>> cause an > >>>>>>>>>>> issue. If > >>>>>>>>>>> you > >>>>>>>>> > >>>>>>>>> aren't > >>>>>>>>> > >>>>>>>>>>> touching the HEADER_LIST anywhere, then I'm > >>>>>>>>>>> not > >>>>>>>>>>> really sure > >>>>>>>>>>> what is > >>>>>>>>>>> going > >>>>>>>>>>> on. > >>>>>>>>>>> Strange. Definitely give it a try though. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Dan > >>>>>>>>>>> > >>>>>>>>>>>> Al > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Jul 22, 2011 at 3:45 PM, Daniel > >>>>>>>>>>>> Kulp > >>>>>>>>>>>> <[email protected]> > >>>>>>>>> > >>>>>>>>> wrote: > >>>>>>>>>>>>> We talked briefly about this at > >>>>>>>>>>>>> lunch today > >>>>>>>>>>>>> (nice to > >>>>>>>>>>>>> meet you > >>>>>>>>>>>>> Al!) but wanted > >>>>>>>>>>>>> to follow up here..... > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> The per-proxy request context can > >>>>>>>>>>>>> definitely > >>>>>>>>>>>>> come > >>>>>>>>>>>>> into play > >>>>>>>>>>>>> here. If any of > >>>>>>>>>>>>> the threads add a header list to the > >>>>>>>>>>>>> RequestContext > >>>>>>>>>>>>> via the > >>>>>>>>>>>>> method > >>>>>>>>>>>>> mentioned > >>>>>>>>> > >>>>>>>>>>>>> in #4 of: > >>>>>>>>> http://cxf.apache.org/faq#FAQ-HowcanIaddsoapheaderst > >>>>>>>>> otherequ > >>>>>>>>> est%2Fre spon > >>>>>>>>> > >>>>>>>>>>>>> se%3F > >>>>>>>>>>>>> > >>>>>>>>>>>>> then those headers would be sent on > >>>>>>>>>>>>> all > >>>>>>>>>>>>> methods > >>>>>>>>>>>>> called on the > >>>>>>>>>>>>> proxy. > >>>>>>>>>>>>> > >>>>>>>>>>>>> What's worse, I THINK that List is > >>>>>>>>>>>>> just > >>>>>>>>>>>>> copied into > >>>>>>>>>>>>> the real > >>>>>>>>>>>>> message > >>>>>>>>>>>>> context > >>>>>>>>>>>>> so any header processing done during > >>>>>>>>>>>>> the > >>>>>>>>>>>>> processing > >>>>>>>>>>>>> of the > >>>>>>>>>>>>> message > >>>>>>>>>>> > >>>>>>>>>>> would > >>>>>>>>>>> > >>>>>>>>>>>>> affect that list, thus affecting all > >>>>>>>>>>>>> threads > >>>>>>>>>>>>> as well > >>>>>>>>>>>>> as future > >>>>>>>>>>>>> method > >>>>>>>>>>>>> calls on > >>>>>>>>>>>>> the same thread. Thinking about > >>>>>>>>>>>>> this, > >>>>>>>>>>>>> it may > >>>>>>>>>>>>> make sense to > >>>>>>>>>>>>> clone > >>>>>>>>>>>>> the > >>>>>>>>>>>>> list > >>>>>>>>>>>>> at the very start of processing > >>>>>>>>>>>>> messages to > >>>>>>>>>>>>> make > >>>>>>>>>>>>> sure the header list > >>>>>>>>>>> > >>>>>>>>>>> in > >>>>>>>>>>> > >>>>>>>>>>>>> the > >>>>>>>>>>>>> request context isn't modified. > >>>>>>>>>>>>> That might > >>>>>>>>>>>>> be > >>>>>>>>>>>>> worth filing a > >>>>>>>>>>>>> jira > >>>>>>>>>>> > >>>>>>>>>>> for. > >>>>>>>>>>> > >>>>>>>>>>>>> For now, I would suggest doing 2 > >>>>>>>>>>>>> things: > >>>>>>>>>>>>> > >>>>>>>>>>>>> 1) Use the thread local request > >>>>>>>>>>>>> contexts: > >>>>>>>>>>>>> > >>>>>>>>>>>>> <jaxws:client id="xService" > >>>>>>>>>>>>> > >>>>>>>>>>>>> serviceClass="com.XS > >>>>>>>>>>>>> ervice" > >>>>>>>>>>>>> address="#xServiceUr > >>>>>>>>>>>>> l" > > >>>>>>>>>>>>> > >>>>>>>>>>>>> <jaxws:properties> > >>>>>>>>>>>>> > >>>>>>>>>>>>> <entry > >>>>>>>>>>>>> key="thread.lo > >>>>>>>>>>>>> cal.requ > >>>>>>>>>>>>> est.con > >>>>>>>>>>>>> text" > >>>>>>>>>>>>> value="true" > >>>>>>>>>>>>> /> > >>>>>>>>>>>>> > >>>>>>>>>>>>> </jaxws:properties> > >>>>>>>>>>>>> > >>>>>>>>>>>>> </jaxws:client> > >>>>>>>>>>>>> > >>>>>>>>>>>>> 2) After returning from any method > >>>>>>>>>>>>> where you > >>>>>>>>>>>>> had set > >>>>>>>>>>>>> a header, > >>>>>>>>>>>>> call: > >>>>>>>>>>>>> context.getMessageContext().remove(H > >>>>>>>>>>>>> eader.HE > >>>>>>>>>>>>> ADER_LIS > >>>>>>>>>>>>> T); > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> That should keep things in a good > >>>>>>>>>>>>> state. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Dan > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Friday, July 22, 2011 10:25:10 AM > >>>>>>>>>>>>> Algirdas Veitas > >>> > >>> wrote: > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We are getting a > >>>>>>>>>>>>>> ConcurrentModificationException > >>>>>>>>>>>>>> exception > >>>>>>>>>>>>>> in our > >>>>>>>>>>>>>> CXF > >>>>>>>>>>>>>> client, when it is processing > >>>>>>>>>>>>>> SOAPHeaders. We > >>>>>>>>>>>>>> did find > >>>>>>>>>>>>>> this > >>>>>>>>>>>>>> following link that describes > >>>>>>>>>>>>>> what is > >>>>>>>>>>>>>> going on > >>>>>>>>> > >>>>>>>>> http://cxf.547215.n5.nabble.com/jira-Created-CXF-276 > >>>>>>>>> 2-Cannot > >>>>>>>>> -deploy- csta > >>>>>>>>> > >>>>>>>>>>>>> -web > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -service-td586662.html#a586663, > >>>>>>>>>>>>>> and we > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>> address that > >>>>>>>>>>>>>> (Header > >>>>>>>>>>>>>> object > >>>>>>>>>>>>>> needs to be unique across > >>>>>>>>>>>>>> threads), but > >>>>>>>>>>>>>> something else > >>>>>>>>>>>>>> strange is > >>>>>>>>>>>>>> happening: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> To start here is our client > >>>>>>>>>>>>>> setup > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> <!-- Web Service Clients --> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> <jaxws:client > >>>>>>>>>>>>>> id="xService" > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> serviceClass="co > >>>>>>>>>>>>>> m.XServi > >>>>>>>>>>>>>> ce" > >>>>>>>>>>>>>> address="#xServi > >>>>>>>>>>>>>> ceUrl" > >>>>>>>>>>>>>> /> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This WSDL that is associate with > >>>>>>>>>>>>>> this > >>>>>>>>>>>>>> client has > >>>>>>>>>>>>>> some > >>>>>>>>>>>>>> methods that > >>>>>>>>>>>>>> DO > >>>>>>>>>>>>> > >>>>>>>>>>>>> have a > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Header and some DO NOT. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> And here is the shortened stack > >>>>>>>>>>>>>> trace: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> <http://newportave.jira.com/wiki > >>>>>>>>>>>>>> /display > >>>>>>>>>>>>>> /21+Jul+ > >>>>>>>>>>>>>> 2011+19/11%3 > >>>>>>>>>>>>>> A46%2C87 4> > >>>>>>>>>>>>>> java.util.ConcurrentModification > >>>>>>>>>>>>>> Exceptio > >>>>>>>>>>>>>> n > >>>>>>>>>>>>>> at > >>>>>>>>> > >>>>>>>>> java.util.AbstractList$Itr.checkForComodification(Ab > >>>>>>>>> stractLi > >>>>>>>>> st.java: 372) > >>>>>>>>> > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.AbstractList$Itr.next( > >>>>>>>>>>>>>> Abstract > >>>>>>>>>>>>>> List.ja > >>>>>>>>>>>>>> va:343) > >>>>>>>>>>>>>> at > >>>>>>>>> > >>>>>>>>> org.apache.cxf.binding.soap.interceptor.SoapOutInter > >>>>>>>>> ceptor.w > >>>>>>>>> riteSoap Enve > >>>>>>>>> > >>>>>>>>>>>>> lope > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Start(SoapOutInterceptor.java:13 > >>>>>>>>>>>>>> 9) at > >>>>>>>>> > >>>>>>>>> org.apache.cxf.binding.soap.interceptor.SoapOutInter > >>>>>>>>> ceptor.h > >>>>>>>>> andleMes sage > >>>>>>>>> > >>>>>>>>>>>>> (Soa > >>>>>>>>>>>>> > >>>>>>>>>>>>>> pOutInterceptor.java:81) at > >>>>>>>>> > >>>>>>>>> org.apache.cxf.binding.soap.interceptor.SoapOutInter > >>>>>>>>> ceptor.h > >>>>>>>>> andleMes sage > >>>>>>>>> > >>>>>>>>>>>>> (Soa > >>>>>>>>>>>>> > >>>>>>>>>>>>>> pOutInterceptor.java:61) at > >>>>>>>>> > >>>>>>>>> org.apache.cxf.phase.PhaseInterceptorChain.doInterce > >>>>>>>>> pt(Phase > >>>>>>>>> Intercep torC > >>>>>>>>> > >>>>>>>>>>>>> hain > >>>>>>>>>>>>> > >>>>>>>>>>>>>> .java:255) at > >>>>>>>>>>>>> > >>>>>>>>>>>>> org.apache.cxf.endpoint.ClientImpl.i > >>>>>>>>>>>>> nvoke(Cl > >>>>>>>>>>>>> ientImpl > >>>>>>>>>>>>> .java:516) > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Here is the rub: This exception > >>>>>>>>>>>>>> is being > >>>>>>>>>>>>>> thrown > >>>>>>>>>>>>>> when we are > >>>>>>>>>>>>>> calling > >>>>>>>>>>>>>> a > >>>>>>>>>>>>> > >>>>>>>>>>>>> method > >>>>>>>>>>>>> > >>>>>>>>>>>>>> that DOES NOT have a Header, but > >>>>>>>>>>>>>> on line > >>>>>>>>>>>>>> 139 in > >>>>>>>>>>>>>> SoapOutInterceptor, > >>>>>>>>>>>>>> it > >>>>>>>>>>>>>> looks like we are trying to > >>>>>>>>>>>>>> process an > >>>>>>>>>>>>>> list that > >>>>>>>>>>>>>> has atleast > >>>>>>>>>>>>>> 1 > >>>>>>>>>>>>>> header. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Am > >>>>>>>>>>>>> > >>>>>>>>>>>>>> pretty certain (this is > >>>>>>>>>>>>>> occurring only > >>>>>>>>>>>>>> in a > >>>>>>>>>>>>>> specific > >>>>>>>>>>>>>> environment) > >>>>>>>>>>>>>> that > >>>>>>>>>>>>>> other methods on this service > >>>>>>>>>>>>>> were made > >>>>>>>>>>>>>> previous > >>>>>>>>>>>>>> to this > >>>>>>>>>>>>>> error that > >>>>>>>>>>>>>> did > >>>>>>>>>>>>>> include a SOAP Header. Am > >>>>>>>>>>>>>> speculating, > >>>>>>>>>>>>>> but is > >>>>>>>>>>>>>> the SOAP > >>>>>>>>>>>>>> Header being > >>>>>>>>>>>>> > >>>>>>>>>>>>> cached > >>>>>>>>>>>>> > >>>>>>>>>>>>>> in the request context and the > >>>>>>>>>>>>>> request > >>>>>>>>>>>>>> context > >>>>>>>>>>>>>> is scoped per > >>>>>>>>>>>>>> client > >>>>>>>>>>>>>> instance as per the FAQ ( > >>>>>>>>>>>>>> http://cxf.apache.org/faq.html#F > >>>>>>>>>>>>>> AQ-AreJA > >>>>>>>>>>>>>> XWSclien > >>>>>>>>>>>>>> tproxiesthre > >>>>>>>>>>>>>> adsafe%3 F)? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> Al > >>>>>>>>>>>>> > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> Daniel Kulp > >>>>>>>>>>>>> [email protected] > >>>>>>>>>>>>> http://dankulp.com/blog > >>>>>>>>>>>>> Talend - http://www.talend.com > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Daniel Kulp > >>>>>>>>>>> [email protected] > >>>>>>>>>>> http://dankulp.com/blog > >>>>>>>>>>> Talend - http://www.talend.com > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Daniel Kulp > >>>>>>>>> [email protected] > >>>>>>>>> http://dankulp.com/blog > >>>>>>>>> Talend - http://www.talend.com > >>>>> > >>>>> -- > >>>>> Daniel Kulp > >>>>> [email protected] > >>>>> http://dankulp.com/blog > >>>>> Talend - http://www.talend.com > >>> > >>> -- > >>> Daniel Kulp > >>> [email protected] > >>> http://dankulp.com/blog > >>> Talend - http://www.talend.com -- Daniel Kulp [email protected] http://dankulp.com/blog Talend - http://www.talend.com
