Thanks for raising this issue, Timothy. I'd like to get to the bottom of it, so I've raised ISIS-1525 [1] referencing this thread.
In 1.13.0 I introduced a threadpool in order to perform some of the initialization in parallel. My guess is that the concurrent modification is arising from this. But I need to look at the code again to remind myself; could be something else completely. Thx Dan [1] https://issues.apache.org/jira/browse/ISIS-1525 On 18 October 2016 at 22:01, Simecsek Timothy <[email protected]> wrote: > Hi Jeroen, > > Thanks a lot for your input, normally hot deployment works fine. My > problem occurs only sometimes and can persist even after killing and > starting tomcat again. > > By the way: > For issues with hot deployment I like VisualVM which ships with JDK and > helped me a lot to identify thread groups/execution services/... that I > started but not properly stopped on application undeploy. > > Regards Timothy > > -----Ursprüngliche Nachricht----- > Von: Jeroen van der Wal [mailto:[email protected]] > Gesendet: Dienstag, 18. Oktober 2016 21:42 > An: users > Betreff: Re: Problems with startup since 1.13.0 > > Hi Timothy, > > I've had issues when hot-deploying wars and had to kill the process to so > now we have a deployment script that stops the tomcat service, waits until > the service has completely shut down, deploys and starts the service again. > > But you could argue it's a hack ;-) > > HTH > > Jeroen > > On 14 October 2016 at 16:42, Simecsek Timothy < > [email protected]> > wrote: > > > Hey, > > > > Has anyone observed problems with running version 1.13.0? > > > > We upgraded a while ago and a bit after that we saw sometimes this > > error when redeploying our application on tomcat: > > 2016-10-14 14:29:21,690 [IsisWicketApplication http-nio-18080-exec-103 > > ERROR] Failed to initialize > > com.google.inject.ProvisionException: Unable to provision, see the > > following errors: > > > > 1) Error in custom provider, java.util.ConcurrentModificationException > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:109) > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:109) > > while locating org.apache.isis.core.runtime.system.session. > > IsisSessionFactory > > for field at org.apache.isis.viewer.wicket. > > viewer.IsisWicketApplication.isisSessionFactory( > > IsisWicketApplication.java:138) > > while locating domainapp.webapp.A1FulfilllmentApplication > > > > 1 error > > at com.google.inject.internal.Errors. > > throwProvisionExceptionIfErrorsExist(Errors.java:482) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:67) > > at com.google.inject.internal.InjectorImpl.injectMembers( > > InjectorImpl.java:984) > > at org.apache.isis.viewer.wicket.viewer.IsisWicketApplication. > > init(IsisWicketApplication.java:279) > > at domainapp.webapp.A1FulfilllmentApplication.init( > > A1FulfilllmentApplication.java:73) > > at org.apache.wicket.Application.initApplication(Application. > > java:823) > > at org.apache.wicket.protocol.http.WicketFilter.init( > > WicketFilter.java:427) > > at org.apache.wicket.protocol.http.WicketFilter.init( > > WicketFilter.java:351) > > at > > org.apache.catalina.core.ApplicationFilterConfig.initFilter( > > ApplicationFilterConfig.java:279) > > at org.apache.catalina.core.ApplicationFilterConfig.getFilter( > > ApplicationFilterConfig.java:260) > > at org.apache.catalina.core.ApplicationFilterConfig.<init> > > (ApplicationFilterConfig.java:105) > > at org.apache.catalina.core.StandardContext.filterStart( > > StandardContext.java:4561) > > at org.apache.catalina.core.StandardContext.startInternal( > > StandardContext.java:5204) > > at org.apache.catalina.util.LifecycleBase.start( > > LifecycleBase.java:152) > > at org.apache.catalina.core.ContainerBase.addChildInternal( > > ContainerBase.java:726) > > at org.apache.catalina.core.ContainerBase.addChild( > > ContainerBase.java:702) > > at org.apache.catalina.core.StandardHost.addChild( > > StandardHost.java:734) > > at org.apache.catalina.startup.HostConfig.deployWAR( > > HostConfig.java:952) > > at org.apache.catalina.startup.HostConfig.deployApps( > > HostConfig.java:473) > > at org.apache.catalina.startup.HostConfig.check(HostConfig. > > java:1617) > > at sun.reflect.GeneratedMethodAccessor3189.invoke(Unknown > Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.tomcat.util.modeler.BaseModelMBean.invoke( > > BaseModelMBean.java:300) > > at > > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke( > > DefaultMBeanServerInterceptor.java:819) > > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke( > > JmxMBeanServer.java:801) > > at org.apache.catalina.manager.ManagerServlet.check( > > ManagerServlet.java:1483) > > at org.apache.catalina.manager.ManagerServlet.deploy( > > ManagerServlet.java:731) > > at org.apache.catalina.manager.ManagerServlet.doPut( > > ManagerServlet.java:424) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:651) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:230) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.tomcat.websocket.server.WsFilter.doFilter( > > WsFilter.java:52) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:192) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.catalina.filters.SetCharacterEncodingFilter. > > doFilter(SetCharacterEncodingFilter.java:108) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:192) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.catalina.core.StandardWrapperValve.invoke( > > StandardWrapperValve.java:198) > > at org.apache.catalina.core.StandardContextValve.invoke( > > StandardContextValve.java:108) > > at org.apache.catalina.authenticator.AuthenticatorBase.invoke( > > AuthenticatorBase.java:632) > > at org.apache.catalina.core.StandardHostValve.invoke( > > StandardHostValve.java:140) > > at org.apache.catalina.valves.ErrorReportValve.invoke( > > ErrorReportValve.java:79) > > at org.apache.catalina.valves.AbstractAccessLogValve.invoke( > > AbstractAccessLogValve.java:620) > > at org.apache.catalina.core.StandardEngineValve.invoke( > > StandardEngineValve.java:87) > > at org.apache.catalina.connector.CoyoteAdapter.service( > > CoyoteAdapter.java:349) > > at org.apache.coyote.http11.Http11Processor.service( > > Http11Processor.java:1110) > > at org.apache.coyote.AbstractProcessorLight.process( > > AbstractProcessorLight.java:66) > > at > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process( > > AbstractProtocol.java:785) > > at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor. > > doRun(NioEndpoint.java:1425) > > at org.apache.tomcat.util.net.SocketProcessorBase.run( > > SocketProcessorBase.java:52) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:617) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( > > TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.ConcurrentModificationException > > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > > at java.util.HashMap$ValueIterator.next(HashMap.java:1466) > > at java.util.Collections$UnmodifiableCollection$1.next( > > Collections.java:1042) > > at org.apache.isis.core.runtime.system.session. > IsisSessionFactory. > > constructServices(IsisSessionFactory.java:151) > > at org.apache.isis.core.runtime.system.session. > > IsisSessionFactoryBuilder.buildSessionFactory(IsisSessionFactoryBuilder. > > java:204) > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:118) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at com.google.inject.internal.ProviderMethod$ > > ReflectionProviderMethod.doProvision(ProviderMethod.java:299) > > at com.google.inject.internal.ProviderMethod.get( > > ProviderMethod.java:172) > > at > > com.google.inject.internal.ProviderInternalFactory.provision( > > ProviderInternalFactory.java:81) > > at com.google.inject.internal.InternalFactoryToInitializable > > Adapter.provision(InternalFactoryToInitializableAdapter.java:53) > > at > > com.google.inject.internal.ProviderInternalFactory.circularGet( > > ProviderInternalFactory.java:61) > > at com.google.inject.internal.InternalFactoryToInitializable > > Adapter.get(InternalFactoryToInitializableAdapter.java:45) > > at com.google.inject.internal.ProviderToInternalFactoryAdapt > > er$1.call(ProviderToInternalFactoryAdapter.java:46) > > at com.google.inject.internal.InjectorImpl.callInContext( > > InjectorImpl.java:1103) > > at com.google.inject.internal.ProviderToInternalFactoryAdapt > > er.get(ProviderToInternalFactoryAdapter.java:40) > > at com.google.inject.internal.SingletonScope$1.get( > > SingletonScope.java:145) > > at com.google.inject.internal.InternalFactoryToProviderAdapt > > er.get(InternalFactoryToProviderAdapter.java:41) > > at com.google.inject.internal.SingleFieldInjector.inject( > > SingleFieldInjector.java:54) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:132) > > at com.google.inject.internal.MembersInjectorImpl$1.call( > > MembersInjectorImpl.java:93) > > at com.google.inject.internal.MembersInjectorImpl$1.call( > > MembersInjectorImpl.java:80) > > at com.google.inject.internal.InjectorImpl.callInContext( > > InjectorImpl.java:1092) > > at > > com.google.inject.internal.MembersInjectorImpl.injectAndNotify( > > MembersInjectorImpl.java:80) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:62) > > ... 54 more > > 2016-10-14 14:29:21,703 [ThreadPoolSupport http-nio-18080-exec-103 > INFO > > ] join'ing 6 tasks: waited 1 milliseconds > > 2016-10-14 14:29:21,703 [WicketFilter http-nio-18080-exec-103 > > ERROR] The initialization of an application with name 'WicketFilter' > > has failed. > > com.google.inject.ProvisionException: Unable to provision, see the > > following errors: > > > > 1) Error in custom provider, java.util.ConcurrentModificationException > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:109) > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:109) > > while locating org.apache.isis.core.runtime.system.session. > > IsisSessionFactory > > for field at org.apache.isis.viewer.wicket. > > viewer.IsisWicketApplication.isisSessionFactory( > > IsisWicketApplication.java:138) > > while locating domainapp.webapp.A1FulfilllmentApplication > > > > 1 error > > at com.google.inject.internal.Errors. > > throwProvisionExceptionIfErrorsExist(Errors.java:482) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:67) > > at com.google.inject.internal.InjectorImpl.injectMembers( > > InjectorImpl.java:984) > > at org.apache.isis.viewer.wicket.viewer.IsisWicketApplication. > > init(IsisWicketApplication.java:279) > > at domainapp.webapp.A1FulfilllmentApplication.init( > > A1FulfilllmentApplication.java:73) > > at org.apache.wicket.Application.initApplication(Application. > > java:823) > > at org.apache.wicket.protocol.http.WicketFilter.init( > > WicketFilter.java:427) > > at org.apache.wicket.protocol.http.WicketFilter.init( > > WicketFilter.java:351) > > at > > org.apache.catalina.core.ApplicationFilterConfig.initFilter( > > ApplicationFilterConfig.java:279) > > at org.apache.catalina.core.ApplicationFilterConfig.getFilter( > > ApplicationFilterConfig.java:260) > > at org.apache.catalina.core.ApplicationFilterConfig.<init> > > (ApplicationFilterConfig.java:105) > > at org.apache.catalina.core.StandardContext.filterStart( > > StandardContext.java:4561) > > at org.apache.catalina.core.StandardContext.startInternal( > > StandardContext.java:5204) > > at org.apache.catalina.util.LifecycleBase.start( > > LifecycleBase.java:152) > > at org.apache.catalina.core.ContainerBase.addChildInternal( > > ContainerBase.java:726) > > at org.apache.catalina.core.ContainerBase.addChild( > > ContainerBase.java:702) > > at org.apache.catalina.core.StandardHost.addChild( > > StandardHost.java:734) > > at org.apache.catalina.startup.HostConfig.deployWAR( > > HostConfig.java:952) > > at org.apache.catalina.startup.HostConfig.deployApps( > > HostConfig.java:473) > > at org.apache.catalina.startup.HostConfig.check(HostConfig. > > java:1617) > > at sun.reflect.GeneratedMethodAccessor3189.invoke(Unknown > Source) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.tomcat.util.modeler.BaseModelMBean.invoke( > > BaseModelMBean.java:300) > > at > > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke( > > DefaultMBeanServerInterceptor.java:819) > > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke( > > JmxMBeanServer.java:801) > > at org.apache.catalina.manager.ManagerServlet.check( > > ManagerServlet.java:1483) > > at org.apache.catalina.manager.ManagerServlet.deploy( > > ManagerServlet.java:731) > > at org.apache.catalina.manager.ManagerServlet.doPut( > > ManagerServlet.java:424) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:651) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:230) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.tomcat.websocket.server.WsFilter.doFilter( > > WsFilter.java:52) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:192) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.catalina.filters.SetCharacterEncodingFilter. > > doFilter(SetCharacterEncodingFilter.java:108) > > at org.apache.catalina.core.ApplicationFilterChain. > > internalDoFilter(ApplicationFilterChain.java:192) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:165) > > at org.apache.catalina.core.StandardWrapperValve.invoke( > > StandardWrapperValve.java:198) > > at org.apache.catalina.core.StandardContextValve.invoke( > > StandardContextValve.java:108) > > at org.apache.catalina.authenticator.AuthenticatorBase.invoke( > > AuthenticatorBase.java:632) > > at org.apache.catalina.core.StandardHostValve.invoke( > > StandardHostValve.java:140) > > at org.apache.catalina.valves.ErrorReportValve.invoke( > > ErrorReportValve.java:79) > > at org.apache.catalina.valves.AbstractAccessLogValve.invoke( > > AbstractAccessLogValve.java:620) > > at org.apache.catalina.core.StandardEngineValve.invoke( > > StandardEngineValve.java:87) > > at org.apache.catalina.connector.CoyoteAdapter.service( > > CoyoteAdapter.java:349) > > at org.apache.coyote.http11.Http11Processor.service( > > Http11Processor.java:1110) > > at org.apache.coyote.AbstractProcessorLight.process( > > AbstractProcessorLight.java:66) > > at > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process( > > AbstractProtocol.java:785) > > at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor. > > doRun(NioEndpoint.java:1425) > > at org.apache.tomcat.util.net.SocketProcessorBase.run( > > SocketProcessorBase.java:52) > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1142) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:617) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run( > > TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.util.ConcurrentModificationException > > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) > > at java.util.HashMap$ValueIterator.next(HashMap.java:1466) > > at java.util.Collections$UnmodifiableCollection$1.next( > > Collections.java:1042) > > at org.apache.isis.core.runtime.system.session. > IsisSessionFactory. > > constructServices(IsisSessionFactory.java:151) > > at org.apache.isis.core.runtime.system.session. > > IsisSessionFactoryBuilder.buildSessionFactory(IsisSessionFactoryBuilder. > > java:204) > > at org.apache.isis.core.runtime.runner.IsisInjectModule. > > provideIsisSessionFactory(IsisInjectModule.java:118) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:62) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at com.google.inject.internal.ProviderMethod$ > > ReflectionProviderMethod.doProvision(ProviderMethod.java:299) > > at com.google.inject.internal.ProviderMethod.get( > > ProviderMethod.java:172) > > at > > com.google.inject.internal.ProviderInternalFactory.provision( > > ProviderInternalFactory.java:81) > > at com.google.inject.internal.InternalFactoryToInitializable > > Adapter.provision(InternalFactoryToInitializableAdapter.java:53) > > at > > com.google.inject.internal.ProviderInternalFactory.circularGet( > > ProviderInternalFactory.java:61) > > at com.google.inject.internal.InternalFactoryToInitializable > > Adapter.get(InternalFactoryToInitializableAdapter.java:45) > > at com.google.inject.internal.ProviderToInternalFactoryAdapt > > er$1.call(ProviderToInternalFactoryAdapter.java:46) > > at com.google.inject.internal.InjectorImpl.callInContext( > > InjectorImpl.java:1103) > > at com.google.inject.internal.ProviderToInternalFactoryAdapt > > er.get(ProviderToInternalFactoryAdapter.java:40) > > at com.google.inject.internal.SingletonScope$1.get( > > SingletonScope.java:145) > > at com.google.inject.internal.InternalFactoryToProviderAdapt > > er.get(InternalFactoryToProviderAdapter.java:41) > > at com.google.inject.internal.SingleFieldInjector.inject( > > SingleFieldInjector.java:54) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:132) > > at com.google.inject.internal.MembersInjectorImpl$1.call( > > MembersInjectorImpl.java:93) > > at com.google.inject.internal.MembersInjectorImpl$1.call( > > MembersInjectorImpl.java:80) > > at com.google.inject.internal.InjectorImpl.callInContext( > > InjectorImpl.java:1092) > > at > > com.google.inject.internal.MembersInjectorImpl.injectAndNotify( > > MembersInjectorImpl.java:80) > > at > > com.google.inject.internal.MembersInjectorImpl.injectMembers( > > MembersInjectorImpl.java:62) > > ... 54 more > > 2016-10-14 14:29:21,720 [Application http-nio-18080-exec-103 > INFO > > ] [WicketFilter] destroy: Wicket core library initializer > > 2016-10-14 14:29:21,720 [Application http-nio-18080-exec-103 > INFO > > ] [WicketFilter] destroy: Wicket extensions initializer > > > > After this the application seems to shutdown, if I executed shutdown > > command for tomcat it isn't successful and the process hangs, only > > option is to kill it. Normally the shutdown command works perfectly! > > Only option is to wait some time and suddenly it starts normally. > > > > > > We observe a similar error sometimes during integration tests. The > > only message is: > > java.lang.RuntimeException: java.util.ConcurrentModificationException > > Caused by: java.util.ConcurrentModificationException > > And after that: > > > > java.lang.RuntimeException: org.apache.isis.core.commons. > exceptions.IsisException: > > SessionFactory already set up > > > > Caused by: org.apache.isis.core.commons.exceptions.IsisException: > > SessionFactory already set up > > > > Solution is to start the build again then it works most of the time. > > > > Has anyone an idea? > > > > Thanks & Regards Timothy > > > > ______________________________________________________________________ > > Disclaimer: This email and any attachments are sent in strictest > > confidence for the sole use of the addressee and may contain legally > > privileged, confidential, and proprietary data. If you are not the > > intended recipient, please advise the sender by replying promptly to > > this email and then delete and destroy this email and any attachments > > without any further use, copying or forwarding. > > ______________________________________________________________________ > Disclaimer: This email and any attachments are sent in strictest confidence > for the sole use of the addressee and may contain legally privileged, > confidential, and proprietary data. If you are not the intended recipient, > please advise the sender by replying promptly to this email and then delete > and destroy this email and any attachments without any further use, copying > or forwarding. >
