Just found this ticket, I think it's a good representation of what I'm seeing.
KARAF-2075 On Sat, Feb 15, 2014 at 11:29 AM, Chris Geer <[email protected]> wrote: > As a follow-up, if I manually go into JMX and set the start level > attribute on the osgi.core framework mbean to 1 it does shutdown the > bundles in the correct order. Then I can stop the server and all is well. > > Chris > > > On Sat, Feb 15, 2014 at 10:56 AM, Chris Geer <[email protected]>wrote: > >> Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still >> destroying lower start level items before higher start level items. I'm >> surprised no one else is noticing this. Here is what I see in my logs >> (leaving out hundreds of error messages). The lines with the * are start >> level 60 and the ** is start level 80. The rest of the start level 80 ones >> are killed even later...after karaf bundles are gone. >> >> 10:46:58,361 | INFO | Thread-2 | Main >> | - - | Karaf shutdown socket: received shutdown command. Stopping >> framework... >> 10:46:58,378 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle locationvalidation-google >> **10:46:58,389 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle operations-service >> 10:46:58,416 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle utils >> *10:46:58,418 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle activemq-broker.xml >> 10:47:00,987 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands >> 10:47:01,038 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle activemq-karaf >> 10:47:01,056 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands >> 10:47:01,067 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean >> 10:47:01,069 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.web >> 10:47:01,071 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev >> 10:47:01,073 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages >> 10:47:01,075 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.log >> 10:47:01,077 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.config >> 10:47:01,081 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.services >> 10:47:01,083 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.management.mbeans.system >> 10:47:01,085 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.diagnostic.command >> 10:47:01,090 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.ssh >> 10:47:01,104 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.admin.management >> 10:47:01,106 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.diagnostic.common >> 10:47:01,120 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.packages >> 10:47:01,127 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.admin.core >> 10:47:01,165 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.admin.command >> 10:47:01,167 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.dev >> 10:47:01,188 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.diagnostic.management >> 10:47:01,190 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.features.command >> 10:47:01,250 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.jaas.command >> 10:47:01,274 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.commands >> 10:47:01,333 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.console >> 10:47:01,365 | INFO | rint Extender: 3 | BlueprintContainerImpl >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle >> org.apache.karaf.shell.log is waiting for namespace handlers [ >> http://karaf.apache.org/xmlns/shell/v1.0.0] >> 10:47:01,372 | INFO | rint Extender: 1 | BlueprintContainerImpl >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle >> org.apache.karaf.shell.web is waiting for namespace handlers [ >> http://karaf.apache.org/xmlns/shell/v1.0.0] >> 10:47:01,441 | INFO | rint Extender: 2 | BlueprintContainerImpl >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle >> org.apache.karaf.shell.osgi is waiting for namespace handlers [ >> http://karaf.apache.org/xmlns/shell/v1.0.0] >> 10:47:01,478 | INFO | rint Extender: 3 | BlueprintContainerImpl >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle >> org.apache.karaf.shell.config is waiting for namespace handlers [ >> http://karaf.apache.org/xmlns/shell/v1.1.0] >> 10:47:01,484 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.web >> 10:47:01,487 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.config >> 10:47:01,490 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.osgi >> 10:47:01,492 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.shell.log >> 10:47:01,496 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle org.apache.karaf.deployer.wrap >> **10:47:01,505 | INFO | FelixShutdown | BlueprintExtender >> | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying >> BlueprintContainer for bundle application-bundle >> >> >> On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[email protected]> wrote: >> >>> Karaf dev team is currently calling a vote to karaf-2.3.4, You may want >>> to try it out and provide feedback >>> >>> -D >>> >>> >>> On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[email protected]>wrote: >>> >>>> Thanks Dan. Since 1.1.1 seems to have never been released I tried to >>>> upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf >>>> 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any >>>> news on when 2.3.4 will be released? >>>> >>>> Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed >>>> this issue? >>>> >>>> Chris >>>> >>>> >>>> On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[email protected]> wrote: >>>> >>>>> see if this help >>>>> >>>>> >>>>> http://karaf.922171.n3.nabble.com/Orderly-shutting-down-services-td4027336.html >>>>> >>>>> -D >>>>> >>>>> >>>>> On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[email protected]>wrote: >>>>> >>>>>> JB, >>>>>> >>>>>> It's the default config.properties files. >>>>>> >>>>>> Chris >>>>>> >>>>>> >>>>>> On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Chris, >>>>>>> >>>>>>> did you change something in the etc config file or it's the default >>>>>>> one ? >>>>>>> >>>>>>> Regards >>>>>>> JB >>>>>>> >>>>>>> >>>>>>> On 02/14/2014 06:54 PM, Chris Geer wrote: >>>>>>> >>>>>>>> We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are >>>>>>>> now >>>>>>>> having some significant issues on shutdown. Previously, everything >>>>>>>> would >>>>>>>> shut down in a normal order but now for some reasons things with >>>>>>>> lower >>>>>>>> start levels are shutting down first. Our big problem is ActiveMQ. >>>>>>>> We >>>>>>>> have our activemq broker starting with a start level of 60 and our >>>>>>>> normal bundles (mostly camel) starting at 80. But on shutdown the >>>>>>>> first >>>>>>>> thing that shows up in the logs is this. >>>>>>>> >>>>>>>> 23:46:38,243 | INFO | Thread-2 | Main >>>>>>>> | - - | Karaf shutdown socket: received shutdown command. >>>>>>>> Stopping >>>>>>>> framework... >>>>>>>> 23:46:38,266 | INFO | FelixShutdown | BlueprintExtender >>>>>>>> | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying >>>>>>>> BlueprintContainer for bundle activemq-broker.xml >>>>>>>> 23:46:38,290 | INFO | FelixShutdown | BrokerService >>>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache >>>>>>>> ActiveMQ >>>>>>>> 5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is >>>>>>>> shutting down >>>>>>>> 23:46:38,368 | INFO | FelixShutdown | TransportConnector >>>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>>>> openwire stopped >>>>>>>> 23:46:38,436 | INFO | FelixShutdown | TransportConnector >>>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>>>> stomp >>>>>>>> stopped >>>>>>>> >>>>>>>> >>>>>>>> As you can imagine that causes problems because now that the broker >>>>>>>> doesn't exist the bundles that use it start throwing errors and >>>>>>>> in-flight transactions are failing. >>>>>>>> >>>>>>>> >>>>>>>> 23:46:38,440 | INFO | : vm://default#0 | PooledConnectionFactory >>>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring >>>>>>>> connection ActiveMQConnection >>>>>>>> {id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID: >>>>>>>> CXTPRO-CG.local-49654-1392332684735-2:8,started=false} >>>>>>>> on IOException: >>>>>>>> org.apache.activemq.transport.TransportDisposedIOException: peer >>>>>>>> (vm://default#1) stopped. >>>>>>>> 23:46:38,444 | WARN | n-notifications] | >>>>>>>> faultJmsMessageListenerContainer | 163 - org.springframework.jms - >>>>>>>> 3.2.4.RELEASE | Setup of JMS message listener invoker failed for >>>>>>>> destination 'permission-notifications' - trying to recover. Cause: >>>>>>>> peer >>>>>>>> (vm://default#3) stopped. >>>>>>>> 23:46:38,444 | WARN | t-notifications] | PooledSession >>>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught >>>>>>>> exception >>>>>>>> trying rollback() when putting session back into the pool, will >>>>>>>> invalidate. javax.jms.IllegalStateException: The Session is closed >>>>>>>> >>>>>>>> >>>>>>>> Is there anything we can do to restore an orderly shutdown order? >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Chris >>>>>>>> >>>>>>>> >>>>>>> -- >>>>>>> Jean-Baptiste Onofré >>>>>>> [email protected] >>>>>>> http://blog.nanthrax.net >>>>>>> Talend - http://www.talend.com >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
