Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still destroying lower start level items before higher start level items. I'm surprised no one else is noticing this. Here is what I see in my logs (leaving out hundreds of error messages). The lines with the * are start level 60 and the ** is start level 80. The rest of the start level 80 ones are killed even later...after karaf bundles are gone.
10:46:58,361 | INFO | Thread-2 | Main | - - | Karaf shutdown socket: received shutdown command. Stopping framework... 10:46:58,378 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle locationvalidation-google **10:46:58,389 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle operations-service 10:46:58,416 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle utils *10:46:58,418 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-broker.xml 10:47:00,987 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands 10:47:01,038 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle activemq-karaf 10:47:01,056 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands 10:47:01,067 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean 10:47:01,069 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.web 10:47:01,071 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev 10:47:01,073 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages 10:47:01,075 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.log 10:47:01,077 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.config 10:47:01,081 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.services 10:47:01,083 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.management.mbeans.system 10:47:01,085 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.command 10:47:01,090 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.ssh 10:47:01,104 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.management 10:47:01,106 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.common 10:47:01,120 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.packages 10:47:01,127 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.core 10:47:01,165 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.admin.command 10:47:01,167 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.dev 10:47:01,188 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.diagnostic.management 10:47:01,190 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.features.command 10:47:01,250 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.jaas.command 10:47:01,274 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.commands 10:47:01,333 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.console 10:47:01,365 | INFO | rint Extender: 3 | BlueprintContainerImpl | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.log is waiting for namespace handlers [ http://karaf.apache.org/xmlns/shell/v1.0.0] 10:47:01,372 | INFO | rint Extender: 1 | BlueprintContainerImpl | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.web is waiting for namespace handlers [ http://karaf.apache.org/xmlns/shell/v1.0.0] 10:47:01,441 | INFO | rint Extender: 2 | BlueprintContainerImpl | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.osgi is waiting for namespace handlers [ http://karaf.apache.org/xmlns/shell/v1.0.0] 10:47:01,478 | INFO | rint Extender: 3 | BlueprintContainerImpl | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle org.apache.karaf.shell.config is waiting for namespace handlers [ http://karaf.apache.org/xmlns/shell/v1.1.0] 10:47:01,484 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.web 10:47:01,487 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.config 10:47:01,490 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.osgi 10:47:01,492 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.shell.log 10:47:01,496 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle org.apache.karaf.deployer.wrap **10:47:01,505 | INFO | FelixShutdown | BlueprintExtender | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying BlueprintContainer for bundle application-bundle On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[email protected]> wrote: > Karaf dev team is currently calling a vote to karaf-2.3.4, You may want to > try it out and provide feedback > > -D > > > On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[email protected]> wrote: > >> Thanks Dan. Since 1.1.1 seems to have never been released I tried to >> upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf >> 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any >> news on when 2.3.4 will be released? >> >> Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this >> issue? >> >> Chris >> >> >> On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[email protected]> wrote: >> >>> see if this help >>> >>> >>> http://karaf.922171.n3.nabble.com/Orderly-shutting-down-services-td4027336.html >>> >>> -D >>> >>> >>> On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[email protected]>wrote: >>> >>>> JB, >>>> >>>> It's the default config.properties files. >>>> >>>> Chris >>>> >>>> >>>> On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré <[email protected] >>>> > wrote: >>>> >>>>> Hi Chris, >>>>> >>>>> did you change something in the etc config file or it's the default >>>>> one ? >>>>> >>>>> Regards >>>>> JB >>>>> >>>>> >>>>> On 02/14/2014 06:54 PM, Chris Geer wrote: >>>>> >>>>>> We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now >>>>>> having some significant issues on shutdown. Previously, everything >>>>>> would >>>>>> shut down in a normal order but now for some reasons things with lower >>>>>> start levels are shutting down first. Our big problem is ActiveMQ. We >>>>>> have our activemq broker starting with a start level of 60 and our >>>>>> normal bundles (mostly camel) starting at 80. But on shutdown the >>>>>> first >>>>>> thing that shows up in the logs is this. >>>>>> >>>>>> 23:46:38,243 | INFO | Thread-2 | Main >>>>>> | - - | Karaf shutdown socket: received shutdown command. >>>>>> Stopping >>>>>> framework... >>>>>> 23:46:38,266 | INFO | FelixShutdown | BlueprintExtender >>>>>> | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying >>>>>> BlueprintContainer for bundle activemq-broker.xml >>>>>> 23:46:38,290 | INFO | FelixShutdown | BrokerService >>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache >>>>>> ActiveMQ >>>>>> 5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is >>>>>> shutting down >>>>>> 23:46:38,368 | INFO | FelixShutdown | TransportConnector >>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>> openwire stopped >>>>>> 23:46:38,436 | INFO | FelixShutdown | TransportConnector >>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>> stomp >>>>>> stopped >>>>>> >>>>>> >>>>>> As you can imagine that causes problems because now that the broker >>>>>> doesn't exist the bundles that use it start throwing errors and >>>>>> in-flight transactions are failing. >>>>>> >>>>>> >>>>>> 23:46:38,440 | INFO | : vm://default#0 | PooledConnectionFactory >>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring >>>>>> connection ActiveMQConnection >>>>>> {id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID: >>>>>> CXTPRO-CG.local-49654-1392332684735-2:8,started=false} >>>>>> on IOException: >>>>>> org.apache.activemq.transport.TransportDisposedIOException: peer >>>>>> (vm://default#1) stopped. >>>>>> 23:46:38,444 | WARN | n-notifications] | >>>>>> faultJmsMessageListenerContainer | 163 - org.springframework.jms - >>>>>> 3.2.4.RELEASE | Setup of JMS message listener invoker failed for >>>>>> destination 'permission-notifications' - trying to recover. Cause: >>>>>> peer >>>>>> (vm://default#3) stopped. >>>>>> 23:46:38,444 | WARN | t-notifications] | PooledSession >>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught >>>>>> exception >>>>>> trying rollback() when putting session back into the pool, will >>>>>> invalidate. javax.jms.IllegalStateException: The Session is closed >>>>>> >>>>>> >>>>>> Is there anything we can do to restore an orderly shutdown order? >>>>>> >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Chris >>>>>> >>>>>> >>>>> -- >>>>> Jean-Baptiste Onofré >>>>> [email protected] >>>>> http://blog.nanthrax.net >>>>> Talend - http://www.talend.com >>>>> >>>> >>>> >>> >> >
