As a follow-up, if I manually go into JMX and set the start level attribute on the osgi.core framework mbean to 1 it does shutdown the bundles in the correct order. Then I can stop the server and all is well.
Chris On Sat, Feb 15, 2014 at 10:56 AM, Chris Geer <[email protected]> wrote: > Well, I upgraded to Karaf 2.3.4 and the problem is no better, it's still > destroying lower start level items before higher start level items. I'm > surprised no one else is noticing this. Here is what I see in my logs > (leaving out hundreds of error messages). The lines with the * are start > level 60 and the ** is start level 80. The rest of the start level 80 ones > are killed even later...after karaf bundles are gone. > > 10:46:58,361 | INFO | Thread-2 | Main > | - - | Karaf shutdown socket: received shutdown command. Stopping > framework... > 10:46:58,378 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle locationvalidation-google > **10:46:58,389 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle operations-service > 10:46:58,416 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle utils > *10:46:58,418 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle activemq-broker.xml > 10:47:00,987 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.camel.karaf.camel-karaf-commands > 10:47:01,038 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle activemq-karaf > 10:47:01,056 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.cxf.karaf.cxf-karaf-commands > 10:47:01,067 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle io.hawt.hawtio-json-schema-mbean > 10:47:01,069 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.web > 10:47:01,071 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.dev > 10:47:01,073 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.packages > 10:47:01,075 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.log > 10:47:01,077 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.config > 10:47:01,081 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.services > 10:47:01,083 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.management.mbeans.system > 10:47:01,085 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.diagnostic.command > 10:47:01,090 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.ssh > 10:47:01,104 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.admin.management > 10:47:01,106 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.diagnostic.common > 10:47:01,120 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.packages > 10:47:01,127 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.admin.core > 10:47:01,165 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.admin.command > 10:47:01,167 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.dev > 10:47:01,188 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.diagnostic.management > 10:47:01,190 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.features.command > 10:47:01,250 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.jaas.command > 10:47:01,274 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.commands > 10:47:01,333 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.console > 10:47:01,365 | INFO | rint Extender: 3 | BlueprintContainerImpl > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle > org.apache.karaf.shell.log is waiting for namespace handlers [ > http://karaf.apache.org/xmlns/shell/v1.0.0] > 10:47:01,372 | INFO | rint Extender: 1 | BlueprintContainerImpl > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle > org.apache.karaf.shell.web is waiting for namespace handlers [ > http://karaf.apache.org/xmlns/shell/v1.0.0] > 10:47:01,441 | INFO | rint Extender: 2 | BlueprintContainerImpl > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle > org.apache.karaf.shell.osgi is waiting for namespace handlers [ > http://karaf.apache.org/xmlns/shell/v1.0.0] > 10:47:01,478 | INFO | rint Extender: 3 | BlueprintContainerImpl > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Bundle > org.apache.karaf.shell.config is waiting for namespace handlers [ > http://karaf.apache.org/xmlns/shell/v1.1.0] > 10:47:01,484 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.web > 10:47:01,487 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.config > 10:47:01,490 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.osgi > 10:47:01,492 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.shell.log > 10:47:01,496 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle org.apache.karaf.deployer.wrap > **10:47:01,505 | INFO | FelixShutdown | BlueprintExtender > | 7 - org.apache.aries.blueprint.core - 1.4.0 | Destroying > BlueprintContainer for bundle application-bundle > > > On Fri, Feb 14, 2014 at 7:14 PM, Dan Tran <[email protected]> wrote: > >> Karaf dev team is currently calling a vote to karaf-2.3.4, You may want >> to try it out and provide feedback >> >> -D >> >> >> On Fri, Feb 14, 2014 at 4:48 PM, Chris Geer <[email protected]>wrote: >> >>> Thanks Dan. Since 1.1.1 seems to have never been released I tried to >>> upgrade to 1.2.0 but that broke the container pretty good. I see that Karaf >>> 2.3.4 has been upgraded to core 1.4.0 so hopefully that will fix it. Any >>> news on when 2.3.4 will be released? >>> >>> Since 1.1.1 isn't even in JIRA, any idea what ticket actually fixed this >>> issue? >>> >>> Chris >>> >>> >>> On Fri, Feb 14, 2014 at 3:20 PM, Dan Tran <[email protected]> wrote: >>> >>>> see if this help >>>> >>>> >>>> http://karaf.922171.n3.nabble.com/Orderly-shutting-down-services-td4027336.html >>>> >>>> -D >>>> >>>> >>>> On Fri, Feb 14, 2014 at 10:02 AM, Chris Geer <[email protected]>wrote: >>>> >>>>> JB, >>>>> >>>>> It's the default config.properties files. >>>>> >>>>> Chris >>>>> >>>>> >>>>> On Fri, Feb 14, 2014 at 10:59 AM, Jean-Baptiste Onofré < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Chris, >>>>>> >>>>>> did you change something in the etc config file or it's the default >>>>>> one ? >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> >>>>>> On 02/14/2014 06:54 PM, Chris Geer wrote: >>>>>> >>>>>>> We just upgraded our container from Karaf 2.2.10 to 2.3.3 and are now >>>>>>> having some significant issues on shutdown. Previously, everything >>>>>>> would >>>>>>> shut down in a normal order but now for some reasons things with >>>>>>> lower >>>>>>> start levels are shutting down first. Our big problem is ActiveMQ. We >>>>>>> have our activemq broker starting with a start level of 60 and our >>>>>>> normal bundles (mostly camel) starting at 80. But on shutdown the >>>>>>> first >>>>>>> thing that shows up in the logs is this. >>>>>>> >>>>>>> 23:46:38,243 | INFO | Thread-2 | Main >>>>>>> | - - | Karaf shutdown socket: received shutdown command. >>>>>>> Stopping >>>>>>> framework... >>>>>>> 23:46:38,266 | INFO | FelixShutdown | BlueprintExtender >>>>>>> | 7 - org.apache.aries.blueprint.core - 1.1.0 | Destroying >>>>>>> BlueprintContainer for bundle activemq-broker.xml >>>>>>> 23:46:38,290 | INFO | FelixShutdown | BrokerService >>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Apache >>>>>>> ActiveMQ >>>>>>> 5.9.0 (default, ID:CXTPRO-CG.local-49654-1392332684735-0:1) is >>>>>>> shutting down >>>>>>> 23:46:38,368 | INFO | FelixShutdown | TransportConnector >>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>>> openwire stopped >>>>>>> 23:46:38,436 | INFO | FelixShutdown | TransportConnector >>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Connector >>>>>>> stomp >>>>>>> stopped >>>>>>> >>>>>>> >>>>>>> As you can imagine that causes problems because now that the broker >>>>>>> doesn't exist the bundles that use it start throwing errors and >>>>>>> in-flight transactions are failing. >>>>>>> >>>>>>> >>>>>>> 23:46:38,440 | INFO | : vm://default#0 | PooledConnectionFactory >>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Expiring >>>>>>> connection ActiveMQConnection >>>>>>> {id=ID:CXTPRO-CG.local-49654-1392332684735-3:1,clientId=ID: >>>>>>> CXTPRO-CG.local-49654-1392332684735-2:8,started=false} >>>>>>> on IOException: >>>>>>> org.apache.activemq.transport.TransportDisposedIOException: peer >>>>>>> (vm://default#1) stopped. >>>>>>> 23:46:38,444 | WARN | n-notifications] | >>>>>>> faultJmsMessageListenerContainer | 163 - org.springframework.jms - >>>>>>> 3.2.4.RELEASE | Setup of JMS message listener invoker failed for >>>>>>> destination 'permission-notifications' - trying to recover. Cause: >>>>>>> peer >>>>>>> (vm://default#3) stopped. >>>>>>> 23:46:38,444 | WARN | t-notifications] | PooledSession >>>>>>> | 133 - org.apache.activemq.activemq-osgi - 5.9.0 | Caught >>>>>>> exception >>>>>>> trying rollback() when putting session back into the pool, will >>>>>>> invalidate. javax.jms.IllegalStateException: The Session is closed >>>>>>> >>>>>>> >>>>>>> Is there anything we can do to restore an orderly shutdown order? >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Chris >>>>>>> >>>>>>> >>>>>> -- >>>>>> Jean-Baptiste Onofré >>>>>> [email protected] >>>>>> http://blog.nanthrax.net >>>>>> Talend - http://www.talend.com >>>>>> >>>>> >>>>> >>>> >>> >> >
