Hi, Great that we have nailed down to the root cause of this and fixed it. This has been a long running issue for WSO2 Message Broker..!
On Fri, Jun 28, 2013 at 10:59 PM, Shammi Jayasinghe <[email protected]> wrote: > Hi, > > According to the offline discussion i had with Prabath, we could figure > out that, there should not be any harm though we removed the shut down hook > registered from the Runtime since we are doing the same thing with OSGI > services. > > So i have created new version for Cassandra and added to svn. > > Thanks > Shammi > > > On Fri, Jun 28, 2013 at 5:58 PM, Shammi Jayasinghe <[email protected]>wrote: > >> Hi , >> >> I think we have found the root cause for this problem. In Cassandra >> Source Code , it registers a shutdown hook in the Runtime as >> >> Runtime.getRuntime().addShutdownHook(drainOnShutdown); >> >> in the Source [1]. Though we get the control with OSGI level, Due to >> this shut down hook registration, It triggers this and it make the >> cassandra server shut down before all of the others. >> >> In this kind of situation , what would be the best possible solution ? >> >> Thanks >> Shammi >> [1] >> https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.1.0/dependencies/cassandra/1.1.3-wso2v3/src/java/org/apache/cassandra/service/StorageService.java >> >> >> On Fri, Jun 28, 2013 at 4:46 PM, Pradeep Fernando <[email protected]>wrote: >> >>> There is a concept call shutdownHandlers as well. There we allow >>> component to register their own handler in to the system, and that handler >>> gets called before we calling OSGiruntime.stop. >>> >>> refer, CarbonCoreServiceComponent#shutdown. >>> >>> will that serve your requirement. Basically you have a handle during >>> Ctrl+C before OSGi runtime gets shut down. Othewise we have to look in to >>> service ordering during shutdown. >>> >>> >>> On Fri, Jun 28, 2013 at 4:37 PM, Pradeep Fernando <[email protected]>wrote: >>> >>>> and we are calling, >>>> >>>> EclipseStarter.shutdown(); >>>> >>>> >>>> >>>> On Fri, Jun 28, 2013 at 4:26 PM, Pradeep Fernando <[email protected]>wrote: >>>> >>>>> Hi >>>>> >>>>> we are calling the shutdownGracefully durign Ctrl+C. - >>>>> CarbonServerManager#addShutdownHook() >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, Jun 28, 2013 at 4:00 PM, Shammi Jayasinghe <[email protected]>wrote: >>>>> >>>>>> Basically What we want is ; >>>>>> >>>>>> We need CassandraServerService up and running until we trigger the >>>>>> shutdown with Deactivate method and it should not be shut down once we >>>>>> press CTRL+C. >>>>>> >>>>>> Is there any way that we can achieve this. >>>>>> >>>>>> Thanks >>>>>> Shammi >>>>>> >>>>>> >>>>>> On Fri, Jun 28, 2013 at 3:45 PM, Ishara Premadasa <[email protected]>wrote: >>>>>> >>>>>>> We experiencing the following issue when shutting down the MB >>>>>>> server. While performing read/write operations on cassandra and if the >>>>>>> server shuts down in middle (by clicking Ctl+C), we close the >>>>>>> CassandraServerService first, before closing the AndesService. This is >>>>>>> done >>>>>>> in order to stop any operations currently running with Cassandra before >>>>>>> shutting down the broker. However ever if this is handled in OSGI >>>>>>> level, at >>>>>>> the server shutdown the following logs are monitored, >>>>>>> >>>>>>> [2013-06-28 14:47:41,745] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook >>>>>>> triggered.... >>>>>>> [2013-06-28 14:47:41,745] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting >>>>>>> down >>>>>>> WSO2 Message Broker... >>>>>>> [2013-06-28 14:47:41,747] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - Starting to switch to >>>>>>> maintenance >>>>>>> mode... >>>>>>> *[2013-06-28 14:47:41,747] INFO {org.apache.cassandra.gms.Gossiper} >>>>>>> - Announcing shutdown >>>>>>> [2013-06-28 14:47:42,748] INFO >>>>>>> {org.apache.cassandra.net.MessagingService} - Waiting for messaging >>>>>>> service >>>>>>> to quiesce >>>>>>> [2013-06-28 14:47:42,749] INFO >>>>>>> {org.apache.cassandra.net.MessagingService} - MessagingService shutting >>>>>>> down server thread. * >>>>>>> [2013-06-28 14:47:44,027] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - Stopped all transport >>>>>>> listeners >>>>>>> [2013-06-28 14:47:44,028] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - Waiting for request service >>>>>>> completion... >>>>>>> [2013-06-28 14:47:44,030] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - All requests have been served. >>>>>>> [2013-06-28 14:47:44,030] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - Waiting for deployment >>>>>>> completion... >>>>>>> [2013-06-28 14:47:44,034] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - All deployment tasks have been >>>>>>> completed. >>>>>>> [2013-06-28 14:47:44,034] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - Waiting for server task >>>>>>> completion... >>>>>>> [2013-06-28 14:47:44,035] INFO >>>>>>> {org.wso2.carbon.core.ServerManagement} - All server tasks have been >>>>>>> completed. >>>>>>> [2013-06-28 14:47:44,035] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Shutting down WSO2 >>>>>>> Message Broker... >>>>>>> [2013-06-28 14:47:44,036] INFO >>>>>>> {org.apache.axis2.transport.tcp.TCPTransportSender} - TCP Sender >>>>>>> Shutdown >>>>>>> [2013-06-28 14:47:44,036] INFO >>>>>>> *{org.wso2.carbon.core.init.CarbonServerManager} >>>>>>> - Shutting down OSGi framework...* >>>>>>> [2013-06-28 14:47:44,198] INFO >>>>>>> {org.wso2.andes.server.registry.ApplicationRegistry} - Shutting down >>>>>>> ApplicationRegistry( >>>>>>> org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry@74607cd0) >>>>>>> >>>>>>> [2013-06-28 14:47:44,198] INFO >>>>>>> {org.wso2.andes.server.registry.ApplicationRegistry} - Shutting down >>>>>>> ApplicationRegistry: >>>>>>> org.wso2.andes.server.registry.ConfigurationFileApplicationRegistry@74607cd0 >>>>>>> [2013-06-28 14:47:44,200] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Stopping all >>>>>>> locally >>>>>>> existing global queue workers >>>>>>> [2013-06-28 14:47:44,200] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_0 >>>>>>> [2013-06-28 14:47:44,200] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_6 >>>>>>> [2013-06-28 14:47:44,200] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_8 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_7 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_5 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_4 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_1 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_3 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_2 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.cluster.GlobalQueueManager} - Removing Global >>>>>>> Queue >>>>>>> Worker for Global Queue : GlobalQueue_9 >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.store.CassandraMessageStore} - Stopping all >>>>>>> current >>>>>>> queue message publishers >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.store.CassandraMessageStore} - Stopping all >>>>>>> current >>>>>>> topic message publishers >>>>>>> [2013-06-28 14:47:44,201] INFO >>>>>>> {org.wso2.andes.server.store.CassandraMessageStore} - Stopping all >>>>>>> global >>>>>>> queue workers locally >>>>>>> [2013-06-28 14:47:44,202] INFO >>>>>>> {org.wso2.andes.server.security.auth.manager.PrincipalDatabaseAuthenticationManager} >>>>>>> - Unregistering UserManagementMBean >>>>>>> [2013-06-28 14:47:44,205] INFO >>>>>>> {org.wso2.carbon.andes.internal.QpidServiceComponent} - Deactivating >>>>>>> Carbonized Cassandra Server... >>>>>>> [2013-06-28 14:47:44,310] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Stopping >>>>>>> CarbonServerManager... >>>>>>> [2013-06-28 14:47:44,342] INFO >>>>>>> {org.wso2.carbon.tomcat.ext.internal.CarbonTomcatServiceComponent} - >>>>>>> Stopping the carbon web-app registered under : / >>>>>>> [2013-06-28 14:47:45,320] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown complete >>>>>>> [2013-06-28 14:47:45,320] INFO >>>>>>> {org.wso2.carbon.core.init.CarbonServerManager} - Halting JVM >>>>>>> >>>>>>> >>>>>>> From the line , {org.apache.cassandra.net.MessagingService} - >>>>>>> MessagingService shutting down server thread, it seems that there are >>>>>>> other >>>>>>> cassandra services running and getting closed before we shut down >>>>>>> cassandra >>>>>>> component in the OSGI. We are getting cassandra connection errors due to >>>>>>> this [1]. >>>>>>> >>>>>>> How can we handle this, as it is needed to halt this thread only >>>>>>> after stopping the OSGI framework? >>>>>>> >>>>>>> [1] https://wso2.org/jira/browse/SS-114 >>>>>>> >>>>>>> -- >>>>>>> Ishara Premasada >>>>>>> Software Engineer, >>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>> *Blog : http://isharapremadasa.blogspot.com/ >>>>>>> Twitter : https://twitter.com/ishadil >>>>>>> Mobile : +94 714445832* >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> [email protected] >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards,* >>>>>> >>>>>> Shammi Jayasinghe* >>>>>> Associate Tech Lead >>>>>> WSO2, Inc.; http://wso2.com, >>>>>> mobile: +94 71 4493085 >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Pradeep Fernando* >>>>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>>>> >>>>> blog: http://pradeepfernando.blogspot.com >>>>> m: +94776603662 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Pradeep Fernando* >>>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>>> >>>> blog: http://pradeepfernando.blogspot.com >>>> m: +94776603662 >>>> >>> >>> >>> >>> -- >>> *Pradeep Fernando* >>> Associate Technical Lead;WSO2 Inc.; http://wso2.com >>> >>> blog: http://pradeepfernando.blogspot.com >>> m: +94776603662 >>> >> >> >> >> -- >> Best Regards,* >> >> Shammi Jayasinghe* >> Associate Tech Lead >> WSO2, Inc.; http://wso2.com, >> mobile: +94 71 4493085 >> >> > > > -- > Best Regards,* > > Shammi Jayasinghe* > Associate Tech Lead > WSO2, Inc.; http://wso2.com, > mobile: +94 71 4493085 > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Hasitha Abeykoon* Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>* * * *
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
