Hi, Prabath is looking into the issue. Prabath, were you able to find out the reason why client transports are closed for Cassandra? Can you shed some light on this?
Thanks On Fri, May 2, 2014 at 11:19 AM, Hasitha Hiranya <[email protected]> wrote: > Looping in PrabathA. > > > On Fri, May 2, 2014 at 11:18 AM, Hasitha Hiranya <[email protected]>wrote: > >> Hi Shameera, >> >> Good catch. >> >> [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} - >> Stopped all transport listeners >> [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} - >> Waiting for request service completion... >> [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} - >> All requests have been served. >> [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} - >> Waiting for deployment completion... >> [2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} >> - Stop listening for CQL clients >> >> What happens when all transport listeners are stopped? >> Stop listening for CQL clients means cassandra will no longer will accept >> requests from a cql client. Most probably same goes with Hector (Thrift) >> also. That might cause these issues. >> >> Thanks >> >> >> On Fri, May 2, 2014 at 11:12 AM, Shameera Rathnayaka >> <[email protected]>wrote: >> >>> Hi HasithaH, >>> >>> In the shutdown logs i could see following line, before start andes >>> deactivation, what does actually mean? does it stop cassandra transport >>> listener? >>> >>> [2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} >>> - Stop listening for CQL clients >>> >>> >>> >>> >>> >>> On Fri, May 2, 2014 at 10:26 AM, Hasitha Hiranya <[email protected]>wrote: >>> >>>> Hi Shameera, >>>> >>>> I have added logs and tested. Full Log is attached at ( >>>> https://drive.google.com/a/wso2.com/file/d/0B57HoxWKqqNnN2FPRE9FeC0yYXM/edit?usp=sharing). >>>> Deactivate of andes service is like follows. >>>> >>>> protected void deactivate(ComponentContext ctx) { >>>> // Unregister QpidService >>>> System.out.println("+++++++++++++++++++Started deactivating >>>> andes"); >>>> System.out.println("++++++++++++Unregistering qpid service"); >>>> try { >>>> if (null != qpidService) { >>>> qpidService.unregister(); >>>> } >>>> } catch (Exception e) {} >>>> System.out.println("+++++++++++++++++Unregistered qpidService"); >>>> // Shutdown the Qpid broker >>>> System.out.println("+++++++++++++++++Shutting down andes"); >>>> ApplicationRegistry.remove(); >>>> System.out.println("+++++++++++done shutting down andes"); >>>> System.out.println("+++++++++++done deactivating of andes >>>> component"); >>>> } >>>> >>>> +++++++++++++++++++Started deactivating andes >>>> ++++++++++++Unregistering qpid service >>>> +++++++++++++++++Unregistered qpidService >>>> +++++++++++++++++Shutting down andes >>>> +++++++++++done shutting down andes >>>> +++++++++++done deactivating of andes component >>>> ++++++++++++++++++++started deactivating cassandra >>>> ++++++++++++++++++done deactivating cassandra >>>> >>>> I have a doubt like is it correct to unregister qpidService before >>>> actually shutting down the broker? >>>> Then I changed the code swapping the order. >>>> >>>> protected void deactivate(ComponentContext ctx) { >>>> // Unregister QpidService >>>> // Shutdown the Qpid broker >>>> ApplicationRegistry.remove(); >>>> try { >>>> if (null != qpidService) { >>>> qpidService.unregister(); >>>> } >>>> } catch (Exception e) {} >>>> } >>>> >>>> >>>> Still errors happened. Order was as follows. >>>> >>>> +++++++++++++++++++Started deactivating andes >>>> +++++++++++++++++++++shutting down andes >>>> +++++++++++done shutting down andes >>>> unregistering qpidservice >>>> +++++++++++++++++Unregistered qpidService >>>> +++++++++++done deactivating of andes component >>>> ++++++++++++++++++++started deactivating cassandra >>>> ++++++++++++++++++done deactivating cassandra >>>> >>>> Pom file has cassandra as a dependency. >>>> >>>> <Import-Package> >>>> org.apache.axis2.*; >>>> version="${axis2.osgi.version.range.qpid}", >>>> org.apache.axiom.*; >>>> version="${axiom.osgi.version.range.qpid}", >>>> >>>> org.wso2.carbon.andes.authentication.service, >>>> org.wso2.carbon.andes.commons, >>>> org.wso2.carbon.andes.commons.registry, >>>> * org.wso2.carbon.cassandra.server; >>>> version="4.2.2",* >>>> *;resolution:=optional >>>> </Import-Package> >>>> >>>> What is going wrong? >>>> >>>> Thanks >>>> >>>> >>>> On Fri, May 2, 2014 at 9:33 AM, Shameera Rathnayaka >>>> <[email protected]>wrote: >>>> >>>>> Hi HasithaH, >>>>> >>>>> Shall we try with log messages to identify service deactivation and >>>>> bundle undeployment order of andes and cassandra ? >>>>> >>>>> Thanks, >>>>> Shameera. >>>>> >>>>> >>>>> On Fri, May 2, 2014 at 9:18 AM, Hasitha Hiranya <[email protected]>wrote: >>>>> >>>>>> During testing I followed following steps. >>>>>> >>>>>> 1. create a topic subscriber >>>>>> 2. publish 1000 msgs >>>>>> 3. wait until the subscriber get 1000 messages and close >>>>>> 4. now underneath MB will still be leisurely deleting the content of >>>>>> removed messages (with timeouts etc) >>>>>> 5. I shutdown the broker by Ctrl+c >>>>>> 6. Now with my above fixes it will delete all records that needs to >>>>>> be deleted before shutting down. >>>>>> >>>>>> I can see when the code is at step 6 MB is saying cassandra is down. >>>>>> Thus before returning from the Close() of message store (hence before >>>>>> returning from deactivte of andes service), cassandra service get >>>>>> disappeared. It boils down to an OSGI problem. >>>>>> >>>>>> @Shameera, >>>>>> >>>>>> I have the dependency to the cassandra bundle as you have suggested >>>>>> in the andes bundle. But seems there is a problem still. Any idea why >>>>>> that >>>>>> happens? >>>>>> >>>>>> >>>>>> On Thu, May 1, 2014 at 10:56 AM, Hasitha Hiranya >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Also in order to stop connection to Cassandra gracefully, we need to >>>>>>> do following. >>>>>>> >>>>>>> cluster.getConnectionManager().shutdown(); >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> On Thu, May 1, 2014 at 10:52 AM, Hasitha Hiranya >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I intend to cleanup graceful shutdown code of WSO2 Message Broker >>>>>>>> in following way. We have to do them as a part of fixing shutdown >>>>>>>> errors. >>>>>>>> We have managed to keep Cassandra until broker service shutdown >>>>>>>> properly in >>>>>>>> OSGI env, but we see problems due to lack of these. >>>>>>>> >>>>>>>> 1. When shutting down we have to flush >>>>>>>> all pubSubMessageContentRemoverTasks, meaning we have to delete all >>>>>>>> acked >>>>>>>> messages for topics, otherwise they will never be removed again (After >>>>>>>> shutting down memory is gone). Concern is we have to wait for timeout >>>>>>>> for >>>>>>>> those messages to happen, which will cause shutting down of MB on hold >>>>>>>> untill all messages are timed out. For now MB will shut down hoping >>>>>>>> some >>>>>>>> other node will clear them up. >>>>>>>> >>>>>>>> 2. Above argument goes with content removal tasks as well. Merely >>>>>>>> stopping deletion thread will not help. >>>>>>>> >>>>>>>> 3. above two tasks should be done AFTER stopping queue/topic >>>>>>>> flusher threads. >>>>>>>> >>>>>>>> 4. When shutting down we have to clear in-memory message status >>>>>>>> (for message count to be correct). >>>>>>>> >>>>>>>> 5. We have to copy back NQ messages back to GQ. >>>>>>>> >>>>>>>> 6. Flush message counts. >>>>>>>> >>>>>>>> @pamod, >>>>>>>> >>>>>>>> You have a fix to flush the message count before shutdown (As we >>>>>>>> update it per message chunks). Is it committed? If so, where is the >>>>>>>> code? >>>>>>>> It should come as point 6. >>>>>>>> >>>>>>>> Apart from point 6 have have done other. Testing now. >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> -- >>>>>>>> *Hasitha Abeykoon* >>>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>>> *cell:* *+94 719363063* >>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Hasitha Abeykoon* >>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>> *cell:* *+94 719363063* >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Hasitha Abeykoon* >>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>> *cell:* *+94 719363063* >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Software Engineer - WSO2 Inc.* >>>>> *email: shameera AT wso2.com <[email protected]> , shameera AT >>>>> apache.org <[email protected]>* >>>>> *phone: +9471 922 1454 <%2B9471%20922%201454>* >>>>> >>>>> *Linked in : * >>>>> http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561 >>>>> *Twitter : *https://twitter.com/Shameera_R >>>>> >>>> >>>> >>>> >>>> -- >>>> *Hasitha Abeykoon* >>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>> *cell:* *+94 719363063* >>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>> >>>> >>> >>> >>> -- >>> *Software Engineer - WSO2 Inc.* >>> *email: shameera AT wso2.com <[email protected]> , shameera AT >>> apache.org <[email protected]>* >>> *phone: +9471 922 1454* >>> >>> *Linked in : *http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561 >>> *Twitter : *https://twitter.com/Shameera_R >>> >> >> >> >> -- >> *Hasitha Abeykoon* >> Senior Software Engineer; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >> >> > > > -- > *Hasitha Abeykoon* > Senior Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
