Looping in PrabathA.
On Fri, May 2, 2014 at 11:18 AM, Hasitha Hiranya <[email protected]> wrote: > Hi Shameera, > > Good catch. > > [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} - > Stopped all transport listeners > [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} - > Waiting for request service completion... > [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} - > All requests have been served. > [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} - > Waiting for deployment completion... > [2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} - > Stop listening for CQL clients > > What happens when all transport listeners are stopped? > Stop listening for CQL clients means cassandra will no longer will accept > requests from a cql client. Most probably same goes with Hector (Thrift) > also. That might cause these issues. > > Thanks > > > On Fri, May 2, 2014 at 11:12 AM, Shameera Rathnayaka <[email protected]>wrote: > >> Hi HasithaH, >> >> In the shutdown logs i could see following line, before start andes >> deactivation, what does actually mean? does it stop cassandra transport >> listener? >> >> [2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} >> - Stop listening for CQL clients >> >> >> >> >> >> On Fri, May 2, 2014 at 10:26 AM, Hasitha Hiranya <[email protected]>wrote: >> >>> Hi Shameera, >>> >>> I have added logs and tested. Full Log is attached at ( >>> https://drive.google.com/a/wso2.com/file/d/0B57HoxWKqqNnN2FPRE9FeC0yYXM/edit?usp=sharing). >>> Deactivate of andes service is like follows. >>> >>> protected void deactivate(ComponentContext ctx) { >>> // Unregister QpidService >>> System.out.println("+++++++++++++++++++Started deactivating >>> andes"); >>> System.out.println("++++++++++++Unregistering qpid service"); >>> try { >>> if (null != qpidService) { >>> qpidService.unregister(); >>> } >>> } catch (Exception e) {} >>> System.out.println("+++++++++++++++++Unregistered qpidService"); >>> // Shutdown the Qpid broker >>> System.out.println("+++++++++++++++++Shutting down andes"); >>> ApplicationRegistry.remove(); >>> System.out.println("+++++++++++done shutting down andes"); >>> System.out.println("+++++++++++done deactivating of andes >>> component"); >>> } >>> >>> +++++++++++++++++++Started deactivating andes >>> ++++++++++++Unregistering qpid service >>> +++++++++++++++++Unregistered qpidService >>> +++++++++++++++++Shutting down andes >>> +++++++++++done shutting down andes >>> +++++++++++done deactivating of andes component >>> ++++++++++++++++++++started deactivating cassandra >>> ++++++++++++++++++done deactivating cassandra >>> >>> I have a doubt like is it correct to unregister qpidService before >>> actually shutting down the broker? >>> Then I changed the code swapping the order. >>> >>> protected void deactivate(ComponentContext ctx) { >>> // Unregister QpidService >>> // Shutdown the Qpid broker >>> ApplicationRegistry.remove(); >>> try { >>> if (null != qpidService) { >>> qpidService.unregister(); >>> } >>> } catch (Exception e) {} >>> } >>> >>> >>> Still errors happened. Order was as follows. >>> >>> +++++++++++++++++++Started deactivating andes >>> +++++++++++++++++++++shutting down andes >>> +++++++++++done shutting down andes >>> unregistering qpidservice >>> +++++++++++++++++Unregistered qpidService >>> +++++++++++done deactivating of andes component >>> ++++++++++++++++++++started deactivating cassandra >>> ++++++++++++++++++done deactivating cassandra >>> >>> Pom file has cassandra as a dependency. >>> >>> <Import-Package> >>> org.apache.axis2.*; >>> version="${axis2.osgi.version.range.qpid}", >>> org.apache.axiom.*; >>> version="${axiom.osgi.version.range.qpid}", >>> org.wso2.carbon.andes.authentication.service, >>> org.wso2.carbon.andes.commons, >>> org.wso2.carbon.andes.commons.registry, >>> * org.wso2.carbon.cassandra.server; >>> version="4.2.2",* >>> *;resolution:=optional >>> </Import-Package> >>> >>> What is going wrong? >>> >>> Thanks >>> >>> >>> On Fri, May 2, 2014 at 9:33 AM, Shameera Rathnayaka >>> <[email protected]>wrote: >>> >>>> Hi HasithaH, >>>> >>>> Shall we try with log messages to identify service deactivation and >>>> bundle undeployment order of andes and cassandra ? >>>> >>>> Thanks, >>>> Shameera. >>>> >>>> >>>> On Fri, May 2, 2014 at 9:18 AM, Hasitha Hiranya <[email protected]>wrote: >>>> >>>>> During testing I followed following steps. >>>>> >>>>> 1. create a topic subscriber >>>>> 2. publish 1000 msgs >>>>> 3. wait until the subscriber get 1000 messages and close >>>>> 4. now underneath MB will still be leisurely deleting the content of >>>>> removed messages (with timeouts etc) >>>>> 5. I shutdown the broker by Ctrl+c >>>>> 6. Now with my above fixes it will delete all records that needs to be >>>>> deleted before shutting down. >>>>> >>>>> I can see when the code is at step 6 MB is saying cassandra is down. >>>>> Thus before returning from the Close() of message store (hence before >>>>> returning from deactivte of andes service), cassandra service get >>>>> disappeared. It boils down to an OSGI problem. >>>>> >>>>> @Shameera, >>>>> >>>>> I have the dependency to the cassandra bundle as you have suggested in >>>>> the andes bundle. But seems there is a problem still. Any idea why that >>>>> happens? >>>>> >>>>> >>>>> On Thu, May 1, 2014 at 10:56 AM, Hasitha Hiranya <[email protected]>wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Also in order to stop connection to Cassandra gracefully, we need to >>>>>> do following. >>>>>> >>>>>> cluster.getConnectionManager().shutdown(); >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> On Thu, May 1, 2014 at 10:52 AM, Hasitha Hiranya >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I intend to cleanup graceful shutdown code of WSO2 Message Broker in >>>>>>> following way. We have to do them as a part of fixing shutdown errors. >>>>>>> We >>>>>>> have managed to keep Cassandra until broker service shutdown properly in >>>>>>> OSGI env, but we see problems due to lack of these. >>>>>>> >>>>>>> 1. When shutting down we have to flush >>>>>>> all pubSubMessageContentRemoverTasks, meaning we have to delete all >>>>>>> acked >>>>>>> messages for topics, otherwise they will never be removed again (After >>>>>>> shutting down memory is gone). Concern is we have to wait for timeout >>>>>>> for >>>>>>> those messages to happen, which will cause shutting down of MB on hold >>>>>>> untill all messages are timed out. For now MB will shut down hoping some >>>>>>> other node will clear them up. >>>>>>> >>>>>>> 2. Above argument goes with content removal tasks as well. Merely >>>>>>> stopping deletion thread will not help. >>>>>>> >>>>>>> 3. above two tasks should be done AFTER stopping queue/topic flusher >>>>>>> threads. >>>>>>> >>>>>>> 4. When shutting down we have to clear in-memory message status (for >>>>>>> message count to be correct). >>>>>>> >>>>>>> 5. We have to copy back NQ messages back to GQ. >>>>>>> >>>>>>> 6. Flush message counts. >>>>>>> >>>>>>> @pamod, >>>>>>> >>>>>>> You have a fix to flush the message count before shutdown (As we >>>>>>> update it per message chunks). Is it committed? If so, where is the >>>>>>> code? >>>>>>> It should come as point 6. >>>>>>> >>>>>>> Apart from point 6 have have done other. Testing now. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> -- >>>>>>> *Hasitha Abeykoon* >>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>> *cell:* *+94 719363063* >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Hasitha Abeykoon* >>>>>> Software Engineer; WSO2, Inc.; http://wso2.com >>>>>> *cell:* *+94 719363063* >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Hasitha Abeykoon* >>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>> *cell:* *+94 719363063* >>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Software Engineer - WSO2 Inc.* >>>> *email: shameera AT wso2.com <[email protected]> , shameera AT >>>> apache.org <[email protected]>* >>>> *phone: +9471 922 1454 <%2B9471%20922%201454>* >>>> >>>> *Linked in : *http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561 >>>> *Twitter : *https://twitter.com/Shameera_R >>>> >>> >>> >>> >>> -- >>> *Hasitha Abeykoon* >>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>> *cell:* *+94 719363063* >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>> >>> >> >> >> -- >> *Software Engineer - WSO2 Inc.* >> *email: shameera AT wso2.com <[email protected]> , shameera AT apache.org >> <[email protected]>* >> *phone: +9471 922 1454* >> >> *Linked in : *http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561 >> *Twitter : *https://twitter.com/Shameera_R >> > > > > -- > *Hasitha Abeykoon* > Senior Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
