Hi Shameera,
Good catch.
[2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} -
Stopped all transport listeners
[2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement} -
Waiting for request service completion...
[2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} -
All requests have been served.
[2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement} -
Waiting for deployment completion...
[2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} -
Stop listening for CQL clients
What happens when all transport listeners are stopped?
Stop listening for CQL clients means cassandra will no longer will accept
requests from a cql client. Most probably same goes with Hector (Thrift)
also. That might cause these issues.
Thanks
On Fri, May 2, 2014 at 11:12 AM, Shameera Rathnayaka <[email protected]>wrote:
> Hi HasithaH,
>
> In the shutdown logs i could see following line, before start andes
> deactivation, what does actually mean? does it stop cassandra transport
> listener?
>
> [2014-05-02 10:12:19,021] INFO {org.apache.cassandra.transport.Server} -
> Stop listening for CQL clients
>
>
>
>
>
> On Fri, May 2, 2014 at 10:26 AM, Hasitha Hiranya <[email protected]>wrote:
>
>> Hi Shameera,
>>
>> I have added logs and tested. Full Log is attached at (
>> https://drive.google.com/a/wso2.com/file/d/0B57HoxWKqqNnN2FPRE9FeC0yYXM/edit?usp=sharing).
>> Deactivate of andes service is like follows.
>>
>> protected void deactivate(ComponentContext ctx) {
>> // Unregister QpidService
>> System.out.println("+++++++++++++++++++Started deactivating
>> andes");
>> System.out.println("++++++++++++Unregistering qpid service");
>> try {
>> if (null != qpidService) {
>> qpidService.unregister();
>> }
>> } catch (Exception e) {}
>> System.out.println("+++++++++++++++++Unregistered qpidService");
>> // Shutdown the Qpid broker
>> System.out.println("+++++++++++++++++Shutting down andes");
>> ApplicationRegistry.remove();
>> System.out.println("+++++++++++done shutting down andes");
>> System.out.println("+++++++++++done deactivating of andes
>> component");
>> }
>>
>> +++++++++++++++++++Started deactivating andes
>> ++++++++++++Unregistering qpid service
>> +++++++++++++++++Unregistered qpidService
>> +++++++++++++++++Shutting down andes
>> +++++++++++done shutting down andes
>> +++++++++++done deactivating of andes component
>> ++++++++++++++++++++started deactivating cassandra
>> ++++++++++++++++++done deactivating cassandra
>>
>> I have a doubt like is it correct to unregister qpidService before
>> actually shutting down the broker?
>> Then I changed the code swapping the order.
>>
>> protected void deactivate(ComponentContext ctx) {
>> // Unregister QpidService
>> // Shutdown the Qpid broker
>> ApplicationRegistry.remove();
>> try {
>> if (null != qpidService) {
>> qpidService.unregister();
>> }
>> } catch (Exception e) {}
>> }
>>
>>
>> Still errors happened. Order was as follows.
>>
>> +++++++++++++++++++Started deactivating andes
>> +++++++++++++++++++++shutting down andes
>> +++++++++++done shutting down andes
>> unregistering qpidservice
>> +++++++++++++++++Unregistered qpidService
>> +++++++++++done deactivating of andes component
>> ++++++++++++++++++++started deactivating cassandra
>> ++++++++++++++++++done deactivating cassandra
>>
>> Pom file has cassandra as a dependency.
>>
>> <Import-Package>
>> org.apache.axis2.*;
>> version="${axis2.osgi.version.range.qpid}",
>> org.apache.axiom.*;
>> version="${axiom.osgi.version.range.qpid}",
>> org.wso2.carbon.andes.authentication.service,
>> org.wso2.carbon.andes.commons,
>> org.wso2.carbon.andes.commons.registry,
>> * org.wso2.carbon.cassandra.server;
>> version="4.2.2",*
>> *;resolution:=optional
>> </Import-Package>
>>
>> What is going wrong?
>>
>> Thanks
>>
>>
>> On Fri, May 2, 2014 at 9:33 AM, Shameera Rathnayaka <[email protected]>wrote:
>>
>>> Hi HasithaH,
>>>
>>> Shall we try with log messages to identify service deactivation and
>>> bundle undeployment order of andes and cassandra ?
>>>
>>> Thanks,
>>> Shameera.
>>>
>>>
>>> On Fri, May 2, 2014 at 9:18 AM, Hasitha Hiranya <[email protected]>wrote:
>>>
>>>> During testing I followed following steps.
>>>>
>>>> 1. create a topic subscriber
>>>> 2. publish 1000 msgs
>>>> 3. wait until the subscriber get 1000 messages and close
>>>> 4. now underneath MB will still be leisurely deleting the content of
>>>> removed messages (with timeouts etc)
>>>> 5. I shutdown the broker by Ctrl+c
>>>> 6. Now with my above fixes it will delete all records that needs to be
>>>> deleted before shutting down.
>>>>
>>>> I can see when the code is at step 6 MB is saying cassandra is down.
>>>> Thus before returning from the Close() of message store (hence before
>>>> returning from deactivte of andes service), cassandra service get
>>>> disappeared. It boils down to an OSGI problem.
>>>>
>>>> @Shameera,
>>>>
>>>> I have the dependency to the cassandra bundle as you have suggested in
>>>> the andes bundle. But seems there is a problem still. Any idea why that
>>>> happens?
>>>>
>>>>
>>>> On Thu, May 1, 2014 at 10:56 AM, Hasitha Hiranya <[email protected]>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Also in order to stop connection to Cassandra gracefully, we need to
>>>>> do following.
>>>>>
>>>>> cluster.getConnectionManager().shutdown();
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> On Thu, May 1, 2014 at 10:52 AM, Hasitha Hiranya <[email protected]>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I intend to cleanup graceful shutdown code of WSO2 Message Broker in
>>>>>> following way. We have to do them as a part of fixing shutdown errors. We
>>>>>> have managed to keep Cassandra until broker service shutdown properly in
>>>>>> OSGI env, but we see problems due to lack of these.
>>>>>>
>>>>>> 1. When shutting down we have to flush
>>>>>> all pubSubMessageContentRemoverTasks, meaning we have to delete all acked
>>>>>> messages for topics, otherwise they will never be removed again (After
>>>>>> shutting down memory is gone). Concern is we have to wait for timeout for
>>>>>> those messages to happen, which will cause shutting down of MB on hold
>>>>>> untill all messages are timed out. For now MB will shut down hoping some
>>>>>> other node will clear them up.
>>>>>>
>>>>>> 2. Above argument goes with content removal tasks as well. Merely
>>>>>> stopping deletion thread will not help.
>>>>>>
>>>>>> 3. above two tasks should be done AFTER stopping queue/topic flusher
>>>>>> threads.
>>>>>>
>>>>>> 4. When shutting down we have to clear in-memory message status (for
>>>>>> message count to be correct).
>>>>>>
>>>>>> 5. We have to copy back NQ messages back to GQ.
>>>>>>
>>>>>> 6. Flush message counts.
>>>>>>
>>>>>> @pamod,
>>>>>>
>>>>>> You have a fix to flush the message count before shutdown (As we
>>>>>> update it per message chunks). Is it committed? If so, where is the code?
>>>>>> It should come as point 6.
>>>>>>
>>>>>> Apart from point 6 have have done other. Testing now.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> --
>>>>>> *Hasitha Abeykoon*
>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com
>>>>>> *cell:* *+94 719363063*
>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Hasitha Abeykoon*
>>>>> Software Engineer; WSO2, Inc.; http://wso2.com
>>>>> *cell:* *+94 719363063*
>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Hasitha Abeykoon*
>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>>>> *cell:* *+94 719363063*
>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Software Engineer - WSO2 Inc.*
>>> *email: shameera AT wso2.com <[email protected]> , shameera AT
>>> apache.org <[email protected]>*
>>> *phone: +9471 922 1454 <%2B9471%20922%201454>*
>>>
>>> *Linked in : *http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561
>>> *Twitter : *https://twitter.com/Shameera_R
>>>
>>
>>
>>
>> --
>> *Hasitha Abeykoon*
>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>> *cell:* *+94 719363063*
>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>
>>
>
>
> --
> *Software Engineer - WSO2 Inc.*
> *email: shameera AT wso2.com <[email protected]> , shameera AT apache.org
> <[email protected]>*
> *phone: +9471 922 1454*
>
> *Linked in : *http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561
> *Twitter : *https://twitter.com/Shameera_R
>
--
*Hasitha Abeykoon*
Senior Software Engineer; WSO2, Inc.; http://wso2.com
*cell:* *+94 719363063*
*blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
