Hi PrabathA et all,
We are still experiencing this issue as reported in [1] when shutting
down. As stated by Hasitha it seems Cassandra message service thread is
shutdown before we perform all data cleaning tasks in MB, as per below
logs.
*[2014-05-08 14:51:15,299] INFO
{org.apache.cassandra.net.MessagingService} - Waiting for messaging
service to quiesce[2014-05-08 14:51:15,303] INFO
{org.apache.cassandra.net.MessagingService} - MessagingService shutting
down server thread.*
[2014-05-08 14:51:15,722] INFO
{org.wso2.andes.server.store.CassandraMessageStore} - Clearing up
Subscription Information
[2014-05-08 14:51:15,722] INFO
{org.wso2.andes.server.cassandra.DefaultClusteringEnabledSubscriptionManager}
- Clearing the Persisted State of Node with ID 0
[2014-05-08 14:51:15,733] ERROR
{org.apache.cassandra.thrift.CustomTThreadPoolServer} - Error occurred
during processing of message.
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:61)
Can we please know the progress of fixing this issue, or else can you
please suggest us what might be the reason for cassandra service getting
closed before deactivation of andes tasks are completed?
Thanks!
Ishara
[1] https://wso2.org/jira/browse/MB-301
On Mon, May 5, 2014 at 9:11 PM, Hasitha Hiranya <[email protected]> wrote:
> Hi,
>
> Following is prabath's explanation
>
> There's no any impact on Cassandra from the Transport Listener Framework.
> Carbon transport listener framework opens up the ports used by the
> transports configured upon the Carbon Server. However, here in this
> particular scenario, Cassandra daemon is responsible for opening up all the
> ports used by a given Cassandra Server. Therefore, Cassandra related ports
> are opened up much later after the transport listener framework is
> initialised. Similarly, when it comes to shutting down the server, the
> Cassandra daemon is stopped before we shut down the listener framework.
>
> So Prabath, how does this affect the OSGI integration of services? How are
> we going to hold Cassandra until Andes is closed gracefully?
>
> Thanks
>
>
> On Sat, May 3, 2014 at 12:59 PM, Hasitha Hiranya <[email protected]>wrote:
>
>> Hi,
>>
>> Prabath is looking into the issue. Prabath, were you able to find out the
>> reason why client transports are closed for Cassandra? Can you shed some
>> light on this?
>>
>> Thanks
>>
>>
>> On Fri, May 2, 2014 at 11:19 AM, Hasitha Hiranya <[email protected]>wrote:
>>
>>> Looping in PrabathA.
>>>
>>>
>>> On Fri, May 2, 2014 at 11:18 AM, Hasitha Hiranya <[email protected]>wrote:
>>>
>>>> Hi Shameera,
>>>>
>>>> Good catch.
>>>>
>>>> [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement}
>>>> - Stopped all transport listeners
>>>> [2014-05-02 10:12:19,015] INFO {org.wso2.carbon.core.ServerManagement}
>>>> - Waiting for request service completion...
>>>> [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement}
>>>> - All requests have been served.
>>>> [2014-05-02 10:12:19,019] INFO {org.wso2.carbon.core.ServerManagement}
>>>> - Waiting for deployment completion...
>>>> [2014-05-02 10:12:19,021] INFO
>>>> {org.apache.cassandra.transport.Server} - Stop listening for CQL clients
>>>>
>>>> What happens when all transport listeners are stopped?
>>>> Stop listening for CQL clients means cassandra will no longer will
>>>> accept requests from a cql client. Most probably same goes with Hector
>>>> (Thrift) also. That might cause these issues.
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Fri, May 2, 2014 at 11:12 AM, Shameera Rathnayaka <[email protected]
>>>> > wrote:
>>>>
>>>>> Hi HasithaH,
>>>>>
>>>>> In the shutdown logs i could see following line, before start andes
>>>>> deactivation, what does actually mean? does it stop cassandra transport
>>>>> listener?
>>>>>
>>>>> [2014-05-02 10:12:19,021] INFO
>>>>> {org.apache.cassandra.transport.Server} - Stop listening for CQL clients
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 2, 2014 at 10:26 AM, Hasitha Hiranya <[email protected]>wrote:
>>>>>
>>>>>> Hi Shameera,
>>>>>>
>>>>>> I have added logs and tested. Full Log is attached at (
>>>>>> https://drive.google.com/a/wso2.com/file/d/0B57HoxWKqqNnN2FPRE9FeC0yYXM/edit?usp=sharing).
>>>>>> Deactivate of andes service is like follows.
>>>>>>
>>>>>> protected void deactivate(ComponentContext ctx) {
>>>>>> // Unregister QpidService
>>>>>> System.out.println("+++++++++++++++++++Started deactivating
>>>>>> andes");
>>>>>> System.out.println("++++++++++++Unregistering qpid service");
>>>>>> try {
>>>>>> if (null != qpidService) {
>>>>>> qpidService.unregister();
>>>>>> }
>>>>>> } catch (Exception e) {}
>>>>>> System.out.println("+++++++++++++++++Unregistered
>>>>>> qpidService");
>>>>>> // Shutdown the Qpid broker
>>>>>> System.out.println("+++++++++++++++++Shutting down andes");
>>>>>> ApplicationRegistry.remove();
>>>>>> System.out.println("+++++++++++done shutting down andes");
>>>>>> System.out.println("+++++++++++done deactivating of andes
>>>>>> component");
>>>>>> }
>>>>>>
>>>>>> +++++++++++++++++++Started deactivating andes
>>>>>> ++++++++++++Unregistering qpid service
>>>>>> +++++++++++++++++Unregistered qpidService
>>>>>> +++++++++++++++++Shutting down andes
>>>>>> +++++++++++done shutting down andes
>>>>>> +++++++++++done deactivating of andes component
>>>>>> ++++++++++++++++++++started deactivating cassandra
>>>>>> ++++++++++++++++++done deactivating cassandra
>>>>>>
>>>>>> I have a doubt like is it correct to unregister qpidService before
>>>>>> actually shutting down the broker?
>>>>>> Then I changed the code swapping the order.
>>>>>>
>>>>>> protected void deactivate(ComponentContext ctx) {
>>>>>> // Unregister QpidService
>>>>>> // Shutdown the Qpid broker
>>>>>> ApplicationRegistry.remove();
>>>>>> try {
>>>>>> if (null != qpidService) {
>>>>>> qpidService.unregister();
>>>>>> }
>>>>>> } catch (Exception e) {}
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Still errors happened. Order was as follows.
>>>>>>
>>>>>> +++++++++++++++++++Started deactivating andes
>>>>>> +++++++++++++++++++++shutting down andes
>>>>>> +++++++++++done shutting down andes
>>>>>> unregistering qpidservice
>>>>>> +++++++++++++++++Unregistered qpidService
>>>>>> +++++++++++done deactivating of andes component
>>>>>> ++++++++++++++++++++started deactivating cassandra
>>>>>> ++++++++++++++++++done deactivating cassandra
>>>>>>
>>>>>> Pom file has cassandra as a dependency.
>>>>>>
>>>>>> <Import-Package>
>>>>>> org.apache.axis2.*;
>>>>>> version="${axis2.osgi.version.range.qpid}",
>>>>>> org.apache.axiom.*;
>>>>>> version="${axiom.osgi.version.range.qpid}",
>>>>>>
>>>>>> org.wso2.carbon.andes.authentication.service,
>>>>>> org.wso2.carbon.andes.commons,
>>>>>> org.wso2.carbon.andes.commons.registry,
>>>>>> * org.wso2.carbon.cassandra.server;
>>>>>> version="4.2.2",*
>>>>>> *;resolution:=optional
>>>>>> </Import-Package>
>>>>>>
>>>>>> What is going wrong?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> On Fri, May 2, 2014 at 9:33 AM, Shameera Rathnayaka <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi HasithaH,
>>>>>>>
>>>>>>> Shall we try with log messages to identify service deactivation and
>>>>>>> bundle undeployment order of andes and cassandra ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Shameera.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, May 2, 2014 at 9:18 AM, Hasitha Hiranya
>>>>>>> <[email protected]>wrote:
>>>>>>>
>>>>>>>> During testing I followed following steps.
>>>>>>>>
>>>>>>>> 1. create a topic subscriber
>>>>>>>> 2. publish 1000 msgs
>>>>>>>> 3. wait until the subscriber get 1000 messages and close
>>>>>>>> 4. now underneath MB will still be leisurely deleting the content
>>>>>>>> of removed messages (with timeouts etc)
>>>>>>>> 5. I shutdown the broker by Ctrl+c
>>>>>>>> 6. Now with my above fixes it will delete all records that needs to
>>>>>>>> be deleted before shutting down.
>>>>>>>>
>>>>>>>> I can see when the code is at step 6 MB is saying cassandra is down.
>>>>>>>> Thus before returning from the Close() of message store (hence
>>>>>>>> before returning from deactivte of andes service), cassandra service
>>>>>>>> get
>>>>>>>> disappeared. It boils down to an OSGI problem.
>>>>>>>>
>>>>>>>> @Shameera,
>>>>>>>>
>>>>>>>> I have the dependency to the cassandra bundle as you have suggested
>>>>>>>> in the andes bundle. But seems there is a problem still. Any idea why
>>>>>>>> that
>>>>>>>> happens?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, May 1, 2014 at 10:56 AM, Hasitha Hiranya <[email protected]
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Also in order to stop connection to Cassandra gracefully, we need
>>>>>>>>> to do following.
>>>>>>>>>
>>>>>>>>> cluster.getConnectionManager().shutdown();
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, May 1, 2014 at 10:52 AM, Hasitha Hiranya <
>>>>>>>>> [email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I intend to cleanup graceful shutdown code of WSO2 Message Broker
>>>>>>>>>> in following way. We have to do them as a part of fixing shutdown
>>>>>>>>>> errors.
>>>>>>>>>> We have managed to keep Cassandra until broker service shutdown
>>>>>>>>>> properly in
>>>>>>>>>> OSGI env, but we see problems due to lack of these.
>>>>>>>>>>
>>>>>>>>>> 1. When shutting down we have to flush
>>>>>>>>>> all pubSubMessageContentRemoverTasks, meaning we have to delete all
>>>>>>>>>> acked
>>>>>>>>>> messages for topics, otherwise they will never be removed again
>>>>>>>>>> (After
>>>>>>>>>> shutting down memory is gone). Concern is we have to wait for
>>>>>>>>>> timeout for
>>>>>>>>>> those messages to happen, which will cause shutting down of MB on
>>>>>>>>>> hold
>>>>>>>>>> untill all messages are timed out. For now MB will shut down hoping
>>>>>>>>>> some
>>>>>>>>>> other node will clear them up.
>>>>>>>>>>
>>>>>>>>>> 2. Above argument goes with content removal tasks as well. Merely
>>>>>>>>>> stopping deletion thread will not help.
>>>>>>>>>>
>>>>>>>>>> 3. above two tasks should be done AFTER stopping queue/topic
>>>>>>>>>> flusher threads.
>>>>>>>>>>
>>>>>>>>>> 4. When shutting down we have to clear in-memory message status
>>>>>>>>>> (for message count to be correct).
>>>>>>>>>>
>>>>>>>>>> 5. We have to copy back NQ messages back to GQ.
>>>>>>>>>>
>>>>>>>>>> 6. Flush message counts.
>>>>>>>>>>
>>>>>>>>>> @pamod,
>>>>>>>>>>
>>>>>>>>>> You have a fix to flush the message count before shutdown (As we
>>>>>>>>>> update it per message chunks). Is it committed? If so, where is the
>>>>>>>>>> code?
>>>>>>>>>> It should come as point 6.
>>>>>>>>>>
>>>>>>>>>> Apart from point 6 have have done other. Testing now.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Hasitha Abeykoon*
>>>>>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com
>>>>>>>>>> *cell:* *+94 719363063*
>>>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Hasitha Abeykoon*
>>>>>>>>> Software Engineer; WSO2, Inc.; http://wso2.com
>>>>>>>>> *cell:* *+94 719363063*
>>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Hasitha Abeykoon*
>>>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>>>>>>>> *cell:* *+94 719363063*
>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Software Engineer - WSO2 Inc.*
>>>>>>> *email: shameera AT wso2.com <[email protected]> , shameera AT
>>>>>>> apache.org <[email protected]>*
>>>>>>> *phone: +9471 922 1454 <%2B9471%20922%201454>*
>>>>>>>
>>>>>>> *Linked in : *
>>>>>>> http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561
>>>>>>> *Twitter : *https://twitter.com/Shameera_R
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Hasitha Abeykoon*
>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>>>>>> *cell:* *+94 719363063*
>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Software Engineer - WSO2 Inc.*
>>>>> *email: shameera AT wso2.com <[email protected]> , shameera AT
>>>>> apache.org <[email protected]>*
>>>>> *phone: +9471 922 1454 <%2B9471%20922%201454>*
>>>>>
>>>>> *Linked in : *
>>>>> http://lk.linkedin.com/pub/shameera-rathnayaka/1a/661/561
>>>>> *Twitter : *https://twitter.com/Shameera_R
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Hasitha Abeykoon*
>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>>>> *cell:* *+94 719363063*
>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Hasitha Abeykoon*
>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>>> *cell:* *+94 719363063*
>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>>
>>>
>>
>>
>> --
>> *Hasitha Abeykoon*
>> Senior Software Engineer; WSO2, Inc.; http://wso2.com
>> *cell:* *+94 719363063*
>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>>
>>
>
>
> --
> *Hasitha Abeykoon*
> Senior Software Engineer; WSO2, Inc.; http://wso2.com
> *cell:* *+94 719363063*
> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>
>
--
Ishara Premasada
Software Engineer,
WSO2 Inc. http://wso2.com/
*Blog : http://isharapremadasa.blogspot.com/
<http://isharapremadasa.blogspot.com/>Twitter :
https://twitter.com/ishadil <https://twitter.com/ishadil>Mobile : +94
714445832*
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
