In order to handle node kill -one other node of the cluster should clear all cluster subscriptions of that node and notify the cluster -when starting a node, if there are cluster subscriptions of the node id assigned delete all of them - no matter cluster or standalone mode -during start do a scan thro all subscriptions and remove any not belonging to the currently living nodes (*because all the nodes in cluster suddenly got killed - very rare - thus will skip*)
other than subscription there are several other things we need to handle for kill or during shutdown ex: unassign all slots owned - should be done in MessageEngine close() I moved all close methods of Andes kernel to AndesKernelBoot class. this will be called before qpid amqp transports are closed. (Broker.shutdown() method of Qpid) Thanks On Sat, Sep 27, 2014 at 5:57 PM, Hasitha Hiranya <[email protected]> wrote: > Hi, > > I got last three working. > > create with sub id= x topic=y. Unsubscribe. Now try sub id= x topic=y - > passed > create with sub id= x topic=y. Unsubscribe. Now try sub id= z topic=y - > passed > create with sub id= x topic=y. Unsubscribe. Now try sub id= x topic=z - > passed > > Done following case as well > create with sub id= x topic=y > create with sub id= z topic=y > create a normal topic subscriber topic=y > create a normal queue subscriber queue=y > > then all queue, topic, durable queues will be there at same time on broker. > > Subscription maps are updated correctly (even though messages did not > received) - passed > > Now need to handle following cases > > - when starting reload durable exchanges/queues/bindings > - when starting sync cluster subscription map only. Keep local > subscription map empty > -is hazlecast assign same node id after a restart? If so, aobce two are > only the cases needs handle. > > -when gracefully shut down close all local subscriptions and send cluster > notifications > -during a kill, we need to delegate above to a different node. > > After above 5, there should be no inconsistancies on subscription model. > > Thanks > > On Sat, Sep 27, 2014 at 5:55 PM, Hasitha Hiranya <[email protected]> > wrote: > >> I got the durable topic scenario working (I am testing with GQ/NQ >> included as Sajini's fixes are on the way) >> So, I am testing subscription perspective only. >> >> Did following tests. >> >> create with sub id= x topic=y. disconnect and try to connect again - >> passed >> create with sub id= x topic=y. kill subscription and try to connect again >> - passed >> create with sub id= x topic=y. try another subscription with same params. >> rejects the subscription - passed >> create with sub id= x topic=y. try another with sub id=z topic=y. Allowed >> - passed >> above mutiple subscribers killed - passed >> create with sub id= x topic=y. Kill it. Then try with sub id= x topic=z. >> rejects the subscription - passed >> create with sub id= x topic=y. Kill it. INSTANTLY create same >> subscription again. Subscription rejected - failed (*do we need to >> address this?*) >> above create subscription again - passed >> create with sub id= x topic=y. Create a normal topc subscription topic=y >> - passed >> kill above individually - passed >> create with sub id= x topic=y. Unsubscribe. Now try sub id= x topic=y - >> failed >> create with sub id= x topic=y. Unsubscribe. Now try sub id= z topic=y - >> failed >> create with sub id= x topic=y. Unsubscribe. Now try sub id= x topic=z - >> failed >> >> Last three failed as qpid did not said to remove bindings and delete >> queue. Maybe an issue with my client. Will check more. >> >> On Sat, Sep 27, 2014 at 5:50 PM, Hasitha Hiranya <[email protected]> >> wrote: >> >>> This change working fine in standalone mode now. According to above >>> logic, if Hazlecast side works there is no reason for MB not to work fine >>> in cluster mode. >>> >>> TODO: >>> >>> 1. Check this in cluster mode >>> 2. Carefully think about durable topic subscriptions >>> 3. handle the case where MB shut down when there are live subscriptions >>> 4. handle case where MB killed when there are live subscribers >>> >>> On Sat, Sep 27, 2014 at 5:50 PM, Hasitha Hiranya <[email protected]> >>> wrote: >>> >>>> I have verified that AndesRecovery handler is working correctly. >>>> After getting it verified I have "short-circuited" cluster changes to >>>> be triggered inside local changes. >>>> So we do not need to write codes specifically for standalone mode. >>>> >>>> On Sat, Sep 27, 2014 at 5:49 PM, Hasitha Hiranya <[email protected]> >>>> wrote: >>>> >>>>> Facts >>>>> We deal with Qpid only thro 2 classes >>>>> >>QpidAndesBridge - qpid tells to kernel >>>>> >>VirtualHostConfigSynchronizer - kernel tells to qpid >>>>> >>>>> Concerns >>>>> Have we not still got rid of CassandraMessageStore class completely?? >>>>> >>>>> On Sat, Sep 27, 2014 at 5:49 PM, Hasitha Hiranya <[email protected]> >>>>> wrote: >>>>> >>>>>> Now we have a concept called subscription disconnect. This >>>>>> specifically come into play during durable topic subscriptions - need to >>>>>> think more >>>>>> >>>>>> In all other occasions we just delete the subscription. No >>>>>> disconnect. >>>>>> >>>>>> On Sat, Sep 27, 2014 at 5:49 PM, Hasitha Hiranya <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> 1. When storing a queue/binding/exchange in DB >>>>>>> 2. and when notifying the change is sent via Hazlecast >>>>>>> >>>>>>> same encoded string is used. So now the recovery from DB is >>>>>>> straightforward. >>>>>>> >>>>>>> On Sat, Sep 27, 2014 at 5:48 PM, Hasitha Hiranya <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Removed following classes >>>>>>>> >>>>>>>> interface ClusterCoordinationManager >>>>>>>> class ClusterwideSubscriptionChangeNotifier >>>>>>>> SubscriptionNotification - now all queues/exchanges/bindings are >>>>>>>> synced by a common notification type called "clusterSubscription" >>>>>>>> >>>>>>>> ClusterSubscription structure - >>>>>>>> >> notification message description >>>>>>>> >> notification body as a encoded string >>>>>>>> >> notification type - add/delete etc >>>>>>>> >>>>>>>> On Sat, Sep 27, 2014 at 5:48 PM, Hasitha Hiranya <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Moved initalizing code of andes kernel to a new class >>>>>>>>> >>>>>>>>> AndesKernelBoot - now Andes kernel startup code lies here. >>>>>>>>> Decoupled from Qpid >>>>>>>>> >>>>>>>>> >>loadConfigurations - done from outside >>>>>>>>> >>startAndesStores - done from outside >>>>>>>>> >>startAndesComponents() >>>>>>>>> >>startHouseKeepingThreads() >>>>>>>>> >>syncNodeWithClusterState() >>>>>>>>> >>registerMBeans() >>>>>>>>> >>startMessaging() >>>>>>>>> >>>>>>>>> On Sat, Sep 27, 2014 at 5:47 PM, Hasitha Hiranya < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Introduced following new classes >>>>>>>>>> >>>>>>>>>> ClusterCoordinationHandler implements QueueListener, >>>>>>>>>> ExchangeListener, BindingListener, SubscriptionListener >>>>>>>>>> >>>>>>>>>> >> one place to handle all cluster coordination >>>>>>>>>> >>>>>>>>>> AndesRecoveryTask - reload from DB time to time in case of >>>>>>>>>> cluster notification miss and simulate cluster notifications - TODO: >>>>>>>>>> to >>>>>>>>>> update default recovery task interval in config file >>>>>>>>>> >>>>>>>>>> AMQPConstructStore - keeps queues/bindings/exchanges - ultimately >>>>>>>>>> refers AndesContextStore >>>>>>>>>> >>>>>>>>>> On Sat, Sep 27, 2014 at 5:46 PM, Hasitha Hiranya < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I began the $subject. >>>>>>>>>>> Idea is to instead of deriving queues/bindings/exchanges from >>>>>>>>>>> subscriptions sync them Independently. >>>>>>>>>>> >>>>>>>>>>> I introduced QueueListener/BindingListener/ExchangeListner in >>>>>>>>>>> the same way as subscription listener >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Hasitha Abeykoon* >>>>>>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>>>>>> *cell:* *+94 719363063* >>>>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Hasitha Abeykoon* >>>>>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>>>>> *cell:* *+94 719363063* >>>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Hasitha Abeykoon* >>>>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>>>> *cell:* *+94 719363063* >>>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Hasitha Abeykoon* >>>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>>> *cell:* *+94 719363063* >>>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Hasitha Abeykoon* >>>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>>> *cell:* *+94 719363063* >>>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Hasitha Abeykoon* >>>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>>> *cell:* *+94 719363063* >>>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Hasitha Abeykoon* >>>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>>> *cell:* *+94 719363063* >>>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Hasitha Abeykoon* >>>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>>> *cell:* *+94 719363063* >>>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>>> >>>> >>> >>> >>> -- >>> *Hasitha Abeykoon* >>> Senior Software Engineer; WSO2, Inc.; http://wso2.com >>> *cell:* *+94 719363063* >>> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >>> >>> >> >> >> -- >> *Hasitha Abeykoon* >> Senior Software Engineer; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >> >> > > > -- > *Hasitha Abeykoon* > Senior Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > -- *Hasitha Abeykoon* Senior Software Engineer; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
