+1 for a rewrite. Started writing the throttle decision engine based on our discussion. Planning to have the first version ready by Monday so that we can have a follow up about the outside APIs etc. I also feel that the current Role, Domain and IP based logic should be re-factored. As far as throttling is concerned (and AFAIK :) ), its only an identifier for ThrottleContext to reference the relevant caller types, but I don't see a reason why those need to participate in throttling decision making.
Regards, Manoj On Sat, Jan 25, 2014 at 7:03 PM, Afkham Azeez <[email protected]> wrote: > Currently, throttling configuration is based on WS-Policy. It would be > good if the new implementation uses a proper rule language. > > > On Sat, Jan 25, 2014 at 6:59 PM, Afkham Azeez <[email protected]> wrote: > >> Given how long it has already taken to understand this legacy code, it >> will be much easier to come up with a very simple design, focusing on the >> problem we are trying to solve, and write it from scratch. To be frank, I >> think it can be done in a week if you fully focus on it. My suggestion was >> to write a throttling decision making engine, and provide nice APIs to the >> outside world. We discussed in detail what the problem at hand is, and it >> is possible to come up with an elegant design & implement this from scratch. >> >> Azeez >> >> >> On Sat, Jan 25, 2014 at 11:53 AM, Manoj Fernando <[email protected]> wrote: >> >>> During a discussion with Srinath and Azeez yesterday, the preference was >>> to rewrite the throttle core with persistence and Hazelcast based >>> replication in mind. I am progressing in that direction and will be >>> reviewing with Srinath periodically. >>> >>> Regards, >>> Manoj >>> >>> >>> On Mon, Jan 13, 2014 at 2:52 PM, Sriskandarajah Suhothayan < >>> [email protected]> wrote: >>> >>>> Siddhi support having Execution Plans, which can be mapped to one of >>>> the current policies. I believe this will reduce the complexity of >>>> the throttling execution logic. >>>> >>>> Suho >>>> >>>> >>>> On Mon, Jan 13, 2014 at 1:34 PM, Manoj Fernando <[email protected]>wrote: >>>> >>>>> Yes, this is something important to consider when we re-write the >>>>> throttle core eventually. However, the persistence logic we want to bring >>>>> in will not have any tight coupling with the throttle core. As per the >>>>> design we have finalized now, the throttle persistence module will >>>>> retrieve >>>>> the counters from the Axis2 context, and as long as the context is updated >>>>> by the core (irrespective of the implementation), the persistence core >>>>> will >>>>> be re-usable. >>>>> >>>>> One thing we should consider is the backward compatibility with >>>>> current throttle policy definitions IF we decide to bring in Siddhi into >>>>> the picture. In the case of API Manager for example, I think users are >>>>> more used to managing policies the way it is done now (i.e. tier.xml), so >>>>> IMO we should continue to support that. Is there such thing as a policy >>>>> definition plugin for Siddhi btw (may be not... right?) ? >>>>> >>>>> Regards, >>>>> Manoj >>>>> >>>>> >>>>> On Fri, Jan 3, 2014 at 4:55 PM, Sriskandarajah Suhothayan < >>>>> [email protected]> wrote: >>>>> >>>>>> Is there any possibility of using Distributed CEP/Siddhi here? >>>>>> Because with Siddhi we can have some flexibility in the way we want to >>>>>> throttle and throttling is a common usecase of CEP. Its underline >>>>>> architecture also uses Hazelcast or Storm for distributed processing. >>>>>> >>>>>> Regards >>>>>> Suho >>>>>> >>>>>> >>>>>> On Tue, Dec 24, 2013 at 8:54 AM, Manoj Fernando <[email protected]>wrote: >>>>>> >>>>>>> +1. Changing caller contexts in to a Hazlecast map would require >>>>>>> some significant changes to the throttle core, which may eventually be >>>>>>> re-written. >>>>>>> >>>>>>> Will update the design. >>>>>>> >>>>>>> Thanks, >>>>>>> Manoj >>>>>>> >>>>>>> >>>>>>> On Mon, Dec 23, 2013 at 4:09 PM, Srinath Perera <[email protected]>wrote: >>>>>>> >>>>>>>> Manoj, above plan look good. >>>>>>>> >>>>>>>> I chatted with Azeez, and we cannot register a Entry listener as I >>>>>>>> mentioned before because hazecast does not support entry listeners for >>>>>>>> atomic long. >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Dec 23, 2013 at 11:15 AM, Manoj Fernando >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> Short update after the discussion with Azeez. >>>>>>>>> >>>>>>>>> - The need to re-write the throttle core is still at large, so the >>>>>>>>> best was to see how we can decouple the persistence logic from the >>>>>>>>> throttle >>>>>>>>> core (at least as much as possible). >>>>>>>>> - A cluster updatable global counter will be included to the >>>>>>>>> ThrottleContext. The idea is that each node will periodically >>>>>>>>> broadcast >>>>>>>>> the local counter info to the members in the cluster and the >>>>>>>>> ThrottleConfiguration will update the value of the Global counter >>>>>>>>> summing >>>>>>>>> up the local counter values. >>>>>>>>> - The ThrottleConfiguration will also push the global counter >>>>>>>>> values to the Axis2 Configuration Context; a K, V pairs identified by >>>>>>>>> the >>>>>>>>> ThrottleContext ID. >>>>>>>>> - A new platform component needs to be written to read the >>>>>>>>> throttle related Axis2 Config Context list and persist them >>>>>>>>> periodically >>>>>>>>> (duration configurable). The throttle core will have no visibility >>>>>>>>> into >>>>>>>>> this persistence logic, so this will be completely decoupled. >>>>>>>>> - So who should do the persistence? We can start with letting all >>>>>>>>> nodes to persist first, but later (or in parallel) we can improve the >>>>>>>>> Hazlecast's leader election (if that's not already there), so that the >>>>>>>>> leader takes the responsibility of persisting. >>>>>>>>> - The counters will be read off the persistence store at the time >>>>>>>>> of Hazlecast Leader election takes place? (An alternative is to load >>>>>>>>> the >>>>>>>>> global counters at the init of ThrottleConfiguration but that means >>>>>>>>> coupling throttle core with persistence.) >>>>>>>>> >>>>>>>>> I will update the design accordingly. >>>>>>>>> >>>>>>>>> Any more thoughts or suggestions? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Manoj >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2013 at 12:30 PM, Manoj Fernando >>>>>>>>> <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> +1. Let me setup a time. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Manoj >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thursday, December 19, 2013, Srinath Perera wrote: >>>>>>>>>> >>>>>>>>>>> We need Azeez's feedback. Shall you, myself, and Azeez chat >>>>>>>>>>> sometime and decide on the first Arch design? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 19, 2013 at 11:55 AM, Manoj Fernando < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Srinath, >>>>>>>>>>> >>>>>>>>>>> That sounds like a much cleaner solution. We can perhaps use >>>>>>>>>>> the native map-store declarative [1] which I think does something >>>>>>>>>>> similar. >>>>>>>>>>> It may sound a little silly to ask... but are we keeping Hazlecast >>>>>>>>>>> active >>>>>>>>>>> on a single node environment as well? :) Otherwise we will have to >>>>>>>>>>> handle >>>>>>>>>>> persistence on a single node in a different way. This is with the >>>>>>>>>>> assumption of needing to persist throttle data on a single node >>>>>>>>>>> environment >>>>>>>>>>> as well (but questioning if we really need to do that is not totally >>>>>>>>>>> invalid IMO). >>>>>>>>>>> >>>>>>>>>>> Shall we go ahead with the Hazlecast option targeting cluster >>>>>>>>>>> deployments then? >>>>>>>>>>> >>>>>>>>>>> - Manoj >>>>>>>>>>> >>>>>>>>>>> [1] https://code.google.com/p/hazelcast/wiki/MapPersistence >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 19, 2013 at 10:51 AM, Srinath Perera < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> One another way to do this use Hazelcast and then use "though >>>>>>>>>>> cache" or "Change listener's" in Hazecast for persistence. >>>>>>>>>>> >>>>>>>>>>> --Srinath >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 17, 2013 at 4:49 PM, Manoj Fernando <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>> +1 for persisting through a single (elected?) node, and let >>>>>>>>>>> Hazlecast do the replication. >>>>>>>>>>> >>>>>>>>>>> I took into consideration the need to persist periodically >>>>>>>>>>> instead of at each and every request (by spawning a separate thread >>>>>>>>>>> that >>>>>>>>>>> has access to the callerContext map)... so yes... we should think >>>>>>>>>>> in the >>>>>>>>>>> same way for replicating the counters across the cluster as well. >>>>>>>>>>> >>>>>>>>>>> Instead of using a global counter, can we perhaps use the last >>>>>>>>>>> updated timestamp of each CallerContext? It's actually not a single >>>>>>>>>>> counter we need to deal with, and each CallerContext instance will >>>>>>>>>>> have >>>>>>>>>>> separate counters mapped to their throttling policy AFAIK. >>>>>>>>>>> Therefore, I >>>>>>>>>>> think its probably better to update CallerContext instances based >>>>>>>>>>> on the >>>>>>>>>>> last update timestamp. >>>>>>>>>>> >>>>>>>>>>> WDYT? >>>>>>>>>>> >>>>>>>>>>> If agree, then I need to figure out how to make delayed >>>>>>>>>>> replication on hazlecast (is it through >>>>>>>>>>> the hazelcast.heartbeat.interval.seconds config item?) >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Manoj >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, Dec 17, 2013 at 4:22 PM, Srinath Perera < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> We need to think it a cluster setup do we need persistence as >>>>>>>>>>> well? As we can have replication using Hazelcast? >>>>>>>>>>> >>>>>>>>>>> If we need persistence, I think it is a good if a single node >>>>>>>>>>> persists the current throttling values, and if that node fails, >>>>>>>>>>> someone >>>>>>>>>>> else takes it place? >>>>>>>>>>> >>>>>>>>>>> Current implementation sync the values across the cluster per >>>>>>>>>>> each message, which introduce significant overhead. I think we >>>>>>>>>>> should go to >>>>>>>>>>> a model where each node collects and update the values once few >>>>>>>>>>> seconds. >>>>>>>>>>> >>>>>>>>>>> idea is >>>>>>>>>>> 1) there is a global counter, that we use to throttle >>>>>>>>>>> 2) Each node keep a global counter, and periodically it update >>>>>>>>>>> the global counter using value in location counter and reset the >>>>>>>>>>> counter >>>>>>>>>>> and read the current global counter. >>>>>>>>>>> 3) Until next update, each node make decisions based on local >>>>>>>>>>> global counter values it has read already >>>>>>>>>>> >>>>>>>>>>> This will mean that the throttling will throttle close to the >>>>>>>>>>> limit, not exactly at the limit. However, IMHO, that is not a >>>>>>>>>>> problem for >>>>>>>>>>> throttling usecase. >>>>>>>>>>> >>>>>>>>>>> --Srinath >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Dec 16, 2013 at 7:20 PM, Manoj Fernando <[email protected] >>>>>>>>>>> > wrot >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Manoj Fernando >>>>>>>>>> Director - Solutions Architecture >>>>>>>>>> >>>>>>>>>> Contact: >>>>>>>>>> LK - +94 112 145345 >>>>>>>>>> Mob: +94 773 759340 >>>>>>>>>> www.wso2.com >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Manoj Fernando >>>>>>>>> Director - Solutions Architecture >>>>>>>>> >>>>>>>>> Contact: >>>>>>>>> LK - +94 112 145345 >>>>>>>>> Mob: +94 773 759340 >>>>>>>>> www.wso2.com >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Srinath Perera, Ph.D. >>>>>>>> http://people.apache.org/~hemapani/ >>>>>>>> http://srinathsview.blogspot.com/ >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Manoj Fernando >>>>>>> Director - Solutions Architecture >>>>>>> >>>>>>> Contact: >>>>>>> LK - +94 112 145345 >>>>>>> Mob: +94 773 759340 >>>>>>> www.wso2.com >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> [email protected] >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *S. Suhothayan * >>>>>> Associate Technical Lead, >>>>>> *WSO2 Inc. *http://wso2.com >>>>>> * <http://wso2.com/>* >>>>>> lean . enterprise . middleware >>>>>> >>>>>> >>>>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/> >>>>>> twitter: >>>>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | >>>>>> linked-in: >>>>>> http://lk.linkedin.com/in/suhothayan >>>>>> <http://lk.linkedin.com/in/suhothayan>* >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> [email protected] >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Manoj Fernando >>>>> Director - Solutions Architecture >>>>> >>>>> Contact: >>>>> LK - +94 112 145345 >>>>> Mob: +94 773 759340 >>>>> www.wso2.com >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *S. Suhothayan* >>>> Associate Technical Lead, >>>> *WSO2 Inc. *http://wso2.com >>>> * <http://wso2.com/>* >>>> lean . enterprise . middleware >>>> >>>> >>>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/> twitter: >>>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>>> http://lk.linkedin.com/in/suhothayan >>>> <http://lk.linkedin.com/in/suhothayan>* >>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Manoj Fernando >>> Director - Solutions Architecture >>> >>> Contact: >>> LK - +94 112 145345 >>> Mob: +94 773 759340 >>> www.wso2.com >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> *Afkham Azeez* >> Director of Architecture; WSO2, Inc.; http://wso2.com >> Member; Apache Software Foundation; http://www.apache.org/ >> * <http://www.apache.org/>* >> *email: **[email protected]* <[email protected]> >> * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * >> *http://blog.afkham.org* <http://blog.afkham.org> >> *twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >> * linked-in: **http://lk.linkedin.com/in/afkhamazeez >> <http://lk.linkedin.com/in/afkhamazeez>* >> >> *Lean . Enterprise . Middleware* >> > > > > -- > *Afkham Azeez* > Director of Architecture; WSO2, Inc.; http://wso2.com > Member; Apache Software Foundation; http://www.apache.org/ > * <http://www.apache.org/>* > *email: **[email protected]* <[email protected]> > * cell: +94 77 3320919 <%2B94%2077%203320919> blog: * > *http://blog.afkham.org* <http://blog.afkham.org> > *twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> > * linked-in: **http://lk.linkedin.com/in/afkhamazeez > <http://lk.linkedin.com/in/afkhamazeez>* > > *Lean . Enterprise . Middleware* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Manoj Fernando Director - Solutions Architecture Contact: LK - +94 112 145345 Mob: +94 773 759340 www.wso2.com
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
